DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing/1: /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tools/rebalancer_tool-test.cc:1203: Failure
Value of: s.ok()
Actual: false
Expected: true
Runtime error: /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 2
Maximum Replica Count | 44
Average Replica Count | 22.875000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 1
Maximum | 3
Average | 2.857143
stderr: W20240503 01:59:58.536991 9363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:58.537266 9363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:58.540885 9363 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 01:59:59.349107 9363 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240503 01:59:59.626546 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9b77b848294749e09f0854d24da5d9f5: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server 99ead02915954b4fab23b033ad6d546b
W20240503 01:59:59.626806 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 72f29547da0b4a95974018c2782cb359: could not find any suitable replica to move from server 492448570cc94ed1879feb2ec3b9051d to server 99ead02915954b4fab23b033ad6d546b
W20240503 01:59:59.626907 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5ccf0adaf2de4b79bbf38c053646e60e: could not find any suitable replica to move from server dec3e4a04d604f7eaaa3f077d081218a to server e1a9251ce1cf4361b8b0ede9fbfc4023
W20240503 01:59:59.626965 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9b77b848294749e09f0854d24da5d9f5: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server 9942306855404cfd891adf227dd65eb2
W20240503 01:59:59.627007 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 14361c7197694d579466ed5d4f2922f7: could not find any suitable replica to move from server 492448570cc94ed1879feb2ec3b9051d to server 99ead02915954b4fab23b033ad6d546b
W20240503 01:59:59.627096 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 296df7c9486144058745436d9b814a78: could not find any suitable replica to move from server 9942306855404cfd891adf227dd65eb2 to server c2ac2eb7cc264047b9ab32fe3a4fc025
W20240503 01:59:59.627197 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 805c97ce87c74e019deaa2acee6a5329: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server e1a9251ce1cf4361b8b0ede9fbfc4023
W20240503 01:59:59.627235 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e0e5e479de824fb18516cfe56d5892c6: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server 99ead02915954b4fab23b033ad6d546b
W20240503 01:59:59.627310 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table eecab7e70bfa4621832bb99bd9ef066c: could not find any suitable replica to move from server 80fd18ef4d604c86b909d5c008a46b4f to server 9942306855404cfd891adf227dd65eb2
W20240503 01:59:59.627360 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6e24d367bc1d434584d8a7363cf538b6: could not find any suitable replica to move from server 80fd18ef4d604c86b909d5c008a46b4f to server 2a01ace80baf4057b6bf1995cf88468b
W20240503 01:59:59.627408 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7455f5480e88494b83e1ac1d180ca4e6: could not find any suitable replica to move from server 492448570cc94ed1879feb2ec3b9051d to server 99ead02915954b4fab23b033ad6d546b
W20240503 01:59:59.627471 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d732c32bc9004748904a109f59db08fd: could not find any suitable replica to move from server dec3e4a04d604f7eaaa3f077d081218a to server 2a01ace80baf4057b6bf1995cf88468b
W20240503 01:59:59.627532 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 029e2476b8c44a8d8cdb98005eb84a4b: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server e1a9251ce1cf4361b8b0ede9fbfc4023
W20240503 01:59:59.627566 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c65135643094446abc8eba15be23cd99: could not find any suitable replica to move from server 492448570cc94ed1879feb2ec3b9051d to server 99ead02915954b4fab23b033ad6d546b
W20240503 01:59:59.627588 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 65ed012903ed4deabf9e7b0dfe803452: could not find any suitable replica to move from server dec3e4a04d604f7eaaa3f077d081218a to server 9942306855404cfd891adf227dd65eb2
W20240503 01:59:59.627617 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4e740b0167ed439291b74d77653ec386: could not find any suitable replica to move from server 80fd18ef4d604c86b909d5c008a46b4f to server e1a9251ce1cf4361b8b0ede9fbfc4023
W20240503 01:59:59.627645 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d93ca01e08fd40d886555f16d21192fc: could not find any suitable replica to move from server 492448570cc94ed1879feb2ec3b9051d to server 99ead02915954b4fab23b033ad6d546b
W20240503 01:59:59.627671 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 250f0720cafa4425b45a0360ebe04a03: could not find any suitable replica to move from server 80fd18ef4d604c86b909d5c008a46b4f to server e1a9251ce1cf4361b8b0ede9fbfc4023
W20240503 01:59:59.627705 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 043b44e4ca1745038882551026afda53: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server 9942306855404cfd891adf227dd65eb2
W20240503 01:59:59.627792 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d930c636ed124c0eb64e1d6ab7c8d106: could not find any suitable replica to move from server 492448570cc94ed1879feb2ec3b9051d to server 2a01ace80baf4057b6bf1995cf88468b
W20240503 01:59:59.627842 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8b571be2b4d14eab84dc1bc1e5b8d53a: could not find any suitable replica to move from server 80fd18ef4d604c86b909d5c008a46b4f to server 9942306855404cfd891adf227dd65eb2
W20240503 01:59:59.627890 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6e24d367bc1d434584d8a7363cf538b6: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server 9942306855404cfd891adf227dd65eb2
W20240503 01:59:59.627974 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7455f5480e88494b83e1ac1d180ca4e6: could not find any suitable replica to move from server 80fd18ef4d604c86b909d5c008a46b4f to server e1a9251ce1cf4361b8b0ede9fbfc4023
W20240503 01:59:59.628016 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5ccf0adaf2de4b79bbf38c053646e60e: could not find any suitable replica to move from server 492448570cc94ed1879feb2ec3b9051d to server 2a01ace80baf4057b6bf1995cf88468b
W20240503 01:59:59.628063 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d732c32bc9004748904a109f59db08fd: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server 9942306855404cfd891adf227dd65eb2
W20240503 01:59:59.628108 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 029e2476b8c44a8d8cdb98005eb84a4b: could not find any suitable replica to move from server dec3e4a04d604f7eaaa3f077d081218a to server 2a01ace80baf4057b6bf1995cf88468b
I20240503 01:59:59.647352 9363 rebalancer_tool.cc:1074] tablet 1fa274c98fc147f29344ed537e2ccf17: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> 'e1a9251ce1cf4361b8b0ede9fbfc4023' move scheduled
I20240503 01:59:59.663408 9363 rebalancer_tool.cc:1074] tablet ceca04391cd446ae932b3485d958237a: '80fd18ef4d604c86b909d5c008a46b4f' -> '9942306855404cfd891adf227dd65eb2' move scheduled
I20240503 01:59:59.688750 9363 rebalancer_tool.cc:1074] tablet 1e445a3cc928429caf9e3c6193a63010: '492448570cc94ed1879feb2ec3b9051d' -> '2a01ace80baf4057b6bf1995cf88468b' move scheduled
I20240503 01:59:59.709553 9363 rebalancer_tool.cc:1074] tablet d82268925a934c559dfcf51a84f03d2e: 'dec3e4a04d604f7eaaa3f077d081218a' -> '99ead02915954b4fab23b033ad6d546b' move scheduled
I20240503 01:59:59.725863 9363 rebalancer_tool.cc:1074] tablet 3416b6107e1848f299ac49ccd88ae7c8: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> 'e1a9251ce1cf4361b8b0ede9fbfc4023' move scheduled
I20240503 01:59:59.742455 9363 rebalancer_tool.cc:1074] tablet 5f176537159143e8a770d65ba63ad62e: '80fd18ef4d604c86b909d5c008a46b4f' -> '9942306855404cfd891adf227dd65eb2' move scheduled
I20240503 01:59:59.762373 9363 rebalancer_tool.cc:1074] tablet 1feaf29d8c6b4842829ce66e0f4ec23b: '492448570cc94ed1879feb2ec3b9051d' -> '2a01ace80baf4057b6bf1995cf88468b' move scheduled
I20240503 01:59:59.804647 9363 rebalancer_tool.cc:1074] tablet 3cf186a0901f420390c8fda54316c5af: 'dec3e4a04d604f7eaaa3f077d081218a' -> '99ead02915954b4fab23b033ad6d546b' move scheduled
I20240503 01:59:59.840487 9363 rebalancer_tool.cc:1074] tablet 46ad540deb624a25b97d3b967fe1f6f4: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> 'e1a9251ce1cf4361b8b0ede9fbfc4023' move scheduled
I20240503 01:59:59.857347 9363 rebalancer_tool.cc:1074] tablet 7a44b6d65bfa43e291b06b0dfd38fab3: '80fd18ef4d604c86b909d5c008a46b4f' -> '9942306855404cfd891adf227dd65eb2' move scheduled
I20240503 01:59:59.875558 9363 rebalancer_tool.cc:1074] tablet 485d1d89a8b94505969d663be92124cd: '492448570cc94ed1879feb2ec3b9051d' -> '99ead02915954b4fab23b033ad6d546b' move scheduled
I20240503 01:59:59.892912 9363 rebalancer_tool.cc:1074] tablet 90ef3584830240acb0d8aad14ae851a6: 'dec3e4a04d604f7eaaa3f077d081218a' -> '2a01ace80baf4057b6bf1995cf88468b' move scheduled
I20240503 01:59:59.931536 9363 rebalancer_tool.cc:1074] tablet 915028b3478341d588ed6031e657c259: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> '9942306855404cfd891adf227dd65eb2' move scheduled
I20240503 01:59:59.953246 9363 rebalancer_tool.cc:1074] tablet 548f43b381f1445c8d3d55705ae5a48a: '80fd18ef4d604c86b909d5c008a46b4f' -> 'e1a9251ce1cf4361b8b0ede9fbfc4023' move scheduled
I20240503 01:59:59.969899 9363 rebalancer_tool.cc:1074] tablet 74ebf5d0fb9a42e6a7ec743a98960e13: '492448570cc94ed1879feb2ec3b9051d' -> '99ead02915954b4fab23b033ad6d546b' move scheduled
I20240503 01:59:59.988325 9363 rebalancer_tool.cc:1074] tablet 8a76d3e2d4264687aa2972c179341b5e: 'dec3e4a04d604f7eaaa3f077d081218a' -> '2a01ace80baf4057b6bf1995cf88468b' move scheduled
I20240503 02:00:00.004880 9363 rebalancer_tool.cc:1074] tablet 2b1d700d09534fa3adcf51bb3c97c8f5: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> '9942306855404cfd891adf227dd65eb2' move scheduled
I20240503 02:00:00.005923 9363 rebalancer_tool.cc:1084] tablet 396a2a8083d240529a2db48519f6139e: '80fd18ef4d604c86b909d5c008a46b4f' -> 'e1a9251ce1cf4361b8b0ede9fbfc4023' move ignored: Not found: 396a2a8083d240529a2db48519f6139e: tablet not found
I20240503 02:00:00.006011 9363 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 02:00:00.296618 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9b77b848294749e09f0854d24da5d9f5: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server 99ead02915954b4fab23b033ad6d546b
W20240503 02:00:00.296752 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 54f1891be3f54f19821aad5d1dfbd85d: could not find any suitable replica to move from server 9942306855404cfd891adf227dd65eb2 to server 80fd18ef4d604c86b909d5c008a46b4f
W20240503 02:00:00.297022 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6e24d367bc1d434584d8a7363cf538b6: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server 2a01ace80baf4057b6bf1995cf88468b
W20240503 02:00:00.297132 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 029e2476b8c44a8d8cdb98005eb84a4b: could not find any suitable replica to move from server dec3e4a04d604f7eaaa3f077d081218a to server e1a9251ce1cf4361b8b0ede9fbfc4023
W20240503 02:00:00.297273 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c65135643094446abc8eba15be23cd99: could not find any suitable replica to move from server 492448570cc94ed1879feb2ec3b9051d to server 9942306855404cfd891adf227dd65eb2
W20240503 02:00:00.297374 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table eecab7e70bfa4621832bb99bd9ef066c: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server 2a01ace80baf4057b6bf1995cf88468b
W20240503 02:00:00.297405 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 54f1891be3f54f19821aad5d1dfbd85d: could not find any suitable replica to move from server dec3e4a04d604f7eaaa3f077d081218a to server c2ac2eb7cc264047b9ab32fe3a4fc025
W20240503 02:00:00.297479 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 805c97ce87c74e019deaa2acee6a5329: could not find any suitable replica to move from server 492448570cc94ed1879feb2ec3b9051d to server 2a01ace80baf4057b6bf1995cf88468b
W20240503 02:00:00.297538 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d732c32bc9004748904a109f59db08fd: could not find any suitable replica to move from server 80fd18ef4d604c86b909d5c008a46b4f to server 99ead02915954b4fab23b033ad6d546b
W20240503 02:00:00.297577 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 14361c7197694d579466ed5d4f2922f7: could not find any suitable replica to move from server 492448570cc94ed1879feb2ec3b9051d to server e1a9251ce1cf4361b8b0ede9fbfc4023
W20240503 02:00:00.297641 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 250f0720cafa4425b45a0360ebe04a03: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server 99ead02915954b4fab23b033ad6d546b
W20240503 02:00:00.297773 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9b77b848294749e09f0854d24da5d9f5: could not find any suitable replica to move from server 492448570cc94ed1879feb2ec3b9051d to server e1a9251ce1cf4361b8b0ede9fbfc4023
W20240503 02:00:00.297820 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 54f1891be3f54f19821aad5d1dfbd85d: could not find any suitable replica to move from server 9942306855404cfd891adf227dd65eb2 to server 492448570cc94ed1879feb2ec3b9051d
W20240503 02:00:00.297855 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 043b44e4ca1745038882551026afda53: could not find any suitable replica to move from server 492448570cc94ed1879feb2ec3b9051d to server 9942306855404cfd891adf227dd65eb2
W20240503 02:00:00.297888 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4e740b0167ed439291b74d77653ec386: could not find any suitable replica to move from server dec3e4a04d604f7eaaa3f077d081218a to server 9942306855404cfd891adf227dd65eb2
W20240503 02:00:00.297932 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7455f5480e88494b83e1ac1d180ca4e6: could not find any suitable replica to move from server dec3e4a04d604f7eaaa3f077d081218a to server e1a9251ce1cf4361b8b0ede9fbfc4023
W20240503 02:00:00.298027 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5ccf0adaf2de4b79bbf38c053646e60e: could not find any suitable replica to move from server 492448570cc94ed1879feb2ec3b9051d to server 99ead02915954b4fab23b033ad6d546b
W20240503 02:00:00.298069 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 65ed012903ed4deabf9e7b0dfe803452: could not find any suitable replica to move from server 492448570cc94ed1879feb2ec3b9051d to server e1a9251ce1cf4361b8b0ede9fbfc4023
W20240503 02:00:00.298117 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d93ca01e08fd40d886555f16d21192fc: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server 99ead02915954b4fab23b033ad6d546b
W20240503 02:00:00.298151 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e0e5e479de824fb18516cfe56d5892c6: could not find any suitable replica to move from server dec3e4a04d604f7eaaa3f077d081218a to server 2a01ace80baf4057b6bf1995cf88468b
W20240503 02:00:00.298183 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 72f29547da0b4a95974018c2782cb359: could not find any suitable replica to move from server 492448570cc94ed1879feb2ec3b9051d to server 2a01ace80baf4057b6bf1995cf88468b
W20240503 02:00:00.298244 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6aaa25ca43754a899cf8589522ff71da: could not find any suitable replica to move from server 80fd18ef4d604c86b909d5c008a46b4f to server 9942306855404cfd891adf227dd65eb2
W20240503 02:00:00.298282 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 029e2476b8c44a8d8cdb98005eb84a4b: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server 2a01ace80baf4057b6bf1995cf88468b
W20240503 02:00:00.298319 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c65135643094446abc8eba15be23cd99: could not find any suitable replica to move from server dec3e4a04d604f7eaaa3f077d081218a to server e1a9251ce1cf4361b8b0ede9fbfc4023
W20240503 02:00:00.298359 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6e24d367bc1d434584d8a7363cf538b6: could not find any suitable replica to move from server 80fd18ef4d604c86b909d5c008a46b4f to server 9942306855404cfd891adf227dd65eb2
W20240503 02:00:00.298390 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9b77b848294749e09f0854d24da5d9f5: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server 99ead02915954b4fab23b033ad6d546b
W20240503 02:00:00.298420 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d732c32bc9004748904a109f59db08fd: could not find any suitable replica to move from server dec3e4a04d604f7eaaa3f077d081218a to server 2a01ace80baf4057b6bf1995cf88468b
W20240503 02:00:00.298451 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d93ca01e08fd40d886555f16d21192fc: could not find any suitable replica to move from server 492448570cc94ed1879feb2ec3b9051d to server e1a9251ce1cf4361b8b0ede9fbfc4023
I20240503 02:00:00.394961 9363 rebalancer_tool.cc:1074] tablet 0b659e9d57db4ebf93c9143938970090: '80fd18ef4d604c86b909d5c008a46b4f' -> 'e1a9251ce1cf4361b8b0ede9fbfc4023' move scheduled
I20240503 02:00:00.493460 9363 rebalancer_tool.cc:1074] tablet 7a4a910a8b9d4134adebb52a73dfae1d: '492448570cc94ed1879feb2ec3b9051d' -> '99ead02915954b4fab23b033ad6d546b' move scheduled
I20240503 02:00:00.537493 9363 rebalancer_tool.cc:1074] tablet 7f637097ad06465cb93c6ff6d97c1d17: 'dec3e4a04d604f7eaaa3f077d081218a' -> '2a01ace80baf4057b6bf1995cf88468b' move scheduled
I20240503 02:00:00.627055 9363 rebalancer_tool.cc:1123] tablet 5f176537159143e8a770d65ba63ad62e: '80fd18ef4d604c86b909d5c008a46b4f' -> '9942306855404cfd891adf227dd65eb2' move is abandoned: Not found: 5f176537159143e8a770d65ba63ad62e: tablet not found
I20240503 02:00:00.754491 9363 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 02:00:01.011598 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9b77b848294749e09f0854d24da5d9f5: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server e1a9251ce1cf4361b8b0ede9fbfc4023
W20240503 02:00:01.011711 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c1969b96b61041869ddd81db8c84bf44: could not find any suitable replica to move from server e1a9251ce1cf4361b8b0ede9fbfc4023 to server dec3e4a04d604f7eaaa3f077d081218a
W20240503 02:00:01.011828 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6e24d367bc1d434584d8a7363cf538b6: could not find any suitable replica to move from server 80fd18ef4d604c86b909d5c008a46b4f to server 2a01ace80baf4057b6bf1995cf88468b
W20240503 02:00:01.011883 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7455f5480e88494b83e1ac1d180ca4e6: could not find any suitable replica to move from server dec3e4a04d604f7eaaa3f077d081218a to server e1a9251ce1cf4361b8b0ede9fbfc4023
W20240503 02:00:01.012025 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 029e2476b8c44a8d8cdb98005eb84a4b: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server e1a9251ce1cf4361b8b0ede9fbfc4023
W20240503 02:00:01.012079 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c65135643094446abc8eba15be23cd99: could not find any suitable replica to move from server 492448570cc94ed1879feb2ec3b9051d to server 9942306855404cfd891adf227dd65eb2
W20240503 02:00:01.012203 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c1969b96b61041869ddd81db8c84bf44: could not find any suitable replica to move from server 99ead02915954b4fab23b033ad6d546b to server c2ac2eb7cc264047b9ab32fe3a4fc025
W20240503 02:00:01.012253 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d732c32bc9004748904a109f59db08fd: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server 9942306855404cfd891adf227dd65eb2
W20240503 02:00:01.012416 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d93ca01e08fd40d886555f16d21192fc: could not find any suitable replica to move from server 492448570cc94ed1879feb2ec3b9051d to server 99ead02915954b4fab23b033ad6d546b
W20240503 02:00:01.012503 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e0e5e479de824fb18516cfe56d5892c6: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server 9942306855404cfd891adf227dd65eb2
W20240503 02:00:01.012589 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4e740b0167ed439291b74d77653ec386: could not find any suitable replica to move from server 80fd18ef4d604c86b909d5c008a46b4f to server e1a9251ce1cf4361b8b0ede9fbfc4023
W20240503 02:00:01.012656 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 043b44e4ca1745038882551026afda53: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server 2a01ace80baf4057b6bf1995cf88468b
W20240503 02:00:01.012701 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9b77b848294749e09f0854d24da5d9f5: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server 99ead02915954b4fab23b033ad6d546b
W20240503 02:00:01.012743 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 805c97ce87c74e019deaa2acee6a5329: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server 9942306855404cfd891adf227dd65eb2
W20240503 02:00:01.012779 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c1969b96b61041869ddd81db8c84bf44: could not find any suitable replica to move from server e1a9251ce1cf4361b8b0ede9fbfc4023 to server 80fd18ef4d604c86b909d5c008a46b4f
W20240503 02:00:01.012840 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5ccf0adaf2de4b79bbf38c053646e60e: could not find any suitable replica to move from server dec3e4a04d604f7eaaa3f077d081218a to server 9942306855404cfd891adf227dd65eb2
W20240503 02:00:01.012883 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 14361c7197694d579466ed5d4f2922f7: could not find any suitable replica to move from server dec3e4a04d604f7eaaa3f077d081218a to server 2a01ace80baf4057b6bf1995cf88468b
W20240503 02:00:01.012914 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 250f0720cafa4425b45a0360ebe04a03: could not find any suitable replica to move from server 80fd18ef4d604c86b909d5c008a46b4f to server e1a9251ce1cf4361b8b0ede9fbfc4023
W20240503 02:00:01.012943 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table eecab7e70bfa4621832bb99bd9ef066c: could not find any suitable replica to move from server 80fd18ef4d604c86b909d5c008a46b4f to server 2a01ace80baf4057b6bf1995cf88468b
W20240503 02:00:01.012979 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e0e5e479de824fb18516cfe56d5892c6: could not find any suitable replica to move from server 492448570cc94ed1879feb2ec3b9051d to server 99ead02915954b4fab23b033ad6d546b
W20240503 02:00:01.013017 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 72f29547da0b4a95974018c2782cb359: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server 2a01ace80baf4057b6bf1995cf88468b
W20240503 02:00:01.013072 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3295265fc41646e7942b5cd0d2d8ab20: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server e1a9251ce1cf4361b8b0ede9fbfc4023
W20240503 02:00:01.013108 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 029e2476b8c44a8d8cdb98005eb84a4b: could not find any suitable replica to move from server dec3e4a04d604f7eaaa3f077d081218a to server 99ead02915954b4fab23b033ad6d546b
W20240503 02:00:01.013141 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6e24d367bc1d434584d8a7363cf538b6: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server e1a9251ce1cf4361b8b0ede9fbfc4023
W20240503 02:00:01.013175 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7455f5480e88494b83e1ac1d180ca4e6: could not find any suitable replica to move from server 80fd18ef4d604c86b909d5c008a46b4f to server 9942306855404cfd891adf227dd65eb2
W20240503 02:00:01.013204 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4e740b0167ed439291b74d77653ec386: could not find any suitable replica to move from server dec3e4a04d604f7eaaa3f077d081218a to server e1a9251ce1cf4361b8b0ede9fbfc4023
W20240503 02:00:01.013238 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 805c97ce87c74e019deaa2acee6a5329: could not find any suitable replica to move from server 492448570cc94ed1879feb2ec3b9051d to server 99ead02915954b4fab23b033ad6d546b
I20240503 02:00:01.034834 9363 rebalancer_tool.cc:1074] tablet 0099654461ca41efa4323fc9f5adab4a: '9942306855404cfd891adf227dd65eb2' -> '80fd18ef4d604c86b909d5c008a46b4f' move scheduled
I20240503 02:00:01.050669 9363 rebalancer_tool.cc:1123] tablet 7f637097ad06465cb93c6ff6d97c1d17: 'dec3e4a04d604f7eaaa3f077d081218a' -> '2a01ace80baf4057b6bf1995cf88468b' move is abandoned: Not found: 7f637097ad06465cb93c6ff6d97c1d17: tablet not found
I20240503 02:00:01.106911 9363 rebalancer_tool.cc:1136] tablet 1fa274c98fc147f29344ed537e2ccf17: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> 'e1a9251ce1cf4361b8b0ede9fbfc4023' move completed: OK
I20240503 02:00:01.142973 9363 rebalancer_tool.cc:1123] tablet 548f43b381f1445c8d3d55705ae5a48a: '80fd18ef4d604c86b909d5c008a46b4f' -> 'e1a9251ce1cf4361b8b0ede9fbfc4023' move is abandoned: Not found: 548f43b381f1445c8d3d55705ae5a48a: tablet not found
I20240503 02:00:01.424549 9363 rebalancer_tool.cc:1074] tablet 9f7ade1e2af24e86b6e54e2b5769e4c0: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> 'e1a9251ce1cf4361b8b0ede9fbfc4023' move scheduled
I20240503 02:00:01.572912 9363 rebalancer_tool.cc:1136] tablet 2b1d700d09534fa3adcf51bb3c97c8f5: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> '9942306855404cfd891adf227dd65eb2' move completed: OK
I20240503 02:00:01.591578 9363 rebalancer_tool.cc:1136] tablet 3416b6107e1848f299ac49ccd88ae7c8: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> 'e1a9251ce1cf4361b8b0ede9fbfc4023' move completed: OK
I20240503 02:00:01.604578 9363 rebalancer_tool.cc:1136] tablet d82268925a934c559dfcf51a84f03d2e: 'dec3e4a04d604f7eaaa3f077d081218a' -> '99ead02915954b4fab23b033ad6d546b' move completed: OK
I20240503 02:00:01.615350 9363 rebalancer_tool.cc:1136] tablet 1e445a3cc928429caf9e3c6193a63010: '492448570cc94ed1879feb2ec3b9051d' -> '2a01ace80baf4057b6bf1995cf88468b' move completed: OK
I20240503 02:00:01.623342 9363 rebalancer_tool.cc:1136] tablet ceca04391cd446ae932b3485d958237a: '80fd18ef4d604c86b909d5c008a46b4f' -> '9942306855404cfd891adf227dd65eb2' move completed: OK
I20240503 02:00:01.630910 9363 rebalancer_tool.cc:1136] tablet 90ef3584830240acb0d8aad14ae851a6: 'dec3e4a04d604f7eaaa3f077d081218a' -> '2a01ace80baf4057b6bf1995cf88468b' move completed: OK
I20240503 02:00:01.637506 9363 rebalancer_tool.cc:1136] tablet 485d1d89a8b94505969d663be92124cd: '492448570cc94ed1879feb2ec3b9051d' -> '99ead02915954b4fab23b033ad6d546b' move completed: OK
I20240503 02:00:01.643836 9363 rebalancer_tool.cc:1136] tablet 1feaf29d8c6b4842829ce66e0f4ec23b: '492448570cc94ed1879feb2ec3b9051d' -> '2a01ace80baf4057b6bf1995cf88468b' move completed: OK
I20240503 02:00:01.652536 9363 rebalancer_tool.cc:1136] tablet 915028b3478341d588ed6031e657c259: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> '9942306855404cfd891adf227dd65eb2' move completed: OK
I20240503 02:00:01.658653 9363 rebalancer_tool.cc:1136] tablet 3cf186a0901f420390c8fda54316c5af: 'dec3e4a04d604f7eaaa3f077d081218a' -> '99ead02915954b4fab23b033ad6d546b' move completed: OK
I20240503 02:00:01.664966 9363 rebalancer_tool.cc:1136] tablet 7a44b6d65bfa43e291b06b0dfd38fab3: '80fd18ef4d604c86b909d5c008a46b4f' -> '9942306855404cfd891adf227dd65eb2' move completed: OK
I20240503 02:00:01.671490 9363 rebalancer_tool.cc:1136] tablet 46ad540deb624a25b97d3b967fe1f6f4: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> 'e1a9251ce1cf4361b8b0ede9fbfc4023' move completed: OK
I20240503 02:00:01.680900 9363 rebalancer_tool.cc:1136] tablet 74ebf5d0fb9a42e6a7ec743a98960e13: '492448570cc94ed1879feb2ec3b9051d' -> '99ead02915954b4fab23b033ad6d546b' move completed: OK
I20240503 02:00:01.718674 9363 rebalancer_tool.cc:1074] tablet 7df4c083550046e883e5e61d3d4e1572: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> '9942306855404cfd891adf227dd65eb2' move scheduled
I20240503 02:00:01.739480 9363 rebalancer_tool.cc:1074] tablet 34120f4948504cbabcd28e0c4f08f9f8: '492448570cc94ed1879feb2ec3b9051d' -> '99ead02915954b4fab23b033ad6d546b' move scheduled
I20240503 02:00:01.756099 9363 rebalancer_tool.cc:1074] tablet 2fc834abc6a94c638d6202b03320a2d2: 'dec3e4a04d604f7eaaa3f077d081218a' -> '2a01ace80baf4057b6bf1995cf88468b' move scheduled
I20240503 02:00:01.757140 9363 rebalancer_tool.cc:1084] tablet ce8bfc835792444e8703e73252b1f638: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> '9942306855404cfd891adf227dd65eb2' move ignored: Not found: ce8bfc835792444e8703e73252b1f638: tablet not found
I20240503 02:00:01.757217 9363 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 02:00:02.025200 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7455f5480e88494b83e1ac1d180ca4e6: could not find any suitable replica to move from server dec3e4a04d604f7eaaa3f077d081218a to server e1a9251ce1cf4361b8b0ede9fbfc4023
W20240503 02:00:02.025489 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0392495bb8a743099943f2bba5f15608: could not find any suitable replica to move from server 9942306855404cfd891adf227dd65eb2 to server c2ac2eb7cc264047b9ab32fe3a4fc025
W20240503 02:00:02.025547 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 805c97ce87c74e019deaa2acee6a5329: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server 9942306855404cfd891adf227dd65eb2
W20240503 02:00:02.025590 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 043b44e4ca1745038882551026afda53: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server 2a01ace80baf4057b6bf1995cf88468b
W20240503 02:00:02.025723 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e0e5e479de824fb18516cfe56d5892c6: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server 9942306855404cfd891adf227dd65eb2
W20240503 02:00:02.025764 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 54f1891be3f54f19821aad5d1dfbd85d: could not find any suitable replica to move from server 2a01ace80baf4057b6bf1995cf88468b to server 492448570cc94ed1879feb2ec3b9051d
W20240503 02:00:02.025794 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table eecab7e70bfa4621832bb99bd9ef066c: could not find any suitable replica to move from server 80fd18ef4d604c86b909d5c008a46b4f to server 99ead02915954b4fab23b033ad6d546b
W20240503 02:00:02.025825 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4e740b0167ed439291b74d77653ec386: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server 2a01ace80baf4057b6bf1995cf88468b
W20240503 02:00:02.025856 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d732c32bc9004748904a109f59db08fd: could not find any suitable replica to move from server 80fd18ef4d604c86b909d5c008a46b4f to server 9942306855404cfd891adf227dd65eb2
W20240503 02:00:02.025925 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c1969b96b61041869ddd81db8c84bf44: could not find any suitable replica to move from server 99ead02915954b4fab23b033ad6d546b to server dec3e4a04d604f7eaaa3f077d081218a
W20240503 02:00:02.025974 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5ccf0adaf2de4b79bbf38c053646e60e: could not find any suitable replica to move from server dec3e4a04d604f7eaaa3f077d081218a to server 2a01ace80baf4057b6bf1995cf88468b
W20240503 02:00:02.026050 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0392495bb8a743099943f2bba5f15608: could not find any suitable replica to move from server 80fd18ef4d604c86b909d5c008a46b4f to server 492448570cc94ed1879feb2ec3b9051d
W20240503 02:00:02.026119 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 54f1891be3f54f19821aad5d1dfbd85d: could not find any suitable replica to move from server dec3e4a04d604f7eaaa3f077d081218a to server 9942306855404cfd891adf227dd65eb2
I20240503 02:00:02.045171 9363 rebalancer_tool.cc:1074] tablet 4c13642db7fc40a5a934aa5e3c2b08fc: '80fd18ef4d604c86b909d5c008a46b4f' -> '2a01ace80baf4057b6bf1995cf88468b' move scheduled
I20240503 02:00:02.061486 9363 rebalancer_tool.cc:1074] tablet 925612b660ec471da5b3c548802ede6e: '9942306855404cfd891adf227dd65eb2' -> 'e1a9251ce1cf4361b8b0ede9fbfc4023' move scheduled
I20240503 02:00:02.081604 9363 rebalancer_tool.cc:1074] tablet 90ef3584830240acb0d8aad14ae851a6: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> '99ead02915954b4fab23b033ad6d546b' move scheduled
I20240503 02:00:02.096154 9363 rebalancer_tool.cc:1074] tablet 46ad540deb624a25b97d3b967fe1f6f4: 'dec3e4a04d604f7eaaa3f077d081218a' -> '9942306855404cfd891adf227dd65eb2' move scheduled
I20240503 02:00:02.177619 9363 rebalancer_tool.cc:1074] tablet 3cf186a0901f420390c8fda54316c5af: '492448570cc94ed1879feb2ec3b9051d' -> '2a01ace80baf4057b6bf1995cf88468b' move scheduled
I20240503 02:00:02.198985 9363 rebalancer_tool.cc:1074] tablet c3485701bf3b48f5ae2e1568afeed133: '80fd18ef4d604c86b909d5c008a46b4f' -> 'e1a9251ce1cf4361b8b0ede9fbfc4023' move scheduled
I20240503 02:00:02.216543 9363 rebalancer_tool.cc:1074] tablet af00a839bb5e402791f87288493efef6: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> '99ead02915954b4fab23b033ad6d546b' move scheduled
I20240503 02:00:02.237360 9363 rebalancer_tool.cc:1074] tablet 1e445a3cc928429caf9e3c6193a63010: 'dec3e4a04d604f7eaaa3f077d081218a' -> '9942306855404cfd891adf227dd65eb2' move scheduled
I20240503 02:00:02.284821 9363 rebalancer_tool.cc:1074] tablet 915028b3478341d588ed6031e657c259: '492448570cc94ed1879feb2ec3b9051d' -> 'e1a9251ce1cf4361b8b0ede9fbfc4023' move scheduled
I20240503 02:00:02.306334 9363 rebalancer_tool.cc:1074] tablet f28586e70bd9473fbdc6b9126b666fa9: '2a01ace80baf4057b6bf1995cf88468b' -> '80fd18ef4d604c86b909d5c008a46b4f' move scheduled
I20240503 02:00:02.323058 9363 rebalancer_tool.cc:1074] tablet ac1632cc5eb94554b1faa7eb715706ef: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> '99ead02915954b4fab23b033ad6d546b' move scheduled
I20240503 02:00:02.420534 9363 rebalancer_tool.cc:1123] tablet 34120f4948504cbabcd28e0c4f08f9f8: '492448570cc94ed1879feb2ec3b9051d' -> '99ead02915954b4fab23b033ad6d546b' move is abandoned: Not found: 34120f4948504cbabcd28e0c4f08f9f8: tablet not found
I20240503 02:00:02.463819 9363 rebalancer_tool.cc:1136] tablet 0b659e9d57db4ebf93c9143938970090: '80fd18ef4d604c86b909d5c008a46b4f' -> 'e1a9251ce1cf4361b8b0ede9fbfc4023' move completed: OK
I20240503 02:00:02.471427 9363 rebalancer_tool.cc:1136] tablet 7a4a910a8b9d4134adebb52a73dfae1d: '492448570cc94ed1879feb2ec3b9051d' -> '99ead02915954b4fab23b033ad6d546b' move completed: OK
I20240503 02:00:02.511894 9363 rebalancer_tool.cc:1074] tablet 2d7244b0dc4d419cb5c3f3677acd70c7: '492448570cc94ed1879feb2ec3b9051d' -> '99ead02915954b4fab23b033ad6d546b' move scheduled
I20240503 02:00:02.530963 9363 rebalancer_tool.cc:1074] tablet 4385d51d11574192883d01376e2d5620: 'dec3e4a04d604f7eaaa3f077d081218a' -> 'e1a9251ce1cf4361b8b0ede9fbfc4023' move scheduled
I20240503 02:00:02.672992 9363 rebalancer_tool.cc:1136] tablet 0099654461ca41efa4323fc9f5adab4a: '9942306855404cfd891adf227dd65eb2' -> '80fd18ef4d604c86b909d5c008a46b4f' move completed: OK
I20240503 02:00:02.692983 9363 rebalancer_tool.cc:1084] tablet 7b392142a4d948e7a36b49314ae15129: '80fd18ef4d604c86b909d5c008a46b4f' -> '9942306855404cfd891adf227dd65eb2' move ignored: Not found: 7b392142a4d948e7a36b49314ae15129: tablet not found
I20240503 02:00:02.693099 9363 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 02:00:03.064880 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4e740b0167ed439291b74d77653ec386: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server 2a01ace80baf4057b6bf1995cf88468b
W20240503 02:00:03.064970 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d732c32bc9004748904a109f59db08fd: could not find any suitable replica to move from server 80fd18ef4d604c86b909d5c008a46b4f to server 9942306855404cfd891adf227dd65eb2
W20240503 02:00:03.065078 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 805c97ce87c74e019deaa2acee6a5329: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server 9942306855404cfd891adf227dd65eb2
W20240503 02:00:03.065155 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5ccf0adaf2de4b79bbf38c053646e60e: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server 2a01ace80baf4057b6bf1995cf88468b
W20240503 02:00:03.065469 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0392495bb8a743099943f2bba5f15608: could not find any suitable replica to move from server 9942306855404cfd891adf227dd65eb2 to server c2ac2eb7cc264047b9ab32fe3a4fc025
W20240503 02:00:03.065640 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 58ca374f38374792b2e80195b318352c: could not find any suitable replica to move from server dec3e4a04d604f7eaaa3f077d081218a to server e1a9251ce1cf4361b8b0ede9fbfc4023
I20240503 02:00:03.098309 9363 rebalancer_tool.cc:1074] tablet cb873018448549ccbe3d14b784e47d26: '80fd18ef4d604c86b909d5c008a46b4f' -> '492448570cc94ed1879feb2ec3b9051d' move scheduled
I20240503 02:00:03.118934 9363 rebalancer_tool.cc:1074] tablet f5ed01740019467d8c1d55cc2fa18ca3: '99ead02915954b4fab23b033ad6d546b' -> '80fd18ef4d604c86b909d5c008a46b4f' move scheduled
I20240503 02:00:03.163864 9363 rebalancer_tool.cc:1074] tablet 3b189fba0d6c461694dda002a3c5eb2f: '492448570cc94ed1879feb2ec3b9051d' -> '9942306855404cfd891adf227dd65eb2' move scheduled
I20240503 02:00:03.297506 9363 rebalancer_tool.cc:1136] tablet 9f7ade1e2af24e86b6e54e2b5769e4c0: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> 'e1a9251ce1cf4361b8b0ede9fbfc4023' move completed: OK
I20240503 02:00:03.311035 9363 rebalancer_tool.cc:1136] tablet 7df4c083550046e883e5e61d3d4e1572: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> '9942306855404cfd891adf227dd65eb2' move completed: OK
I20240503 02:00:03.340684 9363 rebalancer_tool.cc:1136] tablet 8a76d3e2d4264687aa2972c179341b5e: 'dec3e4a04d604f7eaaa3f077d081218a' -> '2a01ace80baf4057b6bf1995cf88468b' move completed: OK
I20240503 02:00:03.355592 9363 rebalancer_tool.cc:1074] tablet d0d5dd42db3e4883af7fd16d8813b550: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> 'e1a9251ce1cf4361b8b0ede9fbfc4023' move scheduled
I20240503 02:00:03.383069 9363 rebalancer_tool.cc:1074] tablet 485d1d89a8b94505969d663be92124cd: 'dec3e4a04d604f7eaaa3f077d081218a' -> '9942306855404cfd891adf227dd65eb2' move scheduled
I20240503 02:00:03.530129 9363 rebalancer_tool.cc:1136] tablet 925612b660ec471da5b3c548802ede6e: '9942306855404cfd891adf227dd65eb2' -> 'e1a9251ce1cf4361b8b0ede9fbfc4023' move completed: OK
I20240503 02:00:03.545480 9363 rebalancer_tool.cc:1123] tablet 4c13642db7fc40a5a934aa5e3c2b08fc: '80fd18ef4d604c86b909d5c008a46b4f' -> '2a01ace80baf4057b6bf1995cf88468b' move is abandoned: Not found: 4c13642db7fc40a5a934aa5e3c2b08fc: tablet not found
I20240503 02:00:03.553505 9363 rebalancer_tool.cc:1136] tablet 2fc834abc6a94c638d6202b03320a2d2: 'dec3e4a04d604f7eaaa3f077d081218a' -> '2a01ace80baf4057b6bf1995cf88468b' move completed: OK
I20240503 02:00:03.594683 9363 rebalancer_tool.cc:1074] tablet 96c395771fd44056abacceb6c1acdb9f: '2a01ace80baf4057b6bf1995cf88468b' -> 'dec3e4a04d604f7eaaa3f077d081218a' move scheduled
I20240503 02:00:03.595628 9363 rebalancer_tool.cc:1084] tablet 0b659e9d57db4ebf93c9143938970090: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> '9942306855404cfd891adf227dd65eb2' move ignored: Not found: 0b659e9d57db4ebf93c9143938970090: tablet not found
I20240503 02:00:03.595695 9363 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 02:00:03.943215 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5ccf0adaf2de4b79bbf38c053646e60e: could not find any suitable replica to move from server dec3e4a04d604f7eaaa3f077d081218a to server 2a01ace80baf4057b6bf1995cf88468b
W20240503 02:00:03.943372 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 805c97ce87c74e019deaa2acee6a5329: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server 9942306855404cfd891adf227dd65eb2
W20240503 02:00:03.943480 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0392495bb8a743099943f2bba5f15608: could not find any suitable replica to move from server 99ead02915954b4fab23b033ad6d546b to server c2ac2eb7cc264047b9ab32fe3a4fc025
W20240503 02:00:03.944453 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 404345ac1896484ca34a8c78c6770131: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server 492448570cc94ed1879feb2ec3b9051d
W20240503 02:00:03.944505 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 72f29547da0b4a95974018c2782cb359: could not find any suitable replica to move from server 80fd18ef4d604c86b909d5c008a46b4f to server 2a01ace80baf4057b6bf1995cf88468b
W20240503 02:00:03.944571 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e43866993a7646be8fc624a6f2150d34: could not find any suitable replica to move from server 9942306855404cfd891adf227dd65eb2 to server c2ac2eb7cc264047b9ab32fe3a4fc025
W20240503 02:00:03.944643 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5ccf0adaf2de4b79bbf38c053646e60e: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server e1a9251ce1cf4361b8b0ede9fbfc4023
W20240503 02:00:03.944693 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4e740b0167ed439291b74d77653ec386: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server dec3e4a04d604f7eaaa3f077d081218a
I20240503 02:00:03.976511 9363 rebalancer_tool.cc:1074] tablet 7df4c083550046e883e5e61d3d4e1572: '9942306855404cfd891adf227dd65eb2' -> '2a01ace80baf4057b6bf1995cf88468b' move scheduled
I20240503 02:00:04.011806 9363 rebalancer_tool.cc:1074] tablet 51b82ef8e4cc474387f6902f2e526ac0: 'e1a9251ce1cf4361b8b0ede9fbfc4023' -> 'c2ac2eb7cc264047b9ab32fe3a4fc025' move scheduled
I20240503 02:00:04.052577 9363 rebalancer_tool.cc:1074] tablet 1fa274c98fc147f29344ed537e2ccf17: '80fd18ef4d604c86b909d5c008a46b4f' -> '2a01ace80baf4057b6bf1995cf88468b' move scheduled
I20240503 02:00:04.121596 9363 rebalancer_tool.cc:1123] tablet af00a839bb5e402791f87288493efef6: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> '99ead02915954b4fab23b033ad6d546b' move is abandoned: Not found: af00a839bb5e402791f87288493efef6: tablet not found
I20240503 02:00:04.148805 9363 rebalancer_tool.cc:1136] tablet 915028b3478341d588ed6031e657c259: '492448570cc94ed1879feb2ec3b9051d' -> 'e1a9251ce1cf4361b8b0ede9fbfc4023' move completed: OK
I20240503 02:00:04.160297 9363 rebalancer_tool.cc:1136] tablet 3cf186a0901f420390c8fda54316c5af: '492448570cc94ed1879feb2ec3b9051d' -> '2a01ace80baf4057b6bf1995cf88468b' move completed: OK
I20240503 02:00:04.185410 9363 rebalancer_tool.cc:1136] tablet 90ef3584830240acb0d8aad14ae851a6: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> '99ead02915954b4fab23b033ad6d546b' move completed: OK
I20240503 02:00:04.216631 9363 rebalancer_tool.cc:1136] tablet 1e445a3cc928429caf9e3c6193a63010: 'dec3e4a04d604f7eaaa3f077d081218a' -> '9942306855404cfd891adf227dd65eb2' move completed: OK
I20240503 02:00:04.226469 9363 rebalancer_tool.cc:1136] tablet 2d7244b0dc4d419cb5c3f3677acd70c7: '492448570cc94ed1879feb2ec3b9051d' -> '99ead02915954b4fab23b033ad6d546b' move completed: OK
I20240503 02:00:04.241597 9363 rebalancer_tool.cc:1136] tablet f28586e70bd9473fbdc6b9126b666fa9: '2a01ace80baf4057b6bf1995cf88468b' -> '80fd18ef4d604c86b909d5c008a46b4f' move completed: OK
I20240503 02:00:04.263391 9363 rebalancer_tool.cc:1136] tablet ac1632cc5eb94554b1faa7eb715706ef: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> '99ead02915954b4fab23b033ad6d546b' move completed: OK
I20240503 02:00:04.271962 9363 rebalancer_tool.cc:1136] tablet 4385d51d11574192883d01376e2d5620: 'dec3e4a04d604f7eaaa3f077d081218a' -> 'e1a9251ce1cf4361b8b0ede9fbfc4023' move completed: OK
I20240503 02:00:04.293192 9363 rebalancer_tool.cc:1074] tablet 3dc213593e4048c99ccbe10161ff8e7f: '492448570cc94ed1879feb2ec3b9051d' -> '99ead02915954b4fab23b033ad6d546b' move scheduled
I20240503 02:00:04.308706 9363 rebalancer_tool.cc:1074] tablet 0ee05e70ccbb496ca3cff08dc38112e8: '2a01ace80baf4057b6bf1995cf88468b' -> 'dec3e4a04d604f7eaaa3f077d081218a' move scheduled
I20240503 02:00:04.325034 9363 rebalancer_tool.cc:1074] tablet d897c21e630d4770832bb10f43778e82: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> 'e1a9251ce1cf4361b8b0ede9fbfc4023' move scheduled
I20240503 02:00:04.341096 9363 rebalancer_tool.cc:1074] tablet 3416b6107e1848f299ac49ccd88ae7c8: '492448570cc94ed1879feb2ec3b9051d' -> '9942306855404cfd891adf227dd65eb2' move scheduled
I20240503 02:00:04.356585 9363 rebalancer_tool.cc:1074] tablet 2e967e377984468d90799041c29882bd: '80fd18ef4d604c86b909d5c008a46b4f' -> '99ead02915954b4fab23b033ad6d546b' move scheduled
I20240503 02:00:04.396317 9363 rebalancer_tool.cc:1074] tablet 067f54011a37432d99157884e70fc601: '2a01ace80baf4057b6bf1995cf88468b' -> '492448570cc94ed1879feb2ec3b9051d' move scheduled
I20240503 02:00:04.416498 9363 rebalancer_tool.cc:1074] tablet 9c8a53980b00421d8c2bd3ae73f0c8cc: 'dec3e4a04d604f7eaaa3f077d081218a' -> 'c2ac2eb7cc264047b9ab32fe3a4fc025' move scheduled
I20240503 02:00:04.962145 9363 rebalancer_tool.cc:1136] tablet d0d5dd42db3e4883af7fd16d8813b550: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> 'e1a9251ce1cf4361b8b0ede9fbfc4023' move completed: OK
I20240503 02:00:05.020974 9363 rebalancer_tool.cc:1136] tablet 3b189fba0d6c461694dda002a3c5eb2f: '492448570cc94ed1879feb2ec3b9051d' -> '9942306855404cfd891adf227dd65eb2' move completed: OK
I20240503 02:00:05.074697 9363 rebalancer_tool.cc:1136] tablet f5ed01740019467d8c1d55cc2fa18ca3: '99ead02915954b4fab23b033ad6d546b' -> '80fd18ef4d604c86b909d5c008a46b4f' move completed: OK
I20240503 02:00:05.120497 9363 rebalancer_tool.cc:1074] tablet 8246488dec204bf6b09e868d7951ac5a: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> 'e1a9251ce1cf4361b8b0ede9fbfc4023' move scheduled
I20240503 02:00:05.203215 9363 rebalancer_tool.cc:1136] tablet 485d1d89a8b94505969d663be92124cd: 'dec3e4a04d604f7eaaa3f077d081218a' -> '9942306855404cfd891adf227dd65eb2' move completed: OK
I20240503 02:00:05.256238 9363 rebalancer_tool.cc:1136] tablet 96c395771fd44056abacceb6c1acdb9f: '2a01ace80baf4057b6bf1995cf88468b' -> 'dec3e4a04d604f7eaaa3f077d081218a' move completed: OK
I20240503 02:00:05.270459 9363 rebalancer_tool.cc:1074] tablet b91a933fb66b4075acb40c5b92ef4824: 'dec3e4a04d604f7eaaa3f077d081218a' -> '9942306855404cfd891adf227dd65eb2' move scheduled
I20240503 02:00:05.300743 9363 rebalancer_tool.cc:1074] tablet 6a82dcc9be5b4c93b5fff03f789350ab: 'e1a9251ce1cf4361b8b0ede9fbfc4023' -> 'dec3e4a04d604f7eaaa3f077d081218a' move scheduled
I20240503 02:00:05.422181 9363 rebalancer_tool.cc:1136] tablet 46ad540deb624a25b97d3b967fe1f6f4: 'dec3e4a04d604f7eaaa3f077d081218a' -> '9942306855404cfd891adf227dd65eb2' move completed: OK
I20240503 02:00:05.518419 9363 rebalancer_tool.cc:1136] tablet 51b82ef8e4cc474387f6902f2e526ac0: 'e1a9251ce1cf4361b8b0ede9fbfc4023' -> 'c2ac2eb7cc264047b9ab32fe3a4fc025' move completed: OK
I20240503 02:00:05.628218 9363 rebalancer_tool.cc:1074] tablet ac7fa0c3722a461c91fb237fd85794a3: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> '99ead02915954b4fab23b033ad6d546b' move scheduled
I20240503 02:00:05.662603 9363 rebalancer_tool.cc:1074] tablet 1feaf29d8c6b4842829ce66e0f4ec23b: 'dec3e4a04d604f7eaaa3f077d081218a' -> 'e1a9251ce1cf4361b8b0ede9fbfc4023' move scheduled
I20240503 02:00:05.811722 9363 rebalancer_tool.cc:1136] tablet 3416b6107e1848f299ac49ccd88ae7c8: '492448570cc94ed1879feb2ec3b9051d' -> '9942306855404cfd891adf227dd65eb2' move completed: OK
I20240503 02:00:05.851882 9363 rebalancer_tool.cc:1136] tablet 3dc213593e4048c99ccbe10161ff8e7f: '492448570cc94ed1879feb2ec3b9051d' -> '99ead02915954b4fab23b033ad6d546b' move completed: OK
I20240503 02:00:05.864341 9363 rebalancer_tool.cc:1136] tablet 1fa274c98fc147f29344ed537e2ccf17: '80fd18ef4d604c86b909d5c008a46b4f' -> '2a01ace80baf4057b6bf1995cf88468b' move completed: OK
I20240503 02:00:05.884183 9363 rebalancer_tool.cc:1136] tablet c3485701bf3b48f5ae2e1568afeed133: '80fd18ef4d604c86b909d5c008a46b4f' -> 'e1a9251ce1cf4361b8b0ede9fbfc4023' move completed: OK
I20240503 02:00:05.890595 9363 rebalancer_tool.cc:1136] tablet 9c8a53980b00421d8c2bd3ae73f0c8cc: 'dec3e4a04d604f7eaaa3f077d081218a' -> 'c2ac2eb7cc264047b9ab32fe3a4fc025' move completed: OK
I20240503 02:00:05.900506 9363 rebalancer_tool.cc:1136] tablet 7df4c083550046e883e5e61d3d4e1572: '9942306855404cfd891adf227dd65eb2' -> '2a01ace80baf4057b6bf1995cf88468b' move completed: OK
I20240503 02:00:05.935667 9363 rebalancer_tool.cc:1074] tablet 51c0137930464635be1e6a97f5e9d952: '9942306855404cfd891adf227dd65eb2' -> 'c2ac2eb7cc264047b9ab32fe3a4fc025' move scheduled
I20240503 02:00:05.961105 9363 rebalancer_tool.cc:1074] tablet 1c721b33983b4c47afd8faa8f155ca84: 'e1a9251ce1cf4361b8b0ede9fbfc4023' -> 'dec3e4a04d604f7eaaa3f077d081218a' move scheduled
I20240503 02:00:06.015184 9363 rebalancer_tool.cc:1136] tablet 2e967e377984468d90799041c29882bd: '80fd18ef4d604c86b909d5c008a46b4f' -> '99ead02915954b4fab23b033ad6d546b' move completed: OK
I20240503 02:00:06.152053 9363 rebalancer_tool.cc:1136] tablet 067f54011a37432d99157884e70fc601: '2a01ace80baf4057b6bf1995cf88468b' -> '492448570cc94ed1879feb2ec3b9051d' move completed: OK
I20240503 02:00:06.525871 9363 rebalancer_tool.cc:1136] tablet 8246488dec204bf6b09e868d7951ac5a: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> 'e1a9251ce1cf4361b8b0ede9fbfc4023' move completed: OK
I20240503 02:00:06.564509 9363 rebalancer_tool.cc:1136] tablet cb873018448549ccbe3d14b784e47d26: '80fd18ef4d604c86b909d5c008a46b4f' -> '492448570cc94ed1879feb2ec3b9051d' move completed: OK
I20240503 02:00:06.880039 9363 rebalancer_tool.cc:1136] tablet b91a933fb66b4075acb40c5b92ef4824: 'dec3e4a04d604f7eaaa3f077d081218a' -> '9942306855404cfd891adf227dd65eb2' move completed: OK
I20240503 02:00:06.917583 9363 rebalancer_tool.cc:1136] tablet d897c21e630d4770832bb10f43778e82: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> 'e1a9251ce1cf4361b8b0ede9fbfc4023' move completed: OK
I20240503 02:00:06.973349 9363 rebalancer_tool.cc:1136] tablet 6a82dcc9be5b4c93b5fff03f789350ab: 'e1a9251ce1cf4361b8b0ede9fbfc4023' -> 'dec3e4a04d604f7eaaa3f077d081218a' move completed: OK
I20240503 02:00:07.037731 9363 rebalancer_tool.cc:1136] tablet 1feaf29d8c6b4842829ce66e0f4ec23b: 'dec3e4a04d604f7eaaa3f077d081218a' -> 'e1a9251ce1cf4361b8b0ede9fbfc4023' move completed: OK
I20240503 02:00:07.070804 9363 rebalancer_tool.cc:1136] tablet ac7fa0c3722a461c91fb237fd85794a3: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> '99ead02915954b4fab23b033ad6d546b' move completed: OK
I20240503 02:00:07.297852 9363 rebalancer_tool.cc:1136] tablet 51c0137930464635be1e6a97f5e9d952: '9942306855404cfd891adf227dd65eb2' -> 'c2ac2eb7cc264047b9ab32fe3a4fc025' move completed: OK
I20240503 02:00:07.539635 9363 rebalancer_tool.cc:1136] tablet 0ee05e70ccbb496ca3cff08dc38112e8: '2a01ace80baf4057b6bf1995cf88468b' -> 'dec3e4a04d604f7eaaa3f077d081218a' move completed: OK
I20240503 02:00:07.772747 9363 rebalancer_tool.cc:1136] tablet 1c721b33983b4c47afd8faa8f155ca84: 'e1a9251ce1cf4361b8b0ede9fbfc4023' -> 'dec3e4a04d604f7eaaa3f077d081218a' move completed: OK
W20240503 02:00:07.984575 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table dd81a75146044b9d9fe1ec1f86a7aebd: could not find any suitable replica to move from server 9942306855404cfd891adf227dd65eb2 to server 80fd18ef4d604c86b909d5c008a46b4f
W20240503 02:00:07.984735 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table dd81a75146044b9d9fe1ec1f86a7aebd: could not find any suitable replica to move from server e1a9251ce1cf4361b8b0ede9fbfc4023 to server 492448570cc94ed1879feb2ec3b9051d
W20240503 02:00:07.984961 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table dd81a75146044b9d9fe1ec1f86a7aebd: could not find any suitable replica to move from server 9942306855404cfd891adf227dd65eb2 to server 99ead02915954b4fab23b033ad6d546b
I20240503 02:00:08.009354 9363 rebalancer_tool.cc:1074] tablet beeea71f02ae4c89a1af13d3584ef70a: '2a01ace80baf4057b6bf1995cf88468b' -> '80fd18ef4d604c86b909d5c008a46b4f' move scheduled
I20240503 02:00:08.039716 9363 rebalancer_tool.cc:1074] tablet 96c395771fd44056abacceb6c1acdb9f: '99ead02915954b4fab23b033ad6d546b' -> 'c2ac2eb7cc264047b9ab32fe3a4fc025' move scheduled
I20240503 02:00:08.061971 9363 rebalancer_tool.cc:1074] tablet b2e62c043adf4a8d9024279d786f596c: '2a01ace80baf4057b6bf1995cf88468b' -> '9942306855404cfd891adf227dd65eb2' move scheduled
I20240503 02:00:08.081597 9363 rebalancer_tool.cc:1074] tablet cea1ea50fa434d31b42451f84f26bc76: '99ead02915954b4fab23b033ad6d546b' -> '492448570cc94ed1879feb2ec3b9051d' move scheduled
I20240503 02:00:08.098927 9363 rebalancer_tool.cc:1074] tablet b20ff3a6f74f4640afebfd4059b58699: '99ead02915954b4fab23b033ad6d546b' -> 'dec3e4a04d604f7eaaa3f077d081218a' move scheduled
I20240503 02:00:08.113816 9363 rebalancer_tool.cc:1074] tablet aa281822047841fab7c58b53d9733e49: 'e1a9251ce1cf4361b8b0ede9fbfc4023' -> 'c2ac2eb7cc264047b9ab32fe3a4fc025' move scheduled
I20240503 02:00:08.132983 9363 rebalancer_tool.cc:1074] tablet ac7fa0c3722a461c91fb237fd85794a3: '80fd18ef4d604c86b909d5c008a46b4f' -> '2a01ace80baf4057b6bf1995cf88468b' move scheduled
I20240503 02:00:08.150460 9363 rebalancer_tool.cc:1074] tablet 1c721b33983b4c47afd8faa8f155ca84: '9942306855404cfd891adf227dd65eb2' -> 'c2ac2eb7cc264047b9ab32fe3a4fc025' move scheduled
I20240503 02:00:08.186134 9363 rebalancer_tool.cc:1074] tablet bd8b9532bd744ecdbe8da66a1225724b: '99ead02915954b4fab23b033ad6d546b' -> 'e1a9251ce1cf4361b8b0ede9fbfc4023' move scheduled
I20240503 02:00:08.207420 9363 rebalancer_tool.cc:1074] tablet e52373099af3428497d176e3fcd8ef27: '2a01ace80baf4057b6bf1995cf88468b' -> '80fd18ef4d604c86b909d5c008a46b4f' move scheduled
I20240503 02:00:08.225680 9363 rebalancer_tool.cc:1074] tablet d73ea6e550a04c04a2695b058e38c566: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> '9942306855404cfd891adf227dd65eb2' move scheduled
I20240503 02:00:08.243218 9363 rebalancer_tool.cc:1074] tablet 9cbb2be1fda74ebe8997123c70629ba3: '99ead02915954b4fab23b033ad6d546b' -> '80fd18ef4d604c86b909d5c008a46b4f' move scheduled
I20240503 02:00:08.296453 9363 rebalancer_tool.cc:1074] tablet 2d7244b0dc4d419cb5c3f3677acd70c7: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> '2a01ace80baf4057b6bf1995cf88468b' move scheduled
W20240503 02:00:08.918651 13514 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 02:00:08.788636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.136:45311 (local address 127.0.0.1:45750)
0503 02:00:08.792921 (+ 4285us) negotiation.cc:107] Waiting for socket to connect
0503 02:00:08.792939 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0503 02:00:08.793046 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 02:00:08.917818 (+124772us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":4179,"thread_start_us":122,"threads_started":1}
I20240503 02:00:08.919864 9363 rebalancer_tool.cc:1123] tablet b2e62c043adf4a8d9024279d786f596c: '2a01ace80baf4057b6bf1995cf88468b' -> '9942306855404cfd891adf227dd65eb2' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:00:08.941910 9363 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): unacceptable health status UNAVAILABLE
I20240503 02:00:09.242514 9179 raft_consensus.cc:1203] T ac7fa0c3722a461c91fb237fd85794a3 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 02:00:09.251286 13626 raft_consensus.cc:482] T be7bbd71b99847068f415cc3d3d49ddd P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:09.251439 13626 raft_consensus.cc:504] T be7bbd71b99847068f415cc3d3d49ddd P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:09.251957 13626 leader_election.cc:288] T be7bbd71b99847068f415cc3d3d49ddd P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:09.252473 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be7bbd71b99847068f415cc3d3d49ddd" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:09.252631 8916 raft_consensus.cc:2454] T be7bbd71b99847068f415cc3d3d49ddd P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:09.252621 6720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be7bbd71b99847068f415cc3d3d49ddd" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 02:00:09.252772 6720 raft_consensus.cc:2454] T be7bbd71b99847068f415cc3d3d49ddd P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:09.252967 6394 leader_election.cc:302] T be7bbd71b99847068f415cc3d3d49ddd P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9942306855404cfd891adf227dd65eb2, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:09.253165 13626 raft_consensus.cc:2781] T be7bbd71b99847068f415cc3d3d49ddd P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:09.253255 13626 raft_consensus.cc:482] T be7bbd71b99847068f415cc3d3d49ddd P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:09.253315 13626 raft_consensus.cc:3037] T be7bbd71b99847068f415cc3d3d49ddd P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:09.254390 13626 raft_consensus.cc:504] T be7bbd71b99847068f415cc3d3d49ddd P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:09.254801 13626 leader_election.cc:288] T be7bbd71b99847068f415cc3d3d49ddd P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Requested vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:09.255148 6720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be7bbd71b99847068f415cc3d3d49ddd" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 02:00:09.255177 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be7bbd71b99847068f415cc3d3d49ddd" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:09.255293 6720 raft_consensus.cc:3037] T be7bbd71b99847068f415cc3d3d49ddd P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:09.255293 8916 raft_consensus.cc:3037] T be7bbd71b99847068f415cc3d3d49ddd P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:09.256351 6720 raft_consensus.cc:2454] T be7bbd71b99847068f415cc3d3d49ddd P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:09.256352 8916 raft_consensus.cc:2454] T be7bbd71b99847068f415cc3d3d49ddd P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:09.256716 6394 leader_election.cc:302] T be7bbd71b99847068f415cc3d3d49ddd P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2ac2eb7cc264047b9ab32fe3a4fc025, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:09.257027 13626 raft_consensus.cc:2781] T be7bbd71b99847068f415cc3d3d49ddd P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:09.257144 13626 raft_consensus.cc:686] T be7bbd71b99847068f415cc3d3d49ddd P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Becoming Leader. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Running, Role: LEADER
I20240503 02:00:09.257329 13626 consensus_queue.cc:237] T be7bbd71b99847068f415cc3d3d49ddd P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:09.259914 6308 catalog_manager.cc:5484] T be7bbd71b99847068f415cc3d3d49ddd P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: term changed from 0 to 1, leader changed from <none> to dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129). New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:09.275573 6309 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:09.275993 8916 raft_consensus.cc:1203] T d73ea6e550a04c04a2695b058e38c566 P 9942306855404cfd891adf227dd65eb2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:00:09.281991 13615 tablet.cc:1722] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:09.282349 13615 tablet.cc:1620] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 80fd18ef4d604c86b909d5c008a46b4f: MemRowSet was empty: no flush needed.
I20240503 02:00:09.282691 13622 tablet.cc:1722] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:09.282961 13622 tablet.cc:1620] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
I20240503 02:00:09.284221 13616 tablet.cc:1722] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:09.284572 13616 tablet.cc:1620] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 99ead02915954b4fab23b033ad6d546b: MemRowSet was empty: no flush needed.
I20240503 02:00:09.289158 6308 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=73d45da493d442e489ee6add747cad2b] alter complete (version 2)
I20240503 02:00:09.334766 13623 raft_consensus.cc:482] T 7cecefe8f1864583be9d0b9c308a78db P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:09.334932 13623 raft_consensus.cc:504] T 7cecefe8f1864583be9d0b9c308a78db P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:09.335323 13623 leader_election.cc:288] T 7cecefe8f1864583be9d0b9c308a78db P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:09.335752 9039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cecefe8f1864583be9d0b9c308a78db" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:09.335739 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cecefe8f1864583be9d0b9c308a78db" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:09.335882 9179 raft_consensus.cc:2454] T 7cecefe8f1864583be9d0b9c308a78db P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 02:00:09.335887 9039 raft_consensus.cc:2454] T 7cecefe8f1864583be9d0b9c308a78db P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 02:00:09.336164 6784 leader_election.cc:302] T 7cecefe8f1864583be9d0b9c308a78db P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 492448570cc94ed1879feb2ec3b9051d; no voters:
I20240503 02:00:09.336354 13623 raft_consensus.cc:2781] T 7cecefe8f1864583be9d0b9c308a78db P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:09.336467 13623 raft_consensus.cc:482] T 7cecefe8f1864583be9d0b9c308a78db P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:09.336503 13623 raft_consensus.cc:3037] T 7cecefe8f1864583be9d0b9c308a78db P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:09.337455 13623 raft_consensus.cc:504] T 7cecefe8f1864583be9d0b9c308a78db P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:09.338119 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cecefe8f1864583be9d0b9c308a78db" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:09.338254 9179 raft_consensus.cc:3037] T 7cecefe8f1864583be9d0b9c308a78db P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:09.339171 9179 raft_consensus.cc:2454] T 7cecefe8f1864583be9d0b9c308a78db P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 02:00:09.339438 13623 leader_election.cc:288] T 7cecefe8f1864583be9d0b9c308a78db P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Requested vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:09.339490 6784 leader_election.cc:302] T 7cecefe8f1864583be9d0b9c308a78db P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 492448570cc94ed1879feb2ec3b9051d; no voters:
I20240503 02:00:09.339615 13623 raft_consensus.cc:2781] T 7cecefe8f1864583be9d0b9c308a78db P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:09.339692 13623 raft_consensus.cc:686] T 7cecefe8f1864583be9d0b9c308a78db P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Becoming Leader. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Running, Role: LEADER
I20240503 02:00:09.339834 13623 consensus_queue.cc:237] T 7cecefe8f1864583be9d0b9c308a78db P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:09.339967 9039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cecefe8f1864583be9d0b9c308a78db" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:09.340085 9039 raft_consensus.cc:3037] T 7cecefe8f1864583be9d0b9c308a78db P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:09.341111 9039 raft_consensus.cc:2454] T 7cecefe8f1864583be9d0b9c308a78db P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 02:00:09.342141 6309 catalog_manager.cc:5484] T 7cecefe8f1864583be9d0b9c308a78db P 492448570cc94ed1879feb2ec3b9051d reported cstate change: term changed from 0 to 1, leader changed from <none> to 492448570cc94ed1879feb2ec3b9051d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:09.374761 13102 raft_consensus.cc:1050] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2: attempting to promote NON_VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 to VOTER
I20240503 02:00:09.375222 13102 consensus_queue.cc:237] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } }
I20240503 02:00:09.377110 6456 raft_consensus.cc:1261] T 96c395771fd44056abacceb6c1acdb9f P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:09.377116 9039 raft_consensus.cc:1261] T 96c395771fd44056abacceb6c1acdb9f P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:09.377161 6720 raft_consensus.cc:1261] T 96c395771fd44056abacceb6c1acdb9f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEARNER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:09.377544 13634 consensus_queue.cc:1035] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:09.377786 13559 consensus_queue.cc:1035] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:09.378131 13018 consensus_queue.cc:1035] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:09.380782 13559 raft_consensus.cc:2932] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } } }
I20240503 02:00:09.382921 6720 raft_consensus.cc:2932] T 96c395771fd44056abacceb6c1acdb9f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } } }
I20240503 02:00:09.382957 6456 raft_consensus.cc:2932] T 96c395771fd44056abacceb6c1acdb9f P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } } }
I20240503 02:00:09.383731 6309 catalog_manager.cc:5484] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2 reported cstate change: config changed from index 5 to 6, c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9942306855404cfd891adf227dd65eb2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:09.389151 8916 consensus_queue.cc:237] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } }
I20240503 02:00:09.390475 6720 raft_consensus.cc:1261] T 96c395771fd44056abacceb6c1acdb9f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:09.390434 6456 raft_consensus.cc:1261] T 96c395771fd44056abacceb6c1acdb9f P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:09.390852 13634 consensus_queue.cc:1035] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:09.391017 13018 consensus_queue.cc:1035] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:09.392221 6589 raft_consensus.cc:1203] T 9cbb2be1fda74ebe8997123c70629ba3 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:00:09.392834 13102 raft_consensus.cc:2932] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } } }
I20240503 02:00:09.393219 6456 raft_consensus.cc:2932] T 96c395771fd44056abacceb6c1acdb9f P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } } }
I20240503 02:00:09.393646 6308 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:09.395148 6719 raft_consensus.cc:2932] T 96c395771fd44056abacceb6c1acdb9f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } } }
I20240503 02:00:09.395861 6303 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 96c395771fd44056abacceb6c1acdb9f with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:09.396762 6309 catalog_manager.cc:5484] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2 reported cstate change: config changed from index 6 to 7, VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "9942306855404cfd891adf227dd65eb2" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:09.400933 9028 tablet_service.cc:1514] Processing DeleteTablet for tablet 96c395771fd44056abacceb6c1acdb9f with delete_type TABLET_DATA_TOMBSTONED (TS 99ead02915954b4fab23b033ad6d546b not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:52584
I20240503 02:00:09.402688 13638 tablet_replica.cc:330] T 96c395771fd44056abacceb6c1acdb9f P 99ead02915954b4fab23b033ad6d546b: stopping tablet replica
I20240503 02:00:09.402710 13615 tablet.cc:1722] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:09.402840 13638 raft_consensus.cc:2229] T 96c395771fd44056abacceb6c1acdb9f P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:09.402974 13638 pending_rounds.cc:62] T 96c395771fd44056abacceb6c1acdb9f P 99ead02915954b4fab23b033ad6d546b: Trying to abort 1 pending ops.
I20240503 02:00:09.403020 13638 pending_rounds.cc:68] T 96c395771fd44056abacceb6c1acdb9f P 99ead02915954b4fab23b033ad6d546b: Aborting op as it isn't in flight: id { term: 1 index: 6 } timestamp: 7023417791999979520 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "96c395771fd44056abacceb6c1acdb9f" old_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: true } } } new_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } } } }
I20240503 02:00:09.403198 13638 raft_consensus.cc:2894] T 96c395771fd44056abacceb6c1acdb9f P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Aborting config change with OpId 1.6: Aborted: Op aborted
I20240503 02:00:09.403496 13622 tablet.cc:1722] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:09.403645 13638 raft_consensus.cc:2866] T 96c395771fd44056abacceb6c1acdb9f P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 02:00:09.403712 13638 raft_consensus.cc:2258] T 96c395771fd44056abacceb6c1acdb9f P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:09.404333 13638 ts_tablet_manager.cc:1875] T 96c395771fd44056abacceb6c1acdb9f P 99ead02915954b4fab23b033ad6d546b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:09.408407 13638 ts_tablet_manager.cc:1888] T 96c395771fd44056abacceb6c1acdb9f P 99ead02915954b4fab23b033ad6d546b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:00:09.408499 13638 log.cc:1196] T 96c395771fd44056abacceb6c1acdb9f P 99ead02915954b4fab23b033ad6d546b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/96c395771fd44056abacceb6c1acdb9f
I20240503 02:00:09.408371 6308 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=73d45da493d442e489ee6add747cad2b] alter complete (version 3)
I20240503 02:00:09.409436 6301 catalog_manager.cc:4830] TS 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157): tablet 96c395771fd44056abacceb6c1acdb9f (table rebalancer_extra_table_6 [id=0392495bb8a743099943f2bba5f15608]) successfully deleted
I20240503 02:00:09.411470 13616 tablet.cc:1722] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:09.464993 13496 raft_consensus.cc:482] T e617d4e31eed42dcb39e03cc40b8e6a6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:09.465134 13496 raft_consensus.cc:504] T e617d4e31eed42dcb39e03cc40b8e6a6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:09.465555 13496 leader_election.cc:288] T e617d4e31eed42dcb39e03cc40b8e6a6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:09.465992 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e617d4e31eed42dcb39e03cc40b8e6a6" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:09.466059 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e617d4e31eed42dcb39e03cc40b8e6a6" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:09.466147 9179 raft_consensus.cc:2454] T e617d4e31eed42dcb39e03cc40b8e6a6 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:09.466204 8916 raft_consensus.cc:2454] T e617d4e31eed42dcb39e03cc40b8e6a6 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:09.466459 6653 leader_election.cc:302] T e617d4e31eed42dcb39e03cc40b8e6a6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:09.466605 13496 raft_consensus.cc:2781] T e617d4e31eed42dcb39e03cc40b8e6a6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:09.466665 13496 raft_consensus.cc:482] T e617d4e31eed42dcb39e03cc40b8e6a6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:09.466703 13496 raft_consensus.cc:3037] T e617d4e31eed42dcb39e03cc40b8e6a6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:09.467612 13496 raft_consensus.cc:504] T e617d4e31eed42dcb39e03cc40b8e6a6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:09.467952 13496 leader_election.cc:288] T e617d4e31eed42dcb39e03cc40b8e6a6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Requested vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:09.468405 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e617d4e31eed42dcb39e03cc40b8e6a6" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:09.468408 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e617d4e31eed42dcb39e03cc40b8e6a6" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:09.468533 8916 raft_consensus.cc:3037] T e617d4e31eed42dcb39e03cc40b8e6a6 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:09.468533 9179 raft_consensus.cc:3037] T e617d4e31eed42dcb39e03cc40b8e6a6 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:09.469285 9179 raft_consensus.cc:2454] T e617d4e31eed42dcb39e03cc40b8e6a6 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:09.469295 8916 raft_consensus.cc:2454] T e617d4e31eed42dcb39e03cc40b8e6a6 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:09.469573 6653 leader_election.cc:302] T e617d4e31eed42dcb39e03cc40b8e6a6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:09.469740 13496 raft_consensus.cc:2781] T e617d4e31eed42dcb39e03cc40b8e6a6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:09.469826 13496 raft_consensus.cc:686] T e617d4e31eed42dcb39e03cc40b8e6a6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Becoming Leader. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Running, Role: LEADER
I20240503 02:00:09.469974 13496 consensus_queue.cc:237] T e617d4e31eed42dcb39e03cc40b8e6a6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:09.472131 6313 catalog_manager.cc:5484] T e617d4e31eed42dcb39e03cc40b8e6a6 P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:09.509586 12922 raft_consensus.cc:1050] T beeea71f02ae4c89a1af13d3584ef70a P 99ead02915954b4fab23b033ad6d546b: attempting to promote NON_VOTER 80fd18ef4d604c86b909d5c008a46b4f to VOTER
I20240503 02:00:09.510079 12922 consensus_queue.cc:237] T beeea71f02ae4c89a1af13d3584ef70a P 99ead02915954b4fab23b033ad6d546b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } }
I20240503 02:00:09.510715 6308 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:09.510838 6308 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:09.511658 9179 raft_consensus.cc:1261] T beeea71f02ae4c89a1af13d3584ef70a P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer 99ead02915954b4fab23b033ad6d546b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:09.512012 6589 raft_consensus.cc:1261] T beeea71f02ae4c89a1af13d3584ef70a P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEARNER]: Refusing update from remote peer 99ead02915954b4fab23b033ad6d546b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:09.512326 13631 consensus_queue.cc:1035] T beeea71f02ae4c89a1af13d3584ef70a P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:09.512580 10473 consensus_queue.cc:1035] T beeea71f02ae4c89a1af13d3584ef70a P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240503 02:00:09.513459 8982 consensus_peers.cc:485] T beeea71f02ae4c89a1af13d3584ef70a P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:09.513489 6308 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet 7a90fdf10e3c4dd8be7d5e1b54e9a1da
I20240503 02:00:09.514161 9028 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a90fdf10e3c4dd8be7d5e1b54e9a1da with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:09 UTC) from {username='slave'} at 127.0.0.1:52584
I20240503 02:00:09.514338 6569 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a90fdf10e3c4dd8be7d5e1b54e9a1da with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:09 UTC) from {username='slave'} at 127.0.0.1:59846
I20240503 02:00:09.514392 9157 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a90fdf10e3c4dd8be7d5e1b54e9a1da with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:09 UTC) from {username='slave'} at 127.0.0.1:51940
I20240503 02:00:09.514489 13638 tablet_replica.cc:330] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 99ead02915954b4fab23b033ad6d546b: stopping tablet replica
I20240503 02:00:09.514778 13644 tablet_replica.cc:330] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 2a01ace80baf4057b6bf1995cf88468b: stopping tablet replica
I20240503 02:00:09.514755 13643 tablet_replica.cc:330] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 02:00:09.515065 13643 raft_consensus.cc:2229] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:09.515204 13644 raft_consensus.cc:2229] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:09.515136 13495 raft_consensus.cc:2932] T beeea71f02ae4c89a1af13d3584ef70a P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:09.515329 13643 raft_consensus.cc:2258] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:09.515347 13644 raft_consensus.cc:2258] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:09.515498 6589 raft_consensus.cc:2932] T beeea71f02ae4c89a1af13d3584ef70a P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:09.515923 13643 ts_tablet_manager.cc:1875] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:09.514896 13638 raft_consensus.cc:2229] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:09.516291 13638 raft_consensus.cc:2258] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:09.516770 13644 ts_tablet_manager.cc:1875] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 2a01ace80baf4057b6bf1995cf88468b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:09.517235 13638 ts_tablet_manager.cc:1875] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 99ead02915954b4fab23b033ad6d546b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:09.517185 9179 raft_consensus.cc:2932] T beeea71f02ae4c89a1af13d3584ef70a P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:09.518862 6308 catalog_manager.cc:5484] T beeea71f02ae4c89a1af13d3584ef70a P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: config changed from index 2 to 3, 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "99ead02915954b4fab23b033ad6d546b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:09.520248 13644 ts_tablet_manager.cc:1888] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 2a01ace80baf4057b6bf1995cf88468b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:09.520327 13644 log.cc:1196] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 2a01ace80baf4057b6bf1995cf88468b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/7a90fdf10e3c4dd8be7d5e1b54e9a1da
I20240503 02:00:09.520591 13644 ts_tablet_manager.cc:1909] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 2a01ace80baf4057b6bf1995cf88468b: Deleting consensus metadata
I20240503 02:00:09.521564 6300 catalog_manager.cc:4830] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079): tablet 7a90fdf10e3c4dd8be7d5e1b54e9a1da (table rebalancer_dynamic_table_new_name [id=73d45da493d442e489ee6add747cad2b]) successfully deleted
I20240503 02:00:09.521803 13643 ts_tablet_manager.cc:1888] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:09.521893 13643 log.cc:1196] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/7a90fdf10e3c4dd8be7d5e1b54e9a1da
I20240503 02:00:09.522207 13643 ts_tablet_manager.cc:1909] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 80fd18ef4d604c86b909d5c008a46b4f: Deleting consensus metadata
I20240503 02:00:09.523046 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet 7a90fdf10e3c4dd8be7d5e1b54e9a1da (table rebalancer_dynamic_table_new_name [id=73d45da493d442e489ee6add747cad2b]) successfully deleted
I20240503 02:00:09.523798 13638 ts_tablet_manager.cc:1888] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 99ead02915954b4fab23b033ad6d546b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:09.523873 13638 log.cc:1196] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 99ead02915954b4fab23b033ad6d546b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/7a90fdf10e3c4dd8be7d5e1b54e9a1da
I20240503 02:00:09.524065 13638 ts_tablet_manager.cc:1909] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 99ead02915954b4fab23b033ad6d546b: Deleting consensus metadata
I20240503 02:00:09.524916 6301 catalog_manager.cc:4830] TS 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157): tablet 7a90fdf10e3c4dd8be7d5e1b54e9a1da (table rebalancer_dynamic_table_new_name [id=73d45da493d442e489ee6add747cad2b]) successfully deleted
I20240503 02:00:09.546100 13631 raft_consensus.cc:1050] T aa281822047841fab7c58b53d9733e49 P 99ead02915954b4fab23b033ad6d546b: attempting to promote NON_VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 to VOTER
I20240503 02:00:09.546546 13631 consensus_queue.cc:237] T aa281822047841fab7c58b53d9733e49 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } }
I20240503 02:00:09.548002 6719 raft_consensus.cc:1261] T aa281822047841fab7c58b53d9733e49 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEARNER]: Refusing update from remote peer 99ead02915954b4fab23b033ad6d546b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:09.548475 13631 consensus_queue.cc:1035] T aa281822047841fab7c58b53d9733e49 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240503 02:00:09.550274 8982 consensus_peers.cc:485] T aa281822047841fab7c58b53d9733e49 P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:09.550779 9179 raft_consensus.cc:1261] T aa281822047841fab7c58b53d9733e49 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer 99ead02915954b4fab23b033ad6d546b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:09.551157 10473 consensus_queue.cc:1035] T aa281822047841fab7c58b53d9733e49 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:09.552815 13634 raft_consensus.cc:1050] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2: attempting to promote NON_VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 to VOTER
I20240503 02:00:09.553207 13634 consensus_queue.cc:237] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } }
I20240503 02:00:09.554414 6719 raft_consensus.cc:1261] T 1c721b33983b4c47afd8faa8f155ca84 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEARNER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:09.554920 6850 raft_consensus.cc:1261] T 1c721b33983b4c47afd8faa8f155ca84 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:09.554921 6456 raft_consensus.cc:1261] T 1c721b33983b4c47afd8faa8f155ca84 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:09.555334 13634 consensus_queue.cc:1035] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:09.555555 13018 consensus_queue.cc:1035] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:09.555740 13102 consensus_queue.cc:1035] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:09.558002 13559 raft_consensus.cc:2932] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } } }
I20240503 02:00:09.559485 6456 raft_consensus.cc:2932] T 1c721b33983b4c47afd8faa8f155ca84 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } } }
I20240503 02:00:09.561419 6313 catalog_manager.cc:5484] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2 reported cstate change: config changed from index 5 to 6, c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9942306855404cfd891adf227dd65eb2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:09.561410 6850 raft_consensus.cc:2932] T 1c721b33983b4c47afd8faa8f155ca84 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } } }
I20240503 02:00:09.561570 6719 raft_consensus.cc:2932] T 1c721b33983b4c47afd8faa8f155ca84 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } } }
I20240503 02:00:09.562937 12907 raft_consensus.cc:482] T b2e62c043adf4a8d9024279d786f596c P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e1a9251ce1cf4361b8b0ede9fbfc4023)
I20240503 02:00:09.563046 12907 raft_consensus.cc:504] T b2e62c043adf4a8d9024279d786f596c P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } }
I20240503 02:00:09.563472 12907 leader_election.cc:288] T b2e62c043adf4a8d9024279d786f596c P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311)
I20240503 02:00:09.566058 6850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2e62c043adf4a8d9024279d786f596c" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
W20240503 02:00:09.566674 9113 leader_election.cc:334] T b2e62c043adf4a8d9024279d786f596c P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:09.566754 9113 leader_election.cc:302] T b2e62c043adf4a8d9024279d786f596c P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b; no voters: 492448570cc94ed1879feb2ec3b9051d, e1a9251ce1cf4361b8b0ede9fbfc4023
I20240503 02:00:09.567063 12907 raft_consensus.cc:2726] T b2e62c043adf4a8d9024279d786f596c P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:09.569295 13631 raft_consensus.cc:2932] T aa281822047841fab7c58b53d9733e49 P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } } }
I20240503 02:00:09.569757 9179 raft_consensus.cc:2932] T aa281822047841fab7c58b53d9733e49 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } } }
I20240503 02:00:09.571523 6719 raft_consensus.cc:2932] T aa281822047841fab7c58b53d9733e49 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } } }
I20240503 02:00:09.572793 6308 catalog_manager.cc:5484] T aa281822047841fab7c58b53d9733e49 P 99ead02915954b4fab23b033ad6d546b reported cstate change: config changed from index 2 to 3, c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "99ead02915954b4fab23b033ad6d546b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:09.574023 12907 raft_consensus.cc:1050] T b20ff3a6f74f4640afebfd4059b58699 P 2a01ace80baf4057b6bf1995cf88468b: attempting to promote NON_VOTER dec3e4a04d604f7eaaa3f077d081218a to VOTER
I20240503 02:00:09.574445 12907 consensus_queue.cc:237] T b20ff3a6f74f4640afebfd4059b58699 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } }
I20240503 02:00:09.575690 6456 raft_consensus.cc:1261] T b20ff3a6f74f4640afebfd4059b58699 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEARNER]: Refusing update from remote peer 2a01ace80baf4057b6bf1995cf88468b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:09.576154 6719 raft_consensus.cc:1261] T b20ff3a6f74f4640afebfd4059b58699 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer 2a01ace80baf4057b6bf1995cf88468b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:09.576613 13629 consensus_queue.cc:1035] T b20ff3a6f74f4640afebfd4059b58699 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:09.576911 13098 consensus_queue.cc:1035] T b20ff3a6f74f4640afebfd4059b58699 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:09.577023 9039 raft_consensus.cc:1261] T b20ff3a6f74f4640afebfd4059b58699 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer 2a01ace80baf4057b6bf1995cf88468b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:09.577605 13098 consensus_queue.cc:1035] T b20ff3a6f74f4640afebfd4059b58699 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:09.578176 9039 consensus_queue.cc:237] T aa281822047841fab7c58b53d9733e49 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } }
I20240503 02:00:09.579339 9179 raft_consensus.cc:1261] T aa281822047841fab7c58b53d9733e49 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer 99ead02915954b4fab23b033ad6d546b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:09.579340 6720 raft_consensus.cc:1261] T aa281822047841fab7c58b53d9733e49 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer 99ead02915954b4fab23b033ad6d546b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:09.579718 10473 consensus_queue.cc:1035] T aa281822047841fab7c58b53d9733e49 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:09.579897 13098 raft_consensus.cc:2932] T b20ff3a6f74f4640afebfd4059b58699 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } }
I20240503 02:00:09.580035 12922 consensus_queue.cc:1035] T aa281822047841fab7c58b53d9733e49 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:09.581183 10473 raft_consensus.cc:2932] T aa281822047841fab7c58b53d9733e49 P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } } }
I20240503 02:00:09.581611 9179 raft_consensus.cc:2932] T aa281822047841fab7c58b53d9733e49 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } } }
I20240503 02:00:09.582527 6720 raft_consensus.cc:2932] T b20ff3a6f74f4640afebfd4059b58699 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } }
I20240503 02:00:09.582901 6301 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet aa281822047841fab7c58b53d9733e49 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:09.583675 6313 catalog_manager.cc:5484] T b20ff3a6f74f4640afebfd4059b58699 P 2a01ace80baf4057b6bf1995cf88468b reported cstate change: config changed from index 2 to 3, dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2a01ace80baf4057b6bf1995cf88468b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:09.585357 6309 catalog_manager.cc:5484] T aa281822047841fab7c58b53d9733e49 P 99ead02915954b4fab23b033ad6d546b reported cstate change: config changed from index 3 to 4, VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "99ead02915954b4fab23b033ad6d546b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:09.585660 6456 raft_consensus.cc:2932] T b20ff3a6f74f4640afebfd4059b58699 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } }
I20240503 02:00:09.587291 9179 consensus_queue.cc:237] T b20ff3a6f74f4640afebfd4059b58699 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } }
I20240503 02:00:09.587875 9048 raft_consensus.cc:2932] T b20ff3a6f74f4640afebfd4059b58699 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } }
I20240503 02:00:09.588675 6719 raft_consensus.cc:1261] T b20ff3a6f74f4640afebfd4059b58699 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer 2a01ace80baf4057b6bf1995cf88468b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:09.589674 6456 raft_consensus.cc:1261] T b20ff3a6f74f4640afebfd4059b58699 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 2a01ace80baf4057b6bf1995cf88468b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:09.589784 12907 consensus_queue.cc:1035] T b20ff3a6f74f4640afebfd4059b58699 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:09.590021 13629 consensus_queue.cc:1035] T b20ff3a6f74f4640afebfd4059b58699 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:00:09.591463 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:09.592409 6720 raft_consensus.cc:2932] T aa281822047841fab7c58b53d9733e49 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } } }
I20240503 02:00:09.595741 13629 raft_consensus.cc:2932] T b20ff3a6f74f4640afebfd4059b58699 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } }
I20240503 02:00:09.595817 6456 raft_consensus.cc:2932] T b20ff3a6f74f4640afebfd4059b58699 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } }
I20240503 02:00:09.596071 6719 raft_consensus.cc:2932] T b20ff3a6f74f4640afebfd4059b58699 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } }
I20240503 02:00:09.597739 6300 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b20ff3a6f74f4640afebfd4059b58699 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:09.598503 6308 catalog_manager.cc:5484] T b20ff3a6f74f4640afebfd4059b58699 P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: config changed from index 3 to 4, VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "2a01ace80baf4057b6bf1995cf88468b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } }
I20240503 02:00:09.600836 9028 tablet_service.cc:1514] Processing DeleteTablet for tablet b20ff3a6f74f4640afebfd4059b58699 with delete_type TABLET_DATA_TOMBSTONED (TS 99ead02915954b4fab23b033ad6d546b not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52584
I20240503 02:00:09.601137 13638 tablet_replica.cc:330] T b20ff3a6f74f4640afebfd4059b58699 P 99ead02915954b4fab23b033ad6d546b: stopping tablet replica
I20240503 02:00:09.601275 13638 raft_consensus.cc:2229] T b20ff3a6f74f4640afebfd4059b58699 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:09.601419 13638 raft_consensus.cc:2258] T b20ff3a6f74f4640afebfd4059b58699 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:09.602030 13638 ts_tablet_manager.cc:1875] T b20ff3a6f74f4640afebfd4059b58699 P 99ead02915954b4fab23b033ad6d546b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:09.604246 13638 ts_tablet_manager.cc:1888] T b20ff3a6f74f4640afebfd4059b58699 P 99ead02915954b4fab23b033ad6d546b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:00:09.604318 13638 log.cc:1196] T b20ff3a6f74f4640afebfd4059b58699 P 99ead02915954b4fab23b033ad6d546b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/b20ff3a6f74f4640afebfd4059b58699
I20240503 02:00:09.604844 6301 catalog_manager.cc:4830] TS 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157): tablet b20ff3a6f74f4640afebfd4059b58699 (table rebalancer_extra_table_18 [id=e035403de1c6423e98f3bb44dcb47f8a]) successfully deleted
I20240503 02:00:09.610656 13298 raft_consensus.cc:1050] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a: attempting to promote NON_VOTER 2a01ace80baf4057b6bf1995cf88468b to VOTER
I20240503 02:00:09.611199 13298 consensus_queue.cc:237] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } }
I20240503 02:00:09.612859 6719 raft_consensus.cc:1261] T 2d7244b0dc4d419cb5c3f3677acd70c7 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:09.613085 9176 raft_consensus.cc:1261] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEARNER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:09.613117 9048 raft_consensus.cc:1261] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:09.613189 13495 raft_consensus.cc:1050] T e52373099af3428497d176e3fcd8ef27 P 99ead02915954b4fab23b033ad6d546b: attempting to promote NON_VOTER 80fd18ef4d604c86b909d5c008a46b4f to VOTER
I20240503 02:00:09.613540 13626 consensus_queue.cc:1035] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:09.613600 13495 consensus_queue.cc:237] T e52373099af3428497d176e3fcd8ef27 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } }
I20240503 02:00:09.613925 13626 consensus_queue.cc:1035] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:09.614063 13657 consensus_queue.cc:1035] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:09.614945 6589 raft_consensus.cc:1261] T e52373099af3428497d176e3fcd8ef27 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEARNER]: Refusing update from remote peer 99ead02915954b4fab23b033ad6d546b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:09.615249 6720 raft_consensus.cc:1261] T e52373099af3428497d176e3fcd8ef27 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer 99ead02915954b4fab23b033ad6d546b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:09.615514 12922 consensus_queue.cc:1035] T e52373099af3428497d176e3fcd8ef27 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:09.615778 10473 consensus_queue.cc:1035] T e52373099af3428497d176e3fcd8ef27 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:09.616504 9178 raft_consensus.cc:1261] T e52373099af3428497d176e3fcd8ef27 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer 99ead02915954b4fab23b033ad6d546b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:09.616525 6308 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:00:09.618618 12922 consensus_queue.cc:1035] T e52373099af3428497d176e3fcd8ef27 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:09.619658 13298 raft_consensus.cc:2932] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:09.620023 10473 raft_consensus.cc:2932] T e52373099af3428497d176e3fcd8ef27 P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:09.621601 9048 raft_consensus.cc:2932] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:09.622074 9179 raft_consensus.cc:2932] T e52373099af3428497d176e3fcd8ef27 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:09.623287 6589 raft_consensus.cc:2932] T e52373099af3428497d176e3fcd8ef27 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:09.623648 9178 raft_consensus.cc:2932] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:09.624460 6309 catalog_manager.cc:5484] T e52373099af3428497d176e3fcd8ef27 P 99ead02915954b4fab23b033ad6d546b reported cstate change: config changed from index 2 to 3, 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "99ead02915954b4fab23b033ad6d546b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:09.625151 6308 catalog_manager.cc:5484] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: config changed from index 7 to 8, 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:09.625960 6719 raft_consensus.cc:2932] T e52373099af3428497d176e3fcd8ef27 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:09.627414 6436 tablet_service.cc:1467] Processing CreateTablet for tablet ad245987cb6c4d0680a1597edc14801b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b3c4040b69424425906498cc28fc97b6]), partition=
I20240503 02:00:09.627642 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad245987cb6c4d0680a1597edc14801b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:09.629142 9157 tablet_service.cc:1467] Processing CreateTablet for tablet ad245987cb6c4d0680a1597edc14801b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b3c4040b69424425906498cc28fc97b6]), partition=
I20240503 02:00:09.629567 9157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad245987cb6c4d0680a1597edc14801b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:09.629766 13607 tablet_bootstrap.cc:492] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:09.626006 6720 raft_consensus.cc:2932] T 2d7244b0dc4d419cb5c3f3677acd70c7 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:09.630642 13607 tablet_bootstrap.cc:654] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:09.631814 8887 tablet_service.cc:1467] Processing CreateTablet for tablet ad245987cb6c4d0680a1597edc14801b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b3c4040b69424425906498cc28fc97b6]), partition=
I20240503 02:00:09.632054 8887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad245987cb6c4d0680a1597edc14801b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:09.632516 9048 consensus_queue.cc:237] T e52373099af3428497d176e3fcd8ef27 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } }
I20240503 02:00:09.633471 6456 consensus_queue.cc:237] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } }
I20240503 02:00:09.633898 6720 raft_consensus.cc:1261] T e52373099af3428497d176e3fcd8ef27 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer 99ead02915954b4fab23b033ad6d546b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:09.634280 12922 consensus_queue.cc:1035] T e52373099af3428497d176e3fcd8ef27 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:09.637229 12922 raft_consensus.cc:2932] T e52373099af3428497d176e3fcd8ef27 P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:09.637585 6720 raft_consensus.cc:2932] T e52373099af3428497d176e3fcd8ef27 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:09.638069 13181 tablet_bootstrap.cc:492] T ad245987cb6c4d0680a1597edc14801b P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:09.638882 13181 tablet_bootstrap.cc:654] T ad245987cb6c4d0680a1597edc14801b P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:09.638936 6301 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e52373099af3428497d176e3fcd8ef27 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:09.639323 6589 raft_consensus.cc:1261] T e52373099af3428497d176e3fcd8ef27 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer 99ead02915954b4fab23b033ad6d546b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:09.639662 9048 raft_consensus.cc:1261] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:09.640041 13495 consensus_queue.cc:1035] T e52373099af3428497d176e3fcd8ef27 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:09.640480 13606 tablet_bootstrap.cc:492] T ad245987cb6c4d0680a1597edc14801b P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:09.640480 13657 consensus_queue.cc:1035] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:09.641371 13606 tablet_bootstrap.cc:654] T ad245987cb6c4d0680a1597edc14801b P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:09.642168 6589 raft_consensus.cc:2932] T e52373099af3428497d176e3fcd8ef27 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:09.642671 13606 tablet_bootstrap.cc:492] T ad245987cb6c4d0680a1597edc14801b P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:09.642760 13606 ts_tablet_manager.cc:1367] T ad245987cb6c4d0680a1597edc14801b P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:09.642987 9178 raft_consensus.cc:1261] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:09.643198 13606 raft_consensus.cc:348] T ad245987cb6c4d0680a1597edc14801b P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:09.643299 13606 raft_consensus.cc:374] T ad245987cb6c4d0680a1597edc14801b P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:09.643330 13606 raft_consensus.cc:729] T ad245987cb6c4d0680a1597edc14801b P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:09.643443 13606 consensus_queue.cc:260] T ad245987cb6c4d0680a1597edc14801b P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:09.643743 13606 ts_tablet_manager.cc:1398] T ad245987cb6c4d0680a1597edc14801b P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:09.643850 13657 consensus_queue.cc:1035] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:09.643841 6313 catalog_manager.cc:5484] T e52373099af3428497d176e3fcd8ef27 P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: config changed from index 3 to 4, VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "99ead02915954b4fab23b033ad6d546b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:09.644470 12190 raft_consensus.cc:2932] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:09.646615 6300 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2d7244b0dc4d419cb5c3f3677acd70c7 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:09.647231 9048 raft_consensus.cc:2932] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:09.647420 6312 catalog_manager.cc:5484] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: config changed from index 8 to 9, VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:09.647907 13607 tablet_bootstrap.cc:492] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:09.648000 13607 ts_tablet_manager.cc:1367] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20240503 02:00:09.648541 13607 raft_consensus.cc:348] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:09.648646 13607 raft_consensus.cc:374] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:09.648676 13607 raft_consensus.cc:729] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:09.648787 13607 consensus_queue.cc:260] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:09.649057 13607 ts_tablet_manager.cc:1398] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:09.650694 6700 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d7244b0dc4d419cb5c3f3677acd70c7 with delete_type TABLET_DATA_TOMBSTONED (TS c2ac2eb7cc264047b9ab32fe3a4fc025 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:40336
I20240503 02:00:09.651727 13626 consensus_queue.cc:1035] T be7bbd71b99847068f415cc3d3d49ddd P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:09.652858 9157 tablet_service.cc:1514] Processing DeleteTablet for tablet e52373099af3428497d176e3fcd8ef27 with delete_type TABLET_DATA_TOMBSTONED (TS 2a01ace80baf4057b6bf1995cf88468b not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:51940
I20240503 02:00:09.653249 13644 tablet_replica.cc:330] T e52373099af3428497d176e3fcd8ef27 P 2a01ace80baf4057b6bf1995cf88468b: stopping tablet replica
I20240503 02:00:09.653386 13644 raft_consensus.cc:2229] T e52373099af3428497d176e3fcd8ef27 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:09.653515 13644 raft_consensus.cc:2258] T e52373099af3428497d176e3fcd8ef27 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:09.654845 13644 ts_tablet_manager.cc:1875] T e52373099af3428497d176e3fcd8ef27 P 2a01ace80baf4057b6bf1995cf88468b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:09.658095 13298 consensus_queue.cc:1035] T be7bbd71b99847068f415cc3d3d49ddd P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:09.662212 13644 ts_tablet_manager.cc:1888] T e52373099af3428497d176e3fcd8ef27 P 2a01ace80baf4057b6bf1995cf88468b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:00:09.662319 13644 log.cc:1196] T e52373099af3428497d176e3fcd8ef27 P 2a01ace80baf4057b6bf1995cf88468b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/e52373099af3428497d176e3fcd8ef27
I20240503 02:00:09.663182 6300 catalog_manager.cc:4830] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079): tablet e52373099af3428497d176e3fcd8ef27 (table rebalancer_extra_table_12 [id=2d54fb96d5ab4898ac53ffc16334115d]) successfully deleted
I20240503 02:00:09.663509 9178 raft_consensus.cc:2932] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:09.665685 13657 raft_consensus.cc:482] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:09.665809 13657 raft_consensus.cc:504] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:09.666196 13657 leader_election.cc:288] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:09.666808 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad245987cb6c4d0680a1597edc14801b" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:09.666954 8916 raft_consensus.cc:2454] T ad245987cb6c4d0680a1597edc14801b P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:09.667150 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad245987cb6c4d0680a1597edc14801b" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:09.667404 6394 leader_election.cc:302] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9942306855404cfd891adf227dd65eb2, dec3e4a04d604f7eaaa3f077d081218a; no voters:
W20240503 02:00:09.667517 6391 leader_election.cc:341] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:00:09.667567 13626 raft_consensus.cc:2781] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:09.667604 13626 raft_consensus.cc:482] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:09.667649 13626 raft_consensus.cc:3037] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:09.668336 13181 tablet_bootstrap.cc:492] T ad245987cb6c4d0680a1597edc14801b P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:09.668463 13181 ts_tablet_manager.cc:1367] T ad245987cb6c4d0680a1597edc14801b P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.030s user 0.001s sys 0.001s
I20240503 02:00:09.668839 13626 raft_consensus.cc:504] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:09.668933 13181 raft_consensus.cc:348] T ad245987cb6c4d0680a1597edc14801b P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:09.669034 13181 raft_consensus.cc:374] T ad245987cb6c4d0680a1597edc14801b P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:09.669072 13181 raft_consensus.cc:729] T ad245987cb6c4d0680a1597edc14801b P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:09.669219 13181 consensus_queue.cc:260] T ad245987cb6c4d0680a1597edc14801b P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:09.669533 13181 ts_tablet_manager.cc:1398] T ad245987cb6c4d0680a1597edc14801b P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:09.669570 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad245987cb6c4d0680a1597edc14801b" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:09.669662 9178 raft_consensus.cc:3037] T ad245987cb6c4d0680a1597edc14801b P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:09.669881 13669 tablet_replica.cc:330] T 2d7244b0dc4d419cb5c3f3677acd70c7 P c2ac2eb7cc264047b9ab32fe3a4fc025: stopping tablet replica
I20240503 02:00:09.670009 13669 raft_consensus.cc:2229] T 2d7244b0dc4d419cb5c3f3677acd70c7 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:09.670226 13669 raft_consensus.cc:2258] T 2d7244b0dc4d419cb5c3f3677acd70c7 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:09.670641 9178 raft_consensus.cc:2454] T ad245987cb6c4d0680a1597edc14801b P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:09.670820 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad245987cb6c4d0680a1597edc14801b" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:09.670946 8916 raft_consensus.cc:3037] T ad245987cb6c4d0680a1597edc14801b P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:09.670957 6391 leader_election.cc:302] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:09.671244 13657 raft_consensus.cc:2781] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:09.671329 13657 raft_consensus.cc:686] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Becoming Leader. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Running, Role: LEADER
I20240503 02:00:09.671459 13657 consensus_queue.cc:237] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:09.671882 13626 leader_election.cc:288] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Requested vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:09.673456 13669 ts_tablet_manager.cc:1875] T 2d7244b0dc4d419cb5c3f3677acd70c7 P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:09.674970 8916 raft_consensus.cc:2454] T ad245987cb6c4d0680a1597edc14801b P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:09.677033 6309 catalog_manager.cc:5484] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: term changed from 0 to 1, leader changed from <none> to dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129). New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:09.677515 13669 ts_tablet_manager.cc:1888] T 2d7244b0dc4d419cb5c3f3677acd70c7 P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 02:00:09.677587 13669 log.cc:1196] T 2d7244b0dc4d419cb5c3f3677acd70c7 P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/2d7244b0dc4d419cb5c3f3677acd70c7
I20240503 02:00:09.678321 6303 catalog_manager.cc:4830] TS c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655): tablet 2d7244b0dc4d419cb5c3f3677acd70c7 (table rebalance_test_table_17 [id=5ccf0adaf2de4b79bbf38c053646e60e]) successfully deleted
I20240503 02:00:09.733016 13496 raft_consensus.cc:1050] T cea1ea50fa434d31b42451f84f26bc76 P c2ac2eb7cc264047b9ab32fe3a4fc025: attempting to promote NON_VOTER 492448570cc94ed1879feb2ec3b9051d to VOTER
I20240503 02:00:09.733390 13496 consensus_queue.cc:237] T cea1ea50fa434d31b42451f84f26bc76 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { promote: false } }
I20240503 02:00:09.734695 12872 raft_consensus.cc:1050] T d73ea6e550a04c04a2695b058e38c566 P c2ac2eb7cc264047b9ab32fe3a4fc025: attempting to promote NON_VOTER 9942306855404cfd891adf227dd65eb2 to VOTER
I20240503 02:00:09.735095 12872 consensus_queue.cc:237] T d73ea6e550a04c04a2695b058e38c566 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } }
I20240503 02:00:09.735167 9048 raft_consensus.cc:1261] T cea1ea50fa434d31b42451f84f26bc76 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:09.735536 13496 consensus_queue.cc:1035] T cea1ea50fa434d31b42451f84f26bc76 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240503 02:00:09.735992 6654 consensus_peers.cc:485] T cea1ea50fa434d31b42451f84f26bc76 P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:09.736331 8916 raft_consensus.cc:1261] T d73ea6e550a04c04a2695b058e38c566 P 9942306855404cfd891adf227dd65eb2 [term 1 LEARNER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:09.736527 6850 raft_consensus.cc:1261] T cea1ea50fa434d31b42451f84f26bc76 P 492448570cc94ed1879feb2ec3b9051d [term 1 LEARNER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:09.736552 6456 raft_consensus.cc:1261] T d73ea6e550a04c04a2695b058e38c566 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:09.736770 12872 consensus_queue.cc:1035] T d73ea6e550a04c04a2695b058e38c566 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:09.737046 13625 consensus_queue.cc:1035] T cea1ea50fa434d31b42451f84f26bc76 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:09.737082 12872 consensus_queue.cc:1035] T d73ea6e550a04c04a2695b058e38c566 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:09.738212 6589 raft_consensus.cc:1261] T d73ea6e550a04c04a2695b058e38c566 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:09.739635 13679 consensus_queue.cc:1035] T d73ea6e550a04c04a2695b058e38c566 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:09.739941 13625 raft_consensus.cc:2932] T cea1ea50fa434d31b42451f84f26bc76 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { promote: false } } }
I20240503 02:00:09.741212 13056 consensus_queue.cc:1035] T 7cecefe8f1864583be9d0b9c308a78db P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:09.741969 13625 raft_consensus.cc:2932] T d73ea6e550a04c04a2695b058e38c566 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:09.742405 6589 raft_consensus.cc:2932] T d73ea6e550a04c04a2695b058e38c566 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:09.743615 6314 catalog_manager.cc:5484] T cea1ea50fa434d31b42451f84f26bc76 P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: config changed from index 2 to 3, 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:09.745079 9048 raft_consensus.cc:2932] T cea1ea50fa434d31b42451f84f26bc76 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { promote: false } } }
I20240503 02:00:09.746120 8916 raft_consensus.cc:2932] T d73ea6e550a04c04a2695b058e38c566 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:09.747133 6850 raft_consensus.cc:2932] T cea1ea50fa434d31b42451f84f26bc76 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { promote: false } } }
I20240503 02:00:09.749919 6309 catalog_manager.cc:5484] T d73ea6e550a04c04a2695b058e38c566 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: config changed from index 4 to 5, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:09.750176 6456 raft_consensus.cc:2932] T d73ea6e550a04c04a2695b058e38c566 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:09.751888 13056 consensus_queue.cc:1035] T 7cecefe8f1864583be9d0b9c308a78db P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:09.782696 6308 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:09.786841 9178 raft_consensus.cc:1261] T ad245987cb6c4d0680a1597edc14801b P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:09.786985 8916 raft_consensus.cc:1261] T ad245987cb6c4d0680a1597edc14801b P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:09.787315 13298 consensus_queue.cc:1035] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:09.787483 13657 consensus_queue.cc:1035] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:09.790390 6308 catalog_manager.cc:5576] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211) does not have the latest schema for tablet ad245987cb6c4d0680a1597edc14801b (table rebalancer_dynamic_table [id=b3c4040b69424425906498cc28fc97b6]). Expected version 1 got 0
I20240503 02:00:09.790912 13690 tablet.cc:1722] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:09.791142 6314 catalog_manager.cc:5576] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079) does not have the latest schema for tablet ad245987cb6c4d0680a1597edc14801b (table rebalancer_dynamic_table [id=b3c4040b69424425906498cc28fc97b6]). Expected version 1 got 0
I20240503 02:00:09.791280 13690 tablet.cc:1620] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a: MemRowSet was empty: no flush needed.
I20240503 02:00:09.792474 13622 tablet.cc:1722] T ad245987cb6c4d0680a1597edc14801b P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:09.792804 13622 tablet.cc:1620] T ad245987cb6c4d0680a1597edc14801b P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
I20240503 02:00:09.795266 6313 catalog_manager.cc:5576] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211) does not have the latest schema for tablet ad245987cb6c4d0680a1597edc14801b (table rebalancer_dynamic_table [id=b3c4040b69424425906498cc28fc97b6]). Expected version 1 got 0
I20240503 02:00:09.795581 6314 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=b3c4040b69424425906498cc28fc97b6] alter complete (version 1)
I20240503 02:00:09.800688 13691 tablet.cc:1722] T ad245987cb6c4d0680a1597edc14801b P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:09.801137 13691 tablet.cc:1620] T ad245987cb6c4d0680a1597edc14801b P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
I20240503 02:00:09.852532 13625 consensus_queue.cc:1035] T e617d4e31eed42dcb39e03cc40b8e6a6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:09.855805 13679 consensus_queue.cc:1035] T e617d4e31eed42dcb39e03cc40b8e6a6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:09.857746 13657 raft_consensus.cc:1050] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a: attempting to promote NON_VOTER 2a01ace80baf4057b6bf1995cf88468b to VOTER
I20240503 02:00:09.858209 13657 consensus_queue.cc:237] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } }
I20240503 02:00:09.859539 9178 raft_consensus.cc:1261] T ac7fa0c3722a461c91fb237fd85794a3 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEARNER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:09.859875 13626 consensus_queue.cc:1035] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:09.860325 9039 raft_consensus.cc:1261] T ac7fa0c3722a461c91fb237fd85794a3 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:09.860365 6589 raft_consensus.cc:1261] T ac7fa0c3722a461c91fb237fd85794a3 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:09.860754 13626 consensus_queue.cc:1035] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:09.860952 13657 consensus_queue.cc:1035] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:09.862726 12190 raft_consensus.cc:2932] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:09.863317 6589 raft_consensus.cc:2932] T ac7fa0c3722a461c91fb237fd85794a3 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:09.863909 13298 raft_consensus.cc:1050] T 9cbb2be1fda74ebe8997123c70629ba3 P dec3e4a04d604f7eaaa3f077d081218a: attempting to promote NON_VOTER 80fd18ef4d604c86b909d5c008a46b4f to VOTER
I20240503 02:00:09.864260 13298 consensus_queue.cc:237] T 9cbb2be1fda74ebe8997123c70629ba3 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } }
I20240503 02:00:09.865931 9048 raft_consensus.cc:1261] T 9cbb2be1fda74ebe8997123c70629ba3 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:09.865885 9039 raft_consensus.cc:2932] T ac7fa0c3722a461c91fb237fd85794a3 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:09.866461 13657 consensus_queue.cc:1035] T 9cbb2be1fda74ebe8997123c70629ba3 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:09.866444 6309 catalog_manager.cc:5484] T ac7fa0c3722a461c91fb237fd85794a3 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: config changed from index 7 to 8, 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:09.867554 8916 raft_consensus.cc:1261] T 9cbb2be1fda74ebe8997123c70629ba3 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:09.867388 9178 raft_consensus.cc:2932] T ac7fa0c3722a461c91fb237fd85794a3 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:09.869542 12190 consensus_queue.cc:1035] T 9cbb2be1fda74ebe8997123c70629ba3 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:09.870330 6589 raft_consensus.cc:1261] T 9cbb2be1fda74ebe8997123c70629ba3 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEARNER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:09.871026 13626 consensus_queue.cc:1035] T 9cbb2be1fda74ebe8997123c70629ba3 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:09.872356 6456 consensus_queue.cc:237] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } }
I20240503 02:00:09.873651 9039 raft_consensus.cc:1261] T ac7fa0c3722a461c91fb237fd85794a3 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:09.873839 9178 raft_consensus.cc:1261] T ac7fa0c3722a461c91fb237fd85794a3 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:09.874102 12190 raft_consensus.cc:2932] T 9cbb2be1fda74ebe8997123c70629ba3 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:09.874325 13626 consensus_queue.cc:1035] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:09.874459 6589 raft_consensus.cc:2932] T 9cbb2be1fda74ebe8997123c70629ba3 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:09.874635 13626 consensus_queue.cc:1035] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:09.876227 8916 raft_consensus.cc:2932] T 9cbb2be1fda74ebe8997123c70629ba3 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:09.876636 13298 raft_consensus.cc:2932] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:09.878655 9039 raft_consensus.cc:2932] T ac7fa0c3722a461c91fb237fd85794a3 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:09.878845 9178 raft_consensus.cc:2932] T ac7fa0c3722a461c91fb237fd85794a3 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:09.879130 6309 catalog_manager.cc:5484] T 9cbb2be1fda74ebe8997123c70629ba3 P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: config changed from index 2 to 3, 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:09.881665 6314 catalog_manager.cc:5484] T ac7fa0c3722a461c91fb237fd85794a3 P 99ead02915954b4fab23b033ad6d546b reported cstate change: config changed from index 8 to 9, VOTER 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:09.883559 6300 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ac7fa0c3722a461c91fb237fd85794a3 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:09.885469 6456 consensus_queue.cc:237] T 9cbb2be1fda74ebe8997123c70629ba3 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } }
I20240503 02:00:09.885434 9048 raft_consensus.cc:2932] T 9cbb2be1fda74ebe8997123c70629ba3 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:09.887266 8916 raft_consensus.cc:1261] T 9cbb2be1fda74ebe8997123c70629ba3 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:09.887324 6569 tablet_service.cc:1514] Processing DeleteTablet for tablet ac7fa0c3722a461c91fb237fd85794a3 with delete_type TABLET_DATA_TOMBSTONED (TS 80fd18ef4d604c86b909d5c008a46b4f not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:59846
I20240503 02:00:09.887599 13643 tablet_replica.cc:330] T ac7fa0c3722a461c91fb237fd85794a3 P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 02:00:09.887748 13643 raft_consensus.cc:2229] T ac7fa0c3722a461c91fb237fd85794a3 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:09.887884 6589 raft_consensus.cc:1261] T 9cbb2be1fda74ebe8997123c70629ba3 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:09.887938 13643 raft_consensus.cc:2258] T ac7fa0c3722a461c91fb237fd85794a3 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:09.888227 12190 consensus_queue.cc:1035] T 9cbb2be1fda74ebe8997123c70629ba3 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:09.888465 13298 consensus_queue.cc:1035] T 9cbb2be1fda74ebe8997123c70629ba3 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:09.888988 13643 ts_tablet_manager.cc:1875] T ac7fa0c3722a461c91fb237fd85794a3 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:09.890179 12190 raft_consensus.cc:2932] T 9cbb2be1fda74ebe8997123c70629ba3 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:09.890650 8916 raft_consensus.cc:2932] T 9cbb2be1fda74ebe8997123c70629ba3 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:09.891294 13643 ts_tablet_manager.cc:1888] T ac7fa0c3722a461c91fb237fd85794a3 P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 02:00:09.891356 13643 log.cc:1196] T ac7fa0c3722a461c91fb237fd85794a3 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/ac7fa0c3722a461c91fb237fd85794a3
I20240503 02:00:09.891466 6300 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9cbb2be1fda74ebe8997123c70629ba3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:09.891847 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet ac7fa0c3722a461c91fb237fd85794a3 (table rebalance_test_table_3 [id=72f29547da0b4a95974018c2782cb359]) successfully deleted
I20240503 02:00:09.894189 6582 raft_consensus.cc:2932] T 9cbb2be1fda74ebe8997123c70629ba3 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:09.894291 6312 catalog_manager.cc:5484] T 9cbb2be1fda74ebe8997123c70629ba3 P 9942306855404cfd891adf227dd65eb2 reported cstate change: config changed from index 3 to 4, VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:09.896818 9028 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cbb2be1fda74ebe8997123c70629ba3 with delete_type TABLET_DATA_TOMBSTONED (TS 99ead02915954b4fab23b033ad6d546b not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52584
I20240503 02:00:09.897073 13638 tablet_replica.cc:330] T 9cbb2be1fda74ebe8997123c70629ba3 P 99ead02915954b4fab23b033ad6d546b: stopping tablet replica
I20240503 02:00:09.897208 13638 raft_consensus.cc:2229] T 9cbb2be1fda74ebe8997123c70629ba3 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:09.897346 13638 raft_consensus.cc:2258] T 9cbb2be1fda74ebe8997123c70629ba3 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:09.897900 13638 ts_tablet_manager.cc:1875] T 9cbb2be1fda74ebe8997123c70629ba3 P 99ead02915954b4fab23b033ad6d546b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:09.899960 13638 ts_tablet_manager.cc:1888] T 9cbb2be1fda74ebe8997123c70629ba3 P 99ead02915954b4fab23b033ad6d546b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:00:09.900036 13638 log.cc:1196] T 9cbb2be1fda74ebe8997123c70629ba3 P 99ead02915954b4fab23b033ad6d546b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/9cbb2be1fda74ebe8997123c70629ba3
I20240503 02:00:09.899950 6312 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:09.900573 6301 catalog_manager.cc:4830] TS 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157): tablet 9cbb2be1fda74ebe8997123c70629ba3 (table rebalancer_extra_table_16 [id=9b52669a75314ef88150c5df1cac862c]) successfully deleted
I20240503 02:00:09.905081 13690 tablet.cc:1722] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:09.905452 13690 tablet.cc:1620] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a: MemRowSet was empty: no flush needed.
I20240503 02:00:09.905565 13622 tablet.cc:1722] T ad245987cb6c4d0680a1597edc14801b P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:09.905848 13622 tablet.cc:1620] T ad245987cb6c4d0680a1597edc14801b P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
I20240503 02:00:09.905910 13691 tablet.cc:1722] T ad245987cb6c4d0680a1597edc14801b P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:09.906210 13691 tablet.cc:1620] T ad245987cb6c4d0680a1597edc14801b P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
I20240503 02:00:09.911020 6308 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=b3c4040b69424425906498cc28fc97b6] alter complete (version 2)
W20240503 02:00:09.960605 6301 catalog_manager.cc:4573] TS e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet aa281822047841fab7c58b53d9733e49: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:10.013221 6314 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:10.016950 6313 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_extra_table_26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:10.018888 13690 tablet.cc:1722] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:10.019968 13691 tablet.cc:1722] T ad245987cb6c4d0680a1597edc14801b P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:10.023487 13622 tablet.cc:1722] T ad245987cb6c4d0680a1597edc14801b P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:10.024416 6831 tablet_service.cc:1467] Processing CreateTablet for tablet 493053c0766648c3bfeaf2490732bbb8 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=e1b413681beb485981324abe8f697565]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:10.024721 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 493053c0766648c3bfeaf2490732bbb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:10.024698 8887 tablet_service.cc:1467] Processing CreateTablet for tablet 493053c0766648c3bfeaf2490732bbb8 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=e1b413681beb485981324abe8f697565]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:10.024946 8887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 493053c0766648c3bfeaf2490732bbb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:10.026382 8897 tablet_service.cc:1467] Processing CreateTablet for tablet 56824b553788471c9a2fcd6ec82d507e (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=e1b413681beb485981324abe8f697565]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:10.026628 8897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56824b553788471c9a2fcd6ec82d507e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:10.026926 8880 tablet_service.cc:1467] Processing CreateTablet for tablet 71b7ef5ff3974a1f836bef781c42d94e (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=e1b413681beb485981324abe8f697565]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:10.026999 13702 tablet_bootstrap.cc:492] T 493053c0766648c3bfeaf2490732bbb8 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:10.027153 8880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71b7ef5ff3974a1f836bef781c42d94e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:10.027235 6569 tablet_service.cc:1467] Processing CreateTablet for tablet 71b7ef5ff3974a1f836bef781c42d94e (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=e1b413681beb485981324abe8f697565]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:10.027491 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71b7ef5ff3974a1f836bef781c42d94e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:10.027993 13702 tablet_bootstrap.cc:654] T 493053c0766648c3bfeaf2490732bbb8 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:10.029736 13704 tablet_bootstrap.cc:492] T 71b7ef5ff3974a1f836bef781c42d94e P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:10.029728 9028 tablet_service.cc:1467] Processing CreateTablet for tablet 493053c0766648c3bfeaf2490732bbb8 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=e1b413681beb485981324abe8f697565]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:10.029992 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 493053c0766648c3bfeaf2490732bbb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:10.029992 9025 tablet_service.cc:1467] Processing CreateTablet for tablet 56824b553788471c9a2fcd6ec82d507e (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=e1b413681beb485981324abe8f697565]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:10.030205 9025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56824b553788471c9a2fcd6ec82d507e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:10.031004 13704 tablet_bootstrap.cc:654] T 71b7ef5ff3974a1f836bef781c42d94e P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:10.032581 13706 tablet_bootstrap.cc:492] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:10.032658 13606 tablet_bootstrap.cc:492] T 493053c0766648c3bfeaf2490732bbb8 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:10.033810 13606 tablet_bootstrap.cc:654] T 493053c0766648c3bfeaf2490732bbb8 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:10.033944 13706 tablet_bootstrap.cc:654] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:10.035709 9157 tablet_service.cc:1467] Processing CreateTablet for tablet 71b7ef5ff3974a1f836bef781c42d94e (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=e1b413681beb485981324abe8f697565]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:10.036016 9157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71b7ef5ff3974a1f836bef781c42d94e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:10.038244 13702 tablet_bootstrap.cc:492] T 493053c0766648c3bfeaf2490732bbb8 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:10.038362 13702 ts_tablet_manager.cc:1367] T 493053c0766648c3bfeaf2490732bbb8 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 02:00:10.038568 13181 tablet_bootstrap.cc:492] T 71b7ef5ff3974a1f836bef781c42d94e P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:10.038867 13702 raft_consensus.cc:348] T 493053c0766648c3bfeaf2490732bbb8 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:10.038976 13702 raft_consensus.cc:374] T 493053c0766648c3bfeaf2490732bbb8 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:10.038992 13706 tablet_bootstrap.cc:492] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:10.039008 13702 raft_consensus.cc:729] T 493053c0766648c3bfeaf2490732bbb8 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:10.039134 13706 ts_tablet_manager.cc:1367] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:00:10.039135 13702 consensus_queue.cc:260] T 493053c0766648c3bfeaf2490732bbb8 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:10.039403 13702 ts_tablet_manager.cc:1398] T 493053c0766648c3bfeaf2490732bbb8 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:10.039499 13181 tablet_bootstrap.cc:654] T 71b7ef5ff3974a1f836bef781c42d94e P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:10.039618 13706 raft_consensus.cc:348] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:10.039738 13706 raft_consensus.cc:374] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:10.039767 13706 raft_consensus.cc:729] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:10.039904 13706 consensus_queue.cc:260] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:10.040151 6308 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=b3c4040b69424425906498cc28fc97b6] alter complete (version 3)
I20240503 02:00:10.040283 13706 ts_tablet_manager.cc:1398] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:10.040580 13706 tablet_bootstrap.cc:492] T 493053c0766648c3bfeaf2490732bbb8 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:10.041554 13706 tablet_bootstrap.cc:654] T 493053c0766648c3bfeaf2490732bbb8 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:10.043586 13181 tablet_bootstrap.cc:492] T 71b7ef5ff3974a1f836bef781c42d94e P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:10.043689 13181 ts_tablet_manager.cc:1367] T 71b7ef5ff3974a1f836bef781c42d94e P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240503 02:00:10.044131 13181 raft_consensus.cc:348] T 71b7ef5ff3974a1f836bef781c42d94e P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:10.044246 13181 raft_consensus.cc:374] T 71b7ef5ff3974a1f836bef781c42d94e P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:10.044279 13181 raft_consensus.cc:729] T 71b7ef5ff3974a1f836bef781c42d94e P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:10.044440 13181 consensus_queue.cc:260] T 71b7ef5ff3974a1f836bef781c42d94e P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:10.044729 13181 ts_tablet_manager.cc:1398] T 71b7ef5ff3974a1f836bef781c42d94e P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:10.046111 13704 tablet_bootstrap.cc:492] T 71b7ef5ff3974a1f836bef781c42d94e P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:10.046227 13704 ts_tablet_manager.cc:1367] T 71b7ef5ff3974a1f836bef781c42d94e P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240503 02:00:10.046710 13704 raft_consensus.cc:348] T 71b7ef5ff3974a1f836bef781c42d94e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:10.046834 13704 raft_consensus.cc:374] T 71b7ef5ff3974a1f836bef781c42d94e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:10.046860 13704 raft_consensus.cc:729] T 71b7ef5ff3974a1f836bef781c42d94e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:10.046978 13704 consensus_queue.cc:260] T 71b7ef5ff3974a1f836bef781c42d94e P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:10.048264 13704 ts_tablet_manager.cc:1398] T 71b7ef5ff3974a1f836bef781c42d94e P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:10.049235 13606 tablet_bootstrap.cc:492] T 493053c0766648c3bfeaf2490732bbb8 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:10.049384 13606 ts_tablet_manager.cc:1367] T 493053c0766648c3bfeaf2490732bbb8 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20240503 02:00:10.049919 13606 raft_consensus.cc:348] T 493053c0766648c3bfeaf2490732bbb8 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:10.050052 13606 raft_consensus.cc:374] T 493053c0766648c3bfeaf2490732bbb8 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:10.050082 13606 raft_consensus.cc:729] T 493053c0766648c3bfeaf2490732bbb8 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:10.050225 13606 consensus_queue.cc:260] T 493053c0766648c3bfeaf2490732bbb8 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:10.050570 13606 ts_tablet_manager.cc:1398] T 493053c0766648c3bfeaf2490732bbb8 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:10.050812 13606 tablet_bootstrap.cc:492] T 56824b553788471c9a2fcd6ec82d507e P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:10.051869 13606 tablet_bootstrap.cc:654] T 56824b553788471c9a2fcd6ec82d507e P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:10.053107 13706 tablet_bootstrap.cc:492] T 493053c0766648c3bfeaf2490732bbb8 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:10.053237 13706 ts_tablet_manager.cc:1367] T 493053c0766648c3bfeaf2490732bbb8 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 02:00:10.053689 13706 raft_consensus.cc:348] T 493053c0766648c3bfeaf2490732bbb8 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:10.053839 13706 raft_consensus.cc:374] T 493053c0766648c3bfeaf2490732bbb8 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:10.053881 13706 raft_consensus.cc:729] T 493053c0766648c3bfeaf2490732bbb8 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:10.054019 13706 consensus_queue.cc:260] T 493053c0766648c3bfeaf2490732bbb8 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:10.055207 13606 tablet_bootstrap.cc:492] T 56824b553788471c9a2fcd6ec82d507e P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:10.055308 13606 ts_tablet_manager.cc:1367] T 56824b553788471c9a2fcd6ec82d507e P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240503 02:00:10.055392 13706 ts_tablet_manager.cc:1398] T 493053c0766648c3bfeaf2490732bbb8 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:10.055790 13606 raft_consensus.cc:348] T 56824b553788471c9a2fcd6ec82d507e P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:10.055895 13606 raft_consensus.cc:374] T 56824b553788471c9a2fcd6ec82d507e P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:10.055919 13606 raft_consensus.cc:729] T 56824b553788471c9a2fcd6ec82d507e P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:10.056028 13606 consensus_queue.cc:260] T 56824b553788471c9a2fcd6ec82d507e P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:10.056269 13606 ts_tablet_manager.cc:1398] T 56824b553788471c9a2fcd6ec82d507e P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:10.056429 13606 tablet_bootstrap.cc:492] T 71b7ef5ff3974a1f836bef781c42d94e P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:10.057359 13606 tablet_bootstrap.cc:654] T 71b7ef5ff3974a1f836bef781c42d94e P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:10.061242 13606 tablet_bootstrap.cc:492] T 71b7ef5ff3974a1f836bef781c42d94e P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:10.063570 13606 ts_tablet_manager.cc:1367] T 71b7ef5ff3974a1f836bef781c42d94e P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.002s
I20240503 02:00:10.064807 13606 raft_consensus.cc:348] T 71b7ef5ff3974a1f836bef781c42d94e P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:10.064940 13606 raft_consensus.cc:374] T 71b7ef5ff3974a1f836bef781c42d94e P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:10.064968 13606 raft_consensus.cc:729] T 71b7ef5ff3974a1f836bef781c42d94e P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:10.065181 13606 consensus_queue.cc:260] T 71b7ef5ff3974a1f836bef781c42d94e P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:10.065466 13606 ts_tablet_manager.cc:1398] T 71b7ef5ff3974a1f836bef781c42d94e P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:10.076464 13495 raft_consensus.cc:482] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:10.076620 13495 raft_consensus.cc:504] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:10.077193 13495 leader_election.cc:288] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:10.078372 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56824b553788471c9a2fcd6ec82d507e" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:10.078526 8916 raft_consensus.cc:2454] T 56824b553788471c9a2fcd6ec82d507e P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
W20240503 02:00:10.078688 8982 leader_election.cc:334] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:10.078887 8984 leader_election.cc:302] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9942306855404cfd891adf227dd65eb2, 99ead02915954b4fab23b033ad6d546b; no voters: e1a9251ce1cf4361b8b0ede9fbfc4023
I20240503 02:00:10.079043 13495 raft_consensus.cc:2781] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:10.079090 13495 raft_consensus.cc:482] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:10.079140 13495 raft_consensus.cc:3037] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:10.080264 13495 raft_consensus.cc:504] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:10.080663 13495 leader_election.cc:288] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Requested vote from peers e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:10.081104 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56824b553788471c9a2fcd6ec82d507e" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:10.081223 8916 raft_consensus.cc:3037] T 56824b553788471c9a2fcd6ec82d507e P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:00:10.081871 8982 leader_election.cc:334] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:10.082250 8916 raft_consensus.cc:2454] T 56824b553788471c9a2fcd6ec82d507e P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:10.082667 8984 leader_election.cc:302] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9942306855404cfd891adf227dd65eb2, 99ead02915954b4fab23b033ad6d546b; no voters: e1a9251ce1cf4361b8b0ede9fbfc4023
I20240503 02:00:10.083447 13495 raft_consensus.cc:2781] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:10.083592 13495 raft_consensus.cc:686] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Becoming Leader. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Running, Role: LEADER
I20240503 02:00:10.083827 13495 consensus_queue.cc:237] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:10.084511 12331 raft_consensus.cc:482] T 71b7ef5ff3974a1f836bef781c42d94e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:10.084681 12331 raft_consensus.cc:504] T 71b7ef5ff3974a1f836bef781c42d94e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:10.085116 12331 leader_election.cc:288] T 71b7ef5ff3974a1f836bef781c42d94e P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:10.085501 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71b7ef5ff3974a1f836bef781c42d94e" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:10.085709 8916 raft_consensus.cc:2454] T 71b7ef5ff3974a1f836bef781c42d94e P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:10.086042 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71b7ef5ff3974a1f836bef781c42d94e" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:10.086241 9178 raft_consensus.cc:2454] T 71b7ef5ff3974a1f836bef781c42d94e P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:10.086449 6525 leader_election.cc:302] T 71b7ef5ff3974a1f836bef781c42d94e P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:10.086620 12331 raft_consensus.cc:2781] T 71b7ef5ff3974a1f836bef781c42d94e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:10.086668 12331 raft_consensus.cc:482] T 71b7ef5ff3974a1f836bef781c42d94e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:10.086701 12331 raft_consensus.cc:3037] T 71b7ef5ff3974a1f836bef781c42d94e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:10.087699 12331 raft_consensus.cc:504] T 71b7ef5ff3974a1f836bef781c42d94e P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:10.088099 12331 leader_election.cc:288] T 71b7ef5ff3974a1f836bef781c42d94e P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Requested vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:10.088519 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71b7ef5ff3974a1f836bef781c42d94e" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:10.088428 6309 catalog_manager.cc:5484] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b reported cstate change: term changed from 0 to 1, leader changed from <none> to 99ead02915954b4fab23b033ad6d546b (127.0.99.134). New cstate: current_term: 1 leader_uuid: "99ead02915954b4fab23b033ad6d546b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:10.088641 9178 raft_consensus.cc:3037] T 71b7ef5ff3974a1f836bef781c42d94e P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:10.089677 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71b7ef5ff3974a1f836bef781c42d94e" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:10.089843 8916 raft_consensus.cc:3037] T 71b7ef5ff3974a1f836bef781c42d94e P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:10.089856 9178 raft_consensus.cc:2454] T 71b7ef5ff3974a1f836bef781c42d94e P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:10.090813 6522 leader_election.cc:302] T 71b7ef5ff3974a1f836bef781c42d94e P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 80fd18ef4d604c86b909d5c008a46b4f; no voters:
I20240503 02:00:10.090917 8916 raft_consensus.cc:2454] T 71b7ef5ff3974a1f836bef781c42d94e P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:10.091032 12331 raft_consensus.cc:2781] T 71b7ef5ff3974a1f836bef781c42d94e P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:10.091145 12331 raft_consensus.cc:686] T 71b7ef5ff3974a1f836bef781c42d94e P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Becoming Leader. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Running, Role: LEADER
I20240503 02:00:10.091286 12331 consensus_queue.cc:237] T 71b7ef5ff3974a1f836bef781c42d94e P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:10.093873 6309 catalog_manager.cc:5484] T 71b7ef5ff3974a1f836bef781c42d94e P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:10.120187 13630 raft_consensus.cc:482] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e1a9251ce1cf4361b8b0ede9fbfc4023)
I20240503 02:00:10.120340 13630 raft_consensus.cc:504] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } }
I20240503 02:00:10.120893 13630 leader_election.cc:288] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311)
I20240503 02:00:10.121506 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2e62c043adf4a8d9024279d786f596c" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:10.121693 9178 raft_consensus.cc:2454] T b2e62c043adf4a8d9024279d786f596c P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 02:00:10.121994 6784 leader_election.cc:302] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 492448570cc94ed1879feb2ec3b9051d; no voters:
W20240503 02:00:10.122108 6785 leader_election.cc:334] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:10.122167 13630 raft_consensus.cc:2781] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:00:10.122220 13630 raft_consensus.cc:482] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Starting leader election (detected failure of leader e1a9251ce1cf4361b8b0ede9fbfc4023)
I20240503 02:00:10.122256 13630 raft_consensus.cc:3037] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:10.123152 13630 raft_consensus.cc:504] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } }
I20240503 02:00:10.123504 13630 leader_election.cc:288] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 2 election: Requested vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311)
I20240503 02:00:10.123987 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2e62c043adf4a8d9024279d786f596c" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:10.124130 9178 raft_consensus.cc:3037] T b2e62c043adf4a8d9024279d786f596c P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:00:10.124656 6785 leader_election.cc:334] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:10.125118 9178 raft_consensus.cc:2454] T b2e62c043adf4a8d9024279d786f596c P 2a01ace80baf4057b6bf1995cf88468b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 2.
I20240503 02:00:10.125422 6784 leader_election.cc:302] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 492448570cc94ed1879feb2ec3b9051d; no voters: e1a9251ce1cf4361b8b0ede9fbfc4023
I20240503 02:00:10.125684 13630 raft_consensus.cc:2781] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:10.125790 13630 raft_consensus.cc:686] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d [term 2 LEADER]: Becoming Leader. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Running, Role: LEADER
I20240503 02:00:10.125953 13630 consensus_queue.cc:237] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } }
I20240503 02:00:10.128043 6309 catalog_manager.cc:5484] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d reported cstate change: term changed from 1 to 2, leader changed from e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) to 492448570cc94ed1879feb2ec3b9051d (127.0.99.132). New cstate: current_term: 2 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:10.148238 6309 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:10.148444 6309 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:10.150705 6309 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet ad245987cb6c4d0680a1597edc14801b
I20240503 02:00:10.151315 8880 tablet_service.cc:1514] Processing DeleteTablet for tablet ad245987cb6c4d0680a1597edc14801b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:10 UTC) from {username='slave'} at 127.0.0.1:40296
I20240503 02:00:10.151346 9157 tablet_service.cc:1514] Processing DeleteTablet for tablet ad245987cb6c4d0680a1597edc14801b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:10 UTC) from {username='slave'} at 127.0.0.1:51940
I20240503 02:00:10.151544 13644 tablet_replica.cc:330] T ad245987cb6c4d0680a1597edc14801b P 2a01ace80baf4057b6bf1995cf88468b: stopping tablet replica
I20240503 02:00:10.151625 6436 tablet_service.cc:1514] Processing DeleteTablet for tablet ad245987cb6c4d0680a1597edc14801b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:10 UTC) from {username='slave'} at 127.0.0.1:51338
I20240503 02:00:10.151669 13717 tablet_replica.cc:330] T ad245987cb6c4d0680a1597edc14801b P 9942306855404cfd891adf227dd65eb2: stopping tablet replica
I20240503 02:00:10.151726 13644 raft_consensus.cc:2229] T ad245987cb6c4d0680a1597edc14801b P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:10.151809 13717 raft_consensus.cc:2229] T ad245987cb6c4d0680a1597edc14801b P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:10.151854 13644 raft_consensus.cc:2258] T ad245987cb6c4d0680a1597edc14801b P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:10.151911 13717 raft_consensus.cc:2258] T ad245987cb6c4d0680a1597edc14801b P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:10.151919 13718 tablet_replica.cc:330] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a: stopping tablet replica
I20240503 02:00:10.152058 13718 raft_consensus.cc:2229] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:10.152269 13718 raft_consensus.cc:2258] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:10.152567 13644 ts_tablet_manager.cc:1875] T ad245987cb6c4d0680a1597edc14801b P 2a01ace80baf4057b6bf1995cf88468b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:10.152590 13717 ts_tablet_manager.cc:1875] T ad245987cb6c4d0680a1597edc14801b P 9942306855404cfd891adf227dd65eb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:10.152786 13718 ts_tablet_manager.cc:1875] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:10.154978 13644 ts_tablet_manager.cc:1888] T ad245987cb6c4d0680a1597edc14801b P 2a01ace80baf4057b6bf1995cf88468b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:10.155256 13644 log.cc:1196] T ad245987cb6c4d0680a1597edc14801b P 2a01ace80baf4057b6bf1995cf88468b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/ad245987cb6c4d0680a1597edc14801b
I20240503 02:00:10.155680 13644 ts_tablet_manager.cc:1909] T ad245987cb6c4d0680a1597edc14801b P 2a01ace80baf4057b6bf1995cf88468b: Deleting consensus metadata
I20240503 02:00:10.160938 6300 catalog_manager.cc:4830] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079): tablet ad245987cb6c4d0680a1597edc14801b (table rebalancer_dynamic_table_new_name [id=b3c4040b69424425906498cc28fc97b6]) successfully deleted
I20240503 02:00:10.161034 13717 ts_tablet_manager.cc:1888] T ad245987cb6c4d0680a1597edc14801b P 9942306855404cfd891adf227dd65eb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:10.161123 13717 log.cc:1196] T ad245987cb6c4d0680a1597edc14801b P 9942306855404cfd891adf227dd65eb2: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/ad245987cb6c4d0680a1597edc14801b
I20240503 02:00:10.161418 13717 ts_tablet_manager.cc:1909] T ad245987cb6c4d0680a1597edc14801b P 9942306855404cfd891adf227dd65eb2: Deleting consensus metadata
I20240503 02:00:10.161842 13718 ts_tablet_manager.cc:1888] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:10.161931 13718 log.cc:1196] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/ad245987cb6c4d0680a1597edc14801b
I20240503 02:00:10.162309 13718 ts_tablet_manager.cc:1909] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a: Deleting consensus metadata
I20240503 02:00:10.162274 6303 catalog_manager.cc:4830] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211): tablet ad245987cb6c4d0680a1597edc14801b (table rebalancer_dynamic_table_new_name [id=b3c4040b69424425906498cc28fc97b6]) successfully deleted
I20240503 02:00:10.162976 6300 catalog_manager.cc:4830] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): tablet ad245987cb6c4d0680a1597edc14801b (table rebalancer_dynamic_table_new_name [id=b3c4040b69424425906498cc28fc97b6]) successfully deleted
I20240503 02:00:10.186240 13630 raft_consensus.cc:482] T 493053c0766648c3bfeaf2490732bbb8 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:10.186375 13630 raft_consensus.cc:504] T 493053c0766648c3bfeaf2490732bbb8 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:10.186815 13630 leader_election.cc:288] T 493053c0766648c3bfeaf2490732bbb8 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:10.187189 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "493053c0766648c3bfeaf2490732bbb8" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:10.187328 8916 raft_consensus.cc:2454] T 493053c0766648c3bfeaf2490732bbb8 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 02:00:10.187585 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "493053c0766648c3bfeaf2490732bbb8" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:10.187604 6787 leader_election.cc:302] T 493053c0766648c3bfeaf2490732bbb8 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:10.187705 9048 raft_consensus.cc:2454] T 493053c0766648c3bfeaf2490732bbb8 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 02:00:10.187800 13630 raft_consensus.cc:2781] T 493053c0766648c3bfeaf2490732bbb8 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:10.187850 13630 raft_consensus.cc:482] T 493053c0766648c3bfeaf2490732bbb8 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:10.187886 13630 raft_consensus.cc:3037] T 493053c0766648c3bfeaf2490732bbb8 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:10.188781 13630 raft_consensus.cc:504] T 493053c0766648c3bfeaf2490732bbb8 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:10.189106 13630 leader_election.cc:288] T 493053c0766648c3bfeaf2490732bbb8 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Requested vote from peers 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:10.189462 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "493053c0766648c3bfeaf2490732bbb8" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:10.189455 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "493053c0766648c3bfeaf2490732bbb8" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:10.189580 9048 raft_consensus.cc:3037] T 493053c0766648c3bfeaf2490732bbb8 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:10.189585 8916 raft_consensus.cc:3037] T 493053c0766648c3bfeaf2490732bbb8 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:10.190550 9048 raft_consensus.cc:2454] T 493053c0766648c3bfeaf2490732bbb8 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 02:00:10.190560 8916 raft_consensus.cc:2454] T 493053c0766648c3bfeaf2490732bbb8 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 02:00:10.190886 6787 leader_election.cc:302] T 493053c0766648c3bfeaf2490732bbb8 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:10.191059 13630 raft_consensus.cc:2781] T 493053c0766648c3bfeaf2490732bbb8 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:10.191147 13630 raft_consensus.cc:686] T 493053c0766648c3bfeaf2490732bbb8 P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Becoming Leader. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Running, Role: LEADER
I20240503 02:00:10.191291 13630 consensus_queue.cc:237] T 493053c0766648c3bfeaf2490732bbb8 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:10.193512 6309 catalog_manager.cc:5484] T 493053c0766648c3bfeaf2490732bbb8 P 492448570cc94ed1879feb2ec3b9051d reported cstate change: term changed from 0 to 1, leader changed from <none> to 492448570cc94ed1879feb2ec3b9051d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } }
W20240503 02:00:10.205881 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:10.252988 6309 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:00:10.258814 6436 tablet_service.cc:1467] Processing CreateTablet for tablet cc19d71f40fb44f1801f1b49dcc76fe0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ee25cc5084614e039daac0216839e837]), partition=
I20240503 02:00:10.258814 9157 tablet_service.cc:1467] Processing CreateTablet for tablet cc19d71f40fb44f1801f1b49dcc76fe0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ee25cc5084614e039daac0216839e837]), partition=
I20240503 02:00:10.259102 9157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc19d71f40fb44f1801f1b49dcc76fe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:10.259104 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc19d71f40fb44f1801f1b49dcc76fe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:10.261520 13181 tablet_bootstrap.cc:492] T cc19d71f40fb44f1801f1b49dcc76fe0 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:10.261698 13721 tablet_bootstrap.cc:492] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:10.262490 13721 tablet_bootstrap.cc:654] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:10.262490 13181 tablet_bootstrap.cc:654] T cc19d71f40fb44f1801f1b49dcc76fe0 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:10.263716 13181 tablet_bootstrap.cc:492] T cc19d71f40fb44f1801f1b49dcc76fe0 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:10.263808 13181 ts_tablet_manager.cc:1367] T cc19d71f40fb44f1801f1b49dcc76fe0 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:00:10.264320 13721 tablet_bootstrap.cc:492] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:10.264287 13181 raft_consensus.cc:348] T cc19d71f40fb44f1801f1b49dcc76fe0 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:10.264428 13721 ts_tablet_manager.cc:1367] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:10.264428 13181 raft_consensus.cc:374] T cc19d71f40fb44f1801f1b49dcc76fe0 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:10.264461 13181 raft_consensus.cc:729] T cc19d71f40fb44f1801f1b49dcc76fe0 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:10.264590 13181 consensus_queue.cc:260] T cc19d71f40fb44f1801f1b49dcc76fe0 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:10.264871 13721 raft_consensus.cc:348] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:10.264966 13721 raft_consensus.cc:374] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:10.264993 13721 raft_consensus.cc:729] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:10.265141 13721 consensus_queue.cc:260] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:10.265409 13721 ts_tablet_manager.cc:1398] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:10.265633 13181 ts_tablet_manager.cc:1398] T cc19d71f40fb44f1801f1b49dcc76fe0 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:10.484159 13298 raft_consensus.cc:482] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:10.484308 13298 raft_consensus.cc:504] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:10.484764 13298 leader_election.cc:288] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:10.485242 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc19d71f40fb44f1801f1b49dcc76fe0" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:10.485386 9178 raft_consensus.cc:2454] T cc19d71f40fb44f1801f1b49dcc76fe0 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:10.485790 6391 leader_election.cc:302] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, dec3e4a04d604f7eaaa3f077d081218a; no voters:
W20240503 02:00:10.485857 6392 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:10.485967 13298 raft_consensus.cc:2781] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:10.486020 13298 raft_consensus.cc:482] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:10.486045 13298 raft_consensus.cc:3037] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:00:10.486725 6392 leader_election.cc:334] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:10.487021 13298 raft_consensus.cc:504] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:10.487346 13298 leader_election.cc:288] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Requested vote from peers e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:10.487886 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc19d71f40fb44f1801f1b49dcc76fe0" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:10.488032 9178 raft_consensus.cc:3037] T cc19d71f40fb44f1801f1b49dcc76fe0 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:00:10.488512 6392 leader_election.cc:334] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:10.489089 9178 raft_consensus.cc:2454] T cc19d71f40fb44f1801f1b49dcc76fe0 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:10.489388 6391 leader_election.cc:302] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, dec3e4a04d604f7eaaa3f077d081218a; no voters: e1a9251ce1cf4361b8b0ede9fbfc4023
I20240503 02:00:10.489557 13298 raft_consensus.cc:2781] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:10.489661 13298 raft_consensus.cc:686] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Becoming Leader. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Running, Role: LEADER
I20240503 02:00:10.489805 13298 consensus_queue.cc:237] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:10.491941 6308 catalog_manager.cc:5484] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: term changed from 0 to 1, leader changed from <none> to dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129). New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:10.551856 12331 consensus_queue.cc:1035] T 71b7ef5ff3974a1f836bef781c42d94e P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:10.556882 12331 consensus_queue.cc:1035] T 71b7ef5ff3974a1f836bef781c42d94e P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:10.595242 6309 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:10.599769 9178 raft_consensus.cc:1261] T cc19d71f40fb44f1801f1b49dcc76fe0 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:00:10.600461 6392 consensus_peers.cc:485] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:10.600556 13298 consensus_queue.cc:1035] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:10.603451 13737 tablet.cc:1722] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:10.603521 6309 catalog_manager.cc:5576] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079) does not have the latest schema for tablet cc19d71f40fb44f1801f1b49dcc76fe0 (table rebalancer_dynamic_table [id=ee25cc5084614e039daac0216839e837]). Expected version 1 got 0
I20240503 02:00:10.603969 13737 tablet.cc:1620] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a: MemRowSet was empty: no flush needed.
I20240503 02:00:10.605180 13738 tablet.cc:1722] T cc19d71f40fb44f1801f1b49dcc76fe0 P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:10.605459 13738 tablet.cc:1620] T cc19d71f40fb44f1801f1b49dcc76fe0 P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
I20240503 02:00:10.605732 6314 catalog_manager.cc:5576] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093) does not have the latest schema for tablet cc19d71f40fb44f1801f1b49dcc76fe0 (table rebalancer_dynamic_table [id=ee25cc5084614e039daac0216839e837]). Expected version 1 got 0
I20240503 02:00:10.607450 6314 catalog_manager.cc:5576] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079) does not have the latest schema for tablet cc19d71f40fb44f1801f1b49dcc76fe0 (table rebalancer_dynamic_table [id=ee25cc5084614e039daac0216839e837]). Expected version 1 got 0
I20240503 02:00:10.608047 13720 consensus_queue.cc:1035] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:00:10.611481 8982 consensus_peers.cc:485] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:10.613070 6314 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=ee25cc5084614e039daac0216839e837] alter complete (version 1)
W20240503 02:00:10.625428 6785 consensus_peers.cc:485] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:10.687713 13056 consensus_queue.cc:1035] T 493053c0766648c3bfeaf2490732bbb8 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:10.692517 13056 consensus_queue.cc:1035] T 493053c0766648c3bfeaf2490732bbb8 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:10.704962 9178 raft_consensus.cc:1261] T b2e62c043adf4a8d9024279d786f596c P 2a01ace80baf4057b6bf1995cf88468b [term 2 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 02:00:10.705418 13056 consensus_queue.cc:1035] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:10.707919 8916 raft_consensus.cc:3037] T b2e62c043adf4a8d9024279d786f596c P 9942306855404cfd891adf227dd65eb2 [term 1 LEARNER]: Advancing to term 2
I20240503 02:00:10.709193 8916 raft_consensus.cc:1261] T b2e62c043adf4a8d9024279d786f596c P 9942306855404cfd891adf227dd65eb2 [term 2 LEARNER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 02:00:10.709611 13630 consensus_queue.cc:1035] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:10.711612 13630 raft_consensus.cc:1050] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d: attempting to promote NON_VOTER 9942306855404cfd891adf227dd65eb2 to VOTER
I20240503 02:00:10.712083 13630 consensus_queue.cc:237] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } }
I20240503 02:00:10.713764 8916 raft_consensus.cc:1261] T b2e62c043adf4a8d9024279d786f596c P 9942306855404cfd891adf227dd65eb2 [term 2 LEARNER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:00:10.714210 9178 raft_consensus.cc:1261] T b2e62c043adf4a8d9024279d786f596c P 2a01ace80baf4057b6bf1995cf88468b [term 2 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:00:10.714187 13746 consensus_queue.cc:1035] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:00:10.714603 6785 consensus_peers.cc:485] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:10.714663 13724 consensus_queue.cc:1035] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:10.715966 6314 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:10.716081 6309 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_extra_table_28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:10.716315 13724 raft_consensus.cc:2932] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:10.718605 8916 raft_consensus.cc:2932] T b2e62c043adf4a8d9024279d786f596c P 9942306855404cfd891adf227dd65eb2 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:10.720443 6314 catalog_manager.cc:5484] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d reported cstate change: config changed from index 2 to 4, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:10.723513 13737 tablet.cc:1722] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:10.723886 13737 tablet.cc:1620] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a: MemRowSet was empty: no flush needed.
I20240503 02:00:10.723801 9179 raft_consensus.cc:2932] T b2e62c043adf4a8d9024279d786f596c P 2a01ace80baf4057b6bf1995cf88468b [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:10.725363 13738 tablet.cc:1722] T cc19d71f40fb44f1801f1b49dcc76fe0 P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:10.725716 13738 tablet.cc:1620] T cc19d71f40fb44f1801f1b49dcc76fe0 P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
I20240503 02:00:10.732615 9028 tablet_service.cc:1467] Processing CreateTablet for tablet 99bc944f874248b5a778971f7a2aaa0c (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=56c5cfdd298a4554989547d44cd96854]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:10.732949 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99bc944f874248b5a778971f7a2aaa0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:10.733107 6700 tablet_service.cc:1467] Processing CreateTablet for tablet 8dfe23ff4f2a456384690fa9e48e51c1 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=56c5cfdd298a4554989547d44cd96854]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:10.733386 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8dfe23ff4f2a456384690fa9e48e51c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:10.734169 9158 tablet_service.cc:1467] Processing CreateTablet for tablet 2dfc375064e24f2ea5bba0ae88535f87 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=56c5cfdd298a4554989547d44cd96854]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:10.734169 9157 tablet_service.cc:1467] Processing CreateTablet for tablet 99bc944f874248b5a778971f7a2aaa0c (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=56c5cfdd298a4554989547d44cd96854]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:10.734876 9158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dfc375064e24f2ea5bba0ae88535f87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:10.735560 13749 tablet_bootstrap.cc:492] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:10.735805 13750 tablet_bootstrap.cc:492] T 8dfe23ff4f2a456384690fa9e48e51c1 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:10.736335 6569 tablet_service.cc:1467] Processing CreateTablet for tablet 8dfe23ff4f2a456384690fa9e48e51c1 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=56c5cfdd298a4554989547d44cd96854]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:10.736529 13749 tablet_bootstrap.cc:654] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:10.736629 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8dfe23ff4f2a456384690fa9e48e51c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:10.737087 13181 tablet_bootstrap.cc:492] T 2dfc375064e24f2ea5bba0ae88535f87 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:10.737998 13181 tablet_bootstrap.cc:654] T 2dfc375064e24f2ea5bba0ae88535f87 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:10.738637 9157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99bc944f874248b5a778971f7a2aaa0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:10.738983 8880 tablet_service.cc:1467] Processing CreateTablet for tablet 2dfc375064e24f2ea5bba0ae88535f87 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=56c5cfdd298a4554989547d44cd96854]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:10.739102 13181 tablet_bootstrap.cc:492] T 2dfc375064e24f2ea5bba0ae88535f87 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:10.739200 13181 ts_tablet_manager.cc:1367] T 2dfc375064e24f2ea5bba0ae88535f87 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:10.739244 8880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dfc375064e24f2ea5bba0ae88535f87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:10.739683 13181 raft_consensus.cc:348] T 2dfc375064e24f2ea5bba0ae88535f87 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:10.739784 13181 raft_consensus.cc:374] T 2dfc375064e24f2ea5bba0ae88535f87 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:10.739812 13181 raft_consensus.cc:729] T 2dfc375064e24f2ea5bba0ae88535f87 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:10.739915 13181 consensus_queue.cc:260] T 2dfc375064e24f2ea5bba0ae88535f87 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:10.740206 13181 ts_tablet_manager.cc:1398] T 2dfc375064e24f2ea5bba0ae88535f87 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:10.740758 9028 tablet_service.cc:1467] Processing CreateTablet for tablet 2dfc375064e24f2ea5bba0ae88535f87 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=56c5cfdd298a4554989547d44cd96854]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:10.741009 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dfc375064e24f2ea5bba0ae88535f87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:10.741443 13754 tablet_bootstrap.cc:492] T 2dfc375064e24f2ea5bba0ae88535f87 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:10.742385 13754 tablet_bootstrap.cc:654] T 2dfc375064e24f2ea5bba0ae88535f87 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
W20240503 02:00:10.744119 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:10.744519 13750 tablet_bootstrap.cc:654] T 8dfe23ff4f2a456384690fa9e48e51c1 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:10.745004 13754 tablet_bootstrap.cc:492] T 2dfc375064e24f2ea5bba0ae88535f87 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:10.745101 13754 ts_tablet_manager.cc:1367] T 2dfc375064e24f2ea5bba0ae88535f87 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240503 02:00:10.745566 13754 raft_consensus.cc:348] T 2dfc375064e24f2ea5bba0ae88535f87 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:10.745677 13754 raft_consensus.cc:374] T 2dfc375064e24f2ea5bba0ae88535f87 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:10.745707 13754 raft_consensus.cc:729] T 2dfc375064e24f2ea5bba0ae88535f87 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:10.745824 13754 consensus_queue.cc:260] T 2dfc375064e24f2ea5bba0ae88535f87 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:10.746116 13754 ts_tablet_manager.cc:1398] T 2dfc375064e24f2ea5bba0ae88535f87 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:10.746665 13750 tablet_bootstrap.cc:492] T 8dfe23ff4f2a456384690fa9e48e51c1 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:10.746752 13750 ts_tablet_manager.cc:1367] T 8dfe23ff4f2a456384690fa9e48e51c1 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.001s
I20240503 02:00:10.747174 13181 tablet_bootstrap.cc:492] T 99bc944f874248b5a778971f7a2aaa0c P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:10.747378 6309 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=ee25cc5084614e039daac0216839e837] alter complete (version 2)
I20240503 02:00:10.747596 13750 raft_consensus.cc:348] T 8dfe23ff4f2a456384690fa9e48e51c1 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:10.747720 13750 raft_consensus.cc:374] T 8dfe23ff4f2a456384690fa9e48e51c1 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:10.747750 13750 raft_consensus.cc:729] T 8dfe23ff4f2a456384690fa9e48e51c1 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:10.747900 13750 consensus_queue.cc:260] T 8dfe23ff4f2a456384690fa9e48e51c1 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:10.748453 13750 ts_tablet_manager.cc:1398] T 8dfe23ff4f2a456384690fa9e48e51c1 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:00:10.748667 13181 tablet_bootstrap.cc:654] T 99bc944f874248b5a778971f7a2aaa0c P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:10.749444 13753 tablet_bootstrap.cc:492] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:10.750490 13753 tablet_bootstrap.cc:654] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:10.751243 13181 tablet_bootstrap.cc:492] T 99bc944f874248b5a778971f7a2aaa0c P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:10.751338 13181 ts_tablet_manager.cc:1367] T 99bc944f874248b5a778971f7a2aaa0c P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20240503 02:00:10.751823 13181 raft_consensus.cc:348] T 99bc944f874248b5a778971f7a2aaa0c P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } }
I20240503 02:00:10.751932 13181 raft_consensus.cc:374] T 99bc944f874248b5a778971f7a2aaa0c P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:10.751957 13181 raft_consensus.cc:729] T 99bc944f874248b5a778971f7a2aaa0c P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:10.752080 13181 consensus_queue.cc:260] T 99bc944f874248b5a778971f7a2aaa0c P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } }
I20240503 02:00:10.752152 13749 tablet_bootstrap.cc:492] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:10.752290 13749 ts_tablet_manager.cc:1367] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.002s
I20240503 02:00:10.752336 13181 ts_tablet_manager.cc:1398] T 99bc944f874248b5a778971f7a2aaa0c P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:10.752864 13749 raft_consensus.cc:348] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } }
I20240503 02:00:10.752967 13749 raft_consensus.cc:374] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:10.752996 13749 raft_consensus.cc:729] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:10.753087 13753 tablet_bootstrap.cc:492] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:10.753132 13749 consensus_queue.cc:260] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } }
I20240503 02:00:10.753214 13753 ts_tablet_manager.cc:1367] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:10.753682 13753 raft_consensus.cc:348] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:10.753811 13753 raft_consensus.cc:374] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:10.753863 13753 raft_consensus.cc:729] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:10.754160 13753 consensus_queue.cc:260] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:10.754837 13749 ts_tablet_manager.cc:1398] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 02:00:10.755088 13753 ts_tablet_manager.cc:1398] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:10.755451 13749 tablet_bootstrap.cc:492] T 2dfc375064e24f2ea5bba0ae88535f87 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:10.757295 13749 tablet_bootstrap.cc:654] T 2dfc375064e24f2ea5bba0ae88535f87 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:10.759279 13749 tablet_bootstrap.cc:492] T 2dfc375064e24f2ea5bba0ae88535f87 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:10.759385 13749 ts_tablet_manager.cc:1367] T 2dfc375064e24f2ea5bba0ae88535f87 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:10.759884 13749 raft_consensus.cc:348] T 2dfc375064e24f2ea5bba0ae88535f87 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:10.760015 13749 raft_consensus.cc:374] T 2dfc375064e24f2ea5bba0ae88535f87 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:10.760046 13749 raft_consensus.cc:729] T 2dfc375064e24f2ea5bba0ae88535f87 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:10.760165 13749 consensus_queue.cc:260] T 2dfc375064e24f2ea5bba0ae88535f87 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:10.761327 13749 ts_tablet_manager.cc:1398] T 2dfc375064e24f2ea5bba0ae88535f87 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20240503 02:00:10.841645 6523 consensus_peers.cc:485] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:00:10.850346 6314 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:10.855607 13737 tablet.cc:1722] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:10.855991 13738 tablet.cc:1722] T cc19d71f40fb44f1801f1b49dcc76fe0 P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:10.860743 6314 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=ee25cc5084614e039daac0216839e837] alter complete (version 3)
I20240503 02:00:10.877092 13720 raft_consensus.cc:482] T 2dfc375064e24f2ea5bba0ae88535f87 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:10.877233 13720 raft_consensus.cc:504] T 2dfc375064e24f2ea5bba0ae88535f87 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:10.877626 13720 leader_election.cc:288] T 2dfc375064e24f2ea5bba0ae88535f87 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:10.878077 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dfc375064e24f2ea5bba0ae88535f87" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:10.878113 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dfc375064e24f2ea5bba0ae88535f87" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:10.878235 9179 raft_consensus.cc:2454] T 2dfc375064e24f2ea5bba0ae88535f87 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:10.878250 8916 raft_consensus.cc:2454] T 2dfc375064e24f2ea5bba0ae88535f87 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:10.878579 8984 leader_election.cc:302] T 2dfc375064e24f2ea5bba0ae88535f87 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9942306855404cfd891adf227dd65eb2, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:10.878762 13720 raft_consensus.cc:2781] T 2dfc375064e24f2ea5bba0ae88535f87 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:10.878819 13720 raft_consensus.cc:482] T 2dfc375064e24f2ea5bba0ae88535f87 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:10.878857 13720 raft_consensus.cc:3037] T 2dfc375064e24f2ea5bba0ae88535f87 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:10.879793 13720 raft_consensus.cc:504] T 2dfc375064e24f2ea5bba0ae88535f87 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:10.880168 13720 leader_election.cc:288] T 2dfc375064e24f2ea5bba0ae88535f87 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Requested vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:10.880549 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dfc375064e24f2ea5bba0ae88535f87" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:10.880563 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dfc375064e24f2ea5bba0ae88535f87" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:10.880681 9179 raft_consensus.cc:3037] T 2dfc375064e24f2ea5bba0ae88535f87 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:10.880681 8916 raft_consensus.cc:3037] T 2dfc375064e24f2ea5bba0ae88535f87 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:10.881676 9179 raft_consensus.cc:2454] T 2dfc375064e24f2ea5bba0ae88535f87 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:10.881676 8916 raft_consensus.cc:2454] T 2dfc375064e24f2ea5bba0ae88535f87 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:10.882017 8981 leader_election.cc:302] T 2dfc375064e24f2ea5bba0ae88535f87 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:10.882186 13720 raft_consensus.cc:2781] T 2dfc375064e24f2ea5bba0ae88535f87 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:10.882305 13720 raft_consensus.cc:686] T 2dfc375064e24f2ea5bba0ae88535f87 P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Becoming Leader. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Running, Role: LEADER
I20240503 02:00:10.882483 13720 consensus_queue.cc:237] T 2dfc375064e24f2ea5bba0ae88535f87 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:10.885130 6312 catalog_manager.cc:5484] T 2dfc375064e24f2ea5bba0ae88535f87 P 99ead02915954b4fab23b033ad6d546b reported cstate change: term changed from 0 to 1, leader changed from <none> to 99ead02915954b4fab23b033ad6d546b (127.0.99.134). New cstate: current_term: 1 leader_uuid: "99ead02915954b4fab23b033ad6d546b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:10.963135 6312 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:10.963304 6312 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:10.965520 6312 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet cc19d71f40fb44f1801f1b49dcc76fe0
I20240503 02:00:10.966337 9157 tablet_service.cc:1514] Processing DeleteTablet for tablet cc19d71f40fb44f1801f1b49dcc76fe0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:10 UTC) from {username='slave'} at 127.0.0.1:51940
W20240503 02:00:10.966892 6301 catalog_manager.cc:4573] TS e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cc19d71f40fb44f1801f1b49dcc76fe0: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:10.966920 6436 tablet_service.cc:1514] Processing DeleteTablet for tablet cc19d71f40fb44f1801f1b49dcc76fe0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:10 UTC) from {username='slave'} at 127.0.0.1:51338
I20240503 02:00:10.967237 13764 tablet_replica.cc:330] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a: stopping tablet replica
I20240503 02:00:10.967386 13764 raft_consensus.cc:2229] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:10.967625 13764 raft_consensus.cc:2258] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:10.968178 13764 ts_tablet_manager.cc:1875] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:10.970297 13764 ts_tablet_manager.cc:1888] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:10.970382 13764 log.cc:1196] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/cc19d71f40fb44f1801f1b49dcc76fe0
I20240503 02:00:10.970675 13764 ts_tablet_manager.cc:1909] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a: Deleting consensus metadata
I20240503 02:00:10.971352 6300 catalog_manager.cc:4830] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): tablet cc19d71f40fb44f1801f1b49dcc76fe0 (table rebalancer_dynamic_table_new_name [id=ee25cc5084614e039daac0216839e837]) successfully deleted
I20240503 02:00:10.976523 13763 tablet_replica.cc:330] T cc19d71f40fb44f1801f1b49dcc76fe0 P 2a01ace80baf4057b6bf1995cf88468b: stopping tablet replica
I20240503 02:00:10.976711 13763 raft_consensus.cc:2229] T cc19d71f40fb44f1801f1b49dcc76fe0 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:10.976862 13763 raft_consensus.cc:2258] T cc19d71f40fb44f1801f1b49dcc76fe0 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:10.977480 13763 ts_tablet_manager.cc:1875] T cc19d71f40fb44f1801f1b49dcc76fe0 P 2a01ace80baf4057b6bf1995cf88468b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:10.979625 13763 ts_tablet_manager.cc:1888] T cc19d71f40fb44f1801f1b49dcc76fe0 P 2a01ace80baf4057b6bf1995cf88468b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:10.979699 13763 log.cc:1196] T cc19d71f40fb44f1801f1b49dcc76fe0 P 2a01ace80baf4057b6bf1995cf88468b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/cc19d71f40fb44f1801f1b49dcc76fe0
I20240503 02:00:10.979907 13763 ts_tablet_manager.cc:1909] T cc19d71f40fb44f1801f1b49dcc76fe0 P 2a01ace80baf4057b6bf1995cf88468b: Deleting consensus metadata
I20240503 02:00:10.981001 6300 catalog_manager.cc:4830] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079): tablet cc19d71f40fb44f1801f1b49dcc76fe0 (table rebalancer_dynamic_table_new_name [id=ee25cc5084614e039daac0216839e837]) successfully deleted
I20240503 02:00:11.067387 6312 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:00:11.072795 9157 tablet_service.cc:1467] Processing CreateTablet for tablet 286073252b944a0898aca512e6f3177a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=03276591ff8a400eb2636d7ca6dc2353]), partition=
I20240503 02:00:11.073014 6831 tablet_service.cc:1467] Processing CreateTablet for tablet 286073252b944a0898aca512e6f3177a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=03276591ff8a400eb2636d7ca6dc2353]), partition=
I20240503 02:00:11.073086 9157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 286073252b944a0898aca512e6f3177a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:11.073241 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 286073252b944a0898aca512e6f3177a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:11.075351 13181 tablet_bootstrap.cc:492] T 286073252b944a0898aca512e6f3177a P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:11.075671 8880 tablet_service.cc:1467] Processing CreateTablet for tablet 286073252b944a0898aca512e6f3177a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=03276591ff8a400eb2636d7ca6dc2353]), partition=
I20240503 02:00:11.075847 13765 tablet_bootstrap.cc:492] T 286073252b944a0898aca512e6f3177a P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:11.076026 8880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 286073252b944a0898aca512e6f3177a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:11.076481 13181 tablet_bootstrap.cc:654] T 286073252b944a0898aca512e6f3177a P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:11.076779 13765 tablet_bootstrap.cc:654] T 286073252b944a0898aca512e6f3177a P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:11.078059 13181 tablet_bootstrap.cc:492] T 286073252b944a0898aca512e6f3177a P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:11.078142 13181 ts_tablet_manager.cc:1367] T 286073252b944a0898aca512e6f3177a P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 02:00:11.078202 13754 tablet_bootstrap.cc:492] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:11.078936 13181 raft_consensus.cc:348] T 286073252b944a0898aca512e6f3177a P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:11.079057 13181 raft_consensus.cc:374] T 286073252b944a0898aca512e6f3177a P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:11.079089 13181 raft_consensus.cc:729] T 286073252b944a0898aca512e6f3177a P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:11.079092 13754 tablet_bootstrap.cc:654] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:11.079210 13181 consensus_queue.cc:260] T 286073252b944a0898aca512e6f3177a P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:11.079480 13181 ts_tablet_manager.cc:1398] T 286073252b944a0898aca512e6f3177a P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:11.080930 13754 tablet_bootstrap.cc:492] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:11.081033 13754 ts_tablet_manager.cc:1367] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:00:11.081552 13754 raft_consensus.cc:348] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:11.081660 13754 raft_consensus.cc:374] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:11.081689 13754 raft_consensus.cc:729] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:11.081810 13754 consensus_queue.cc:260] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:11.082091 13754 ts_tablet_manager.cc:1398] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:11.083158 13765 tablet_bootstrap.cc:492] T 286073252b944a0898aca512e6f3177a P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:11.083317 13765 ts_tablet_manager.cc:1367] T 286073252b944a0898aca512e6f3177a P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 02:00:11.083912 13765 raft_consensus.cc:348] T 286073252b944a0898aca512e6f3177a P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:11.084110 13765 raft_consensus.cc:374] T 286073252b944a0898aca512e6f3177a P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:11.084185 13765 raft_consensus.cc:729] T 286073252b944a0898aca512e6f3177a P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:11.084365 13765 consensus_queue.cc:260] T 286073252b944a0898aca512e6f3177a P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:11.084769 13765 ts_tablet_manager.cc:1398] T 286073252b944a0898aca512e6f3177a P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20240503 02:00:11.090085 8851 consensus_peers.cc:485] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:11.096484 8851 consensus_peers.cc:485] T 067f54011a37432d99157884e70fc601 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:00:11.134701 12331 raft_consensus.cc:482] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:11.134850 12331 raft_consensus.cc:504] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:11.135337 12331 leader_election.cc:288] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311)
I20240503 02:00:11.135766 6719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dfe23ff4f2a456384690fa9e48e51c1" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 02:00:11.135916 6719 raft_consensus.cc:2454] T 8dfe23ff4f2a456384690fa9e48e51c1 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:11.136233 6525 leader_election.cc:302] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
W20240503 02:00:11.136406 6523 leader_election.cc:334] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:11.136413 12331 raft_consensus.cc:2781] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:11.136493 12331 raft_consensus.cc:482] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:11.136538 12331 raft_consensus.cc:3037] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:11.137398 12331 raft_consensus.cc:504] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:11.137753 12331 leader_election.cc:288] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Requested vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311)
I20240503 02:00:11.138114 6719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dfe23ff4f2a456384690fa9e48e51c1" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 02:00:11.138242 6719 raft_consensus.cc:3037] T 8dfe23ff4f2a456384690fa9e48e51c1 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:11.139035 6719 raft_consensus.cc:2454] T 8dfe23ff4f2a456384690fa9e48e51c1 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:11.139335 6525 leader_election.cc:302] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:11.139501 12331 raft_consensus.cc:2781] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:11.139570 12331 raft_consensus.cc:686] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Becoming Leader. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Running, Role: LEADER
I20240503 02:00:11.139707 12331 consensus_queue.cc:237] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
W20240503 02:00:11.139734 6523 leader_election.cc:334] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:11.142150 6312 catalog_manager.cc:5484] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } }
W20240503 02:00:11.160152 6654 consensus_peers.cc:485] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:11.165657 6523 consensus_peers.cc:485] T 4385d51d11574192883d01376e2d5620 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:11.169885 6785 consensus_peers.cc:485] T c3485701bf3b48f5ae2e1568afeed133 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:00:11.190601 12922 raft_consensus.cc:482] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:11.190730 12922 raft_consensus.cc:504] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } }
I20240503 02:00:11.191128 12922 leader_election.cc:288] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311)
I20240503 02:00:11.191573 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99bc944f874248b5a778971f7a2aaa0c" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:11.191746 9179 raft_consensus.cc:2454] T 99bc944f874248b5a778971f7a2aaa0c P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:11.192066 8981 leader_election.cc:302] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 99ead02915954b4fab23b033ad6d546b; no voters:
W20240503 02:00:11.192206 8982 leader_election.cc:334] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:11.192258 12922 raft_consensus.cc:2781] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:11.192296 12922 raft_consensus.cc:482] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:11.192318 12922 raft_consensus.cc:3037] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:11.193259 12922 raft_consensus.cc:504] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } }
I20240503 02:00:11.193600 12922 leader_election.cc:288] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Requested vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311)
I20240503 02:00:11.193924 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99bc944f874248b5a778971f7a2aaa0c" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:11.194053 9179 raft_consensus.cc:3037] T 99bc944f874248b5a778971f7a2aaa0c P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:00:11.194561 8982 leader_election.cc:334] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:11.194974 9179 raft_consensus.cc:2454] T 99bc944f874248b5a778971f7a2aaa0c P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:11.195246 8981 leader_election.cc:302] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 99ead02915954b4fab23b033ad6d546b; no voters: e1a9251ce1cf4361b8b0ede9fbfc4023
I20240503 02:00:11.195391 12922 raft_consensus.cc:2781] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:11.195475 12922 raft_consensus.cc:686] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Becoming Leader. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Running, Role: LEADER
I20240503 02:00:11.195614 12922 consensus_queue.cc:237] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } }
I20240503 02:00:11.213079 6312 catalog_manager.cc:5484] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b reported cstate change: term changed from 0 to 1, leader changed from <none> to 99ead02915954b4fab23b033ad6d546b (127.0.99.134). New cstate: current_term: 1 leader_uuid: "99ead02915954b4fab23b033ad6d546b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } health_report { overall_health: UNKNOWN } } }
W20240503 02:00:11.232805 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
W20240503 02:00:11.260202 6523 consensus_peers.cc:485] T 925612b660ec471da5b3c548802ede6e P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:11.262396 6785 consensus_peers.cc:485] T 46ad540deb624a25b97d3b967fe1f6f4 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:00:11.269199 12922 consensus_queue.cc:1035] T 2dfc375064e24f2ea5bba0ae88535f87 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:11.273387 12922 consensus_queue.cc:1035] T 2dfc375064e24f2ea5bba0ae88535f87 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:00:11.281561 6654 consensus_peers.cc:485] T ba17d4d3e0754b2798eb95484d37229a P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:00:11.283671 13102 raft_consensus.cc:482] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:11.283797 13102 raft_consensus.cc:504] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:11.284194 13102 leader_election.cc:288] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:11.284698 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286073252b944a0898aca512e6f3177a" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:11.284698 6850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286073252b944a0898aca512e6f3177a" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 02:00:11.284849 6850 raft_consensus.cc:2454] T 286073252b944a0898aca512e6f3177a P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 0.
I20240503 02:00:11.284849 9179 raft_consensus.cc:2454] T 286073252b944a0898aca512e6f3177a P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 0.
I20240503 02:00:11.285157 8853 leader_election.cc:302] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:11.285391 13102 raft_consensus.cc:2781] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:11.285442 13102 raft_consensus.cc:482] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:11.285476 13102 raft_consensus.cc:3037] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:11.286321 13102 raft_consensus.cc:504] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:11.286695 13102 leader_election.cc:288] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 election: Requested vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:11.287073 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286073252b944a0898aca512e6f3177a" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:11.287232 9179 raft_consensus.cc:3037] T 286073252b944a0898aca512e6f3177a P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:11.287263 6850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286073252b944a0898aca512e6f3177a" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 02:00:11.287386 6850 raft_consensus.cc:3037] T 286073252b944a0898aca512e6f3177a P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:11.288122 9179 raft_consensus.cc:2454] T 286073252b944a0898aca512e6f3177a P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 1.
I20240503 02:00:11.288127 6850 raft_consensus.cc:2454] T 286073252b944a0898aca512e6f3177a P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 1.
I20240503 02:00:11.288434 8853 leader_election.cc:302] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:11.288610 13102 raft_consensus.cc:2781] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:11.288691 13102 raft_consensus.cc:686] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2 [term 1 LEADER]: Becoming Leader. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Running, Role: LEADER
I20240503 02:00:11.288861 13102 consensus_queue.cc:237] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:11.290953 6312 catalog_manager.cc:5484] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9942306855404cfd891adf227dd65eb2 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "9942306855404cfd891adf227dd65eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: HEALTHY } } }
W20240503 02:00:11.297222 6523 consensus_peers.cc:485] T d0d5dd42db3e4883af7fd16d8813b550 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:11.357892 6523 consensus_peers.cc:485] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:11.376219 8851 consensus_peers.cc:485] T 7da8f614a70e4177b70e7c2f31d22a10 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:11.379478 8982 consensus_peers.cc:485] T 13756a6e44814fe4b09a6b0f087546c3 P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:11.401546 6785 consensus_peers.cc:485] T 8246488dec204bf6b09e868d7951ac5a P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:00:11.414672 6312 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:11.418668 6850 raft_consensus.cc:1261] T 286073252b944a0898aca512e6f3177a P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:11.419095 9179 raft_consensus.cc:1261] T 286073252b944a0898aca512e6f3177a P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:11.419153 13102 consensus_queue.cc:1035] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:11.419526 13102 consensus_queue.cc:1035] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:11.422124 13781 tablet.cc:1722] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:11.422173 6312 catalog_manager.cc:5576] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719) does not have the latest schema for tablet 286073252b944a0898aca512e6f3177a (table rebalancer_dynamic_table [id=03276591ff8a400eb2636d7ca6dc2353]). Expected version 1 got 0
I20240503 02:00:11.422575 13781 tablet.cc:1620] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
I20240503 02:00:11.424854 6314 catalog_manager.cc:5576] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079) does not have the latest schema for tablet 286073252b944a0898aca512e6f3177a (table rebalancer_dynamic_table [id=03276591ff8a400eb2636d7ca6dc2353]). Expected version 1 got 0
I20240503 02:00:11.425967 13783 tablet.cc:1722] T 286073252b944a0898aca512e6f3177a P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:11.427362 13784 tablet.cc:1722] T 286073252b944a0898aca512e6f3177a P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:11.427781 13784 tablet.cc:1620] T 286073252b944a0898aca512e6f3177a P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
I20240503 02:00:11.427866 6309 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=03276591ff8a400eb2636d7ca6dc2353] alter complete (version 1)
I20240503 02:00:11.428416 6314 catalog_manager.cc:5576] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079) does not have the latest schema for tablet 286073252b944a0898aca512e6f3177a (table rebalancer_dynamic_table [id=03276591ff8a400eb2636d7ca6dc2353]). Expected version 1 got 0
I20240503 02:00:11.430562 13783 tablet.cc:1620] T 286073252b944a0898aca512e6f3177a P 492448570cc94ed1879feb2ec3b9051d: MemRowSet was empty: no flush needed.
W20240503 02:00:11.440701 6785 consensus_peers.cc:485] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:00:11.533891 6314 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:11.539758 13781 tablet.cc:1722] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:11.539884 13784 tablet.cc:1722] T 286073252b944a0898aca512e6f3177a P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:11.540084 13783 tablet.cc:1722] T 286073252b944a0898aca512e6f3177a P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:11.540148 13781 tablet.cc:1620] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
I20240503 02:00:11.540194 13784 tablet.cc:1620] T 286073252b944a0898aca512e6f3177a P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
I20240503 02:00:11.540349 13783 tablet.cc:1620] T 286073252b944a0898aca512e6f3177a P 492448570cc94ed1879feb2ec3b9051d: MemRowSet was empty: no flush needed.
I20240503 02:00:11.546074 6309 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=03276591ff8a400eb2636d7ca6dc2353] alter complete (version 2)
W20240503 02:00:11.563577 9113 consensus_peers.cc:485] T f44b59277c3b47adbd4dd6aff9350af1 P 2a01ace80baf4057b6bf1995cf88468b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:11.573029 8851 consensus_peers.cc:485] T f28586e70bd9473fbdc6b9126b666fa9 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:11.574934 9113 consensus_peers.cc:485] T b1740b877c1e4372a0337d99f5bcab0e P 2a01ace80baf4057b6bf1995cf88468b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:11.581688 8851 consensus_peers.cc:485] T 5f51fe5f799f4e479b62f5917a2c2606 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:11.605531 6785 consensus_peers.cc:485] T d897c21e630d4770832bb10f43778e82 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:11.638689 8982 consensus_peers.cc:485] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:11.648957 6312 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:11.655812 13781 tablet.cc:1722] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:11.656965 13783 tablet.cc:1722] T 286073252b944a0898aca512e6f3177a P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:11.657963 13784 tablet.cc:1722] T 286073252b944a0898aca512e6f3177a P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:11.664798 6309 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=03276591ff8a400eb2636d7ca6dc2353] alter complete (version 3)
W20240503 02:00:11.682772 8982 consensus_peers.cc:485] T 1a7ee79312ea4bad816264375ca5bb2d P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:00:11.685608 12922 consensus_queue.cc:1035] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:11.700259 12331 consensus_queue.cc:1035] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:00:11.704149 6523 consensus_peers.cc:485] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:11.764353 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:11.766753 6312 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:11.766929 6312 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:11.769258 6312 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet 286073252b944a0898aca512e6f3177a
I20240503 02:00:11.770069 8880 tablet_service.cc:1514] Processing DeleteTablet for tablet 286073252b944a0898aca512e6f3177a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:11 UTC) from {username='slave'} at 127.0.0.1:40296
I20240503 02:00:11.770135 6831 tablet_service.cc:1514] Processing DeleteTablet for tablet 286073252b944a0898aca512e6f3177a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:11 UTC) from {username='slave'} at 127.0.0.1:52812
I20240503 02:00:11.770488 9157 tablet_service.cc:1514] Processing DeleteTablet for tablet 286073252b944a0898aca512e6f3177a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:11 UTC) from {username='slave'} at 127.0.0.1:51940
I20240503 02:00:11.770776 13794 tablet_replica.cc:330] T 286073252b944a0898aca512e6f3177a P 2a01ace80baf4057b6bf1995cf88468b: stopping tablet replica
I20240503 02:00:11.770934 13794 raft_consensus.cc:2229] T 286073252b944a0898aca512e6f3177a P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:11.771085 13794 raft_consensus.cc:2258] T 286073252b944a0898aca512e6f3177a P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:11.780067 13793 tablet_replica.cc:330] T 286073252b944a0898aca512e6f3177a P 492448570cc94ed1879feb2ec3b9051d: stopping tablet replica
I20240503 02:00:11.780248 13793 raft_consensus.cc:2229] T 286073252b944a0898aca512e6f3177a P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:11.780431 13793 raft_consensus.cc:2258] T 286073252b944a0898aca512e6f3177a P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:11.780689 13792 tablet_replica.cc:330] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2: stopping tablet replica
I20240503 02:00:11.780840 13792 raft_consensus.cc:2229] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:11.781067 13792 raft_consensus.cc:2258] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:11.781605 13793 ts_tablet_manager.cc:1875] T 286073252b944a0898aca512e6f3177a P 492448570cc94ed1879feb2ec3b9051d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:11.781730 13792 ts_tablet_manager.cc:1875] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:11.783849 13794 ts_tablet_manager.cc:1875] T 286073252b944a0898aca512e6f3177a P 2a01ace80baf4057b6bf1995cf88468b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:11.784215 13793 ts_tablet_manager.cc:1888] T 286073252b944a0898aca512e6f3177a P 492448570cc94ed1879feb2ec3b9051d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:11.784298 13793 log.cc:1196] T 286073252b944a0898aca512e6f3177a P 492448570cc94ed1879feb2ec3b9051d: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/286073252b944a0898aca512e6f3177a
I20240503 02:00:11.784346 13792 ts_tablet_manager.cc:1888] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:11.784484 13792 log.cc:1196] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/286073252b944a0898aca512e6f3177a
I20240503 02:00:11.784710 13793 ts_tablet_manager.cc:1909] T 286073252b944a0898aca512e6f3177a P 492448570cc94ed1879feb2ec3b9051d: Deleting consensus metadata
I20240503 02:00:11.784770 13792 ts_tablet_manager.cc:1909] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2: Deleting consensus metadata
I20240503 02:00:11.785611 6303 catalog_manager.cc:4830] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211): tablet 286073252b944a0898aca512e6f3177a (table rebalancer_dynamic_table_new_name [id=03276591ff8a400eb2636d7ca6dc2353]) successfully deleted
I20240503 02:00:11.785799 6303 catalog_manager.cc:4830] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719): tablet 286073252b944a0898aca512e6f3177a (table rebalancer_dynamic_table_new_name [id=03276591ff8a400eb2636d7ca6dc2353]) successfully deleted
I20240503 02:00:11.786248 13794 ts_tablet_manager.cc:1888] T 286073252b944a0898aca512e6f3177a P 2a01ace80baf4057b6bf1995cf88468b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:11.786321 13794 log.cc:1196] T 286073252b944a0898aca512e6f3177a P 2a01ace80baf4057b6bf1995cf88468b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/286073252b944a0898aca512e6f3177a
I20240503 02:00:11.786530 13794 ts_tablet_manager.cc:1909] T 286073252b944a0898aca512e6f3177a P 2a01ace80baf4057b6bf1995cf88468b: Deleting consensus metadata
I20240503 02:00:11.787319 6300 catalog_manager.cc:4830] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079): tablet 286073252b944a0898aca512e6f3177a (table rebalancer_dynamic_table_new_name [id=03276591ff8a400eb2636d7ca6dc2353]) successfully deleted
I20240503 02:00:11.795964 6312 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_extra_table_30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:11.802423 8880 tablet_service.cc:1467] Processing CreateTablet for tablet 5625259b3ac54b4ea639e2823902aee5 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=21fd6878ffc94acaa25ab621ead27223]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:11.802762 8880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5625259b3ac54b4ea639e2823902aee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:11.802850 8897 tablet_service.cc:1467] Processing CreateTablet for tablet e8466d03df4c4ed8bf571bffd936f515 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=21fd6878ffc94acaa25ab621ead27223]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:11.803103 8897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8466d03df4c4ed8bf571bffd936f515. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:11.803051 9028 tablet_service.cc:1467] Processing CreateTablet for tablet d244f21f8c6a4ab296e9a08f50d9c990 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=21fd6878ffc94acaa25ab621ead27223]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:11.803304 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d244f21f8c6a4ab296e9a08f50d9c990. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:11.805178 6700 tablet_service.cc:1467] Processing CreateTablet for tablet d244f21f8c6a4ab296e9a08f50d9c990 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=21fd6878ffc94acaa25ab621ead27223]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:11.805491 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d244f21f8c6a4ab296e9a08f50d9c990. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:11.805554 6438 tablet_service.cc:1467] Processing CreateTablet for tablet e8466d03df4c4ed8bf571bffd936f515 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=21fd6878ffc94acaa25ab621ead27223]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:11.805809 6438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8466d03df4c4ed8bf571bffd936f515. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:11.805805 13796 tablet_bootstrap.cc:492] T d244f21f8c6a4ab296e9a08f50d9c990 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:11.807004 13796 tablet_bootstrap.cc:654] T d244f21f8c6a4ab296e9a08f50d9c990 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:11.807544 13797 tablet_bootstrap.cc:492] T e8466d03df4c4ed8bf571bffd936f515 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:11.808056 6436 tablet_service.cc:1467] Processing CreateTablet for tablet 5625259b3ac54b4ea639e2823902aee5 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=21fd6878ffc94acaa25ab621ead27223]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:11.808821 13798 tablet_bootstrap.cc:492] T d244f21f8c6a4ab296e9a08f50d9c990 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:11.808835 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5625259b3ac54b4ea639e2823902aee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:11.808877 9158 tablet_service.cc:1467] Processing CreateTablet for tablet e8466d03df4c4ed8bf571bffd936f515 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=21fd6878ffc94acaa25ab621ead27223]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:11.809146 9158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8466d03df4c4ed8bf571bffd936f515. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:11.809842 13798 tablet_bootstrap.cc:654] T d244f21f8c6a4ab296e9a08f50d9c990 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:11.810004 13797 tablet_bootstrap.cc:654] T e8466d03df4c4ed8bf571bffd936f515 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:11.811108 13797 tablet_bootstrap.cc:492] T e8466d03df4c4ed8bf571bffd936f515 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:11.811244 13797 ts_tablet_manager.cc:1367] T e8466d03df4c4ed8bf571bffd936f515 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:11.811424 9159 tablet_service.cc:1467] Processing CreateTablet for tablet d244f21f8c6a4ab296e9a08f50d9c990 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=21fd6878ffc94acaa25ab621ead27223]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:11.811515 13795 tablet_bootstrap.cc:492] T 5625259b3ac54b4ea639e2823902aee5 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:11.811698 9159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d244f21f8c6a4ab296e9a08f50d9c990. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:11.811733 13797 raft_consensus.cc:348] T e8466d03df4c4ed8bf571bffd936f515 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:11.811841 13797 raft_consensus.cc:374] T e8466d03df4c4ed8bf571bffd936f515 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:11.811872 13797 raft_consensus.cc:729] T e8466d03df4c4ed8bf571bffd936f515 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:11.811988 13797 consensus_queue.cc:260] T e8466d03df4c4ed8bf571bffd936f515 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:11.812785 13797 ts_tablet_manager.cc:1398] T e8466d03df4c4ed8bf571bffd936f515 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:11.812813 13795 tablet_bootstrap.cc:654] T 5625259b3ac54b4ea639e2823902aee5 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:11.812983 13797 tablet_bootstrap.cc:492] T 5625259b3ac54b4ea639e2823902aee5 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:11.814216 13798 tablet_bootstrap.cc:492] T d244f21f8c6a4ab296e9a08f50d9c990 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:11.814304 13798 ts_tablet_manager.cc:1367] T d244f21f8c6a4ab296e9a08f50d9c990 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:00:11.814435 13797 tablet_bootstrap.cc:654] T 5625259b3ac54b4ea639e2823902aee5 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:11.814769 13798 raft_consensus.cc:348] T d244f21f8c6a4ab296e9a08f50d9c990 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:11.814880 13798 raft_consensus.cc:374] T d244f21f8c6a4ab296e9a08f50d9c990 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:11.814911 13798 raft_consensus.cc:729] T d244f21f8c6a4ab296e9a08f50d9c990 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:11.815037 13798 consensus_queue.cc:260] T d244f21f8c6a4ab296e9a08f50d9c990 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:11.815313 13798 ts_tablet_manager.cc:1398] T d244f21f8c6a4ab296e9a08f50d9c990 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:11.815407 9157 tablet_service.cc:1467] Processing CreateTablet for tablet 5625259b3ac54b4ea639e2823902aee5 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=21fd6878ffc94acaa25ab621ead27223]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:11.815791 9157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5625259b3ac54b4ea639e2823902aee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:11.821483 13801 tablet_bootstrap.cc:492] T e8466d03df4c4ed8bf571bffd936f515 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:11.822060 13796 tablet_bootstrap.cc:492] T d244f21f8c6a4ab296e9a08f50d9c990 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:11.822187 13796 ts_tablet_manager.cc:1367] T d244f21f8c6a4ab296e9a08f50d9c990 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240503 02:00:11.822767 13796 raft_consensus.cc:348] T d244f21f8c6a4ab296e9a08f50d9c990 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:11.822914 13796 raft_consensus.cc:374] T d244f21f8c6a4ab296e9a08f50d9c990 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:11.822948 13796 raft_consensus.cc:729] T d244f21f8c6a4ab296e9a08f50d9c990 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:11.823083 13796 consensus_queue.cc:260] T d244f21f8c6a4ab296e9a08f50d9c990 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:11.823418 13796 ts_tablet_manager.cc:1398] T d244f21f8c6a4ab296e9a08f50d9c990 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 02:00:11.824713 13801 tablet_bootstrap.cc:654] T e8466d03df4c4ed8bf571bffd936f515 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:11.826112 13795 tablet_bootstrap.cc:492] T 5625259b3ac54b4ea639e2823902aee5 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:11.826234 13795 ts_tablet_manager.cc:1367] T 5625259b3ac54b4ea639e2823902aee5 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240503 02:00:11.826395 13801 tablet_bootstrap.cc:492] T e8466d03df4c4ed8bf571bffd936f515 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:11.826489 13801 ts_tablet_manager.cc:1367] T e8466d03df4c4ed8bf571bffd936f515 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:00:11.826766 13795 raft_consensus.cc:348] T 5625259b3ac54b4ea639e2823902aee5 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:11.826900 13795 raft_consensus.cc:374] T 5625259b3ac54b4ea639e2823902aee5 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:11.827067 13795 raft_consensus.cc:729] T 5625259b3ac54b4ea639e2823902aee5 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:11.827299 13795 consensus_queue.cc:260] T 5625259b3ac54b4ea639e2823902aee5 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:11.827822 13801 raft_consensus.cc:348] T e8466d03df4c4ed8bf571bffd936f515 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:11.828044 13801 raft_consensus.cc:374] T e8466d03df4c4ed8bf571bffd936f515 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:11.828326 13801 raft_consensus.cc:729] T e8466d03df4c4ed8bf571bffd936f515 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:11.828617 13801 consensus_queue.cc:260] T e8466d03df4c4ed8bf571bffd936f515 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:11.828743 13795 ts_tablet_manager.cc:1398] T 5625259b3ac54b4ea639e2823902aee5 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:11.829104 13795 tablet_bootstrap.cc:492] T e8466d03df4c4ed8bf571bffd936f515 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:11.829104 13801 ts_tablet_manager.cc:1398] T e8466d03df4c4ed8bf571bffd936f515 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:11.829298 13801 tablet_bootstrap.cc:492] T 5625259b3ac54b4ea639e2823902aee5 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:11.830183 13795 tablet_bootstrap.cc:654] T e8466d03df4c4ed8bf571bffd936f515 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:11.830724 13801 tablet_bootstrap.cc:654] T 5625259b3ac54b4ea639e2823902aee5 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:11.831463 13797 tablet_bootstrap.cc:492] T 5625259b3ac54b4ea639e2823902aee5 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:11.831657 13797 ts_tablet_manager.cc:1367] T 5625259b3ac54b4ea639e2823902aee5 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240503 02:00:11.832895 13801 tablet_bootstrap.cc:492] T 5625259b3ac54b4ea639e2823902aee5 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:11.833369 13801 ts_tablet_manager.cc:1367] T 5625259b3ac54b4ea639e2823902aee5 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:11.834008 13801 raft_consensus.cc:348] T 5625259b3ac54b4ea639e2823902aee5 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:11.834194 13801 raft_consensus.cc:374] T 5625259b3ac54b4ea639e2823902aee5 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:11.834265 13801 raft_consensus.cc:729] T 5625259b3ac54b4ea639e2823902aee5 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:11.834463 13801 consensus_queue.cc:260] T 5625259b3ac54b4ea639e2823902aee5 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:11.834667 13797 raft_consensus.cc:348] T 5625259b3ac54b4ea639e2823902aee5 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:11.834851 13797 raft_consensus.cc:374] T 5625259b3ac54b4ea639e2823902aee5 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:11.834905 13801 ts_tablet_manager.cc:1398] T 5625259b3ac54b4ea639e2823902aee5 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:11.835143 13797 raft_consensus.cc:729] T 5625259b3ac54b4ea639e2823902aee5 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:11.835242 13801 tablet_bootstrap.cc:492] T d244f21f8c6a4ab296e9a08f50d9c990 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:11.836436 13801 tablet_bootstrap.cc:654] T d244f21f8c6a4ab296e9a08f50d9c990 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:11.836997 13795 tablet_bootstrap.cc:492] T e8466d03df4c4ed8bf571bffd936f515 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:11.837111 13795 ts_tablet_manager.cc:1367] T e8466d03df4c4ed8bf571bffd936f515 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 02:00:11.837015 13797 consensus_queue.cc:260] T 5625259b3ac54b4ea639e2823902aee5 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:11.837430 13797 ts_tablet_manager.cc:1398] T 5625259b3ac54b4ea639e2823902aee5 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:00:11.837610 13795 raft_consensus.cc:348] T e8466d03df4c4ed8bf571bffd936f515 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:11.837725 13795 raft_consensus.cc:374] T e8466d03df4c4ed8bf571bffd936f515 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:11.837754 13795 raft_consensus.cc:729] T e8466d03df4c4ed8bf571bffd936f515 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:11.837884 13795 consensus_queue.cc:260] T e8466d03df4c4ed8bf571bffd936f515 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:11.838153 13795 ts_tablet_manager.cc:1398] T e8466d03df4c4ed8bf571bffd936f515 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:11.841984 13801 tablet_bootstrap.cc:492] T d244f21f8c6a4ab296e9a08f50d9c990 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:11.842078 13801 ts_tablet_manager.cc:1367] T d244f21f8c6a4ab296e9a08f50d9c990 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:00:11.842489 13801 raft_consensus.cc:348] T d244f21f8c6a4ab296e9a08f50d9c990 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:11.842589 13801 raft_consensus.cc:374] T d244f21f8c6a4ab296e9a08f50d9c990 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:11.842618 13801 raft_consensus.cc:729] T d244f21f8c6a4ab296e9a08f50d9c990 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:11.842716 13801 consensus_queue.cc:260] T d244f21f8c6a4ab296e9a08f50d9c990 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:11.842984 13801 ts_tablet_manager.cc:1398] T d244f21f8c6a4ab296e9a08f50d9c990 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:11.871536 6314 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:00:11.877476 9028 tablet_service.cc:1467] Processing CreateTablet for tablet 141d454209d3476c93071c291e16a44c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=55bf5f42f2d9482bad4ceb69fb30c3e5]), partition=
I20240503 02:00:11.877791 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 141d454209d3476c93071c291e16a44c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:11.878105 8897 tablet_service.cc:1467] Processing CreateTablet for tablet 141d454209d3476c93071c291e16a44c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=55bf5f42f2d9482bad4ceb69fb30c3e5]), partition=
I20240503 02:00:11.878317 8897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 141d454209d3476c93071c291e16a44c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:11.880143 13796 tablet_bootstrap.cc:492] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:11.880362 6569 tablet_service.cc:1467] Processing CreateTablet for tablet 141d454209d3476c93071c291e16a44c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=55bf5f42f2d9482bad4ceb69fb30c3e5]), partition=
I20240503 02:00:11.880625 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 141d454209d3476c93071c291e16a44c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:11.880838 13795 tablet_bootstrap.cc:492] T 141d454209d3476c93071c291e16a44c P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:11.881297 13796 tablet_bootstrap.cc:654] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:11.881884 13795 tablet_bootstrap.cc:654] T 141d454209d3476c93071c291e16a44c P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:11.882944 13795 tablet_bootstrap.cc:492] T 141d454209d3476c93071c291e16a44c P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:11.882972 13811 tablet_bootstrap.cc:492] T 141d454209d3476c93071c291e16a44c P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:11.883139 13795 ts_tablet_manager.cc:1367] T 141d454209d3476c93071c291e16a44c P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:11.883206 13796 tablet_bootstrap.cc:492] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:11.883296 13796 ts_tablet_manager.cc:1367] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:11.883728 13796 raft_consensus.cc:348] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:11.883708 13795 raft_consensus.cc:348] T 141d454209d3476c93071c291e16a44c P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:11.883834 13796 raft_consensus.cc:374] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:11.883863 13811 tablet_bootstrap.cc:654] T 141d454209d3476c93071c291e16a44c P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:11.883867 13795 raft_consensus.cc:374] T 141d454209d3476c93071c291e16a44c P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:11.883884 13796 raft_consensus.cc:729] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:11.883908 13795 raft_consensus.cc:729] T 141d454209d3476c93071c291e16a44c P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:11.884013 13796 consensus_queue.cc:260] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:11.884101 13795 consensus_queue.cc:260] T 141d454209d3476c93071c291e16a44c P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:11.884308 13796 ts_tablet_manager.cc:1398] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:11.885303 13795 ts_tablet_manager.cc:1398] T 141d454209d3476c93071c291e16a44c P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:11.886150 13811 tablet_bootstrap.cc:492] T 141d454209d3476c93071c291e16a44c P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:11.886284 13811 ts_tablet_manager.cc:1367] T 141d454209d3476c93071c291e16a44c P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:00:11.886790 13811 raft_consensus.cc:348] T 141d454209d3476c93071c291e16a44c P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:11.886935 13811 raft_consensus.cc:374] T 141d454209d3476c93071c291e16a44c P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:11.886981 13811 raft_consensus.cc:729] T 141d454209d3476c93071c291e16a44c P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:11.887141 13811 consensus_queue.cc:260] T 141d454209d3476c93071c291e16a44c P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:11.887445 13811 ts_tablet_manager.cc:1398] T 141d454209d3476c93071c291e16a44c P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:11.921594 13791 raft_consensus.cc:482] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:11.921730 13791 raft_consensus.cc:504] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:11.922191 13791 leader_election.cc:288] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:11.922555 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141d454209d3476c93071c291e16a44c" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 02:00:11.922657 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141d454209d3476c93071c291e16a44c" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:11.922724 6589 raft_consensus.cc:2454] T 141d454209d3476c93071c291e16a44c P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:11.922789 8916 raft_consensus.cc:2454] T 141d454209d3476c93071c291e16a44c P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:11.923044 8983 leader_election.cc:302] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:11.923254 13791 raft_consensus.cc:2781] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:11.923305 13791 raft_consensus.cc:482] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:11.923339 13791 raft_consensus.cc:3037] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:11.924515 13791 raft_consensus.cc:504] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:11.924994 13791 leader_election.cc:288] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Requested vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:11.925323 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141d454209d3476c93071c291e16a44c" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:11.925323 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141d454209d3476c93071c291e16a44c" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 02:00:11.925451 8916 raft_consensus.cc:3037] T 141d454209d3476c93071c291e16a44c P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:11.925451 6589 raft_consensus.cc:3037] T 141d454209d3476c93071c291e16a44c P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:11.926462 8916 raft_consensus.cc:2454] T 141d454209d3476c93071c291e16a44c P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:11.926462 6589 raft_consensus.cc:2454] T 141d454209d3476c93071c291e16a44c P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:11.926774 8984 leader_election.cc:302] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9942306855404cfd891adf227dd65eb2, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:11.926991 13791 raft_consensus.cc:2781] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:11.927093 13791 raft_consensus.cc:686] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Becoming Leader. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Running, Role: LEADER
I20240503 02:00:11.927240 13791 consensus_queue.cc:237] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:11.929544 6314 catalog_manager.cc:5484] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b reported cstate change: term changed from 0 to 1, leader changed from <none> to 99ead02915954b4fab23b033ad6d546b (127.0.99.134). New cstate: current_term: 1 leader_uuid: "99ead02915954b4fab23b033ad6d546b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } }
W20240503 02:00:11.960851 8982 consensus_peers.cc:485] T beeea71f02ae4c89a1af13d3584ef70a P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:00:11.998099 13776 raft_consensus.cc:482] T e8466d03df4c4ed8bf571bffd936f515 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:11.998248 13776 raft_consensus.cc:504] T e8466d03df4c4ed8bf571bffd936f515 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:11.998700 13776 leader_election.cc:288] T e8466d03df4c4ed8bf571bffd936f515 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:11.999089 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8466d03df4c4ed8bf571bffd936f515" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:11.999143 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8466d03df4c4ed8bf571bffd936f515" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 02:00:11.999228 8916 raft_consensus.cc:2454] T e8466d03df4c4ed8bf571bffd936f515 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:11.999325 6456 raft_consensus.cc:2454] T e8466d03df4c4ed8bf571bffd936f515 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:11.999482 9115 leader_election.cc:302] T e8466d03df4c4ed8bf571bffd936f515 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:11.999677 13776 raft_consensus.cc:2781] T e8466d03df4c4ed8bf571bffd936f515 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:11.999729 13776 raft_consensus.cc:482] T e8466d03df4c4ed8bf571bffd936f515 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:11.999764 13776 raft_consensus.cc:3037] T e8466d03df4c4ed8bf571bffd936f515 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:12.000902 13776 raft_consensus.cc:504] T e8466d03df4c4ed8bf571bffd936f515 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:12.001307 13776 leader_election.cc:288] T e8466d03df4c4ed8bf571bffd936f515 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Requested vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:12.001657 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8466d03df4c4ed8bf571bffd936f515" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 02:00:12.001690 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8466d03df4c4ed8bf571bffd936f515" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:12.001778 6456 raft_consensus.cc:3037] T e8466d03df4c4ed8bf571bffd936f515 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:12.001822 8916 raft_consensus.cc:3037] T e8466d03df4c4ed8bf571bffd936f515 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:12.002611 6456 raft_consensus.cc:2454] T e8466d03df4c4ed8bf571bffd936f515 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:12.002918 8916 raft_consensus.cc:2454] T e8466d03df4c4ed8bf571bffd936f515 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:12.003052 9112 leader_election.cc:302] T e8466d03df4c4ed8bf571bffd936f515 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:12.003583 13776 raft_consensus.cc:2781] T e8466d03df4c4ed8bf571bffd936f515 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:12.003854 13776 raft_consensus.cc:686] T e8466d03df4c4ed8bf571bffd936f515 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Becoming Leader. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Running, Role: LEADER
I20240503 02:00:12.004048 13776 consensus_queue.cc:237] T e8466d03df4c4ed8bf571bffd936f515 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:12.006366 6314 catalog_manager.cc:5484] T e8466d03df4c4ed8bf571bffd936f515 P 2a01ace80baf4057b6bf1995cf88468b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135). New cstate: current_term: 1 leader_uuid: "2a01ace80baf4057b6bf1995cf88468b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:12.034338 6314 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:12.038699 8916 raft_consensus.cc:1261] T 141d454209d3476c93071c291e16a44c P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Refusing update from remote peer 99ead02915954b4fab23b033ad6d546b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:12.038699 6589 raft_consensus.cc:1261] T 141d454209d3476c93071c291e16a44c P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer 99ead02915954b4fab23b033ad6d546b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:12.039188 12922 raft_consensus.cc:482] T d244f21f8c6a4ab296e9a08f50d9c990 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:12.039301 12922 raft_consensus.cc:504] T d244f21f8c6a4ab296e9a08f50d9c990 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:12.039295 13791 consensus_queue.cc:1035] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:12.039481 13720 consensus_queue.cc:1035] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:12.039754 12922 leader_election.cc:288] T d244f21f8c6a4ab296e9a08f50d9c990 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:12.040277 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d244f21f8c6a4ab296e9a08f50d9c990" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:12.040441 9179 raft_consensus.cc:2454] T d244f21f8c6a4ab296e9a08f50d9c990 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:12.040699 8981 leader_election.cc:302] T d244f21f8c6a4ab296e9a08f50d9c990 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:12.040855 13791 raft_consensus.cc:2781] T d244f21f8c6a4ab296e9a08f50d9c990 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:12.040939 13791 raft_consensus.cc:482] T d244f21f8c6a4ab296e9a08f50d9c990 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:12.040983 13791 raft_consensus.cc:3037] T d244f21f8c6a4ab296e9a08f50d9c990 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:12.041100 6719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d244f21f8c6a4ab296e9a08f50d9c990" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 02:00:12.041245 6719 raft_consensus.cc:2454] T d244f21f8c6a4ab296e9a08f50d9c990 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:12.041859 13791 raft_consensus.cc:504] T d244f21f8c6a4ab296e9a08f50d9c990 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:12.042222 13791 leader_election.cc:288] T d244f21f8c6a4ab296e9a08f50d9c990 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Requested vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:12.042531 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d244f21f8c6a4ab296e9a08f50d9c990" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:12.042646 9179 raft_consensus.cc:3037] T d244f21f8c6a4ab296e9a08f50d9c990 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:12.042820 6719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d244f21f8c6a4ab296e9a08f50d9c990" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 02:00:12.042930 6719 raft_consensus.cc:3037] T d244f21f8c6a4ab296e9a08f50d9c990 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:12.043416 6314 catalog_manager.cc:5576] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211) does not have the latest schema for tablet 141d454209d3476c93071c291e16a44c (table rebalancer_dynamic_table [id=55bf5f42f2d9482bad4ceb69fb30c3e5]). Expected version 1 got 0
I20240503 02:00:12.043895 9179 raft_consensus.cc:2454] T d244f21f8c6a4ab296e9a08f50d9c990 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:12.043895 6719 raft_consensus.cc:2454] T d244f21f8c6a4ab296e9a08f50d9c990 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:12.044214 8981 leader_election.cc:302] T d244f21f8c6a4ab296e9a08f50d9c990 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:12.044363 13791 raft_consensus.cc:2781] T d244f21f8c6a4ab296e9a08f50d9c990 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:12.044476 13791 raft_consensus.cc:686] T d244f21f8c6a4ab296e9a08f50d9c990 P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Becoming Leader. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Running, Role: LEADER
I20240503 02:00:12.044631 13791 consensus_queue.cc:237] T d244f21f8c6a4ab296e9a08f50d9c990 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:12.046669 6314 catalog_manager.cc:5576] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863) does not have the latest schema for tablet 141d454209d3476c93071c291e16a44c (table rebalancer_dynamic_table [id=55bf5f42f2d9482bad4ceb69fb30c3e5]). Expected version 1 got 0
I20240503 02:00:12.047564 6313 catalog_manager.cc:5484] T d244f21f8c6a4ab296e9a08f50d9c990 P 99ead02915954b4fab23b033ad6d546b reported cstate change: term changed from 0 to 1, leader changed from <none> to 99ead02915954b4fab23b033ad6d546b (127.0.99.134). New cstate: current_term: 1 leader_uuid: "99ead02915954b4fab23b033ad6d546b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:12.047950 13820 tablet.cc:1722] T 141d454209d3476c93071c291e16a44c P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:12.048202 13781 tablet.cc:1722] T 141d454209d3476c93071c291e16a44c P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:12.048343 13820 tablet.cc:1620] T 141d454209d3476c93071c291e16a44c P 80fd18ef4d604c86b909d5c008a46b4f: MemRowSet was empty: no flush needed.
I20240503 02:00:12.048538 13781 tablet.cc:1620] T 141d454209d3476c93071c291e16a44c P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
I20240503 02:00:12.046614 13819 tablet.cc:1722] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:12.053874 6314 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=55bf5f42f2d9482bad4ceb69fb30c3e5] alter complete (version 1)
I20240503 02:00:12.054417 13819 tablet.cc:1620] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b: MemRowSet was empty: no flush needed.
I20240503 02:00:12.157114 6309 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:12.164134 13819 tablet.cc:1722] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:12.164857 13819 tablet.cc:1620] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b: MemRowSet was empty: no flush needed.
I20240503 02:00:12.164999 13781 tablet.cc:1722] T 141d454209d3476c93071c291e16a44c P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:12.165298 13781 tablet.cc:1620] T 141d454209d3476c93071c291e16a44c P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
I20240503 02:00:12.167824 13814 raft_consensus.cc:482] T 5625259b3ac54b4ea639e2823902aee5 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:12.168004 13814 raft_consensus.cc:504] T 5625259b3ac54b4ea639e2823902aee5 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:12.168642 13814 leader_election.cc:288] T 5625259b3ac54b4ea639e2823902aee5 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 02:00:12.169306 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5625259b3ac54b4ea639e2823902aee5" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 02:00:12.169440 6456 raft_consensus.cc:2454] T 5625259b3ac54b4ea639e2823902aee5 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:12.169471 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5625259b3ac54b4ea639e2823902aee5" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:12.169612 8916 raft_consensus.cc:2454] T 5625259b3ac54b4ea639e2823902aee5 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:12.169592 13820 tablet.cc:1722] T 141d454209d3476c93071c291e16a44c P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:12.169867 9115 leader_election.cc:302] T 5625259b3ac54b4ea639e2823902aee5 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:12.169909 13820 tablet.cc:1620] T 141d454209d3476c93071c291e16a44c P 80fd18ef4d604c86b909d5c008a46b4f: MemRowSet was empty: no flush needed.
I20240503 02:00:12.170428 13814 raft_consensus.cc:2781] T 5625259b3ac54b4ea639e2823902aee5 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:12.170534 13814 raft_consensus.cc:482] T 5625259b3ac54b4ea639e2823902aee5 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:12.170621 13814 raft_consensus.cc:3037] T 5625259b3ac54b4ea639e2823902aee5 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:12.173409 13814 raft_consensus.cc:504] T 5625259b3ac54b4ea639e2823902aee5 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:12.173636 6314 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=55bf5f42f2d9482bad4ceb69fb30c3e5] alter complete (version 2)
I20240503 02:00:12.175136 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5625259b3ac54b4ea639e2823902aee5" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:12.175469 8916 raft_consensus.cc:3037] T 5625259b3ac54b4ea639e2823902aee5 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:12.176617 8916 raft_consensus.cc:2454] T 5625259b3ac54b4ea639e2823902aee5 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:12.176645 13814 leader_election.cc:288] T 5625259b3ac54b4ea639e2823902aee5 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Requested vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 02:00:12.176918 9115 leader_election.cc:302] T 5625259b3ac54b4ea639e2823902aee5 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:12.177119 13814 raft_consensus.cc:2781] T 5625259b3ac54b4ea639e2823902aee5 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:12.177079 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5625259b3ac54b4ea639e2823902aee5" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 02:00:12.177217 6456 raft_consensus.cc:3037] T 5625259b3ac54b4ea639e2823902aee5 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:12.177210 13814 raft_consensus.cc:686] T 5625259b3ac54b4ea639e2823902aee5 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Becoming Leader. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Running, Role: LEADER
I20240503 02:00:12.177359 13814 consensus_queue.cc:237] T 5625259b3ac54b4ea639e2823902aee5 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:12.178584 6456 raft_consensus.cc:2454] T 5625259b3ac54b4ea639e2823902aee5 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:12.179916 6309 catalog_manager.cc:5484] T 5625259b3ac54b4ea639e2823902aee5 P 2a01ace80baf4057b6bf1995cf88468b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135). New cstate: current_term: 1 leader_uuid: "2a01ace80baf4057b6bf1995cf88468b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } }
W20240503 02:00:12.205469 6301 catalog_manager.cc:4573] TS e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cc19d71f40fb44f1801f1b49dcc76fe0: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
W20240503 02:00:12.264587 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:12.276485 6309 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:12.282011 13819 tablet.cc:1722] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:12.282416 13781 tablet.cc:1722] T 141d454209d3476c93071c291e16a44c P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:12.282420 13820 tablet.cc:1722] T 141d454209d3476c93071c291e16a44c P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:12.287559 6313 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=55bf5f42f2d9482bad4ceb69fb30c3e5] alter complete (version 3)
W20240503 02:00:12.309191 6654 consensus_peers.cc:485] T cea1ea50fa434d31b42451f84f26bc76 P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:00:12.390223 6309 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:12.390383 6309 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:12.392812 6309 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet 141d454209d3476c93071c291e16a44c
I20240503 02:00:12.393450 6569 tablet_service.cc:1514] Processing DeleteTablet for tablet 141d454209d3476c93071c291e16a44c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:12 UTC) from {username='slave'} at 127.0.0.1:59846
I20240503 02:00:12.393529 8897 tablet_service.cc:1514] Processing DeleteTablet for tablet 141d454209d3476c93071c291e16a44c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:12 UTC) from {username='slave'} at 127.0.0.1:40296
I20240503 02:00:12.393770 9028 tablet_service.cc:1514] Processing DeleteTablet for tablet 141d454209d3476c93071c291e16a44c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:12 UTC) from {username='slave'} at 127.0.0.1:52584
I20240503 02:00:12.393791 13822 tablet_replica.cc:330] T 141d454209d3476c93071c291e16a44c P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 02:00:12.393955 13822 raft_consensus.cc:2229] T 141d454209d3476c93071c291e16a44c P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:12.393966 13823 tablet_replica.cc:330] T 141d454209d3476c93071c291e16a44c P 9942306855404cfd891adf227dd65eb2: stopping tablet replica
I20240503 02:00:12.394045 13824 tablet_replica.cc:330] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b: stopping tablet replica
I20240503 02:00:12.394081 13822 raft_consensus.cc:2258] T 141d454209d3476c93071c291e16a44c P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:12.394104 13823 raft_consensus.cc:2229] T 141d454209d3476c93071c291e16a44c P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:12.394167 13824 raft_consensus.cc:2229] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:12.394274 13823 raft_consensus.cc:2258] T 141d454209d3476c93071c291e16a44c P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:12.394410 13824 raft_consensus.cc:2258] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:12.394670 13822 ts_tablet_manager.cc:1875] T 141d454209d3476c93071c291e16a44c P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:12.394879 13823 ts_tablet_manager.cc:1875] T 141d454209d3476c93071c291e16a44c P 9942306855404cfd891adf227dd65eb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:12.394940 13824 ts_tablet_manager.cc:1875] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:12.397526 13824 ts_tablet_manager.cc:1888] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:12.397658 13824 log.cc:1196] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/141d454209d3476c93071c291e16a44c
I20240503 02:00:12.397799 13822 ts_tablet_manager.cc:1888] T 141d454209d3476c93071c291e16a44c P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:12.397917 13822 log.cc:1196] T 141d454209d3476c93071c291e16a44c P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/141d454209d3476c93071c291e16a44c
I20240503 02:00:12.398124 13823 ts_tablet_manager.cc:1888] T 141d454209d3476c93071c291e16a44c P 9942306855404cfd891adf227dd65eb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:12.398108 13824 ts_tablet_manager.cc:1909] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b: Deleting consensus metadata
I20240503 02:00:12.398195 13823 log.cc:1196] T 141d454209d3476c93071c291e16a44c P 9942306855404cfd891adf227dd65eb2: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/141d454209d3476c93071c291e16a44c
I20240503 02:00:12.398458 13823 ts_tablet_manager.cc:1909] T 141d454209d3476c93071c291e16a44c P 9942306855404cfd891adf227dd65eb2: Deleting consensus metadata
I20240503 02:00:12.398512 13822 ts_tablet_manager.cc:1909] T 141d454209d3476c93071c291e16a44c P 80fd18ef4d604c86b909d5c008a46b4f: Deleting consensus metadata
I20240503 02:00:12.399319 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet 141d454209d3476c93071c291e16a44c (table rebalancer_dynamic_table_new_name [id=55bf5f42f2d9482bad4ceb69fb30c3e5]) successfully deleted
I20240503 02:00:12.399319 6303 catalog_manager.cc:4830] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211): tablet 141d454209d3476c93071c291e16a44c (table rebalancer_dynamic_table_new_name [id=55bf5f42f2d9482bad4ceb69fb30c3e5]) successfully deleted
I20240503 02:00:12.399724 6301 catalog_manager.cc:4830] TS 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157): tablet 141d454209d3476c93071c291e16a44c (table rebalancer_dynamic_table_new_name [id=55bf5f42f2d9482bad4ceb69fb30c3e5]) successfully deleted
I20240503 02:00:12.429766 13814 consensus_queue.cc:1035] T e8466d03df4c4ed8bf571bffd936f515 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:12.433298 13814 consensus_queue.cc:1035] T e8466d03df4c4ed8bf571bffd936f515 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:12.494925 6314 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:00:12.500411 8897 tablet_service.cc:1467] Processing CreateTablet for tablet 7b69d44bbbfc408c8d46f54d21f8a427 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=271a62be2be84ada961794cdfcb28433]), partition=
I20240503 02:00:12.500403 6831 tablet_service.cc:1467] Processing CreateTablet for tablet 7b69d44bbbfc408c8d46f54d21f8a427 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=271a62be2be84ada961794cdfcb28433]), partition=
I20240503 02:00:12.500412 6569 tablet_service.cc:1467] Processing CreateTablet for tablet 7b69d44bbbfc408c8d46f54d21f8a427 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=271a62be2be84ada961794cdfcb28433]), partition=
I20240503 02:00:12.500699 8897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b69d44bbbfc408c8d46f54d21f8a427. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:12.500697 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b69d44bbbfc408c8d46f54d21f8a427. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:12.500697 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b69d44bbbfc408c8d46f54d21f8a427. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:12.502714 13830 tablet_bootstrap.cc:492] T 7b69d44bbbfc408c8d46f54d21f8a427 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:12.503940 13830 tablet_bootstrap.cc:654] T 7b69d44bbbfc408c8d46f54d21f8a427 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:12.504058 13831 tablet_bootstrap.cc:492] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:12.504271 13832 tablet_bootstrap.cc:492] T 7b69d44bbbfc408c8d46f54d21f8a427 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:12.505255 13832 tablet_bootstrap.cc:654] T 7b69d44bbbfc408c8d46f54d21f8a427 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:12.505255 13831 tablet_bootstrap.cc:654] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:12.505967 13830 tablet_bootstrap.cc:492] T 7b69d44bbbfc408c8d46f54d21f8a427 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:12.506084 13830 ts_tablet_manager.cc:1367] T 7b69d44bbbfc408c8d46f54d21f8a427 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:12.506438 13831 tablet_bootstrap.cc:492] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:12.506552 13831 ts_tablet_manager.cc:1367] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:12.506592 13832 tablet_bootstrap.cc:492] T 7b69d44bbbfc408c8d46f54d21f8a427 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:12.506554 13830 raft_consensus.cc:348] T 7b69d44bbbfc408c8d46f54d21f8a427 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:12.506680 13830 raft_consensus.cc:374] T 7b69d44bbbfc408c8d46f54d21f8a427 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:12.506698 13832 ts_tablet_manager.cc:1367] T 7b69d44bbbfc408c8d46f54d21f8a427 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:12.506717 13830 raft_consensus.cc:729] T 7b69d44bbbfc408c8d46f54d21f8a427 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:12.506842 13830 consensus_queue.cc:260] T 7b69d44bbbfc408c8d46f54d21f8a427 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:12.507048 13831 raft_consensus.cc:348] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:12.507135 13830 ts_tablet_manager.cc:1398] T 7b69d44bbbfc408c8d46f54d21f8a427 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:12.507222 13831 raft_consensus.cc:374] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:12.507254 13832 raft_consensus.cc:348] T 7b69d44bbbfc408c8d46f54d21f8a427 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:12.507315 13831 raft_consensus.cc:729] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:12.507388 13832 raft_consensus.cc:374] T 7b69d44bbbfc408c8d46f54d21f8a427 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:12.507448 13832 raft_consensus.cc:729] T 7b69d44bbbfc408c8d46f54d21f8a427 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:12.507498 13831 consensus_queue.cc:260] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:12.507580 13832 consensus_queue.cc:260] T 7b69d44bbbfc408c8d46f54d21f8a427 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:12.507787 13831 ts_tablet_manager.cc:1398] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:12.510342 13832 ts_tablet_manager.cc:1398] T 7b69d44bbbfc408c8d46f54d21f8a427 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 02:00:12.562429 13814 consensus_queue.cc:1035] T 5625259b3ac54b4ea639e2823902aee5 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:12.567183 13814 consensus_queue.cc:1035] T 5625259b3ac54b4ea639e2823902aee5 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:12.568960 13791 consensus_queue.cc:1035] T d244f21f8c6a4ab296e9a08f50d9c990 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:12.572906 13720 consensus_queue.cc:1035] T d244f21f8c6a4ab296e9a08f50d9c990 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:12.691711 12331 raft_consensus.cc:482] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:12.691861 12331 raft_consensus.cc:504] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:12.692270 12331 leader_election.cc:288] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:12.692777 6850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b69d44bbbfc408c8d46f54d21f8a427" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 02:00:12.692775 8915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b69d44bbbfc408c8d46f54d21f8a427" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:12.692920 6850 raft_consensus.cc:2454] T 7b69d44bbbfc408c8d46f54d21f8a427 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:12.692929 8915 raft_consensus.cc:2454] T 7b69d44bbbfc408c8d46f54d21f8a427 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:12.693221 6525 leader_election.cc:302] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:12.693400 12331 raft_consensus.cc:2781] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:12.693485 12331 raft_consensus.cc:482] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:12.693536 12331 raft_consensus.cc:3037] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:12.694661 12331 raft_consensus.cc:504] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:12.695031 12331 leader_election.cc:288] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Requested vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:12.695436 8915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b69d44bbbfc408c8d46f54d21f8a427" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:12.695466 6850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b69d44bbbfc408c8d46f54d21f8a427" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 02:00:12.695554 8915 raft_consensus.cc:3037] T 7b69d44bbbfc408c8d46f54d21f8a427 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:12.695593 6850 raft_consensus.cc:3037] T 7b69d44bbbfc408c8d46f54d21f8a427 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:12.696717 8915 raft_consensus.cc:2454] T 7b69d44bbbfc408c8d46f54d21f8a427 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:12.696743 6850 raft_consensus.cc:2454] T 7b69d44bbbfc408c8d46f54d21f8a427 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:12.697089 6525 leader_election.cc:302] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:12.697294 12331 raft_consensus.cc:2781] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:12.697386 12331 raft_consensus.cc:686] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Becoming Leader. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Running, Role: LEADER
I20240503 02:00:12.697691 12331 consensus_queue.cc:237] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:12.699725 6312 catalog_manager.cc:5484] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:12.767403 6312 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_extra_table_32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:12.774739 6436 tablet_service.cc:1467] Processing CreateTablet for tablet 71c44cb6883a495797823588f7c2b4a3 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=89c19ab2e5c145068b45e07be7e5ca99]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:12.775092 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71c44cb6883a495797823588f7c2b4a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:12.775086 9025 tablet_service.cc:1467] Processing CreateTablet for tablet ccf7a4ad8b1d44efa6548772c271d497 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=89c19ab2e5c145068b45e07be7e5ca99]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:12.775326 9025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccf7a4ad8b1d44efa6548772c271d497. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:12.776718 9028 tablet_service.cc:1467] Processing CreateTablet for tablet 9ba716e7e1a04275be92428ccbd651b8 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=89c19ab2e5c145068b45e07be7e5ca99]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:12.776993 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ba716e7e1a04275be92428ccbd651b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:12.779417 9159 tablet_service.cc:1467] Processing CreateTablet for tablet 71c44cb6883a495797823588f7c2b4a3 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=89c19ab2e5c145068b45e07be7e5ca99]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:12.779706 9159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71c44cb6883a495797823588f7c2b4a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:12.781716 13846 tablet_bootstrap.cc:492] T 71c44cb6883a495797823588f7c2b4a3 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:12.781967 9157 tablet_service.cc:1467] Processing CreateTablet for tablet 9ba716e7e1a04275be92428ccbd651b8 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=89c19ab2e5c145068b45e07be7e5ca99]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:12.782224 9157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ba716e7e1a04275be92428ccbd651b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:12.782974 13846 tablet_bootstrap.cc:654] T 71c44cb6883a495797823588f7c2b4a3 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:12.783753 13844 tablet_bootstrap.cc:492] T ccf7a4ad8b1d44efa6548772c271d497 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
W20240503 02:00:12.783754 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:12.784508 13846 tablet_bootstrap.cc:492] T 71c44cb6883a495797823588f7c2b4a3 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:12.784740 9158 tablet_service.cc:1467] Processing CreateTablet for tablet ccf7a4ad8b1d44efa6548772c271d497 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=89c19ab2e5c145068b45e07be7e5ca99]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:12.784799 13846 ts_tablet_manager.cc:1367] T 71c44cb6883a495797823588f7c2b4a3 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:12.785027 9158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccf7a4ad8b1d44efa6548772c271d497. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:12.785339 13846 raft_consensus.cc:348] T 71c44cb6883a495797823588f7c2b4a3 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:12.785436 13846 raft_consensus.cc:374] T 71c44cb6883a495797823588f7c2b4a3 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:12.785461 13846 raft_consensus.cc:729] T 71c44cb6883a495797823588f7c2b4a3 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:12.785578 13846 consensus_queue.cc:260] T 71c44cb6883a495797823588f7c2b4a3 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:12.785874 13846 ts_tablet_manager.cc:1398] T 71c44cb6883a495797823588f7c2b4a3 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:12.786052 13846 tablet_bootstrap.cc:492] T 9ba716e7e1a04275be92428ccbd651b8 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:12.786608 13844 tablet_bootstrap.cc:654] T ccf7a4ad8b1d44efa6548772c271d497 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:12.786952 13846 tablet_bootstrap.cc:654] T 9ba716e7e1a04275be92428ccbd651b8 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:12.787817 13844 tablet_bootstrap.cc:492] T ccf7a4ad8b1d44efa6548772c271d497 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:12.787922 13844 ts_tablet_manager.cc:1367] T ccf7a4ad8b1d44efa6548772c271d497 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:12.788347 13844 raft_consensus.cc:348] T ccf7a4ad8b1d44efa6548772c271d497 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:12.788509 13844 raft_consensus.cc:374] T ccf7a4ad8b1d44efa6548772c271d497 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:12.788549 13844 raft_consensus.cc:729] T ccf7a4ad8b1d44efa6548772c271d497 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:12.788686 13844 consensus_queue.cc:260] T ccf7a4ad8b1d44efa6548772c271d497 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:12.788990 13844 ts_tablet_manager.cc:1398] T ccf7a4ad8b1d44efa6548772c271d497 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:12.789163 13844 tablet_bootstrap.cc:492] T 9ba716e7e1a04275be92428ccbd651b8 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:12.789165 6831 tablet_service.cc:1467] Processing CreateTablet for tablet 71c44cb6883a495797823588f7c2b4a3 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=89c19ab2e5c145068b45e07be7e5ca99]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:12.789180 8897 tablet_service.cc:1467] Processing CreateTablet for tablet 9ba716e7e1a04275be92428ccbd651b8 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=89c19ab2e5c145068b45e07be7e5ca99]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:12.789448 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71c44cb6883a495797823588f7c2b4a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:12.789458 8897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ba716e7e1a04275be92428ccbd651b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:12.790076 13844 tablet_bootstrap.cc:654] T 9ba716e7e1a04275be92428ccbd651b8 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:12.791591 13832 tablet_bootstrap.cc:492] T 71c44cb6883a495797823588f7c2b4a3 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:12.792481 13830 tablet_bootstrap.cc:492] T 9ba716e7e1a04275be92428ccbd651b8 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:12.792615 13832 tablet_bootstrap.cc:654] T 71c44cb6883a495797823588f7c2b4a3 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:12.793370 13830 tablet_bootstrap.cc:654] T 9ba716e7e1a04275be92428ccbd651b8 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:12.793613 13844 tablet_bootstrap.cc:492] T 9ba716e7e1a04275be92428ccbd651b8 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:12.793613 13832 tablet_bootstrap.cc:492] T 71c44cb6883a495797823588f7c2b4a3 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:12.793692 13844 ts_tablet_manager.cc:1367] T 9ba716e7e1a04275be92428ccbd651b8 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 02:00:12.793692 13832 ts_tablet_manager.cc:1367] T 71c44cb6883a495797823588f7c2b4a3 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:12.794147 13832 raft_consensus.cc:348] T 71c44cb6883a495797823588f7c2b4a3 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:12.794147 13844 raft_consensus.cc:348] T 9ba716e7e1a04275be92428ccbd651b8 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:12.794253 13832 raft_consensus.cc:374] T 71c44cb6883a495797823588f7c2b4a3 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:12.794261 13844 raft_consensus.cc:374] T 9ba716e7e1a04275be92428ccbd651b8 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:12.794284 13832 raft_consensus.cc:729] T 71c44cb6883a495797823588f7c2b4a3 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:12.794291 13844 raft_consensus.cc:729] T 9ba716e7e1a04275be92428ccbd651b8 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:12.794411 13832 consensus_queue.cc:260] T 71c44cb6883a495797823588f7c2b4a3 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:12.794421 13844 consensus_queue.cc:260] T 9ba716e7e1a04275be92428ccbd651b8 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:12.794695 13844 ts_tablet_manager.cc:1398] T 9ba716e7e1a04275be92428ccbd651b8 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:12.794703 13832 ts_tablet_manager.cc:1398] T 71c44cb6883a495797823588f7c2b4a3 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:12.797761 13845 tablet_bootstrap.cc:492] T 71c44cb6883a495797823588f7c2b4a3 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:12.800113 8897 tablet_service.cc:1467] Processing CreateTablet for tablet ccf7a4ad8b1d44efa6548772c271d497 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=89c19ab2e5c145068b45e07be7e5ca99]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:12.800455 8897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccf7a4ad8b1d44efa6548772c271d497. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:12.801244 13845 tablet_bootstrap.cc:654] T 71c44cb6883a495797823588f7c2b4a3 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:12.802744 13846 tablet_bootstrap.cc:492] T 9ba716e7e1a04275be92428ccbd651b8 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:12.802953 13846 ts_tablet_manager.cc:1367] T 9ba716e7e1a04275be92428ccbd651b8 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240503 02:00:12.803028 13845 tablet_bootstrap.cc:492] T 71c44cb6883a495797823588f7c2b4a3 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:12.803117 13845 ts_tablet_manager.cc:1367] T 71c44cb6883a495797823588f7c2b4a3 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 02:00:12.803587 13845 raft_consensus.cc:348] T 71c44cb6883a495797823588f7c2b4a3 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:12.803586 13846 raft_consensus.cc:348] T 9ba716e7e1a04275be92428ccbd651b8 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:12.803710 13845 raft_consensus.cc:374] T 71c44cb6883a495797823588f7c2b4a3 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:12.803747 13845 raft_consensus.cc:729] T 71c44cb6883a495797823588f7c2b4a3 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:12.803843 13846 raft_consensus.cc:374] T 9ba716e7e1a04275be92428ccbd651b8 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:12.803917 13846 raft_consensus.cc:729] T 9ba716e7e1a04275be92428ccbd651b8 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:12.803882 13845 consensus_queue.cc:260] T 71c44cb6883a495797823588f7c2b4a3 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:12.804433 13846 consensus_queue.cc:260] T 9ba716e7e1a04275be92428ccbd651b8 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:12.804781 13830 tablet_bootstrap.cc:492] T 9ba716e7e1a04275be92428ccbd651b8 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:12.804891 13830 ts_tablet_manager.cc:1367] T 9ba716e7e1a04275be92428ccbd651b8 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20240503 02:00:12.805114 13845 ts_tablet_manager.cc:1398] T 71c44cb6883a495797823588f7c2b4a3 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:12.805397 13830 raft_consensus.cc:348] T 9ba716e7e1a04275be92428ccbd651b8 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:12.805514 13830 raft_consensus.cc:374] T 9ba716e7e1a04275be92428ccbd651b8 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:12.805523 13846 ts_tablet_manager.cc:1398] T 9ba716e7e1a04275be92428ccbd651b8 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:12.805542 13830 raft_consensus.cc:729] T 9ba716e7e1a04275be92428ccbd651b8 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:12.805703 13846 tablet_bootstrap.cc:492] T ccf7a4ad8b1d44efa6548772c271d497 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:12.805747 13830 consensus_queue.cc:260] T 9ba716e7e1a04275be92428ccbd651b8 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:12.806300 13830 ts_tablet_manager.cc:1398] T 9ba716e7e1a04275be92428ccbd651b8 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:12.806607 13830 tablet_bootstrap.cc:492] T ccf7a4ad8b1d44efa6548772c271d497 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:12.806682 13846 tablet_bootstrap.cc:654] T ccf7a4ad8b1d44efa6548772c271d497 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:12.807884 13830 tablet_bootstrap.cc:654] T ccf7a4ad8b1d44efa6548772c271d497 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:12.812252 13846 tablet_bootstrap.cc:492] T ccf7a4ad8b1d44efa6548772c271d497 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:12.812325 13830 tablet_bootstrap.cc:492] T ccf7a4ad8b1d44efa6548772c271d497 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:12.812482 13846 ts_tablet_manager.cc:1367] T ccf7a4ad8b1d44efa6548772c271d497 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:00:12.812494 13830 ts_tablet_manager.cc:1367] T ccf7a4ad8b1d44efa6548772c271d497 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20240503 02:00:12.812966 13846 raft_consensus.cc:348] T ccf7a4ad8b1d44efa6548772c271d497 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:12.812989 13830 raft_consensus.cc:348] T ccf7a4ad8b1d44efa6548772c271d497 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:12.813076 13846 raft_consensus.cc:374] T ccf7a4ad8b1d44efa6548772c271d497 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:12.813105 13846 raft_consensus.cc:729] T ccf7a4ad8b1d44efa6548772c271d497 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:12.813103 13830 raft_consensus.cc:374] T ccf7a4ad8b1d44efa6548772c271d497 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:12.813138 13830 raft_consensus.cc:729] T ccf7a4ad8b1d44efa6548772c271d497 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:12.813222 13846 consensus_queue.cc:260] T ccf7a4ad8b1d44efa6548772c271d497 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:12.813257 13830 consensus_queue.cc:260] T ccf7a4ad8b1d44efa6548772c271d497 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:12.813508 13846 ts_tablet_manager.cc:1398] T ccf7a4ad8b1d44efa6548772c271d497 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:12.813608 13830 ts_tablet_manager.cc:1398] T ccf7a4ad8b1d44efa6548772c271d497 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:12.834424 13102 raft_consensus.cc:482] T ccf7a4ad8b1d44efa6548772c271d497 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:12.834561 13102 raft_consensus.cc:504] T ccf7a4ad8b1d44efa6548772c271d497 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:12.834980 13102 leader_election.cc:288] T ccf7a4ad8b1d44efa6548772c271d497 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:12.835331 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccf7a4ad8b1d44efa6548772c271d497" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:12.835331 9039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccf7a4ad8b1d44efa6548772c271d497" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:12.835465 9039 raft_consensus.cc:2454] T ccf7a4ad8b1d44efa6548772c271d497 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 0.
I20240503 02:00:12.835464 9179 raft_consensus.cc:2454] T ccf7a4ad8b1d44efa6548772c271d497 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 0.
I20240503 02:00:12.835896 8850 leader_election.cc:302] T ccf7a4ad8b1d44efa6548772c271d497 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:12.836093 13102 raft_consensus.cc:2781] T ccf7a4ad8b1d44efa6548772c271d497 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:12.836150 13102 raft_consensus.cc:482] T ccf7a4ad8b1d44efa6548772c271d497 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:12.836189 13102 raft_consensus.cc:3037] T ccf7a4ad8b1d44efa6548772c271d497 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:12.837275 13102 raft_consensus.cc:504] T ccf7a4ad8b1d44efa6548772c271d497 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:12.837646 13102 leader_election.cc:288] T ccf7a4ad8b1d44efa6548772c271d497 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 election: Requested vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:12.837951 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccf7a4ad8b1d44efa6548772c271d497" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:12.838060 9179 raft_consensus.cc:3037] T ccf7a4ad8b1d44efa6548772c271d497 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:12.838238 9039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccf7a4ad8b1d44efa6548772c271d497" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:12.838430 9039 raft_consensus.cc:3037] T ccf7a4ad8b1d44efa6548772c271d497 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:12.839102 9179 raft_consensus.cc:2454] T ccf7a4ad8b1d44efa6548772c271d497 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 1.
I20240503 02:00:12.839286 9039 raft_consensus.cc:2454] T ccf7a4ad8b1d44efa6548772c271d497 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 1.
I20240503 02:00:12.839527 8850 leader_election.cc:302] T ccf7a4ad8b1d44efa6548772c271d497 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:12.839697 13102 raft_consensus.cc:2781] T ccf7a4ad8b1d44efa6548772c271d497 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:12.839849 13102 raft_consensus.cc:686] T ccf7a4ad8b1d44efa6548772c271d497 P 9942306855404cfd891adf227dd65eb2 [term 1 LEADER]: Becoming Leader. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Running, Role: LEADER
I20240503 02:00:12.840013 13102 consensus_queue.cc:237] T ccf7a4ad8b1d44efa6548772c271d497 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:12.842272 6309 catalog_manager.cc:5484] T ccf7a4ad8b1d44efa6548772c271d497 P 9942306855404cfd891adf227dd65eb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9942306855404cfd891adf227dd65eb2 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "9942306855404cfd891adf227dd65eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:12.844110 6313 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:12.849185 8915 raft_consensus.cc:1261] T 7b69d44bbbfc408c8d46f54d21f8a427 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:12.849179 6850 raft_consensus.cc:1261] T 7b69d44bbbfc408c8d46f54d21f8a427 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:12.849578 13836 consensus_queue.cc:1035] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:12.849745 12331 consensus_queue.cc:1035] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:12.852850 6313 catalog_manager.cc:5576] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719) does not have the latest schema for tablet 7b69d44bbbfc408c8d46f54d21f8a427 (table rebalancer_dynamic_table [id=271a62be2be84ada961794cdfcb28433]). Expected version 1 got 0
I20240503 02:00:12.853242 6309 catalog_manager.cc:5576] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211) does not have the latest schema for tablet 7b69d44bbbfc408c8d46f54d21f8a427 (table rebalancer_dynamic_table [id=271a62be2be84ada961794cdfcb28433]). Expected version 1 got 0
I20240503 02:00:12.854534 13863 tablet.cc:1722] T 7b69d44bbbfc408c8d46f54d21f8a427 P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:12.855180 13863 tablet.cc:1620] T 7b69d44bbbfc408c8d46f54d21f8a427 P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
I20240503 02:00:12.855885 13862 tablet.cc:1722] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:12.856161 6312 catalog_manager.cc:5576] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211) does not have the latest schema for tablet 7b69d44bbbfc408c8d46f54d21f8a427 (table rebalancer_dynamic_table [id=271a62be2be84ada961794cdfcb28433]). Expected version 1 got 0
I20240503 02:00:12.856441 13862 tablet.cc:1620] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f: MemRowSet was empty: no flush needed.
I20240503 02:00:12.856650 6308 catalog_manager.cc:5576] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719) does not have the latest schema for tablet 7b69d44bbbfc408c8d46f54d21f8a427 (table rebalancer_dynamic_table [id=271a62be2be84ada961794cdfcb28433]). Expected version 1 got 0
I20240503 02:00:12.857821 6313 catalog_manager.cc:5576] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863) does not have the latest schema for tablet 7b69d44bbbfc408c8d46f54d21f8a427 (table rebalancer_dynamic_table [id=271a62be2be84ada961794cdfcb28433]). Expected version 1 got 0
I20240503 02:00:12.858912 13864 tablet.cc:1722] T 7b69d44bbbfc408c8d46f54d21f8a427 P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:12.859302 13864 tablet.cc:1620] T 7b69d44bbbfc408c8d46f54d21f8a427 P 492448570cc94ed1879feb2ec3b9051d: MemRowSet was empty: no flush needed.
I20240503 02:00:12.862383 6313 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=271a62be2be84ada961794cdfcb28433] alter complete (version 1)
I20240503 02:00:12.865540 13814 raft_consensus.cc:482] T 71c44cb6883a495797823588f7c2b4a3 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:12.865689 13814 raft_consensus.cc:504] T 71c44cb6883a495797823588f7c2b4a3 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:12.866145 13814 leader_election.cc:288] T 71c44cb6883a495797823588f7c2b4a3 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:12.866674 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71c44cb6883a495797823588f7c2b4a3" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 02:00:12.866724 6850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71c44cb6883a495797823588f7c2b4a3" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 02:00:12.866820 6456 raft_consensus.cc:2454] T 71c44cb6883a495797823588f7c2b4a3 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:12.866888 6850 raft_consensus.cc:2454] T 71c44cb6883a495797823588f7c2b4a3 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:12.867170 9115 leader_election.cc:302] T 71c44cb6883a495797823588f7c2b4a3 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 492448570cc94ed1879feb2ec3b9051d; no voters:
I20240503 02:00:12.867393 13814 raft_consensus.cc:2781] T 71c44cb6883a495797823588f7c2b4a3 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:12.867447 13814 raft_consensus.cc:482] T 71c44cb6883a495797823588f7c2b4a3 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:12.867493 13814 raft_consensus.cc:3037] T 71c44cb6883a495797823588f7c2b4a3 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:12.868476 13814 raft_consensus.cc:504] T 71c44cb6883a495797823588f7c2b4a3 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:12.868842 13814 leader_election.cc:288] T 71c44cb6883a495797823588f7c2b4a3 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Requested vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:12.869197 6850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71c44cb6883a495797823588f7c2b4a3" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 02:00:12.869217 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71c44cb6883a495797823588f7c2b4a3" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 02:00:12.869307 6850 raft_consensus.cc:3037] T 71c44cb6883a495797823588f7c2b4a3 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:12.869319 6456 raft_consensus.cc:3037] T 71c44cb6883a495797823588f7c2b4a3 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:12.870095 6850 raft_consensus.cc:2454] T 71c44cb6883a495797823588f7c2b4a3 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:12.870115 6456 raft_consensus.cc:2454] T 71c44cb6883a495797823588f7c2b4a3 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:12.870416 9112 leader_election.cc:302] T 71c44cb6883a495797823588f7c2b4a3 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:12.870602 13814 raft_consensus.cc:2781] T 71c44cb6883a495797823588f7c2b4a3 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:12.870702 13814 raft_consensus.cc:686] T 71c44cb6883a495797823588f7c2b4a3 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Becoming Leader. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Running, Role: LEADER
I20240503 02:00:12.870839 13814 consensus_queue.cc:237] T 71c44cb6883a495797823588f7c2b4a3 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:12.873275 6308 catalog_manager.cc:5484] T 71c44cb6883a495797823588f7c2b4a3 P 2a01ace80baf4057b6bf1995cf88468b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135). New cstate: current_term: 1 leader_uuid: "2a01ace80baf4057b6bf1995cf88468b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:12.964897 6308 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:12.971239 13862 tablet.cc:1722] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:12.971530 13864 tablet.cc:1722] T 7b69d44bbbfc408c8d46f54d21f8a427 P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:12.971648 13862 tablet.cc:1620] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f: MemRowSet was empty: no flush needed.
I20240503 02:00:12.971812 13864 tablet.cc:1620] T 7b69d44bbbfc408c8d46f54d21f8a427 P 492448570cc94ed1879feb2ec3b9051d: MemRowSet was empty: no flush needed.
I20240503 02:00:12.973825 13863 tablet.cc:1722] T 7b69d44bbbfc408c8d46f54d21f8a427 P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:12.975112 13863 tablet.cc:1620] T 7b69d44bbbfc408c8d46f54d21f8a427 P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
I20240503 02:00:12.979889 6308 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=271a62be2be84ada961794cdfcb28433] alter complete (version 2)
I20240503 02:00:13.020646 13786 raft_consensus.cc:482] T 9ba716e7e1a04275be92428ccbd651b8 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:13.020782 13786 raft_consensus.cc:504] T 9ba716e7e1a04275be92428ccbd651b8 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:13.021239 13786 leader_election.cc:288] T 9ba716e7e1a04275be92428ccbd651b8 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:13.021615 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ba716e7e1a04275be92428ccbd651b8" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:13.021667 9039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ba716e7e1a04275be92428ccbd651b8" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:13.021776 9039 raft_consensus.cc:2454] T 9ba716e7e1a04275be92428ccbd651b8 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 0.
I20240503 02:00:13.021773 9178 raft_consensus.cc:2454] T 9ba716e7e1a04275be92428ccbd651b8 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 0.
I20240503 02:00:13.022079 8850 leader_election.cc:302] T 9ba716e7e1a04275be92428ccbd651b8 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:13.022466 13786 raft_consensus.cc:2781] T 9ba716e7e1a04275be92428ccbd651b8 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:13.022529 13786 raft_consensus.cc:482] T 9ba716e7e1a04275be92428ccbd651b8 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:13.022573 13786 raft_consensus.cc:3037] T 9ba716e7e1a04275be92428ccbd651b8 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:13.023808 13786 raft_consensus.cc:504] T 9ba716e7e1a04275be92428ccbd651b8 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:13.024166 13786 leader_election.cc:288] T 9ba716e7e1a04275be92428ccbd651b8 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 election: Requested vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:13.024549 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ba716e7e1a04275be92428ccbd651b8" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:13.024580 9039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ba716e7e1a04275be92428ccbd651b8" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:13.024679 9178 raft_consensus.cc:3037] T 9ba716e7e1a04275be92428ccbd651b8 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:13.024699 9039 raft_consensus.cc:3037] T 9ba716e7e1a04275be92428ccbd651b8 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:13.025497 9178 raft_consensus.cc:2454] T 9ba716e7e1a04275be92428ccbd651b8 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 1.
I20240503 02:00:13.025833 9039 raft_consensus.cc:2454] T 9ba716e7e1a04275be92428ccbd651b8 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 1.
I20240503 02:00:13.026515 8851 leader_election.cc:302] T 9ba716e7e1a04275be92428ccbd651b8 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9942306855404cfd891adf227dd65eb2, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:13.026701 13786 raft_consensus.cc:2781] T 9ba716e7e1a04275be92428ccbd651b8 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:13.026772 13786 raft_consensus.cc:686] T 9ba716e7e1a04275be92428ccbd651b8 P 9942306855404cfd891adf227dd65eb2 [term 1 LEADER]: Becoming Leader. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Running, Role: LEADER
I20240503 02:00:13.026901 13786 consensus_queue.cc:237] T 9ba716e7e1a04275be92428ccbd651b8 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:13.029790 6308 catalog_manager.cc:5484] T 9ba716e7e1a04275be92428ccbd651b8 P 9942306855404cfd891adf227dd65eb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9942306855404cfd891adf227dd65eb2 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "9942306855404cfd891adf227dd65eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:13.081164 6308 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:13.086980 13862 tablet.cc:1722] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:13.087666 13864 tablet.cc:1722] T 7b69d44bbbfc408c8d46f54d21f8a427 P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:13.088770 13863 tablet.cc:1722] T 7b69d44bbbfc408c8d46f54d21f8a427 P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:13.093823 6308 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=271a62be2be84ada961794cdfcb28433] alter complete (version 3)
I20240503 02:00:13.195286 6309 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:13.195598 6309 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:13.197645 6309 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet 7b69d44bbbfc408c8d46f54d21f8a427
I20240503 02:00:13.198364 6831 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b69d44bbbfc408c8d46f54d21f8a427 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:13 UTC) from {username='slave'} at 127.0.0.1:52812
I20240503 02:00:13.198518 6569 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b69d44bbbfc408c8d46f54d21f8a427 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:13 UTC) from {username='slave'} at 127.0.0.1:59846
I20240503 02:00:13.198676 8897 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b69d44bbbfc408c8d46f54d21f8a427 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:13 UTC) from {username='slave'} at 127.0.0.1:40296
I20240503 02:00:13.198825 13871 tablet_replica.cc:330] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 02:00:13.198941 13870 tablet_replica.cc:330] T 7b69d44bbbfc408c8d46f54d21f8a427 P 492448570cc94ed1879feb2ec3b9051d: stopping tablet replica
I20240503 02:00:13.198974 13871 raft_consensus.cc:2229] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:13.199097 13870 raft_consensus.cc:2229] T 7b69d44bbbfc408c8d46f54d21f8a427 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:13.199190 13871 raft_consensus.cc:2258] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:13.199226 13870 raft_consensus.cc:2258] T 7b69d44bbbfc408c8d46f54d21f8a427 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:13.199735 13871 ts_tablet_manager.cc:1875] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:13.199829 13870 ts_tablet_manager.cc:1875] T 7b69d44bbbfc408c8d46f54d21f8a427 P 492448570cc94ed1879feb2ec3b9051d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:13.200588 13872 tablet_replica.cc:330] T 7b69d44bbbfc408c8d46f54d21f8a427 P 9942306855404cfd891adf227dd65eb2: stopping tablet replica
I20240503 02:00:13.200803 13872 raft_consensus.cc:2229] T 7b69d44bbbfc408c8d46f54d21f8a427 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:13.201001 13872 raft_consensus.cc:2258] T 7b69d44bbbfc408c8d46f54d21f8a427 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:13.201647 13872 ts_tablet_manager.cc:1875] T 7b69d44bbbfc408c8d46f54d21f8a427 P 9942306855404cfd891adf227dd65eb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:13.202155 13871 ts_tablet_manager.cc:1888] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:13.202164 13870 ts_tablet_manager.cc:1888] T 7b69d44bbbfc408c8d46f54d21f8a427 P 492448570cc94ed1879feb2ec3b9051d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:13.202231 13870 log.cc:1196] T 7b69d44bbbfc408c8d46f54d21f8a427 P 492448570cc94ed1879feb2ec3b9051d: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/7b69d44bbbfc408c8d46f54d21f8a427
I20240503 02:00:13.202301 13871 log.cc:1196] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/7b69d44bbbfc408c8d46f54d21f8a427
I20240503 02:00:13.202648 13870 ts_tablet_manager.cc:1909] T 7b69d44bbbfc408c8d46f54d21f8a427 P 492448570cc94ed1879feb2ec3b9051d: Deleting consensus metadata
I20240503 02:00:13.202747 13871 ts_tablet_manager.cc:1909] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting consensus metadata
I20240503 02:00:13.203570 6303 catalog_manager.cc:4830] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719): tablet 7b69d44bbbfc408c8d46f54d21f8a427 (table rebalancer_dynamic_table_new_name [id=271a62be2be84ada961794cdfcb28433]) successfully deleted
I20240503 02:00:13.203570 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet 7b69d44bbbfc408c8d46f54d21f8a427 (table rebalancer_dynamic_table_new_name [id=271a62be2be84ada961794cdfcb28433]) successfully deleted
I20240503 02:00:13.204069 13872 ts_tablet_manager.cc:1888] T 7b69d44bbbfc408c8d46f54d21f8a427 P 9942306855404cfd891adf227dd65eb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:13.204135 13872 log.cc:1196] T 7b69d44bbbfc408c8d46f54d21f8a427 P 9942306855404cfd891adf227dd65eb2: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/7b69d44bbbfc408c8d46f54d21f8a427
I20240503 02:00:13.204339 13872 ts_tablet_manager.cc:1909] T 7b69d44bbbfc408c8d46f54d21f8a427 P 9942306855404cfd891adf227dd65eb2: Deleting consensus metadata
I20240503 02:00:13.205103 6303 catalog_manager.cc:4830] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211): tablet 7b69d44bbbfc408c8d46f54d21f8a427 (table rebalancer_dynamic_table_new_name [id=271a62be2be84ada961794cdfcb28433]) successfully deleted
W20240503 02:00:13.243845 6301 catalog_manager.cc:4573] TS e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cc19d71f40fb44f1801f1b49dcc76fe0: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:13.273573 13776 consensus_queue.cc:1035] T 71c44cb6883a495797823588f7c2b4a3 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:13.278100 13814 consensus_queue.cc:1035] T 71c44cb6883a495797823588f7c2b4a3 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:13.279635 13786 consensus_queue.cc:1035] T ccf7a4ad8b1d44efa6548772c271d497 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:13.283886 13878 consensus_queue.cc:1035] T ccf7a4ad8b1d44efa6548772c271d497 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20240503 02:00:13.289288 8982 consensus_peers.cc:485] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:00:13.299371 6309 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:00:13.304796 8897 tablet_service.cc:1467] Processing CreateTablet for tablet 7b37a8876d0349f4a6885868f63ee6ce (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2e477496b2bb4a5d8f1af5475a756447]), partition=
I20240503 02:00:13.304885 6569 tablet_service.cc:1467] Processing CreateTablet for tablet 7b37a8876d0349f4a6885868f63ee6ce (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2e477496b2bb4a5d8f1af5475a756447]), partition=
I20240503 02:00:13.305109 8897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b37a8876d0349f4a6885868f63ee6ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:13.305145 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b37a8876d0349f4a6885868f63ee6ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:13.307025 13830 tablet_bootstrap.cc:492] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:13.307142 13881 tablet_bootstrap.cc:492] T 7b37a8876d0349f4a6885868f63ee6ce P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:13.307292 9028 tablet_service.cc:1467] Processing CreateTablet for tablet 7b37a8876d0349f4a6885868f63ee6ce (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2e477496b2bb4a5d8f1af5475a756447]), partition=
I20240503 02:00:13.307557 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b37a8876d0349f4a6885868f63ee6ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:13.308048 13881 tablet_bootstrap.cc:654] T 7b37a8876d0349f4a6885868f63ee6ce P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:13.308162 13830 tablet_bootstrap.cc:654] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:13.309885 13830 tablet_bootstrap.cc:492] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:13.310011 13830 ts_tablet_manager.cc:1367] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:13.310052 13884 tablet_bootstrap.cc:492] T 7b37a8876d0349f4a6885868f63ee6ce P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:13.310499 13830 raft_consensus.cc:348] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:13.310606 13830 raft_consensus.cc:374] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:13.310612 13881 tablet_bootstrap.cc:492] T 7b37a8876d0349f4a6885868f63ee6ce P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:13.310626 13830 raft_consensus.cc:729] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:13.310703 13881 ts_tablet_manager.cc:1367] T 7b37a8876d0349f4a6885868f63ee6ce P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 02:00:13.310741 13830 consensus_queue.cc:260] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:13.311003 13830 ts_tablet_manager.cc:1398] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:13.311141 13881 raft_consensus.cc:348] T 7b37a8876d0349f4a6885868f63ee6ce P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:13.311240 13881 raft_consensus.cc:374] T 7b37a8876d0349f4a6885868f63ee6ce P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:13.311267 13881 raft_consensus.cc:729] T 7b37a8876d0349f4a6885868f63ee6ce P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:13.311398 13881 consensus_queue.cc:260] T 7b37a8876d0349f4a6885868f63ee6ce P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:13.311691 13881 ts_tablet_manager.cc:1398] T 7b37a8876d0349f4a6885868f63ee6ce P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:13.313949 13884 tablet_bootstrap.cc:654] T 7b37a8876d0349f4a6885868f63ee6ce P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:13.315470 13884 tablet_bootstrap.cc:492] T 7b37a8876d0349f4a6885868f63ee6ce P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:13.315608 13884 ts_tablet_manager.cc:1367] T 7b37a8876d0349f4a6885868f63ee6ce P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:00:13.316128 13884 raft_consensus.cc:348] T 7b37a8876d0349f4a6885868f63ee6ce P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:13.316258 13884 raft_consensus.cc:374] T 7b37a8876d0349f4a6885868f63ee6ce P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:13.316304 13884 raft_consensus.cc:729] T 7b37a8876d0349f4a6885868f63ee6ce P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:13.316457 13884 consensus_queue.cc:260] T 7b37a8876d0349f4a6885868f63ee6ce P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:13.316720 13884 ts_tablet_manager.cc:1398] T 7b37a8876d0349f4a6885868f63ee6ce P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20240503 02:00:13.374754 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:13.375975 13878 raft_consensus.cc:482] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:13.376111 13878 raft_consensus.cc:504] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:13.376549 13878 leader_election.cc:288] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:13.376981 6582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b37a8876d0349f4a6885868f63ee6ce" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 02:00:13.376983 9039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b37a8876d0349f4a6885868f63ee6ce" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:13.377117 6582 raft_consensus.cc:2454] T 7b37a8876d0349f4a6885868f63ee6ce P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 0.
I20240503 02:00:13.377127 9039 raft_consensus.cc:2454] T 7b37a8876d0349f4a6885868f63ee6ce P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 0.
I20240503 02:00:13.377398 8851 leader_election.cc:302] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9942306855404cfd891adf227dd65eb2, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:13.377553 13878 raft_consensus.cc:2781] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:13.377611 13878 raft_consensus.cc:482] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:13.377655 13878 raft_consensus.cc:3037] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:13.378937 13878 raft_consensus.cc:504] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:13.379333 13878 leader_election.cc:288] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 election: Requested vote from peers 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:13.379854 6582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b37a8876d0349f4a6885868f63ee6ce" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 02:00:13.379866 9039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b37a8876d0349f4a6885868f63ee6ce" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:13.380010 6582 raft_consensus.cc:3037] T 7b37a8876d0349f4a6885868f63ee6ce P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:13.380023 9039 raft_consensus.cc:3037] T 7b37a8876d0349f4a6885868f63ee6ce P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:13.381127 6582 raft_consensus.cc:2454] T 7b37a8876d0349f4a6885868f63ee6ce P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 1.
I20240503 02:00:13.381121 9039 raft_consensus.cc:2454] T 7b37a8876d0349f4a6885868f63ee6ce P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 1.
I20240503 02:00:13.381450 8852 leader_election.cc:302] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:13.381655 13878 raft_consensus.cc:2781] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:13.381752 13878 raft_consensus.cc:686] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2 [term 1 LEADER]: Becoming Leader. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Running, Role: LEADER
I20240503 02:00:13.381927 13878 consensus_queue.cc:237] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:13.384775 6308 catalog_manager.cc:5484] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9942306855404cfd891adf227dd65eb2 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "9942306855404cfd891adf227dd65eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } }
W20240503 02:00:13.500560 6785 consensus_peers.cc:485] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:00:13.500980 13878 consensus_queue.cc:1035] T 9ba716e7e1a04275be92428ccbd651b8 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:00:13.503530 8851 consensus_peers.cc:485] T 067f54011a37432d99157884e70fc601 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:00:13.506847 6313 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:13.508611 13878 consensus_queue.cc:1035] T 9ba716e7e1a04275be92428ccbd651b8 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:13.512068 6582 raft_consensus.cc:1261] T 7b37a8876d0349f4a6885868f63ee6ce P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:13.512101 9039 raft_consensus.cc:1261] T 7b37a8876d0349f4a6885868f63ee6ce P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:13.512571 13102 consensus_queue.cc:1035] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:13.512976 13102 consensus_queue.cc:1035] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:13.515846 6313 catalog_manager.cc:5576] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863) does not have the latest schema for tablet 7b37a8876d0349f4a6885868f63ee6ce (table rebalancer_dynamic_table [id=2e477496b2bb4a5d8f1af5475a756447]). Expected version 1 got 0
I20240503 02:00:13.516033 13863 tablet.cc:1722] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:13.516414 13863 tablet.cc:1620] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
I20240503 02:00:13.516746 13862 tablet.cc:1722] T 7b37a8876d0349f4a6885868f63ee6ce P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:13.517066 13862 tablet.cc:1620] T 7b37a8876d0349f4a6885868f63ee6ce P 80fd18ef4d604c86b909d5c008a46b4f: MemRowSet was empty: no flush needed.
I20240503 02:00:13.520326 6313 catalog_manager.cc:5576] TS 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157) does not have the latest schema for tablet 7b37a8876d0349f4a6885868f63ee6ce (table rebalancer_dynamic_table [id=2e477496b2bb4a5d8f1af5475a756447]). Expected version 1 got 0
I20240503 02:00:13.521487 13892 tablet.cc:1722] T 7b37a8876d0349f4a6885868f63ee6ce P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:13.522586 13892 tablet.cc:1620] T 7b37a8876d0349f4a6885868f63ee6ce P 99ead02915954b4fab23b033ad6d546b: MemRowSet was empty: no flush needed.
I20240503 02:00:13.523502 6308 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=2e477496b2bb4a5d8f1af5475a756447] alter complete (version 1)
W20240503 02:00:13.526813 6523 consensus_peers.cc:485] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:00:13.535724 6309 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_extra_table_34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:13.542019 6831 tablet_service.cc:1467] Processing CreateTablet for tablet 849308d0b7e744f7946c892dc444f1f0 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=dc0ffce47c184e1f816da7595fee0912]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:13.542029 6569 tablet_service.cc:1467] Processing CreateTablet for tablet 849308d0b7e744f7946c892dc444f1f0 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=dc0ffce47c184e1f816da7595fee0912]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:13.542160 9028 tablet_service.cc:1467] Processing CreateTablet for tablet 849308d0b7e744f7946c892dc444f1f0 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=dc0ffce47c184e1f816da7595fee0912]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:13.542363 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 849308d0b7e744f7946c892dc444f1f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:13.542371 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 849308d0b7e744f7946c892dc444f1f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:13.542397 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 849308d0b7e744f7946c892dc444f1f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:13.544625 13881 tablet_bootstrap.cc:492] T 849308d0b7e744f7946c892dc444f1f0 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:13.544690 13893 tablet_bootstrap.cc:492] T 849308d0b7e744f7946c892dc444f1f0 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:13.545730 13881 tablet_bootstrap.cc:654] T 849308d0b7e744f7946c892dc444f1f0 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:13.545647 9157 tablet_service.cc:1467] Processing CreateTablet for tablet 603eb0789cfb44ea930f447cd2190e6a (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=dc0ffce47c184e1f816da7595fee0912]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:13.545751 13893 tablet_bootstrap.cc:654] T 849308d0b7e744f7946c892dc444f1f0 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:13.545647 9158 tablet_service.cc:1467] Processing CreateTablet for tablet e1770e6ac12d420eadb2be38ba010bba (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=dc0ffce47c184e1f816da7595fee0912]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:13.545956 9157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 603eb0789cfb44ea930f447cd2190e6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:13.546144 9158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1770e6ac12d420eadb2be38ba010bba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:13.546437 13884 tablet_bootstrap.cc:492] T 849308d0b7e744f7946c892dc444f1f0 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:13.547458 13884 tablet_bootstrap.cc:654] T 849308d0b7e744f7946c892dc444f1f0 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:13.548269 13897 tablet_bootstrap.cc:492] T 603eb0789cfb44ea930f447cd2190e6a P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:13.549579 6700 tablet_service.cc:1467] Processing CreateTablet for tablet 603eb0789cfb44ea930f447cd2190e6a (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=dc0ffce47c184e1f816da7595fee0912]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:13.549579 6831 tablet_service.cc:1467] Processing CreateTablet for tablet 603eb0789cfb44ea930f447cd2190e6a (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=dc0ffce47c184e1f816da7595fee0912]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:13.549702 13897 tablet_bootstrap.cc:654] T 603eb0789cfb44ea930f447cd2190e6a P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:13.549865 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 603eb0789cfb44ea930f447cd2190e6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:13.549865 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 603eb0789cfb44ea930f447cd2190e6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:13.549579 6699 tablet_service.cc:1467] Processing CreateTablet for tablet e1770e6ac12d420eadb2be38ba010bba (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=dc0ffce47c184e1f816da7595fee0912]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:13.553565 6699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1770e6ac12d420eadb2be38ba010bba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:13.554760 13881 tablet_bootstrap.cc:492] T 849308d0b7e744f7946c892dc444f1f0 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:13.554855 13881 ts_tablet_manager.cc:1367] T 849308d0b7e744f7946c892dc444f1f0 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.002s
I20240503 02:00:13.555330 13881 raft_consensus.cc:348] T 849308d0b7e744f7946c892dc444f1f0 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:13.555454 13881 raft_consensus.cc:374] T 849308d0b7e744f7946c892dc444f1f0 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:13.555483 13881 raft_consensus.cc:729] T 849308d0b7e744f7946c892dc444f1f0 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:13.555624 13881 consensus_queue.cc:260] T 849308d0b7e744f7946c892dc444f1f0 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:13.555898 13881 ts_tablet_manager.cc:1398] T 849308d0b7e744f7946c892dc444f1f0 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:13.556310 6436 tablet_service.cc:1467] Processing CreateTablet for tablet e1770e6ac12d420eadb2be38ba010bba (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=dc0ffce47c184e1f816da7595fee0912]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:13.556640 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1770e6ac12d420eadb2be38ba010bba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:13.556741 13897 tablet_bootstrap.cc:492] T 603eb0789cfb44ea930f447cd2190e6a P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:13.556842 13897 ts_tablet_manager.cc:1367] T 603eb0789cfb44ea930f447cd2190e6a P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 02:00:13.557314 13893 tablet_bootstrap.cc:492] T 849308d0b7e744f7946c892dc444f1f0 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:13.557307 13897 raft_consensus.cc:348] T 603eb0789cfb44ea930f447cd2190e6a P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:13.557405 13893 ts_tablet_manager.cc:1367] T 849308d0b7e744f7946c892dc444f1f0 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 02:00:13.557417 13897 raft_consensus.cc:374] T 603eb0789cfb44ea930f447cd2190e6a P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:13.557448 13897 raft_consensus.cc:729] T 603eb0789cfb44ea930f447cd2190e6a P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:13.557571 13897 consensus_queue.cc:260] T 603eb0789cfb44ea930f447cd2190e6a P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:13.557863 13897 ts_tablet_manager.cc:1398] T 603eb0789cfb44ea930f447cd2190e6a P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:13.557929 13893 raft_consensus.cc:348] T 849308d0b7e744f7946c892dc444f1f0 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:13.558036 13893 raft_consensus.cc:374] T 849308d0b7e744f7946c892dc444f1f0 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:13.558046 13897 tablet_bootstrap.cc:492] T e1770e6ac12d420eadb2be38ba010bba P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:13.558068 13893 raft_consensus.cc:729] T 849308d0b7e744f7946c892dc444f1f0 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:13.558187 13893 consensus_queue.cc:260] T 849308d0b7e744f7946c892dc444f1f0 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:13.558446 13893 ts_tablet_manager.cc:1398] T 849308d0b7e744f7946c892dc444f1f0 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:13.558624 13893 tablet_bootstrap.cc:492] T 603eb0789cfb44ea930f447cd2190e6a P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:13.560034 13893 tablet_bootstrap.cc:654] T 603eb0789cfb44ea930f447cd2190e6a P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:13.562052 13899 tablet_bootstrap.cc:492] T 603eb0789cfb44ea930f447cd2190e6a P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:13.563341 13901 tablet_bootstrap.cc:492] T e1770e6ac12d420eadb2be38ba010bba P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:13.564112 13884 tablet_bootstrap.cc:492] T 849308d0b7e744f7946c892dc444f1f0 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:13.564234 13884 ts_tablet_manager.cc:1367] T 849308d0b7e744f7946c892dc444f1f0 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.002s
I20240503 02:00:13.564785 13884 raft_consensus.cc:348] T 849308d0b7e744f7946c892dc444f1f0 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:13.564904 13884 raft_consensus.cc:374] T 849308d0b7e744f7946c892dc444f1f0 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:13.564934 13884 raft_consensus.cc:729] T 849308d0b7e744f7946c892dc444f1f0 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:13.565060 13884 consensus_queue.cc:260] T 849308d0b7e744f7946c892dc444f1f0 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:13.565362 13884 ts_tablet_manager.cc:1398] T 849308d0b7e744f7946c892dc444f1f0 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:13.565876 13901 tablet_bootstrap.cc:654] T e1770e6ac12d420eadb2be38ba010bba P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
W20240503 02:00:13.566018 6785 consensus_peers.cc:485] T c3485701bf3b48f5ae2e1568afeed133 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:00:13.566108 13899 tablet_bootstrap.cc:654] T 603eb0789cfb44ea930f447cd2190e6a P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:13.567034 13897 tablet_bootstrap.cc:654] T e1770e6ac12d420eadb2be38ba010bba P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:13.568221 13899 tablet_bootstrap.cc:492] T 603eb0789cfb44ea930f447cd2190e6a P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:13.568323 13899 ts_tablet_manager.cc:1367] T 603eb0789cfb44ea930f447cd2190e6a P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:00:13.568818 13899 raft_consensus.cc:348] T 603eb0789cfb44ea930f447cd2190e6a P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:13.568933 13899 raft_consensus.cc:374] T 603eb0789cfb44ea930f447cd2190e6a P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:13.568959 13899 raft_consensus.cc:729] T 603eb0789cfb44ea930f447cd2190e6a P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:13.569075 13899 consensus_queue.cc:260] T 603eb0789cfb44ea930f447cd2190e6a P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:13.569348 13899 ts_tablet_manager.cc:1398] T 603eb0789cfb44ea930f447cd2190e6a P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:13.569892 13899 tablet_bootstrap.cc:492] T e1770e6ac12d420eadb2be38ba010bba P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:13.570876 13899 tablet_bootstrap.cc:654] T e1770e6ac12d420eadb2be38ba010bba P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:13.572850 13901 tablet_bootstrap.cc:492] T e1770e6ac12d420eadb2be38ba010bba P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:13.572973 13901 ts_tablet_manager.cc:1367] T e1770e6ac12d420eadb2be38ba010bba P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 02:00:13.573482 13901 raft_consensus.cc:348] T e1770e6ac12d420eadb2be38ba010bba P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:13.573606 13901 raft_consensus.cc:374] T e1770e6ac12d420eadb2be38ba010bba P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:13.573638 13901 raft_consensus.cc:729] T e1770e6ac12d420eadb2be38ba010bba P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:13.573644 13899 tablet_bootstrap.cc:492] T e1770e6ac12d420eadb2be38ba010bba P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:13.573757 13901 consensus_queue.cc:260] T e1770e6ac12d420eadb2be38ba010bba P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:13.574038 13901 ts_tablet_manager.cc:1398] T e1770e6ac12d420eadb2be38ba010bba P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:13.572211 13897 tablet_bootstrap.cc:492] T e1770e6ac12d420eadb2be38ba010bba P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:13.577260 13897 ts_tablet_manager.cc:1367] T e1770e6ac12d420eadb2be38ba010bba P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
W20240503 02:00:13.577387 6654 consensus_peers.cc:485] T ba17d4d3e0754b2798eb95484d37229a P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:00:13.573724 13899 ts_tablet_manager.cc:1367] T e1770e6ac12d420eadb2be38ba010bba P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 02:00:13.578169 13899 raft_consensus.cc:348] T e1770e6ac12d420eadb2be38ba010bba P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:13.578787 13897 raft_consensus.cc:348] T e1770e6ac12d420eadb2be38ba010bba P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:13.578902 13897 raft_consensus.cc:374] T e1770e6ac12d420eadb2be38ba010bba P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:13.578982 13897 raft_consensus.cc:729] T e1770e6ac12d420eadb2be38ba010bba P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:13.579144 13897 consensus_queue.cc:260] T e1770e6ac12d420eadb2be38ba010bba P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:13.579201 13893 tablet_bootstrap.cc:492] T 603eb0789cfb44ea930f447cd2190e6a P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:13.579281 13899 raft_consensus.cc:374] T e1770e6ac12d420eadb2be38ba010bba P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:13.579303 13893 ts_tablet_manager.cc:1367] T 603eb0789cfb44ea930f447cd2190e6a P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.000s
I20240503 02:00:13.579373 13899 raft_consensus.cc:729] T e1770e6ac12d420eadb2be38ba010bba P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:13.579535 13899 consensus_queue.cc:260] T e1770e6ac12d420eadb2be38ba010bba P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:13.579756 13893 raft_consensus.cc:348] T 603eb0789cfb44ea930f447cd2190e6a P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:13.579891 13893 raft_consensus.cc:374] T 603eb0789cfb44ea930f447cd2190e6a P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:13.579924 13893 raft_consensus.cc:729] T 603eb0789cfb44ea930f447cd2190e6a P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:13.580046 13893 consensus_queue.cc:260] T 603eb0789cfb44ea930f447cd2190e6a P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:13.580356 13893 ts_tablet_manager.cc:1398] T 603eb0789cfb44ea930f447cd2190e6a P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:13.582379 13897 ts_tablet_manager.cc:1398] T e1770e6ac12d420eadb2be38ba010bba P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 02:00:13.582769 13899 ts_tablet_manager.cc:1398] T e1770e6ac12d420eadb2be38ba010bba P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
W20240503 02:00:13.600319 6785 consensus_peers.cc:485] T 46ad540deb624a25b97d3b967fe1f6f4 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:13.610780 8851 consensus_peers.cc:485] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:00:13.622576 13814 raft_consensus.cc:482] T 603eb0789cfb44ea930f447cd2190e6a P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:13.622720 13814 raft_consensus.cc:504] T 603eb0789cfb44ea930f447cd2190e6a P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:13.623135 13814 leader_election.cc:288] T 603eb0789cfb44ea930f447cd2190e6a P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:13.623431 13887 raft_consensus.cc:482] T 849308d0b7e744f7946c892dc444f1f0 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:13.623544 13887 raft_consensus.cc:504] T 849308d0b7e744f7946c892dc444f1f0 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:13.623644 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "603eb0789cfb44ea930f447cd2190e6a" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 02:00:13.623668 6720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "603eb0789cfb44ea930f447cd2190e6a" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 02:00:13.623788 6720 raft_consensus.cc:2454] T 603eb0789cfb44ea930f447cd2190e6a P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:13.623808 6849 raft_consensus.cc:2454] T 603eb0789cfb44ea930f447cd2190e6a P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:13.623978 13887 leader_election.cc:288] T 849308d0b7e744f7946c892dc444f1f0 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:13.624224 9115 leader_election.cc:302] T 603eb0789cfb44ea930f447cd2190e6a P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 492448570cc94ed1879feb2ec3b9051d; no voters:
I20240503 02:00:13.624402 13814 raft_consensus.cc:2781] T 603eb0789cfb44ea930f447cd2190e6a P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:13.624363 6582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "849308d0b7e744f7946c892dc444f1f0" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 02:00:13.624341 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "849308d0b7e744f7946c892dc444f1f0" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:13.624454 13814 raft_consensus.cc:482] T 603eb0789cfb44ea930f447cd2190e6a P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:13.624487 13814 raft_consensus.cc:3037] T 603eb0789cfb44ea930f447cd2190e6a P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:13.624514 9048 raft_consensus.cc:2454] T 849308d0b7e744f7946c892dc444f1f0 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 02:00:13.624514 6582 raft_consensus.cc:2454] T 849308d0b7e744f7946c892dc444f1f0 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 02:00:13.624765 6785 leader_election.cc:302] T 849308d0b7e744f7946c892dc444f1f0 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:13.624917 13887 raft_consensus.cc:2781] T 849308d0b7e744f7946c892dc444f1f0 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:13.624969 13887 raft_consensus.cc:482] T 849308d0b7e744f7946c892dc444f1f0 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:13.625005 13887 raft_consensus.cc:3037] T 849308d0b7e744f7946c892dc444f1f0 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:13.625568 13814 raft_consensus.cc:504] T 603eb0789cfb44ea930f447cd2190e6a P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:13.625808 13887 raft_consensus.cc:504] T 849308d0b7e744f7946c892dc444f1f0 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:13.625900 13814 leader_election.cc:288] T 603eb0789cfb44ea930f447cd2190e6a P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Requested vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:13.626132 13887 leader_election.cc:288] T 849308d0b7e744f7946c892dc444f1f0 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Requested vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:13.626227 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "603eb0789cfb44ea930f447cd2190e6a" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 02:00:13.626328 6849 raft_consensus.cc:3037] T 603eb0789cfb44ea930f447cd2190e6a P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:13.626302 6720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "603eb0789cfb44ea930f447cd2190e6a" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 02:00:13.626417 6720 raft_consensus.cc:3037] T 603eb0789cfb44ea930f447cd2190e6a P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:13.626526 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "849308d0b7e744f7946c892dc444f1f0" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:13.626624 9048 raft_consensus.cc:3037] T 849308d0b7e744f7946c892dc444f1f0 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:13.627351 6849 raft_consensus.cc:2454] T 603eb0789cfb44ea930f447cd2190e6a P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:13.627351 6720 raft_consensus.cc:2454] T 603eb0789cfb44ea930f447cd2190e6a P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:13.627398 9048 raft_consensus.cc:2454] T 849308d0b7e744f7946c892dc444f1f0 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 02:00:13.627710 6785 leader_election.cc:302] T 849308d0b7e744f7946c892dc444f1f0 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:13.627817 6582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "849308d0b7e744f7946c892dc444f1f0" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 02:00:13.627875 13887 raft_consensus.cc:2781] T 849308d0b7e744f7946c892dc444f1f0 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:13.627939 13887 raft_consensus.cc:686] T 849308d0b7e744f7946c892dc444f1f0 P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Becoming Leader. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Running, Role: LEADER
I20240503 02:00:13.627919 9115 leader_election.cc:302] T 603eb0789cfb44ea930f447cd2190e6a P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 492448570cc94ed1879feb2ec3b9051d; no voters:
I20240503 02:00:13.627951 6582 raft_consensus.cc:3037] T 849308d0b7e744f7946c892dc444f1f0 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:13.628072 13887 consensus_queue.cc:237] T 849308d0b7e744f7946c892dc444f1f0 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:13.628304 13814 raft_consensus.cc:2781] T 603eb0789cfb44ea930f447cd2190e6a P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:13.628409 13814 raft_consensus.cc:686] T 603eb0789cfb44ea930f447cd2190e6a P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Becoming Leader. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Running, Role: LEADER
I20240503 02:00:13.628540 13814 consensus_queue.cc:237] T 603eb0789cfb44ea930f447cd2190e6a P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:13.630204 6582 raft_consensus.cc:2454] T 849308d0b7e744f7946c892dc444f1f0 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 02:00:13.630582 6309 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:13.630722 6313 catalog_manager.cc:5484] T 849308d0b7e744f7946c892dc444f1f0 P 492448570cc94ed1879feb2ec3b9051d reported cstate change: term changed from 0 to 1, leader changed from <none> to 492448570cc94ed1879feb2ec3b9051d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:13.630909 6308 catalog_manager.cc:5484] T 603eb0789cfb44ea930f447cd2190e6a P 2a01ace80baf4057b6bf1995cf88468b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135). New cstate: current_term: 1 leader_uuid: "2a01ace80baf4057b6bf1995cf88468b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:13.637346 13863 tablet.cc:1722] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:13.637733 13863 tablet.cc:1620] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
I20240503 02:00:13.637759 13862 tablet.cc:1722] T 7b37a8876d0349f4a6885868f63ee6ce P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:13.638075 13862 tablet.cc:1620] T 7b37a8876d0349f4a6885868f63ee6ce P 80fd18ef4d604c86b909d5c008a46b4f: MemRowSet was empty: no flush needed.
I20240503 02:00:13.638712 13892 tablet.cc:1722] T 7b37a8876d0349f4a6885868f63ee6ce P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:13.639004 13892 tablet.cc:1620] T 7b37a8876d0349f4a6885868f63ee6ce P 99ead02915954b4fab23b033ad6d546b: MemRowSet was empty: no flush needed.
I20240503 02:00:13.645073 6313 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=2e477496b2bb4a5d8f1af5475a756447] alter complete (version 2)
W20240503 02:00:13.692667 6523 consensus_peers.cc:485] T d0d5dd42db3e4883af7fd16d8813b550 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:00:13.699349 13755 raft_consensus.cc:482] T e1770e6ac12d420eadb2be38ba010bba P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:13.699498 13755 raft_consensus.cc:504] T e1770e6ac12d420eadb2be38ba010bba P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:13.699930 13755 leader_election.cc:288] T e1770e6ac12d420eadb2be38ba010bba P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:13.700323 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1770e6ac12d420eadb2be38ba010bba" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 02:00:13.700506 6456 raft_consensus.cc:2454] T e1770e6ac12d420eadb2be38ba010bba P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:13.700542 6720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1770e6ac12d420eadb2be38ba010bba" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 02:00:13.700666 6720 raft_consensus.cc:2454] T e1770e6ac12d420eadb2be38ba010bba P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:13.700891 9112 leader_election.cc:302] T e1770e6ac12d420eadb2be38ba010bba P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:13.701084 13755 raft_consensus.cc:2781] T e1770e6ac12d420eadb2be38ba010bba P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:13.701141 13755 raft_consensus.cc:482] T e1770e6ac12d420eadb2be38ba010bba P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:13.701169 13755 raft_consensus.cc:3037] T e1770e6ac12d420eadb2be38ba010bba P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:13.702076 13755 raft_consensus.cc:504] T e1770e6ac12d420eadb2be38ba010bba P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:13.702448 13755 leader_election.cc:288] T e1770e6ac12d420eadb2be38ba010bba P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Requested vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:13.702960 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1770e6ac12d420eadb2be38ba010bba" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 02:00:13.702962 6720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1770e6ac12d420eadb2be38ba010bba" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 02:00:13.703078 6720 raft_consensus.cc:3037] T e1770e6ac12d420eadb2be38ba010bba P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:13.703084 6456 raft_consensus.cc:3037] T e1770e6ac12d420eadb2be38ba010bba P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:13.703965 6456 raft_consensus.cc:2454] T e1770e6ac12d420eadb2be38ba010bba P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:13.704078 6720 raft_consensus.cc:2454] T e1770e6ac12d420eadb2be38ba010bba P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:13.704447 9112 leader_election.cc:302] T e1770e6ac12d420eadb2be38ba010bba P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:13.704653 13814 raft_consensus.cc:2781] T e1770e6ac12d420eadb2be38ba010bba P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:13.704723 13814 raft_consensus.cc:686] T e1770e6ac12d420eadb2be38ba010bba P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Becoming Leader. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Running, Role: LEADER
I20240503 02:00:13.704866 13814 consensus_queue.cc:237] T e1770e6ac12d420eadb2be38ba010bba P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:13.707132 6313 catalog_manager.cc:5484] T e1770e6ac12d420eadb2be38ba010bba P 2a01ace80baf4057b6bf1995cf88468b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135). New cstate: current_term: 1 leader_uuid: "2a01ace80baf4057b6bf1995cf88468b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } }
W20240503 02:00:13.743497 6654 consensus_peers.cc:485] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:00:13.747087 6313 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:13.752944 13863 tablet.cc:1722] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:13.753482 13862 tablet.cc:1722] T 7b37a8876d0349f4a6885868f63ee6ce P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:13.753602 13892 tablet.cc:1722] T 7b37a8876d0349f4a6885868f63ee6ce P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:13.759773 6313 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=2e477496b2bb4a5d8f1af5475a756447] alter complete (version 3)
W20240503 02:00:13.760648 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
W20240503 02:00:13.857681 9113 consensus_peers.cc:485] T f44b59277c3b47adbd4dd6aff9350af1 P 2a01ace80baf4057b6bf1995cf88468b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:13.858315 6785 consensus_peers.cc:485] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:00:13.861311 6313 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:13.861462 6313 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:13.863600 6313 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet 7b37a8876d0349f4a6885868f63ee6ce
I20240503 02:00:13.864300 6569 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b37a8876d0349f4a6885868f63ee6ce with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:13 UTC) from {username='slave'} at 127.0.0.1:59846
I20240503 02:00:13.864692 13913 tablet_replica.cc:330] T 7b37a8876d0349f4a6885868f63ee6ce P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 02:00:13.864913 13913 raft_consensus.cc:2229] T 7b37a8876d0349f4a6885868f63ee6ce P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:13.864953 8897 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b37a8876d0349f4a6885868f63ee6ce with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:13 UTC) from {username='slave'} at 127.0.0.1:40296
I20240503 02:00:13.865073 13913 raft_consensus.cc:2258] T 7b37a8876d0349f4a6885868f63ee6ce P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:13.865481 13914 tablet_replica.cc:330] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2: stopping tablet replica
I20240503 02:00:13.865622 13914 raft_consensus.cc:2229] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:13.865866 13914 raft_consensus.cc:2258] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:13.865896 9028 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b37a8876d0349f4a6885868f63ee6ce with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:13 UTC) from {username='slave'} at 127.0.0.1:52584
I20240503 02:00:13.866168 13913 ts_tablet_manager.cc:1875] T 7b37a8876d0349f4a6885868f63ee6ce P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:13.866288 13915 tablet_replica.cc:330] T 7b37a8876d0349f4a6885868f63ee6ce P 99ead02915954b4fab23b033ad6d546b: stopping tablet replica
I20240503 02:00:13.866426 13915 raft_consensus.cc:2229] T 7b37a8876d0349f4a6885868f63ee6ce P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:13.866429 13914 ts_tablet_manager.cc:1875] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:13.866555 13915 raft_consensus.cc:2258] T 7b37a8876d0349f4a6885868f63ee6ce P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:13.867090 13915 ts_tablet_manager.cc:1875] T 7b37a8876d0349f4a6885868f63ee6ce P 99ead02915954b4fab23b033ad6d546b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:13.868983 13915 ts_tablet_manager.cc:1888] T 7b37a8876d0349f4a6885868f63ee6ce P 99ead02915954b4fab23b033ad6d546b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:13.868963 13913 ts_tablet_manager.cc:1888] T 7b37a8876d0349f4a6885868f63ee6ce P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:13.868974 13914 ts_tablet_manager.cc:1888] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:13.869056 13915 log.cc:1196] T 7b37a8876d0349f4a6885868f63ee6ce P 99ead02915954b4fab23b033ad6d546b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/7b37a8876d0349f4a6885868f63ee6ce
I20240503 02:00:13.869057 13914 log.cc:1196] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/7b37a8876d0349f4a6885868f63ee6ce
I20240503 02:00:13.869057 13913 log.cc:1196] T 7b37a8876d0349f4a6885868f63ee6ce P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/7b37a8876d0349f4a6885868f63ee6ce
I20240503 02:00:13.869316 13915 ts_tablet_manager.cc:1909] T 7b37a8876d0349f4a6885868f63ee6ce P 99ead02915954b4fab23b033ad6d546b: Deleting consensus metadata
I20240503 02:00:13.869324 13914 ts_tablet_manager.cc:1909] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2: Deleting consensus metadata
I20240503 02:00:13.869357 13913 ts_tablet_manager.cc:1909] T 7b37a8876d0349f4a6885868f63ee6ce P 80fd18ef4d604c86b909d5c008a46b4f: Deleting consensus metadata
I20240503 02:00:13.870093 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet 7b37a8876d0349f4a6885868f63ee6ce (table rebalancer_dynamic_table_new_name [id=2e477496b2bb4a5d8f1af5475a756447]) successfully deleted
I20240503 02:00:13.870160 6303 catalog_manager.cc:4830] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211): tablet 7b37a8876d0349f4a6885868f63ee6ce (table rebalancer_dynamic_table_new_name [id=2e477496b2bb4a5d8f1af5475a756447]) successfully deleted
I20240503 02:00:13.870532 6301 catalog_manager.cc:4830] TS 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157): tablet 7b37a8876d0349f4a6885868f63ee6ce (table rebalancer_dynamic_table_new_name [id=2e477496b2bb4a5d8f1af5475a756447]) successfully deleted
W20240503 02:00:13.879943 6785 consensus_peers.cc:485] T d897c21e630d4770832bb10f43778e82 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:13.913137 6523 consensus_peers.cc:485] T 925612b660ec471da5b3c548802ede6e P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:13.928684 8851 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 02:00:13.938019 6785 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240503 02:00:13.940716 6654 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 02:00:13.945075 8982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240503 02:00:13.965508 6313 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
W20240503 02:00:13.970527 6785 consensus_peers.cc:485] T 8246488dec204bf6b09e868d7951ac5a P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:00:13.972178 6436 tablet_service.cc:1467] Processing CreateTablet for tablet 308236ce104a41baa84cf1d8e6f68c0e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d3615bb8f356419e9ff1adfb5631ddc8]), partition=
I20240503 02:00:13.972326 6569 tablet_service.cc:1467] Processing CreateTablet for tablet 308236ce104a41baa84cf1d8e6f68c0e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d3615bb8f356419e9ff1adfb5631ddc8]), partition=
I20240503 02:00:13.972496 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 308236ce104a41baa84cf1d8e6f68c0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:13.972616 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 308236ce104a41baa84cf1d8e6f68c0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:13.974699 13901 tablet_bootstrap.cc:492] T 308236ce104a41baa84cf1d8e6f68c0e P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:13.974804 13881 tablet_bootstrap.cc:492] T 308236ce104a41baa84cf1d8e6f68c0e P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:13.975472 9028 tablet_service.cc:1467] Processing CreateTablet for tablet 308236ce104a41baa84cf1d8e6f68c0e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d3615bb8f356419e9ff1adfb5631ddc8]), partition=
I20240503 02:00:13.975685 13881 tablet_bootstrap.cc:654] T 308236ce104a41baa84cf1d8e6f68c0e P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:13.975687 13901 tablet_bootstrap.cc:654] T 308236ce104a41baa84cf1d8e6f68c0e P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:13.975711 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 308236ce104a41baa84cf1d8e6f68c0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:13.977161 13881 tablet_bootstrap.cc:492] T 308236ce104a41baa84cf1d8e6f68c0e P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:13.977241 13881 ts_tablet_manager.cc:1367] T 308236ce104a41baa84cf1d8e6f68c0e P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:00:13.977665 13881 raft_consensus.cc:348] T 308236ce104a41baa84cf1d8e6f68c0e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:13.977762 13881 raft_consensus.cc:374] T 308236ce104a41baa84cf1d8e6f68c0e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:13.977787 13881 raft_consensus.cc:729] T 308236ce104a41baa84cf1d8e6f68c0e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:13.977907 13881 consensus_queue.cc:260] T 308236ce104a41baa84cf1d8e6f68c0e P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:13.978049 13884 tablet_bootstrap.cc:492] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:13.978152 13881 ts_tablet_manager.cc:1398] T 308236ce104a41baa84cf1d8e6f68c0e P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:13.979136 13884 tablet_bootstrap.cc:654] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:13.980881 13901 tablet_bootstrap.cc:492] T 308236ce104a41baa84cf1d8e6f68c0e P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:13.980990 13901 ts_tablet_manager.cc:1367] T 308236ce104a41baa84cf1d8e6f68c0e P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:00:13.981508 13901 raft_consensus.cc:348] T 308236ce104a41baa84cf1d8e6f68c0e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:13.981631 13901 raft_consensus.cc:374] T 308236ce104a41baa84cf1d8e6f68c0e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:13.981660 13901 raft_consensus.cc:729] T 308236ce104a41baa84cf1d8e6f68c0e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:13.981798 13901 consensus_queue.cc:260] T 308236ce104a41baa84cf1d8e6f68c0e P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:13.983023 13901 ts_tablet_manager.cc:1398] T 308236ce104a41baa84cf1d8e6f68c0e P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:13.985600 13884 tablet_bootstrap.cc:492] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:13.985723 13884 ts_tablet_manager.cc:1367] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20240503 02:00:13.986181 13884 raft_consensus.cc:348] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:13.986303 13884 raft_consensus.cc:374] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:13.986338 13884 raft_consensus.cc:729] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:13.986464 13884 consensus_queue.cc:260] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:13.986752 13884 ts_tablet_manager.cc:1398] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20240503 02:00:13.991488 6523 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 02:00:13.992280 6523 consensus_peers.cc:485] T 4385d51d11574192883d01376e2d5620 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:14.004704 6523 consensus_peers.cc:485] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:14.015327 8851 consensus_peers.cc:485] T 5f51fe5f799f4e479b62f5917a2c2606 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:00:14.037117 13887 consensus_queue.cc:1035] T 849308d0b7e744f7946c892dc444f1f0 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:00:14.037592 8851 consensus_peers.cc:485] T f28586e70bd9473fbdc6b9126b666fa9 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:00:14.041064 13887 consensus_queue.cc:1035] T 849308d0b7e744f7946c892dc444f1f0 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:00:14.089943 9113 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 02:00:14.090626 9113 consensus_peers.cc:485] T b1740b877c1e4372a0337d99f5bcab0e P 2a01ace80baf4057b6bf1995cf88468b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:14.108647 8851 consensus_peers.cc:485] T 7da8f614a70e4177b70e7c2f31d22a10 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:14.110893 8982 consensus_peers.cc:485] T 13756a6e44814fe4b09a6b0f087546c3 P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:00:14.113708 13924 raft_consensus.cc:482] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:14.113852 13924 raft_consensus.cc:504] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:14.114260 13924 leader_election.cc:288] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:14.114712 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308236ce104a41baa84cf1d8e6f68c0e" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 02:00:14.114867 6456 raft_consensus.cc:2454] T 308236ce104a41baa84cf1d8e6f68c0e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:14.115018 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308236ce104a41baa84cf1d8e6f68c0e" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 02:00:14.115144 6589 raft_consensus.cc:2454] T 308236ce104a41baa84cf1d8e6f68c0e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:14.115396 8983 leader_election.cc:302] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:14.115589 13924 raft_consensus.cc:2781] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:14.115646 13924 raft_consensus.cc:482] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:14.115690 13924 raft_consensus.cc:3037] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:14.116561 13924 raft_consensus.cc:504] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:14.116943 13924 leader_election.cc:288] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Requested vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:14.117357 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308236ce104a41baa84cf1d8e6f68c0e" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 02:00:14.117345 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308236ce104a41baa84cf1d8e6f68c0e" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 02:00:14.117477 6456 raft_consensus.cc:3037] T 308236ce104a41baa84cf1d8e6f68c0e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:14.117502 6589 raft_consensus.cc:3037] T 308236ce104a41baa84cf1d8e6f68c0e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:14.118299 6456 raft_consensus.cc:2454] T 308236ce104a41baa84cf1d8e6f68c0e P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:14.118352 6589 raft_consensus.cc:2454] T 308236ce104a41baa84cf1d8e6f68c0e P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:14.118680 8983 leader_election.cc:302] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, 99ead02915954b4fab23b033ad6d546b, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:14.118984 13924 raft_consensus.cc:2781] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:14.119081 13924 raft_consensus.cc:686] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Becoming Leader. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Running, Role: LEADER
I20240503 02:00:14.119261 13924 consensus_queue.cc:237] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
W20240503 02:00:14.120415 6523 consensus_peers.cc:485] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:00:14.122707 6309 catalog_manager.cc:5484] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b reported cstate change: term changed from 0 to 1, leader changed from <none> to 99ead02915954b4fab23b033ad6d546b (127.0.99.134). New cstate: current_term: 1 leader_uuid: "99ead02915954b4fab23b033ad6d546b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } }
W20240503 02:00:14.139240 8982 consensus_peers.cc:485] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:14.173719 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:14.200642 13814 consensus_queue.cc:1035] T e1770e6ac12d420eadb2be38ba010bba P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:14.205544 13814 consensus_queue.cc:1035] T e1770e6ac12d420eadb2be38ba010bba P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:14.221732 6309 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_extra_table_36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:14.227696 6569 tablet_service.cc:1467] Processing CreateTablet for tablet 0fc624ba9c6940a3a7eedcf9a7d79dde (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=5d8b1e5b1c3a44018823bf52234cd57b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:14.228041 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fc624ba9c6940a3a7eedcf9a7d79dde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:14.228369 6831 tablet_service.cc:1467] Processing CreateTablet for tablet 37190356d40d40c7aed6607b87c676fd (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=5d8b1e5b1c3a44018823bf52234cd57b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:14.228680 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37190356d40d40c7aed6607b87c676fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:14.228719 6699 tablet_service.cc:1467] Processing CreateTablet for tablet 0fc624ba9c6940a3a7eedcf9a7d79dde (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=5d8b1e5b1c3a44018823bf52234cd57b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:14.228967 6699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fc624ba9c6940a3a7eedcf9a7d79dde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:14.228950 9025 tablet_service.cc:1467] Processing CreateTablet for tablet d9c6a7e041014192b2a8e9bced641391 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=5d8b1e5b1c3a44018823bf52234cd57b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:14.229220 9025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9c6a7e041014192b2a8e9bced641391. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:14.230664 13881 tablet_bootstrap.cc:492] T 0fc624ba9c6940a3a7eedcf9a7d79dde P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:14.231206 6829 tablet_service.cc:1467] Processing CreateTablet for tablet d9c6a7e041014192b2a8e9bced641391 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=5d8b1e5b1c3a44018823bf52234cd57b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:14.231336 9028 tablet_service.cc:1467] Processing CreateTablet for tablet 37190356d40d40c7aed6607b87c676fd (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=5d8b1e5b1c3a44018823bf52234cd57b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:14.231482 6829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9c6a7e041014192b2a8e9bced641391. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:14.231578 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37190356d40d40c7aed6607b87c676fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:14.231712 13881 tablet_bootstrap.cc:654] T 0fc624ba9c6940a3a7eedcf9a7d79dde P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:14.232079 13933 tablet_bootstrap.cc:492] T 0fc624ba9c6940a3a7eedcf9a7d79dde P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:14.233112 13933 tablet_bootstrap.cc:654] T 0fc624ba9c6940a3a7eedcf9a7d79dde P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:14.233583 13884 tablet_bootstrap.cc:492] T d9c6a7e041014192b2a8e9bced641391 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:14.234525 13884 tablet_bootstrap.cc:654] T d9c6a7e041014192b2a8e9bced641391 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:14.234866 6699 tablet_service.cc:1467] Processing CreateTablet for tablet d9c6a7e041014192b2a8e9bced641391 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=5d8b1e5b1c3a44018823bf52234cd57b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:14.235077 13881 tablet_bootstrap.cc:492] T 0fc624ba9c6940a3a7eedcf9a7d79dde P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:14.235126 6699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9c6a7e041014192b2a8e9bced641391. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:14.235167 13881 ts_tablet_manager.cc:1367] T 0fc624ba9c6940a3a7eedcf9a7d79dde P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240503 02:00:14.235662 13881 raft_consensus.cc:348] T 0fc624ba9c6940a3a7eedcf9a7d79dde P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:14.235780 13881 raft_consensus.cc:374] T 0fc624ba9c6940a3a7eedcf9a7d79dde P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:14.235810 13881 raft_consensus.cc:729] T 0fc624ba9c6940a3a7eedcf9a7d79dde P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:14.235932 13881 consensus_queue.cc:260] T 0fc624ba9c6940a3a7eedcf9a7d79dde P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:14.235978 8897 tablet_service.cc:1467] Processing CreateTablet for tablet 0fc624ba9c6940a3a7eedcf9a7d79dde (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=5d8b1e5b1c3a44018823bf52234cd57b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:14.236235 8897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fc624ba9c6940a3a7eedcf9a7d79dde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:14.236227 13881 ts_tablet_manager.cc:1398] T 0fc624ba9c6940a3a7eedcf9a7d79dde P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:14.236320 13884 tablet_bootstrap.cc:492] T d9c6a7e041014192b2a8e9bced641391 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:14.236428 13884 ts_tablet_manager.cc:1367] T d9c6a7e041014192b2a8e9bced641391 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:00:14.236894 13884 raft_consensus.cc:348] T d9c6a7e041014192b2a8e9bced641391 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:14.236994 13884 raft_consensus.cc:374] T d9c6a7e041014192b2a8e9bced641391 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:14.237020 13884 raft_consensus.cc:729] T d9c6a7e041014192b2a8e9bced641391 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:14.237088 9158 tablet_service.cc:1467] Processing CreateTablet for tablet 37190356d40d40c7aed6607b87c676fd (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=5d8b1e5b1c3a44018823bf52234cd57b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:14.237135 13884 consensus_queue.cc:260] T d9c6a7e041014192b2a8e9bced641391 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:14.237334 9158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37190356d40d40c7aed6607b87c676fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:14.237408 13884 ts_tablet_manager.cc:1398] T d9c6a7e041014192b2a8e9bced641391 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:14.237833 13776 consensus_queue.cc:1035] T 603eb0789cfb44ea930f447cd2190e6a P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:14.239691 13884 tablet_bootstrap.cc:492] T 37190356d40d40c7aed6607b87c676fd P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:14.241334 13884 tablet_bootstrap.cc:654] T 37190356d40d40c7aed6607b87c676fd P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:14.241257 13814 consensus_queue.cc:1035] T 603eb0789cfb44ea930f447cd2190e6a P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:14.242273 6309 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:14.245647 13932 tablet_bootstrap.cc:492] T 37190356d40d40c7aed6607b87c676fd P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:14.246040 13939 tablet_bootstrap.cc:492] T 37190356d40d40c7aed6607b87c676fd P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:14.246220 13884 tablet_bootstrap.cc:492] T 37190356d40d40c7aed6607b87c676fd P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:14.246313 13884 ts_tablet_manager.cc:1367] T 37190356d40d40c7aed6607b87c676fd P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:00:14.248346 13932 tablet_bootstrap.cc:654] T 37190356d40d40c7aed6607b87c676fd P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:14.249073 13939 tablet_bootstrap.cc:654] T 37190356d40d40c7aed6607b87c676fd P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:14.249894 13884 raft_consensus.cc:348] T 37190356d40d40c7aed6607b87c676fd P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:14.250341 13884 raft_consensus.cc:374] T 37190356d40d40c7aed6607b87c676fd P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:14.250440 13884 raft_consensus.cc:729] T 37190356d40d40c7aed6607b87c676fd P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:14.250777 13884 consensus_queue.cc:260] T 37190356d40d40c7aed6607b87c676fd P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:14.251274 13884 ts_tablet_manager.cc:1398] T 37190356d40d40c7aed6607b87c676fd P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240503 02:00:14.251797 6458 raft_consensus.cc:1261] T 308236ce104a41baa84cf1d8e6f68c0e P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 99ead02915954b4fab23b033ad6d546b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:14.252208 6589 raft_consensus.cc:1261] T 308236ce104a41baa84cf1d8e6f68c0e P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer 99ead02915954b4fab23b033ad6d546b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:14.252936 13924 consensus_queue.cc:1035] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:14.253211 13937 tablet_bootstrap.cc:492] T 0fc624ba9c6940a3a7eedcf9a7d79dde P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:14.253995 13924 consensus_queue.cc:1035] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:14.255728 6308 catalog_manager.cc:5576] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863) does not have the latest schema for tablet 308236ce104a41baa84cf1d8e6f68c0e (table rebalancer_dynamic_table [id=d3615bb8f356419e9ff1adfb5631ddc8]). Expected version 1 got 0
I20240503 02:00:14.256233 13933 tablet_bootstrap.cc:492] T 0fc624ba9c6940a3a7eedcf9a7d79dde P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:14.256331 13933 ts_tablet_manager.cc:1367] T 0fc624ba9c6940a3a7eedcf9a7d79dde P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.000s
I20240503 02:00:14.256882 13933 raft_consensus.cc:348] T 0fc624ba9c6940a3a7eedcf9a7d79dde P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:14.257331 13933 raft_consensus.cc:374] T 0fc624ba9c6940a3a7eedcf9a7d79dde P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:14.257411 13932 tablet_bootstrap.cc:492] T 37190356d40d40c7aed6607b87c676fd P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:14.257570 13932 ts_tablet_manager.cc:1367] T 37190356d40d40c7aed6607b87c676fd P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 02:00:14.258093 13932 raft_consensus.cc:348] T 37190356d40d40c7aed6607b87c676fd P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:14.258229 6309 catalog_manager.cc:5576] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093) does not have the latest schema for tablet 308236ce104a41baa84cf1d8e6f68c0e (table rebalancer_dynamic_table [id=d3615bb8f356419e9ff1adfb5631ddc8]). Expected version 1 got 0
I20240503 02:00:14.258915 13937 tablet_bootstrap.cc:654] T 0fc624ba9c6940a3a7eedcf9a7d79dde P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:14.259405 13933 raft_consensus.cc:729] T 0fc624ba9c6940a3a7eedcf9a7d79dde P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:14.259485 13932 raft_consensus.cc:374] T 37190356d40d40c7aed6607b87c676fd P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:14.259555 13932 raft_consensus.cc:729] T 37190356d40d40c7aed6607b87c676fd P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:14.259706 13932 consensus_queue.cc:260] T 37190356d40d40c7aed6607b87c676fd P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:14.259765 13933 consensus_queue.cc:260] T 0fc624ba9c6940a3a7eedcf9a7d79dde P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:14.260032 13932 ts_tablet_manager.cc:1398] T 37190356d40d40c7aed6607b87c676fd P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:14.260174 13933 ts_tablet_manager.cc:1398] T 0fc624ba9c6940a3a7eedcf9a7d79dde P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 02:00:14.260219 13932 tablet_bootstrap.cc:492] T d9c6a7e041014192b2a8e9bced641391 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:14.261260 13932 tablet_bootstrap.cc:654] T d9c6a7e041014192b2a8e9bced641391 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:14.261518 13933 tablet_bootstrap.cc:492] T d9c6a7e041014192b2a8e9bced641391 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:14.262668 13933 tablet_bootstrap.cc:654] T d9c6a7e041014192b2a8e9bced641391 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:14.262797 13937 tablet_bootstrap.cc:492] T 0fc624ba9c6940a3a7eedcf9a7d79dde P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:14.262822 13932 tablet_bootstrap.cc:492] T d9c6a7e041014192b2a8e9bced641391 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:14.262905 13932 ts_tablet_manager.cc:1367] T d9c6a7e041014192b2a8e9bced641391 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:14.262934 13937 ts_tablet_manager.cc:1367] T 0fc624ba9c6940a3a7eedcf9a7d79dde P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 02:00:14.263460 13937 raft_consensus.cc:348] T 0fc624ba9c6940a3a7eedcf9a7d79dde P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:14.263459 13932 raft_consensus.cc:348] T d9c6a7e041014192b2a8e9bced641391 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:14.263588 13932 raft_consensus.cc:374] T d9c6a7e041014192b2a8e9bced641391 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:14.263724 13932 raft_consensus.cc:729] T d9c6a7e041014192b2a8e9bced641391 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:14.263901 13932 consensus_queue.cc:260] T d9c6a7e041014192b2a8e9bced641391 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:14.264250 13932 ts_tablet_manager.cc:1398] T d9c6a7e041014192b2a8e9bced641391 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:14.263624 13937 raft_consensus.cc:374] T 0fc624ba9c6940a3a7eedcf9a7d79dde P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:14.264726 13937 raft_consensus.cc:729] T 0fc624ba9c6940a3a7eedcf9a7d79dde P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:14.266183 13937 consensus_queue.cc:260] T 0fc624ba9c6940a3a7eedcf9a7d79dde P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:14.266649 13933 tablet_bootstrap.cc:492] T d9c6a7e041014192b2a8e9bced641391 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:14.266745 13933 ts_tablet_manager.cc:1367] T d9c6a7e041014192b2a8e9bced641391 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240503 02:00:14.267230 13933 raft_consensus.cc:348] T d9c6a7e041014192b2a8e9bced641391 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:14.267334 13933 raft_consensus.cc:374] T d9c6a7e041014192b2a8e9bced641391 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:14.267361 13933 raft_consensus.cc:729] T d9c6a7e041014192b2a8e9bced641391 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:14.267611 13950 tablet.cc:1722] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:14.267935 13950 tablet.cc:1620] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b: MemRowSet was empty: no flush needed.
I20240503 02:00:14.269068 13933 consensus_queue.cc:260] T d9c6a7e041014192b2a8e9bced641391 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:14.270792 13933 ts_tablet_manager.cc:1398] T d9c6a7e041014192b2a8e9bced641391 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.004s user 0.000s sys 0.000s
I20240503 02:00:14.271836 6309 catalog_manager.cc:5576] TS 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157) does not have the latest schema for tablet 308236ce104a41baa84cf1d8e6f68c0e (table rebalancer_dynamic_table [id=d3615bb8f356419e9ff1adfb5631ddc8]). Expected version 1 got 0
I20240503 02:00:14.272539 13939 tablet_bootstrap.cc:492] T 37190356d40d40c7aed6607b87c676fd P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:14.272639 13939 ts_tablet_manager.cc:1367] T 37190356d40d40c7aed6607b87c676fd P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.000s
I20240503 02:00:14.273192 13939 raft_consensus.cc:348] T 37190356d40d40c7aed6607b87c676fd P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:14.273308 13939 raft_consensus.cc:374] T 37190356d40d40c7aed6607b87c676fd P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:14.273339 13939 raft_consensus.cc:729] T 37190356d40d40c7aed6607b87c676fd P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:14.273455 13939 consensus_queue.cc:260] T 37190356d40d40c7aed6607b87c676fd P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:14.273749 13939 ts_tablet_manager.cc:1398] T 37190356d40d40c7aed6607b87c676fd P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:14.274029 13937 ts_tablet_manager.cc:1398] T 0fc624ba9c6940a3a7eedcf9a7d79dde P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20240503 02:00:14.275005 6313 catalog_manager.cc:5576] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863) does not have the latest schema for tablet 308236ce104a41baa84cf1d8e6f68c0e (table rebalancer_dynamic_table [id=d3615bb8f356419e9ff1adfb5631ddc8]). Expected version 1 got 0
I20240503 02:00:14.275112 13951 tablet.cc:1722] T 308236ce104a41baa84cf1d8e6f68c0e P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:14.276151 13951 tablet.cc:1620] T 308236ce104a41baa84cf1d8e6f68c0e P dec3e4a04d604f7eaaa3f077d081218a: MemRowSet was empty: no flush needed.
I20240503 02:00:14.277295 6312 catalog_manager.cc:5576] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093) does not have the latest schema for tablet 308236ce104a41baa84cf1d8e6f68c0e (table rebalancer_dynamic_table [id=d3615bb8f356419e9ff1adfb5631ddc8]). Expected version 1 got 0
I20240503 02:00:14.278475 6308 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=d3615bb8f356419e9ff1adfb5631ddc8] alter complete (version 1)
I20240503 02:00:14.280073 13952 tablet.cc:1722] T 308236ce104a41baa84cf1d8e6f68c0e P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:14.280490 13952 tablet.cc:1620] T 308236ce104a41baa84cf1d8e6f68c0e P 80fd18ef4d604c86b909d5c008a46b4f: MemRowSet was empty: no flush needed.
I20240503 02:00:14.308444 13911 raft_consensus.cc:482] T 0fc624ba9c6940a3a7eedcf9a7d79dde P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:14.308594 13911 raft_consensus.cc:504] T 0fc624ba9c6940a3a7eedcf9a7d79dde P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:14.309015 13911 leader_election.cc:288] T 0fc624ba9c6940a3a7eedcf9a7d79dde P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:14.309453 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc624ba9c6940a3a7eedcf9a7d79dde" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:14.309515 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc624ba9c6940a3a7eedcf9a7d79dde" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 02:00:14.309593 8916 raft_consensus.cc:2454] T 0fc624ba9c6940a3a7eedcf9a7d79dde P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:14.309665 6589 raft_consensus.cc:2454] T 0fc624ba9c6940a3a7eedcf9a7d79dde P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:14.309937 6656 leader_election.cc:302] T 0fc624ba9c6940a3a7eedcf9a7d79dde P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9942306855404cfd891adf227dd65eb2, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:14.310214 13911 raft_consensus.cc:2781] T 0fc624ba9c6940a3a7eedcf9a7d79dde P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:14.310276 13911 raft_consensus.cc:482] T 0fc624ba9c6940a3a7eedcf9a7d79dde P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:14.310302 13911 raft_consensus.cc:3037] T 0fc624ba9c6940a3a7eedcf9a7d79dde P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:14.311195 13911 raft_consensus.cc:504] T 0fc624ba9c6940a3a7eedcf9a7d79dde P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:14.311623 13911 leader_election.cc:288] T 0fc624ba9c6940a3a7eedcf9a7d79dde P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Requested vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:14.312065 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc624ba9c6940a3a7eedcf9a7d79dde" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:14.312193 8916 raft_consensus.cc:3037] T 0fc624ba9c6940a3a7eedcf9a7d79dde P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:14.312191 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc624ba9c6940a3a7eedcf9a7d79dde" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 02:00:14.312330 6589 raft_consensus.cc:3037] T 0fc624ba9c6940a3a7eedcf9a7d79dde P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:14.313182 8916 raft_consensus.cc:2454] T 0fc624ba9c6940a3a7eedcf9a7d79dde P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:14.313359 6589 raft_consensus.cc:2454] T 0fc624ba9c6940a3a7eedcf9a7d79dde P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:14.313490 6656 leader_election.cc:302] T 0fc624ba9c6940a3a7eedcf9a7d79dde P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9942306855404cfd891adf227dd65eb2, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:14.313690 13911 raft_consensus.cc:2781] T 0fc624ba9c6940a3a7eedcf9a7d79dde P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:14.313864 13911 raft_consensus.cc:686] T 0fc624ba9c6940a3a7eedcf9a7d79dde P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Becoming Leader. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Running, Role: LEADER
I20240503 02:00:14.314019 13911 consensus_queue.cc:237] T 0fc624ba9c6940a3a7eedcf9a7d79dde P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:14.316128 6309 catalog_manager.cc:5484] T 0fc624ba9c6940a3a7eedcf9a7d79dde P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } }
W20240503 02:00:14.324738 6301 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111) [suppressed 56 similar messages]
W20240503 02:00:14.325449 6301 catalog_manager.cc:4573] TS e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): CreateTablet RPC failed for tablet 56824b553788471c9a2fcd6ec82d507e: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
W20240503 02:00:14.329160 6654 consensus_peers.cc:485] T cea1ea50fa434d31b42451f84f26bc76 P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:00:14.383385 6309 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:14.394821 13952 tablet.cc:1722] T 308236ce104a41baa84cf1d8e6f68c0e P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:14.395202 13952 tablet.cc:1620] T 308236ce104a41baa84cf1d8e6f68c0e P 80fd18ef4d604c86b909d5c008a46b4f: MemRowSet was empty: no flush needed.
I20240503 02:00:14.395514 13950 tablet.cc:1722] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:14.395833 13950 tablet.cc:1620] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b: MemRowSet was empty: no flush needed.
I20240503 02:00:14.401247 13951 tablet.cc:1722] T 308236ce104a41baa84cf1d8e6f68c0e P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:14.401621 13951 tablet.cc:1620] T 308236ce104a41baa84cf1d8e6f68c0e P dec3e4a04d604f7eaaa3f077d081218a: MemRowSet was empty: no flush needed.
I20240503 02:00:14.408622 6312 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=d3615bb8f356419e9ff1adfb5631ddc8] alter complete (version 2)
I20240503 02:00:14.409536 13679 raft_consensus.cc:482] T d9c6a7e041014192b2a8e9bced641391 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:14.409643 13679 raft_consensus.cc:504] T d9c6a7e041014192b2a8e9bced641391 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:14.410024 13679 leader_election.cc:288] T d9c6a7e041014192b2a8e9bced641391 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:14.410410 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9c6a7e041014192b2a8e9bced641391" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 02:00:14.410547 6851 raft_consensus.cc:2454] T d9c6a7e041014192b2a8e9bced641391 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:14.410604 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9c6a7e041014192b2a8e9bced641391" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:14.410735 9048 raft_consensus.cc:2454] T d9c6a7e041014192b2a8e9bced641391 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:14.410986 6656 leader_election.cc:302] T d9c6a7e041014192b2a8e9bced641391 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:14.411259 13679 raft_consensus.cc:2781] T d9c6a7e041014192b2a8e9bced641391 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:14.411319 13679 raft_consensus.cc:482] T d9c6a7e041014192b2a8e9bced641391 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:14.411346 13679 raft_consensus.cc:3037] T d9c6a7e041014192b2a8e9bced641391 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:14.412189 13679 raft_consensus.cc:504] T d9c6a7e041014192b2a8e9bced641391 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:14.412603 13679 leader_election.cc:288] T d9c6a7e041014192b2a8e9bced641391 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Requested vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:14.412968 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9c6a7e041014192b2a8e9bced641391" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:14.412989 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9c6a7e041014192b2a8e9bced641391" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 02:00:14.413081 9048 raft_consensus.cc:3037] T d9c6a7e041014192b2a8e9bced641391 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:14.413100 6851 raft_consensus.cc:3037] T d9c6a7e041014192b2a8e9bced641391 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:14.413931 6851 raft_consensus.cc:2454] T d9c6a7e041014192b2a8e9bced641391 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:14.413928 9048 raft_consensus.cc:2454] T d9c6a7e041014192b2a8e9bced641391 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:14.414350 6654 leader_election.cc:302] T d9c6a7e041014192b2a8e9bced641391 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99ead02915954b4fab23b033ad6d546b, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:14.414914 13679 raft_consensus.cc:2781] T d9c6a7e041014192b2a8e9bced641391 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:14.415007 13679 raft_consensus.cc:686] T d9c6a7e041014192b2a8e9bced641391 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Becoming Leader. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Running, Role: LEADER
I20240503 02:00:14.415151 13679 consensus_queue.cc:237] T d9c6a7e041014192b2a8e9bced641391 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:14.417402 6308 catalog_manager.cc:5484] T d9c6a7e041014192b2a8e9bced641391 P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } }
W20240503 02:00:14.470124 8982 consensus_peers.cc:485] T beeea71f02ae4c89a1af13d3584ef70a P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:14.483481 8982 consensus_peers.cc:485] T 1a7ee79312ea4bad816264375ca5bb2d P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:00:14.512534 6308 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:14.516469 13924 raft_consensus.cc:482] T 37190356d40d40c7aed6607b87c676fd P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:14.516589 13924 raft_consensus.cc:504] T 37190356d40d40c7aed6607b87c676fd P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:14.516976 13924 leader_election.cc:288] T 37190356d40d40c7aed6607b87c676fd P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:14.517905 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37190356d40d40c7aed6607b87c676fd" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:14.518059 9178 raft_consensus.cc:2454] T 37190356d40d40c7aed6607b87c676fd P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:14.518431 8981 leader_election.cc:302] T 37190356d40d40c7aed6607b87c676fd P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:14.518571 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37190356d40d40c7aed6607b87c676fd" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 02:00:14.518733 6851 raft_consensus.cc:2454] T 37190356d40d40c7aed6607b87c676fd P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:14.518739 13952 tablet.cc:1722] T 308236ce104a41baa84cf1d8e6f68c0e P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:14.519153 13900 raft_consensus.cc:2781] T 37190356d40d40c7aed6607b87c676fd P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:14.519212 13900 raft_consensus.cc:482] T 37190356d40d40c7aed6607b87c676fd P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:14.519251 13900 raft_consensus.cc:3037] T 37190356d40d40c7aed6607b87c676fd P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:14.519395 13950 tablet.cc:1722] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:14.520215 13900 raft_consensus.cc:504] T 37190356d40d40c7aed6607b87c676fd P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:14.520635 13900 leader_election.cc:288] T 37190356d40d40c7aed6607b87c676fd P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Requested vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:14.522167 13951 tablet.cc:1722] T 308236ce104a41baa84cf1d8e6f68c0e P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:14.522986 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37190356d40d40c7aed6607b87c676fd" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:14.523136 9178 raft_consensus.cc:3037] T 37190356d40d40c7aed6607b87c676fd P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:14.524210 9178 raft_consensus.cc:2454] T 37190356d40d40c7aed6607b87c676fd P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:14.524590 8981 leader_election.cc:302] T 37190356d40d40c7aed6607b87c676fd P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:14.525272 13956 raft_consensus.cc:2781] T 37190356d40d40c7aed6607b87c676fd P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:14.525417 13956 raft_consensus.cc:686] T 37190356d40d40c7aed6607b87c676fd P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Becoming Leader. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Running, Role: LEADER
I20240503 02:00:14.525609 13956 consensus_queue.cc:237] T 37190356d40d40c7aed6607b87c676fd P 99ead02915954b4fab23b033ad6d546b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:14.526402 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37190356d40d40c7aed6607b87c676fd" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 02:00:14.526533 6851 raft_consensus.cc:3037] T 37190356d40d40c7aed6607b87c676fd P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:14.527149 6309 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=d3615bb8f356419e9ff1adfb5631ddc8] alter complete (version 3)
I20240503 02:00:14.527714 6851 raft_consensus.cc:2454] T 37190356d40d40c7aed6607b87c676fd P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:14.529774 6308 catalog_manager.cc:5484] T 37190356d40d40c7aed6607b87c676fd P 99ead02915954b4fab23b033ad6d546b reported cstate change: term changed from 0 to 1, leader changed from <none> to 99ead02915954b4fab23b033ad6d546b (127.0.99.134). New cstate: current_term: 1 leader_uuid: "99ead02915954b4fab23b033ad6d546b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:14.632862 6308 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:14.633026 6308 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:14.635115 6308 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet 308236ce104a41baa84cf1d8e6f68c0e
I20240503 02:00:14.635776 9028 tablet_service.cc:1514] Processing DeleteTablet for tablet 308236ce104a41baa84cf1d8e6f68c0e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:14 UTC) from {username='slave'} at 127.0.0.1:52584
I20240503 02:00:14.635859 6569 tablet_service.cc:1514] Processing DeleteTablet for tablet 308236ce104a41baa84cf1d8e6f68c0e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:14 UTC) from {username='slave'} at 127.0.0.1:59846
I20240503 02:00:14.635861 6436 tablet_service.cc:1514] Processing DeleteTablet for tablet 308236ce104a41baa84cf1d8e6f68c0e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:14 UTC) from {username='slave'} at 127.0.0.1:51338
I20240503 02:00:14.636179 13961 tablet_replica.cc:330] T 308236ce104a41baa84cf1d8e6f68c0e P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 02:00:14.636196 13962 tablet_replica.cc:330] T 308236ce104a41baa84cf1d8e6f68c0e P dec3e4a04d604f7eaaa3f077d081218a: stopping tablet replica
I20240503 02:00:14.636344 13960 tablet_replica.cc:330] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b: stopping tablet replica
I20240503 02:00:14.636341 13961 raft_consensus.cc:2229] T 308236ce104a41baa84cf1d8e6f68c0e P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:14.636355 13962 raft_consensus.cc:2229] T 308236ce104a41baa84cf1d8e6f68c0e P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:14.636497 13960 raft_consensus.cc:2229] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:14.636560 13962 raft_consensus.cc:2258] T 308236ce104a41baa84cf1d8e6f68c0e P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:14.636497 13961 raft_consensus.cc:2258] T 308236ce104a41baa84cf1d8e6f68c0e P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:14.636965 13960 raft_consensus.cc:2258] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:14.637156 13962 ts_tablet_manager.cc:1875] T 308236ce104a41baa84cf1d8e6f68c0e P dec3e4a04d604f7eaaa3f077d081218a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:14.637166 13961 ts_tablet_manager.cc:1875] T 308236ce104a41baa84cf1d8e6f68c0e P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:14.637501 13960 ts_tablet_manager.cc:1875] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:14.639355 13961 ts_tablet_manager.cc:1888] T 308236ce104a41baa84cf1d8e6f68c0e P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:14.639439 13961 log.cc:1196] T 308236ce104a41baa84cf1d8e6f68c0e P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/308236ce104a41baa84cf1d8e6f68c0e
I20240503 02:00:14.639617 13962 ts_tablet_manager.cc:1888] T 308236ce104a41baa84cf1d8e6f68c0e P dec3e4a04d604f7eaaa3f077d081218a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:14.639742 13961 ts_tablet_manager.cc:1909] T 308236ce104a41baa84cf1d8e6f68c0e P 80fd18ef4d604c86b909d5c008a46b4f: Deleting consensus metadata
I20240503 02:00:14.639907 13960 ts_tablet_manager.cc:1888] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:14.639945 13962 log.cc:1196] T 308236ce104a41baa84cf1d8e6f68c0e P dec3e4a04d604f7eaaa3f077d081218a: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/308236ce104a41baa84cf1d8e6f68c0e
I20240503 02:00:14.639989 13960 log.cc:1196] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/308236ce104a41baa84cf1d8e6f68c0e
I20240503 02:00:14.640235 13960 ts_tablet_manager.cc:1909] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b: Deleting consensus metadata
I20240503 02:00:14.640275 13962 ts_tablet_manager.cc:1909] T 308236ce104a41baa84cf1d8e6f68c0e P dec3e4a04d604f7eaaa3f077d081218a: Deleting consensus metadata
I20240503 02:00:14.640877 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet 308236ce104a41baa84cf1d8e6f68c0e (table rebalancer_dynamic_table_new_name [id=d3615bb8f356419e9ff1adfb5631ddc8]) successfully deleted
I20240503 02:00:14.641057 6300 catalog_manager.cc:4830] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): tablet 308236ce104a41baa84cf1d8e6f68c0e (table rebalancer_dynamic_table_new_name [id=d3615bb8f356419e9ff1adfb5631ddc8]) successfully deleted
I20240503 02:00:14.641381 6301 catalog_manager.cc:4830] TS 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157): tablet 308236ce104a41baa84cf1d8e6f68c0e (table rebalancer_dynamic_table_new_name [id=d3615bb8f356419e9ff1adfb5631ddc8]) successfully deleted
W20240503 02:00:14.727628 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:14.737404 6308 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:00:14.743417 8897 tablet_service.cc:1467] Processing CreateTablet for tablet aaec090e44114ab38e49cc24814453fe (DEFAULT_TABLE table=rebalancer_dynamic_table [id=66c21d25deab4cceb8c1e9ff09440b2b]), partition=
I20240503 02:00:14.743556 6569 tablet_service.cc:1467] Processing CreateTablet for tablet aaec090e44114ab38e49cc24814453fe (DEFAULT_TABLE table=rebalancer_dynamic_table [id=66c21d25deab4cceb8c1e9ff09440b2b]), partition=
I20240503 02:00:14.743690 8897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaec090e44114ab38e49cc24814453fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:14.743817 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaec090e44114ab38e49cc24814453fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:14.745245 9028 tablet_service.cc:1467] Processing CreateTablet for tablet aaec090e44114ab38e49cc24814453fe (DEFAULT_TABLE table=rebalancer_dynamic_table [id=66c21d25deab4cceb8c1e9ff09440b2b]), partition=
I20240503 02:00:14.745554 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaec090e44114ab38e49cc24814453fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:14.746312 13963 tablet_bootstrap.cc:492] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:14.746320 13937 tablet_bootstrap.cc:492] T aaec090e44114ab38e49cc24814453fe P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:14.747273 13937 tablet_bootstrap.cc:654] T aaec090e44114ab38e49cc24814453fe P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:14.747339 13963 tablet_bootstrap.cc:654] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:14.747831 13884 tablet_bootstrap.cc:492] T aaec090e44114ab38e49cc24814453fe P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:14.748667 13963 tablet_bootstrap.cc:492] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:14.748772 13963 ts_tablet_manager.cc:1367] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:14.749096 13884 tablet_bootstrap.cc:654] T aaec090e44114ab38e49cc24814453fe P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:14.749375 13937 tablet_bootstrap.cc:492] T aaec090e44114ab38e49cc24814453fe P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:14.749460 13937 ts_tablet_manager.cc:1367] T aaec090e44114ab38e49cc24814453fe P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:14.749629 13963 raft_consensus.cc:348] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:14.749872 13963 raft_consensus.cc:374] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:14.749920 13963 raft_consensus.cc:729] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:14.749928 13937 raft_consensus.cc:348] T aaec090e44114ab38e49cc24814453fe P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:14.750047 13937 raft_consensus.cc:374] T aaec090e44114ab38e49cc24814453fe P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:14.750087 13937 raft_consensus.cc:729] T aaec090e44114ab38e49cc24814453fe P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:14.750169 13963 consensus_queue.cc:260] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:14.750207 13937 consensus_queue.cc:260] T aaec090e44114ab38e49cc24814453fe P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:14.750473 13937 ts_tablet_manager.cc:1398] T aaec090e44114ab38e49cc24814453fe P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:14.751610 13963 ts_tablet_manager.cc:1398] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:00:14.751928 13884 tablet_bootstrap.cc:492] T aaec090e44114ab38e49cc24814453fe P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:14.752043 13884 ts_tablet_manager.cc:1367] T aaec090e44114ab38e49cc24814453fe P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:14.752586 13884 raft_consensus.cc:348] T aaec090e44114ab38e49cc24814453fe P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:14.752714 13884 raft_consensus.cc:374] T aaec090e44114ab38e49cc24814453fe P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:14.752753 13884 raft_consensus.cc:729] T aaec090e44114ab38e49cc24814453fe P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:14.752878 13884 consensus_queue.cc:260] T aaec090e44114ab38e49cc24814453fe P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:14.753152 13884 ts_tablet_manager.cc:1398] T aaec090e44114ab38e49cc24814453fe P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:14.827914 13969 consensus_queue.cc:1035] T d9c6a7e041014192b2a8e9bced641391 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:14.832425 13969 consensus_queue.cc:1035] T d9c6a7e041014192b2a8e9bced641391 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:14.887717 13969 consensus_queue.cc:1035] T 0fc624ba9c6940a3a7eedcf9a7d79dde P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:14.891083 13911 consensus_queue.cc:1035] T 0fc624ba9c6940a3a7eedcf9a7d79dde P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:14.923871 13943 consensus_queue.cc:1035] T 37190356d40d40c7aed6607b87c676fd P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:14.927531 13956 consensus_queue.cc:1035] T 37190356d40d40c7aed6607b87c676fd P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:15.045089 13978 raft_consensus.cc:482] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:15.045279 13978 raft_consensus.cc:504] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:15.045509 6309 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_extra_table_38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:15.045835 13978 leader_election.cc:288] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:15.046497 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaec090e44114ab38e49cc24814453fe" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:15.046509 8915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaec090e44114ab38e49cc24814453fe" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:15.046635 9048 raft_consensus.cc:2454] T aaec090e44114ab38e49cc24814453fe P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:15.046651 8915 raft_consensus.cc:2454] T aaec090e44114ab38e49cc24814453fe P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:15.047029 6525 leader_election.cc:302] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:15.047300 13978 raft_consensus.cc:2781] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:15.047351 13978 raft_consensus.cc:482] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:15.047376 13978 raft_consensus.cc:3037] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:15.048588 13978 raft_consensus.cc:504] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:15.049018 13978 leader_election.cc:288] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Requested vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:15.049515 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaec090e44114ab38e49cc24814453fe" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:15.049661 9048 raft_consensus.cc:3037] T aaec090e44114ab38e49cc24814453fe P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:15.050861 9048 raft_consensus.cc:2454] T aaec090e44114ab38e49cc24814453fe P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:15.050969 8915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaec090e44114ab38e49cc24814453fe" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:15.051091 8915 raft_consensus.cc:3037] T aaec090e44114ab38e49cc24814453fe P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:15.051235 6523 leader_election.cc:302] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:15.052194 8915 raft_consensus.cc:2454] T aaec090e44114ab38e49cc24814453fe P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:15.053357 9028 tablet_service.cc:1467] Processing CreateTablet for tablet 5ca282a13b714b63954c2812308904b4 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=f84ba9cbcf8f4e91af818979cee96b3c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:15.053364 8880 tablet_service.cc:1467] Processing CreateTablet for tablet 5ca282a13b714b63954c2812308904b4 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=f84ba9cbcf8f4e91af818979cee96b3c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:15.053689 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ca282a13b714b63954c2812308904b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:15.053709 8880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ca282a13b714b63954c2812308904b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:15.054262 6438 tablet_service.cc:1467] Processing CreateTablet for tablet 6f2e5973824b4d7a8b0d31faa622e6aa (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=f84ba9cbcf8f4e91af818979cee96b3c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:15.054545 6438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f2e5973824b4d7a8b0d31faa622e6aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:15.055925 13884 tablet_bootstrap.cc:492] T 5ca282a13b714b63954c2812308904b4 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:15.055959 6436 tablet_service.cc:1467] Processing CreateTablet for tablet 01b63d6a79314a86bd64f7618c86716d (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=f84ba9cbcf8f4e91af818979cee96b3c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:15.056219 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01b63d6a79314a86bd64f7618c86716d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:15.056954 13884 tablet_bootstrap.cc:654] T 5ca282a13b714b63954c2812308904b4 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:15.057562 9028 tablet_service.cc:1467] Processing CreateTablet for tablet 6f2e5973824b4d7a8b0d31faa622e6aa (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=f84ba9cbcf8f4e91af818979cee96b3c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:15.057823 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f2e5973824b4d7a8b0d31faa622e6aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:15.058452 13937 tablet_bootstrap.cc:492] T 5ca282a13b714b63954c2812308904b4 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:15.058586 13884 tablet_bootstrap.cc:492] T 5ca282a13b714b63954c2812308904b4 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:15.058660 13884 ts_tablet_manager.cc:1367] T 5ca282a13b714b63954c2812308904b4 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:15.059034 13978 raft_consensus.cc:2781] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:15.059134 13884 raft_consensus.cc:348] T 5ca282a13b714b63954c2812308904b4 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:15.059207 13978 raft_consensus.cc:686] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Becoming Leader. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Running, Role: LEADER
I20240503 02:00:15.059239 13884 raft_consensus.cc:374] T 5ca282a13b714b63954c2812308904b4 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:15.059270 13884 raft_consensus.cc:729] T 5ca282a13b714b63954c2812308904b4 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:15.059347 13978 consensus_queue.cc:237] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:15.059379 13884 consensus_queue.cc:260] T 5ca282a13b714b63954c2812308904b4 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:15.059468 13937 tablet_bootstrap.cc:654] T 5ca282a13b714b63954c2812308904b4 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:15.059669 13884 ts_tablet_manager.cc:1398] T 5ca282a13b714b63954c2812308904b4 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:15.060227 6569 tablet_service.cc:1467] Processing CreateTablet for tablet 01b63d6a79314a86bd64f7618c86716d (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=f84ba9cbcf8f4e91af818979cee96b3c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:15.060449 6567 tablet_service.cc:1467] Processing CreateTablet for tablet 5ca282a13b714b63954c2812308904b4 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=f84ba9cbcf8f4e91af818979cee96b3c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:15.060614 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01b63d6a79314a86bd64f7618c86716d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:15.053357 8897 tablet_service.cc:1467] Processing CreateTablet for tablet 01b63d6a79314a86bd64f7618c86716d (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=f84ba9cbcf8f4e91af818979cee96b3c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:15.060767 6567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ca282a13b714b63954c2812308904b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:15.060899 8897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01b63d6a79314a86bd64f7618c86716d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:15.061363 13884 tablet_bootstrap.cc:492] T 6f2e5973824b4d7a8b0d31faa622e6aa P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:15.062307 13884 tablet_bootstrap.cc:654] T 6f2e5973824b4d7a8b0d31faa622e6aa P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:15.063079 13963 tablet_bootstrap.cc:492] T 5ca282a13b714b63954c2812308904b4 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:15.064007 13937 tablet_bootstrap.cc:492] T 5ca282a13b714b63954c2812308904b4 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:15.064100 13937 ts_tablet_manager.cc:1367] T 5ca282a13b714b63954c2812308904b4 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:00:15.064241 13963 tablet_bootstrap.cc:654] T 5ca282a13b714b63954c2812308904b4 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:15.064641 13937 raft_consensus.cc:348] T 5ca282a13b714b63954c2812308904b4 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:15.064769 13937 raft_consensus.cc:374] T 5ca282a13b714b63954c2812308904b4 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:15.064800 13937 raft_consensus.cc:729] T 5ca282a13b714b63954c2812308904b4 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:15.064916 13937 consensus_queue.cc:260] T 5ca282a13b714b63954c2812308904b4 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:15.065204 13937 ts_tablet_manager.cc:1398] T 5ca282a13b714b63954c2812308904b4 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:15.065371 13937 tablet_bootstrap.cc:492] T 01b63d6a79314a86bd64f7618c86716d P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:15.066399 13937 tablet_bootstrap.cc:654] T 01b63d6a79314a86bd64f7618c86716d P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:15.066676 9158 tablet_service.cc:1467] Processing CreateTablet for tablet 6f2e5973824b4d7a8b0d31faa622e6aa (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=f84ba9cbcf8f4e91af818979cee96b3c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:15.066970 9158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f2e5973824b4d7a8b0d31faa622e6aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:15.067739 13937 tablet_bootstrap.cc:492] T 01b63d6a79314a86bd64f7618c86716d P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:15.067826 13937 ts_tablet_manager.cc:1367] T 01b63d6a79314a86bd64f7618c86716d P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:15.068353 13937 raft_consensus.cc:348] T 01b63d6a79314a86bd64f7618c86716d P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:15.068488 13937 raft_consensus.cc:374] T 01b63d6a79314a86bd64f7618c86716d P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:15.068519 13937 raft_consensus.cc:729] T 01b63d6a79314a86bd64f7618c86716d P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:15.068639 13937 consensus_queue.cc:260] T 01b63d6a79314a86bd64f7618c86716d P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:15.068928 13937 ts_tablet_manager.cc:1398] T 01b63d6a79314a86bd64f7618c86716d P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:15.069126 13963 tablet_bootstrap.cc:492] T 5ca282a13b714b63954c2812308904b4 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:15.069242 13963 ts_tablet_manager.cc:1367] T 5ca282a13b714b63954c2812308904b4 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:00:15.069710 13963 raft_consensus.cc:348] T 5ca282a13b714b63954c2812308904b4 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:15.069820 13963 raft_consensus.cc:374] T 5ca282a13b714b63954c2812308904b4 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:15.069852 13963 raft_consensus.cc:729] T 5ca282a13b714b63954c2812308904b4 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:15.069988 13963 consensus_queue.cc:260] T 5ca282a13b714b63954c2812308904b4 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:15.070295 13963 ts_tablet_manager.cc:1398] T 5ca282a13b714b63954c2812308904b4 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:15.070493 13963 tablet_bootstrap.cc:492] T 01b63d6a79314a86bd64f7618c86716d P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:15.071184 13992 tablet_bootstrap.cc:492] T 6f2e5973824b4d7a8b0d31faa622e6aa P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:15.071533 13963 tablet_bootstrap.cc:654] T 01b63d6a79314a86bd64f7618c86716d P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:15.072237 13992 tablet_bootstrap.cc:654] T 6f2e5973824b4d7a8b0d31faa622e6aa P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:15.074033 13985 tablet_bootstrap.cc:492] T 6f2e5973824b4d7a8b0d31faa622e6aa P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:15.075141 13985 tablet_bootstrap.cc:654] T 6f2e5973824b4d7a8b0d31faa622e6aa P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:15.075188 6309 catalog_manager.cc:5484] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:15.076663 13985 tablet_bootstrap.cc:492] T 6f2e5973824b4d7a8b0d31faa622e6aa P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:15.076864 13985 ts_tablet_manager.cc:1367] T 6f2e5973824b4d7a8b0d31faa622e6aa P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:15.077266 13963 tablet_bootstrap.cc:492] T 01b63d6a79314a86bd64f7618c86716d P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:15.077374 13963 ts_tablet_manager.cc:1367] T 01b63d6a79314a86bd64f7618c86716d P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:00:15.077425 13985 raft_consensus.cc:348] T 6f2e5973824b4d7a8b0d31faa622e6aa P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:15.077534 13985 raft_consensus.cc:374] T 6f2e5973824b4d7a8b0d31faa622e6aa P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:15.077562 13985 raft_consensus.cc:729] T 6f2e5973824b4d7a8b0d31faa622e6aa P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:15.077703 13985 consensus_queue.cc:260] T 6f2e5973824b4d7a8b0d31faa622e6aa P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:15.077855 13963 raft_consensus.cc:348] T 01b63d6a79314a86bd64f7618c86716d P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:15.077956 13963 raft_consensus.cc:374] T 01b63d6a79314a86bd64f7618c86716d P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:15.077987 13963 raft_consensus.cc:729] T 01b63d6a79314a86bd64f7618c86716d P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:15.078002 13985 ts_tablet_manager.cc:1398] T 6f2e5973824b4d7a8b0d31faa622e6aa P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:15.078104 13963 consensus_queue.cc:260] T 01b63d6a79314a86bd64f7618c86716d P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:15.078387 13963 ts_tablet_manager.cc:1398] T 01b63d6a79314a86bd64f7618c86716d P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:15.078572 13985 tablet_bootstrap.cc:492] T 01b63d6a79314a86bd64f7618c86716d P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:15.081099 13884 tablet_bootstrap.cc:492] T 6f2e5973824b4d7a8b0d31faa622e6aa P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:15.081236 13884 ts_tablet_manager.cc:1367] T 6f2e5973824b4d7a8b0d31faa622e6aa P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20240503 02:00:15.081511 13985 tablet_bootstrap.cc:654] T 01b63d6a79314a86bd64f7618c86716d P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:15.081772 13884 raft_consensus.cc:348] T 6f2e5973824b4d7a8b0d31faa622e6aa P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:15.081889 13884 raft_consensus.cc:374] T 6f2e5973824b4d7a8b0d31faa622e6aa P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:15.081918 13884 raft_consensus.cc:729] T 6f2e5973824b4d7a8b0d31faa622e6aa P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:15.082049 13884 consensus_queue.cc:260] T 6f2e5973824b4d7a8b0d31faa622e6aa P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:15.082334 13884 ts_tablet_manager.cc:1398] T 6f2e5973824b4d7a8b0d31faa622e6aa P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:15.084327 13992 tablet_bootstrap.cc:492] T 6f2e5973824b4d7a8b0d31faa622e6aa P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:15.084555 13992 ts_tablet_manager.cc:1367] T 6f2e5973824b4d7a8b0d31faa622e6aa P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.002s
I20240503 02:00:15.084699 13985 tablet_bootstrap.cc:492] T 01b63d6a79314a86bd64f7618c86716d P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:15.084812 13985 ts_tablet_manager.cc:1367] T 01b63d6a79314a86bd64f7618c86716d P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:00:15.085175 13992 raft_consensus.cc:348] T 6f2e5973824b4d7a8b0d31faa622e6aa P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:15.085379 13992 raft_consensus.cc:374] T 6f2e5973824b4d7a8b0d31faa622e6aa P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:15.085472 13985 raft_consensus.cc:348] T 01b63d6a79314a86bd64f7618c86716d P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:15.085579 13985 raft_consensus.cc:374] T 01b63d6a79314a86bd64f7618c86716d P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:15.085608 13985 raft_consensus.cc:729] T 01b63d6a79314a86bd64f7618c86716d P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:15.085738 13985 consensus_queue.cc:260] T 01b63d6a79314a86bd64f7618c86716d P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:15.086014 13985 ts_tablet_manager.cc:1398] T 01b63d6a79314a86bd64f7618c86716d P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:15.086318 13992 raft_consensus.cc:729] T 6f2e5973824b4d7a8b0d31faa622e6aa P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:15.086510 13992 consensus_queue.cc:260] T 6f2e5973824b4d7a8b0d31faa622e6aa P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:15.086925 13992 ts_tablet_manager.cc:1398] T 6f2e5973824b4d7a8b0d31faa622e6aa P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
W20240503 02:00:15.203860 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:15.213878 6312 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:15.219329 9048 raft_consensus.cc:1261] T aaec090e44114ab38e49cc24814453fe P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:15.219503 8916 raft_consensus.cc:1261] T aaec090e44114ab38e49cc24814453fe P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:15.219957 13954 consensus_queue.cc:1035] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:15.220353 13997 consensus_queue.cc:1035] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:15.223644 14001 tablet.cc:1722] T aaec090e44114ab38e49cc24814453fe P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:15.224112 14001 tablet.cc:1620] T aaec090e44114ab38e49cc24814453fe P 99ead02915954b4fab23b033ad6d546b: MemRowSet was empty: no flush needed.
I20240503 02:00:15.224539 6309 catalog_manager.cc:5576] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211) does not have the latest schema for tablet aaec090e44114ab38e49cc24814453fe (table rebalancer_dynamic_table [id=66c21d25deab4cceb8c1e9ff09440b2b]). Expected version 1 got 0
I20240503 02:00:15.225278 6308 catalog_manager.cc:5576] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863) does not have the latest schema for tablet aaec090e44114ab38e49cc24814453fe (table rebalancer_dynamic_table [id=66c21d25deab4cceb8c1e9ff09440b2b]). Expected version 1 got 0
I20240503 02:00:15.225908 6312 catalog_manager.cc:5576] TS 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157) does not have the latest schema for tablet aaec090e44114ab38e49cc24814453fe (table rebalancer_dynamic_table [id=66c21d25deab4cceb8c1e9ff09440b2b]). Expected version 1 got 0
I20240503 02:00:15.226426 14000 tablet.cc:1722] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:15.227753 14000 tablet.cc:1620] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f: MemRowSet was empty: no flush needed.
I20240503 02:00:15.228147 14002 tablet.cc:1722] T aaec090e44114ab38e49cc24814453fe P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:15.228492 14002 tablet.cc:1620] T aaec090e44114ab38e49cc24814453fe P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
I20240503 02:00:15.230638 6308 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=66c21d25deab4cceb8c1e9ff09440b2b] alter complete (version 1)
I20240503 02:00:15.232579 6308 catalog_manager.cc:5576] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211) does not have the latest schema for tablet aaec090e44114ab38e49cc24814453fe (table rebalancer_dynamic_table [id=66c21d25deab4cceb8c1e9ff09440b2b]). Expected version 1 got 0
I20240503 02:00:15.238692 13991 raft_consensus.cc:482] T 01b63d6a79314a86bd64f7618c86716d P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:15.238822 13991 raft_consensus.cc:504] T 01b63d6a79314a86bd64f7618c86716d P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:15.239284 13991 leader_election.cc:288] T 01b63d6a79314a86bd64f7618c86716d P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:15.239675 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b63d6a79314a86bd64f7618c86716d" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:15.239677 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b63d6a79314a86bd64f7618c86716d" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 02:00:15.239799 8916 raft_consensus.cc:2454] T 01b63d6a79314a86bd64f7618c86716d P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:15.239812 6589 raft_consensus.cc:2454] T 01b63d6a79314a86bd64f7618c86716d P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:15.240075 6394 leader_election.cc:302] T 01b63d6a79314a86bd64f7618c86716d P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9942306855404cfd891adf227dd65eb2, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:15.240242 13991 raft_consensus.cc:2781] T 01b63d6a79314a86bd64f7618c86716d P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:15.240293 13991 raft_consensus.cc:482] T 01b63d6a79314a86bd64f7618c86716d P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:15.240322 13991 raft_consensus.cc:3037] T 01b63d6a79314a86bd64f7618c86716d P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:15.241400 13991 raft_consensus.cc:504] T 01b63d6a79314a86bd64f7618c86716d P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:15.241782 13991 leader_election.cc:288] T 01b63d6a79314a86bd64f7618c86716d P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Requested vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:15.242087 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b63d6a79314a86bd64f7618c86716d" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:15.242173 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b63d6a79314a86bd64f7618c86716d" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 02:00:15.242234 8916 raft_consensus.cc:3037] T 01b63d6a79314a86bd64f7618c86716d P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:15.242282 6589 raft_consensus.cc:3037] T 01b63d6a79314a86bd64f7618c86716d P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:15.243299 6589 raft_consensus.cc:2454] T 01b63d6a79314a86bd64f7618c86716d P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:15.243485 8916 raft_consensus.cc:2454] T 01b63d6a79314a86bd64f7618c86716d P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:15.243775 6393 leader_election.cc:302] T 01b63d6a79314a86bd64f7618c86716d P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:15.244030 13991 raft_consensus.cc:2781] T 01b63d6a79314a86bd64f7618c86716d P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:15.244125 13991 raft_consensus.cc:686] T 01b63d6a79314a86bd64f7618c86716d P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Becoming Leader. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Running, Role: LEADER
I20240503 02:00:15.244264 13991 consensus_queue.cc:237] T 01b63d6a79314a86bd64f7618c86716d P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:15.246294 6312 catalog_manager.cc:5484] T 01b63d6a79314a86bd64f7618c86716d P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: term changed from 0 to 1, leader changed from <none> to dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129). New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:15.248795 13956 raft_consensus.cc:482] T 6f2e5973824b4d7a8b0d31faa622e6aa P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:15.248911 13956 raft_consensus.cc:504] T 6f2e5973824b4d7a8b0d31faa622e6aa P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:15.249281 13956 leader_election.cc:288] T 6f2e5973824b4d7a8b0d31faa622e6aa P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 02:00:15.249752 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f2e5973824b4d7a8b0d31faa622e6aa" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 02:00:15.249895 6458 raft_consensus.cc:2454] T 6f2e5973824b4d7a8b0d31faa622e6aa P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:15.250128 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f2e5973824b4d7a8b0d31faa622e6aa" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:15.250241 9179 raft_consensus.cc:2454] T 6f2e5973824b4d7a8b0d31faa622e6aa P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:15.250299 8981 leader_election.cc:302] T 6f2e5973824b4d7a8b0d31faa622e6aa P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99ead02915954b4fab23b033ad6d546b, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:15.250653 13956 raft_consensus.cc:2781] T 6f2e5973824b4d7a8b0d31faa622e6aa P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:15.250720 13956 raft_consensus.cc:482] T 6f2e5973824b4d7a8b0d31faa622e6aa P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:15.250764 13956 raft_consensus.cc:3037] T 6f2e5973824b4d7a8b0d31faa622e6aa P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:15.251708 13956 raft_consensus.cc:504] T 6f2e5973824b4d7a8b0d31faa622e6aa P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:15.252075 13956 leader_election.cc:288] T 6f2e5973824b4d7a8b0d31faa622e6aa P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Requested vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 02:00:15.252501 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f2e5973824b4d7a8b0d31faa622e6aa" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:15.252501 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f2e5973824b4d7a8b0d31faa622e6aa" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 02:00:15.252624 9179 raft_consensus.cc:3037] T 6f2e5973824b4d7a8b0d31faa622e6aa P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:15.252625 6458 raft_consensus.cc:3037] T 6f2e5973824b4d7a8b0d31faa622e6aa P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:15.253624 6458 raft_consensus.cc:2454] T 6f2e5973824b4d7a8b0d31faa622e6aa P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:15.253623 9179 raft_consensus.cc:2454] T 6f2e5973824b4d7a8b0d31faa622e6aa P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:15.253957 8981 leader_election.cc:302] T 6f2e5973824b4d7a8b0d31faa622e6aa P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:15.254156 13956 raft_consensus.cc:2781] T 6f2e5973824b4d7a8b0d31faa622e6aa P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:15.254242 13956 raft_consensus.cc:686] T 6f2e5973824b4d7a8b0d31faa622e6aa P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Becoming Leader. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Running, Role: LEADER
I20240503 02:00:15.254390 13956 consensus_queue.cc:237] T 6f2e5973824b4d7a8b0d31faa622e6aa P 99ead02915954b4fab23b033ad6d546b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:15.256824 6308 catalog_manager.cc:5484] T 6f2e5973824b4d7a8b0d31faa622e6aa P 99ead02915954b4fab23b033ad6d546b reported cstate change: term changed from 0 to 1, leader changed from <none> to 99ead02915954b4fab23b033ad6d546b (127.0.99.134). New cstate: current_term: 1 leader_uuid: "99ead02915954b4fab23b033ad6d546b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:15.292024 13102 raft_consensus.cc:482] T 5ca282a13b714b63954c2812308904b4 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:15.292158 13102 raft_consensus.cc:504] T 5ca282a13b714b63954c2812308904b4 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:15.292697 13102 leader_election.cc:288] T 5ca282a13b714b63954c2812308904b4 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:15.293143 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ca282a13b714b63954c2812308904b4" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 02:00:15.293143 9039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ca282a13b714b63954c2812308904b4" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:15.293298 9039 raft_consensus.cc:2454] T 5ca282a13b714b63954c2812308904b4 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 0.
I20240503 02:00:15.293298 6589 raft_consensus.cc:2454] T 5ca282a13b714b63954c2812308904b4 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 0.
I20240503 02:00:15.293583 8852 leader_election.cc:302] T 5ca282a13b714b63954c2812308904b4 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:15.293757 13102 raft_consensus.cc:2781] T 5ca282a13b714b63954c2812308904b4 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:15.293817 13102 raft_consensus.cc:482] T 5ca282a13b714b63954c2812308904b4 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:15.293844 13102 raft_consensus.cc:3037] T 5ca282a13b714b63954c2812308904b4 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:15.294896 13102 raft_consensus.cc:504] T 5ca282a13b714b63954c2812308904b4 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:15.295289 13102 leader_election.cc:288] T 5ca282a13b714b63954c2812308904b4 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 election: Requested vote from peers 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:15.295629 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ca282a13b714b63954c2812308904b4" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 02:00:15.295751 6589 raft_consensus.cc:3037] T 5ca282a13b714b63954c2812308904b4 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:15.295725 9039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ca282a13b714b63954c2812308904b4" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:15.295876 9039 raft_consensus.cc:3037] T 5ca282a13b714b63954c2812308904b4 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:15.296705 6589 raft_consensus.cc:2454] T 5ca282a13b714b63954c2812308904b4 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 1.
I20240503 02:00:15.296875 9039 raft_consensus.cc:2454] T 5ca282a13b714b63954c2812308904b4 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 1.
I20240503 02:00:15.296995 8852 leader_election.cc:302] T 5ca282a13b714b63954c2812308904b4 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:15.297168 13102 raft_consensus.cc:2781] T 5ca282a13b714b63954c2812308904b4 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:15.297255 13102 raft_consensus.cc:686] T 5ca282a13b714b63954c2812308904b4 P 9942306855404cfd891adf227dd65eb2 [term 1 LEADER]: Becoming Leader. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Running, Role: LEADER
I20240503 02:00:15.297395 13102 consensus_queue.cc:237] T 5ca282a13b714b63954c2812308904b4 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:15.299533 6308 catalog_manager.cc:5484] T 5ca282a13b714b63954c2812308904b4 P 9942306855404cfd891adf227dd65eb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9942306855404cfd891adf227dd65eb2 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "9942306855404cfd891adf227dd65eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:15.336645 6312 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:15.342357 14000 tablet.cc:1722] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:15.342721 14000 tablet.cc:1620] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f: MemRowSet was empty: no flush needed.
I20240503 02:00:15.342720 14001 tablet.cc:1722] T aaec090e44114ab38e49cc24814453fe P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:15.343060 14001 tablet.cc:1620] T aaec090e44114ab38e49cc24814453fe P 99ead02915954b4fab23b033ad6d546b: MemRowSet was empty: no flush needed.
I20240503 02:00:15.343438 14002 tablet.cc:1722] T aaec090e44114ab38e49cc24814453fe P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:15.343731 14002 tablet.cc:1620] T aaec090e44114ab38e49cc24814453fe P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
I20240503 02:00:15.349637 6309 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=66c21d25deab4cceb8c1e9ff09440b2b] alter complete (version 2)
I20240503 02:00:15.452988 6308 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:15.458621 14000 tablet.cc:1722] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:15.459162 14001 tablet.cc:1722] T aaec090e44114ab38e49cc24814453fe P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:15.459447 14002 tablet.cc:1722] T aaec090e44114ab38e49cc24814453fe P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:15.464571 6309 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=66c21d25deab4cceb8c1e9ff09440b2b] alter complete (version 3)
I20240503 02:00:15.568696 6309 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:15.568892 6309 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:15.570972 6309 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet aaec090e44114ab38e49cc24814453fe
I20240503 02:00:15.571699 9028 tablet_service.cc:1514] Processing DeleteTablet for tablet aaec090e44114ab38e49cc24814453fe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:15 UTC) from {username='slave'} at 127.0.0.1:52584
I20240503 02:00:15.571699 8897 tablet_service.cc:1514] Processing DeleteTablet for tablet aaec090e44114ab38e49cc24814453fe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:15 UTC) from {username='slave'} at 127.0.0.1:40296
I20240503 02:00:15.571985 6569 tablet_service.cc:1514] Processing DeleteTablet for tablet aaec090e44114ab38e49cc24814453fe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:15 UTC) from {username='slave'} at 127.0.0.1:59846
I20240503 02:00:15.572063 14007 tablet_replica.cc:330] T aaec090e44114ab38e49cc24814453fe P 9942306855404cfd891adf227dd65eb2: stopping tablet replica
I20240503 02:00:15.572229 14007 raft_consensus.cc:2229] T aaec090e44114ab38e49cc24814453fe P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:15.572309 14009 tablet_replica.cc:330] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 02:00:15.572410 14007 raft_consensus.cc:2258] T aaec090e44114ab38e49cc24814453fe P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:15.572472 14009 raft_consensus.cc:2229] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:15.572774 14009 raft_consensus.cc:2258] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:15.573262 14007 ts_tablet_manager.cc:1875] T aaec090e44114ab38e49cc24814453fe P 9942306855404cfd891adf227dd65eb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:15.573349 14009 ts_tablet_manager.cc:1875] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:15.573419 14008 tablet_replica.cc:330] T aaec090e44114ab38e49cc24814453fe P 99ead02915954b4fab23b033ad6d546b: stopping tablet replica
I20240503 02:00:15.573575 14008 raft_consensus.cc:2229] T aaec090e44114ab38e49cc24814453fe P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:15.573700 14008 raft_consensus.cc:2258] T aaec090e44114ab38e49cc24814453fe P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:15.574219 14008 ts_tablet_manager.cc:1875] T aaec090e44114ab38e49cc24814453fe P 99ead02915954b4fab23b033ad6d546b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:15.576042 14007 ts_tablet_manager.cc:1888] T aaec090e44114ab38e49cc24814453fe P 9942306855404cfd891adf227dd65eb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:15.576051 14009 ts_tablet_manager.cc:1888] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:15.576138 14007 log.cc:1196] T aaec090e44114ab38e49cc24814453fe P 9942306855404cfd891adf227dd65eb2: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/aaec090e44114ab38e49cc24814453fe
I20240503 02:00:15.576138 14009 log.cc:1196] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/aaec090e44114ab38e49cc24814453fe
I20240503 02:00:15.576177 14008 ts_tablet_manager.cc:1888] T aaec090e44114ab38e49cc24814453fe P 99ead02915954b4fab23b033ad6d546b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:15.576448 14007 ts_tablet_manager.cc:1909] T aaec090e44114ab38e49cc24814453fe P 9942306855404cfd891adf227dd65eb2: Deleting consensus metadata
I20240503 02:00:15.576469 14008 log.cc:1196] T aaec090e44114ab38e49cc24814453fe P 99ead02915954b4fab23b033ad6d546b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/aaec090e44114ab38e49cc24814453fe
I20240503 02:00:15.576534 14009 ts_tablet_manager.cc:1909] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f: Deleting consensus metadata
I20240503 02:00:15.576958 14008 ts_tablet_manager.cc:1909] T aaec090e44114ab38e49cc24814453fe P 99ead02915954b4fab23b033ad6d546b: Deleting consensus metadata
I20240503 02:00:15.577279 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet aaec090e44114ab38e49cc24814453fe (table rebalancer_dynamic_table_new_name [id=66c21d25deab4cceb8c1e9ff09440b2b]) successfully deleted
I20240503 02:00:15.577392 6303 catalog_manager.cc:4830] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211): tablet aaec090e44114ab38e49cc24814453fe (table rebalancer_dynamic_table_new_name [id=66c21d25deab4cceb8c1e9ff09440b2b]) successfully deleted
I20240503 02:00:15.577641 6301 catalog_manager.cc:4830] TS 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157): tablet aaec090e44114ab38e49cc24814453fe (table rebalancer_dynamic_table_new_name [id=66c21d25deab4cceb8c1e9ff09440b2b]) successfully deleted
I20240503 02:00:15.629885 13829 consensus_queue.cc:1035] T 01b63d6a79314a86bd64f7618c86716d P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:15.633460 13991 consensus_queue.cc:1035] T 01b63d6a79314a86bd64f7618c86716d P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:15.672569 6312 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
W20240503 02:00:15.676455 8982 consensus_peers.cc:485] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:00:15.680270 6829 tablet_service.cc:1467] Processing CreateTablet for tablet 61ea3cc712c54205955c7e585d02025f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=73b1d9c7686d4af099d8797f3009a419]), partition=
I20240503 02:00:15.680258 9028 tablet_service.cc:1467] Processing CreateTablet for tablet 61ea3cc712c54205955c7e585d02025f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=73b1d9c7686d4af099d8797f3009a419]), partition=
I20240503 02:00:15.680258 6699 tablet_service.cc:1467] Processing CreateTablet for tablet 61ea3cc712c54205955c7e585d02025f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=73b1d9c7686d4af099d8797f3009a419]), partition=
I20240503 02:00:15.680579 6699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61ea3cc712c54205955c7e585d02025f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:15.680578 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61ea3cc712c54205955c7e585d02025f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:15.680578 6829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61ea3cc712c54205955c7e585d02025f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:15.683624 14014 tablet_bootstrap.cc:492] T 61ea3cc712c54205955c7e585d02025f P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:15.683671 14015 tablet_bootstrap.cc:492] T 61ea3cc712c54205955c7e585d02025f P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:15.684055 14016 tablet_bootstrap.cc:492] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:15.684828 14014 tablet_bootstrap.cc:654] T 61ea3cc712c54205955c7e585d02025f P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:15.685510 14015 tablet_bootstrap.cc:654] T 61ea3cc712c54205955c7e585d02025f P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:15.685660 14016 tablet_bootstrap.cc:654] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:15.687355 14016 tablet_bootstrap.cc:492] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:15.687465 14016 ts_tablet_manager.cc:1367] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:15.687940 14014 tablet_bootstrap.cc:492] T 61ea3cc712c54205955c7e585d02025f P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:15.687889 14016 raft_consensus.cc:348] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:15.687989 14016 raft_consensus.cc:374] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:15.688009 14016 raft_consensus.cc:729] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:15.688035 14014 ts_tablet_manager.cc:1367] T 61ea3cc712c54205955c7e585d02025f P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:15.688119 14016 consensus_queue.cc:260] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:15.688417 14016 ts_tablet_manager.cc:1398] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:15.688511 14014 raft_consensus.cc:348] T 61ea3cc712c54205955c7e585d02025f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:15.688617 14014 raft_consensus.cc:374] T 61ea3cc712c54205955c7e585d02025f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:15.688647 14014 raft_consensus.cc:729] T 61ea3cc712c54205955c7e585d02025f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:15.688764 14014 consensus_queue.cc:260] T 61ea3cc712c54205955c7e585d02025f P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:15.689038 14015 tablet_bootstrap.cc:492] T 61ea3cc712c54205955c7e585d02025f P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:15.689054 14014 ts_tablet_manager.cc:1398] T 61ea3cc712c54205955c7e585d02025f P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:15.689139 14015 ts_tablet_manager.cc:1367] T 61ea3cc712c54205955c7e585d02025f P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:00:15.689702 14015 raft_consensus.cc:348] T 61ea3cc712c54205955c7e585d02025f P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:15.689833 14015 raft_consensus.cc:374] T 61ea3cc712c54205955c7e585d02025f P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:15.689868 14015 raft_consensus.cc:729] T 61ea3cc712c54205955c7e585d02025f P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:15.689998 14015 consensus_queue.cc:260] T 61ea3cc712c54205955c7e585d02025f P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:15.690327 14015 ts_tablet_manager.cc:1398] T 61ea3cc712c54205955c7e585d02025f P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240503 02:00:15.698967 6785 consensus_peers.cc:485] T c3485701bf3b48f5ae2e1568afeed133 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:00:15.711455 13956 raft_consensus.cc:482] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:15.711589 13956 raft_consensus.cc:504] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:15.712083 13956 leader_election.cc:288] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:15.712553 6719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61ea3cc712c54205955c7e585d02025f" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 02:00:15.712612 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61ea3cc712c54205955c7e585d02025f" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 02:00:15.712714 6719 raft_consensus.cc:2454] T 61ea3cc712c54205955c7e585d02025f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:15.712726 6849 raft_consensus.cc:2454] T 61ea3cc712c54205955c7e585d02025f P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:15.712996 8984 leader_election.cc:302] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:15.713169 13956 raft_consensus.cc:2781] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:15.713229 13956 raft_consensus.cc:482] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:15.713255 13956 raft_consensus.cc:3037] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:15.714259 13956 raft_consensus.cc:504] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:15.714599 13956 leader_election.cc:288] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Requested vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:15.715015 6719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61ea3cc712c54205955c7e585d02025f" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 02:00:15.715016 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61ea3cc712c54205955c7e585d02025f" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 02:00:15.715147 6849 raft_consensus.cc:3037] T 61ea3cc712c54205955c7e585d02025f P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:15.715145 6719 raft_consensus.cc:3037] T 61ea3cc712c54205955c7e585d02025f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:15.715929 6849 raft_consensus.cc:2454] T 61ea3cc712c54205955c7e585d02025f P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:15.715929 6719 raft_consensus.cc:2454] T 61ea3cc712c54205955c7e585d02025f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:15.716328 8984 leader_election.cc:302] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99ead02915954b4fab23b033ad6d546b, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:15.716647 13956 raft_consensus.cc:2781] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:15.716738 13956 raft_consensus.cc:686] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Becoming Leader. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Running, Role: LEADER
I20240503 02:00:15.716884 13956 consensus_queue.cc:237] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:15.719502 6308 catalog_manager.cc:5484] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b reported cstate change: term changed from 0 to 1, leader changed from <none> to 99ead02915954b4fab23b033ad6d546b (127.0.99.134). New cstate: current_term: 1 leader_uuid: "99ead02915954b4fab23b033ad6d546b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:15.725975 13102 consensus_queue.cc:1035] T 5ca282a13b714b63954c2812308904b4 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:15.726390 13900 consensus_queue.cc:1035] T 6f2e5973824b4d7a8b0d31faa622e6aa P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:15.729998 13943 consensus_queue.cc:1035] T 6f2e5973824b4d7a8b0d31faa622e6aa P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:15.730335 13910 consensus_queue.cc:1035] T 5ca282a13b714b63954c2812308904b4 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:00:15.732780 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:15.803247 6312 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_extra_table_40"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:15.809813 6699 tablet_service.cc:1467] Processing CreateTablet for tablet 0f53daddf5fd44a3a9d8febcf0792ce8 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=4b0e62f32b2e4d32b6c0c4ef08932d08]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:15.810143 6699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f53daddf5fd44a3a9d8febcf0792ce8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:15.810724 9158 tablet_service.cc:1467] Processing CreateTablet for tablet 350fc92a3e034564800e8fbd8f6914a5 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=4b0e62f32b2e4d32b6c0c4ef08932d08]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:15.810900 6438 tablet_service.cc:1467] Processing CreateTablet for tablet 4966bb684477438e96d337e2320810f9 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=4b0e62f32b2e4d32b6c0c4ef08932d08]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:15.810982 9158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 350fc92a3e034564800e8fbd8f6914a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:15.811175 6438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4966bb684477438e96d337e2320810f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:15.812521 14014 tablet_bootstrap.cc:492] T 0f53daddf5fd44a3a9d8febcf0792ce8 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:15.812788 6436 tablet_service.cc:1467] Processing CreateTablet for tablet 0f53daddf5fd44a3a9d8febcf0792ce8 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=4b0e62f32b2e4d32b6c0c4ef08932d08]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:15.813058 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f53daddf5fd44a3a9d8febcf0792ce8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:15.813738 14014 tablet_bootstrap.cc:654] T 0f53daddf5fd44a3a9d8febcf0792ce8 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:15.814190 6569 tablet_service.cc:1467] Processing CreateTablet for tablet 4966bb684477438e96d337e2320810f9 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=4b0e62f32b2e4d32b6c0c4ef08932d08]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:15.814453 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4966bb684477438e96d337e2320810f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:15.814550 9028 tablet_service.cc:1467] Processing CreateTablet for tablet 0f53daddf5fd44a3a9d8febcf0792ce8 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=4b0e62f32b2e4d32b6c0c4ef08932d08]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:15.814600 14027 tablet_bootstrap.cc:492] T 350fc92a3e034564800e8fbd8f6914a5 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:15.814786 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f53daddf5fd44a3a9d8febcf0792ce8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:15.815768 14027 tablet_bootstrap.cc:654] T 350fc92a3e034564800e8fbd8f6914a5 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:15.816882 14016 tablet_bootstrap.cc:492] T 0f53daddf5fd44a3a9d8febcf0792ce8 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:15.816962 14030 tablet_bootstrap.cc:492] T 4966bb684477438e96d337e2320810f9 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:15.816978 9025 tablet_service.cc:1467] Processing CreateTablet for tablet 4966bb684477438e96d337e2320810f9 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=4b0e62f32b2e4d32b6c0c4ef08932d08]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:15.817195 9025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4966bb684477438e96d337e2320810f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:15.817801 14016 tablet_bootstrap.cc:654] T 0f53daddf5fd44a3a9d8febcf0792ce8 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:15.817991 6699 tablet_service.cc:1467] Processing CreateTablet for tablet 350fc92a3e034564800e8fbd8f6914a5 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=4b0e62f32b2e4d32b6c0c4ef08932d08]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:15.818289 6699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 350fc92a3e034564800e8fbd8f6914a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:15.818679 14030 tablet_bootstrap.cc:654] T 4966bb684477438e96d337e2320810f9 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:15.819365 6829 tablet_service.cc:1467] Processing CreateTablet for tablet 350fc92a3e034564800e8fbd8f6914a5 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=4b0e62f32b2e4d32b6c0c4ef08932d08]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:15.819675 6829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 350fc92a3e034564800e8fbd8f6914a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:15.820235 14014 tablet_bootstrap.cc:492] T 0f53daddf5fd44a3a9d8febcf0792ce8 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:15.820326 14014 ts_tablet_manager.cc:1367] T 0f53daddf5fd44a3a9d8febcf0792ce8 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 02:00:15.820936 14014 raft_consensus.cc:348] T 0f53daddf5fd44a3a9d8febcf0792ce8 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:15.821056 14014 raft_consensus.cc:374] T 0f53daddf5fd44a3a9d8febcf0792ce8 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:15.821101 14014 raft_consensus.cc:729] T 0f53daddf5fd44a3a9d8febcf0792ce8 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:15.821316 14014 consensus_queue.cc:260] T 0f53daddf5fd44a3a9d8febcf0792ce8 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:15.821372 14026 tablet_bootstrap.cc:492] T 4966bb684477438e96d337e2320810f9 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:15.821636 14014 ts_tablet_manager.cc:1398] T 0f53daddf5fd44a3a9d8febcf0792ce8 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:15.821808 14014 tablet_bootstrap.cc:492] T 350fc92a3e034564800e8fbd8f6914a5 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:15.822012 14015 tablet_bootstrap.cc:492] T 350fc92a3e034564800e8fbd8f6914a5 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:15.822599 14026 tablet_bootstrap.cc:654] T 4966bb684477438e96d337e2320810f9 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:15.822726 14014 tablet_bootstrap.cc:654] T 350fc92a3e034564800e8fbd8f6914a5 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:15.822947 14015 tablet_bootstrap.cc:654] T 350fc92a3e034564800e8fbd8f6914a5 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:15.824085 14027 tablet_bootstrap.cc:492] T 350fc92a3e034564800e8fbd8f6914a5 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:15.824175 14027 ts_tablet_manager.cc:1367] T 350fc92a3e034564800e8fbd8f6914a5 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 02:00:15.824363 14016 tablet_bootstrap.cc:492] T 0f53daddf5fd44a3a9d8febcf0792ce8 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:15.824477 14016 ts_tablet_manager.cc:1367] T 0f53daddf5fd44a3a9d8febcf0792ce8 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240503 02:00:15.824723 14027 raft_consensus.cc:348] T 350fc92a3e034564800e8fbd8f6914a5 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:15.824829 14027 raft_consensus.cc:374] T 350fc92a3e034564800e8fbd8f6914a5 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:15.824863 14027 raft_consensus.cc:729] T 350fc92a3e034564800e8fbd8f6914a5 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:15.824928 14016 raft_consensus.cc:348] T 0f53daddf5fd44a3a9d8febcf0792ce8 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:15.825021 14016 raft_consensus.cc:374] T 0f53daddf5fd44a3a9d8febcf0792ce8 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:15.824982 14027 consensus_queue.cc:260] T 350fc92a3e034564800e8fbd8f6914a5 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:15.825049 14016 raft_consensus.cc:729] T 0f53daddf5fd44a3a9d8febcf0792ce8 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:15.825155 14016 consensus_queue.cc:260] T 0f53daddf5fd44a3a9d8febcf0792ce8 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:15.825274 14027 ts_tablet_manager.cc:1398] T 350fc92a3e034564800e8fbd8f6914a5 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:15.825423 14016 ts_tablet_manager.cc:1398] T 0f53daddf5fd44a3a9d8febcf0792ce8 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:15.825588 14016 tablet_bootstrap.cc:492] T 4966bb684477438e96d337e2320810f9 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:15.826531 14016 tablet_bootstrap.cc:654] T 4966bb684477438e96d337e2320810f9 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:15.827327 14014 tablet_bootstrap.cc:492] T 350fc92a3e034564800e8fbd8f6914a5 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:15.827430 14014 ts_tablet_manager.cc:1367] T 350fc92a3e034564800e8fbd8f6914a5 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:00:15.825867 14030 tablet_bootstrap.cc:492] T 4966bb684477438e96d337e2320810f9 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:15.827651 14030 ts_tablet_manager.cc:1367] T 4966bb684477438e96d337e2320810f9 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 02:00:15.828298 14014 raft_consensus.cc:348] T 350fc92a3e034564800e8fbd8f6914a5 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:15.828303 14030 raft_consensus.cc:348] T 4966bb684477438e96d337e2320810f9 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:15.828449 14030 raft_consensus.cc:374] T 4966bb684477438e96d337e2320810f9 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:15.828498 14030 raft_consensus.cc:729] T 4966bb684477438e96d337e2320810f9 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:15.828481 14014 raft_consensus.cc:374] T 350fc92a3e034564800e8fbd8f6914a5 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:15.828573 14014 raft_consensus.cc:729] T 350fc92a3e034564800e8fbd8f6914a5 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:15.828629 14030 consensus_queue.cc:260] T 4966bb684477438e96d337e2320810f9 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:15.828747 14014 consensus_queue.cc:260] T 350fc92a3e034564800e8fbd8f6914a5 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:15.829591 14030 ts_tablet_manager.cc:1398] T 4966bb684477438e96d337e2320810f9 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:15.830127 14014 ts_tablet_manager.cc:1398] T 350fc92a3e034564800e8fbd8f6914a5 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:00:15.831409 14026 tablet_bootstrap.cc:492] T 4966bb684477438e96d337e2320810f9 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:15.831581 14026 ts_tablet_manager.cc:1367] T 4966bb684477438e96d337e2320810f9 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 02:00:15.832198 14015 tablet_bootstrap.cc:492] T 350fc92a3e034564800e8fbd8f6914a5 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:15.832298 14015 ts_tablet_manager.cc:1367] T 350fc92a3e034564800e8fbd8f6914a5 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 02:00:15.832473 14026 raft_consensus.cc:348] T 4966bb684477438e96d337e2320810f9 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:15.832579 14026 raft_consensus.cc:374] T 4966bb684477438e96d337e2320810f9 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:15.832609 14026 raft_consensus.cc:729] T 4966bb684477438e96d337e2320810f9 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:15.832769 14026 consensus_queue.cc:260] T 4966bb684477438e96d337e2320810f9 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:15.832808 14015 raft_consensus.cc:348] T 350fc92a3e034564800e8fbd8f6914a5 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:15.832895 14015 raft_consensus.cc:374] T 350fc92a3e034564800e8fbd8f6914a5 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:15.832944 14015 raft_consensus.cc:729] T 350fc92a3e034564800e8fbd8f6914a5 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:15.833063 14026 ts_tablet_manager.cc:1398] T 4966bb684477438e96d337e2320810f9 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:15.833047 14015 consensus_queue.cc:260] T 350fc92a3e034564800e8fbd8f6914a5 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:15.833329 14015 ts_tablet_manager.cc:1398] T 350fc92a3e034564800e8fbd8f6914a5 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:15.833959 14026 tablet_bootstrap.cc:492] T 0f53daddf5fd44a3a9d8febcf0792ce8 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:15.836145 14026 tablet_bootstrap.cc:654] T 0f53daddf5fd44a3a9d8febcf0792ce8 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:15.838500 6308 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:15.838979 14026 tablet_bootstrap.cc:492] T 0f53daddf5fd44a3a9d8febcf0792ce8 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:15.839103 14026 ts_tablet_manager.cc:1367] T 0f53daddf5fd44a3a9d8febcf0792ce8 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:00:15.839623 14026 raft_consensus.cc:348] T 0f53daddf5fd44a3a9d8febcf0792ce8 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:15.839960 14026 raft_consensus.cc:374] T 0f53daddf5fd44a3a9d8febcf0792ce8 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:15.840035 14026 raft_consensus.cc:729] T 0f53daddf5fd44a3a9d8febcf0792ce8 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:15.840193 14026 consensus_queue.cc:260] T 0f53daddf5fd44a3a9d8febcf0792ce8 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:15.840528 14026 ts_tablet_manager.cc:1398] T 0f53daddf5fd44a3a9d8febcf0792ce8 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:15.843935 6719 raft_consensus.cc:1261] T 61ea3cc712c54205955c7e585d02025f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer 99ead02915954b4fab23b033ad6d546b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:15.844754 6849 raft_consensus.cc:1261] T 61ea3cc712c54205955c7e585d02025f P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer 99ead02915954b4fab23b033ad6d546b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:15.845125 13943 consensus_queue.cc:1035] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:15.845404 13956 consensus_queue.cc:1035] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:15.850194 6309 catalog_manager.cc:5576] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719) does not have the latest schema for tablet 61ea3cc712c54205955c7e585d02025f (table rebalancer_dynamic_table [id=73b1d9c7686d4af099d8797f3009a419]). Expected version 1 got 0
I20240503 02:00:15.850281 14001 tablet.cc:1722] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:15.850651 14001 tablet.cc:1620] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b: MemRowSet was empty: no flush needed.
I20240503 02:00:15.850764 14042 tablet.cc:1722] T 61ea3cc712c54205955c7e585d02025f P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:15.851094 14042 tablet.cc:1620] T 61ea3cc712c54205955c7e585d02025f P 492448570cc94ed1879feb2ec3b9051d: MemRowSet was empty: no flush needed.
I20240503 02:00:15.854935 6313 catalog_manager.cc:5576] TS c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655) does not have the latest schema for tablet 61ea3cc712c54205955c7e585d02025f (table rebalancer_dynamic_table [id=73b1d9c7686d4af099d8797f3009a419]). Expected version 1 got 0
I20240503 02:00:15.855268 6308 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=73b1d9c7686d4af099d8797f3009a419] alter complete (version 1)
I20240503 02:00:15.855544 14043 tablet.cc:1722] T 61ea3cc712c54205955c7e585d02025f P c2ac2eb7cc264047b9ab32fe3a4fc025: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:15.855876 14043 tablet.cc:1620] T 61ea3cc712c54205955c7e585d02025f P c2ac2eb7cc264047b9ab32fe3a4fc025: MemRowSet was empty: no flush needed.
I20240503 02:00:15.856918 14016 tablet_bootstrap.cc:492] T 4966bb684477438e96d337e2320810f9 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:15.857033 14016 ts_tablet_manager.cc:1367] T 4966bb684477438e96d337e2320810f9 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.031s user 0.000s sys 0.002s
I20240503 02:00:15.857538 14016 raft_consensus.cc:348] T 4966bb684477438e96d337e2320810f9 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:15.857630 14016 raft_consensus.cc:374] T 4966bb684477438e96d337e2320810f9 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:15.857650 14016 raft_consensus.cc:729] T 4966bb684477438e96d337e2320810f9 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:15.857757 14016 consensus_queue.cc:260] T 4966bb684477438e96d337e2320810f9 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:15.858070 14016 ts_tablet_manager.cc:1398] T 4966bb684477438e96d337e2320810f9 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:15.879215 13900 raft_consensus.cc:482] T 0f53daddf5fd44a3a9d8febcf0792ce8 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:15.879393 13900 raft_consensus.cc:504] T 0f53daddf5fd44a3a9d8febcf0792ce8 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:15.880091 13900 leader_election.cc:288] T 0f53daddf5fd44a3a9d8febcf0792ce8 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 02:00:15.880550 6719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f53daddf5fd44a3a9d8febcf0792ce8" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 02:00:15.880532 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f53daddf5fd44a3a9d8febcf0792ce8" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 02:00:15.880699 6458 raft_consensus.cc:2454] T 0f53daddf5fd44a3a9d8febcf0792ce8 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:15.880702 6719 raft_consensus.cc:2454] T 0f53daddf5fd44a3a9d8febcf0792ce8 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:15.881039 8981 leader_election.cc:302] T 0f53daddf5fd44a3a9d8febcf0792ce8 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99ead02915954b4fab23b033ad6d546b, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:15.881242 13900 raft_consensus.cc:2781] T 0f53daddf5fd44a3a9d8febcf0792ce8 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:15.881314 13900 raft_consensus.cc:482] T 0f53daddf5fd44a3a9d8febcf0792ce8 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:15.881369 13900 raft_consensus.cc:3037] T 0f53daddf5fd44a3a9d8febcf0792ce8 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:15.882229 13900 raft_consensus.cc:504] T 0f53daddf5fd44a3a9d8febcf0792ce8 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:15.882715 13900 leader_election.cc:288] T 0f53daddf5fd44a3a9d8febcf0792ce8 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Requested vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 02:00:15.883172 6719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f53daddf5fd44a3a9d8febcf0792ce8" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 02:00:15.883172 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f53daddf5fd44a3a9d8febcf0792ce8" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 02:00:15.883313 6719 raft_consensus.cc:3037] T 0f53daddf5fd44a3a9d8febcf0792ce8 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:15.883313 6458 raft_consensus.cc:3037] T 0f53daddf5fd44a3a9d8febcf0792ce8 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:15.883318 13829 raft_consensus.cc:482] T 4966bb684477438e96d337e2320810f9 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:15.883493 13829 raft_consensus.cc:504] T 4966bb684477438e96d337e2320810f9 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:15.883957 13829 leader_election.cc:288] T 4966bb684477438e96d337e2320810f9 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:15.884346 6582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4966bb684477438e96d337e2320810f9" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 02:00:15.884346 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4966bb684477438e96d337e2320810f9" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:15.884457 6458 raft_consensus.cc:2454] T 0f53daddf5fd44a3a9d8febcf0792ce8 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:15.884500 9048 raft_consensus.cc:2454] T 4966bb684477438e96d337e2320810f9 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:15.884500 6582 raft_consensus.cc:2454] T 4966bb684477438e96d337e2320810f9 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:15.884649 6719 raft_consensus.cc:2454] T 0f53daddf5fd44a3a9d8febcf0792ce8 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:15.884785 8981 leader_election.cc:302] T 0f53daddf5fd44a3a9d8febcf0792ce8 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99ead02915954b4fab23b033ad6d546b, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:15.884887 6393 leader_election.cc:302] T 4966bb684477438e96d337e2320810f9 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:15.885075 13829 raft_consensus.cc:2781] T 4966bb684477438e96d337e2320810f9 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:15.885139 13829 raft_consensus.cc:482] T 4966bb684477438e96d337e2320810f9 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:15.885182 13829 raft_consensus.cc:3037] T 4966bb684477438e96d337e2320810f9 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:15.885399 13900 raft_consensus.cc:2781] T 0f53daddf5fd44a3a9d8febcf0792ce8 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:15.885481 13900 raft_consensus.cc:686] T 0f53daddf5fd44a3a9d8febcf0792ce8 P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Becoming Leader. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Running, Role: LEADER
I20240503 02:00:15.885604 13900 consensus_queue.cc:237] T 0f53daddf5fd44a3a9d8febcf0792ce8 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:15.886224 13829 raft_consensus.cc:504] T 4966bb684477438e96d337e2320810f9 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:15.886531 13829 leader_election.cc:288] T 4966bb684477438e96d337e2320810f9 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Requested vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:15.886878 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4966bb684477438e96d337e2320810f9" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:15.886973 9048 raft_consensus.cc:3037] T 4966bb684477438e96d337e2320810f9 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:15.887266 6582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4966bb684477438e96d337e2320810f9" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 02:00:15.887385 6582 raft_consensus.cc:3037] T 4966bb684477438e96d337e2320810f9 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:15.887883 9048 raft_consensus.cc:2454] T 4966bb684477438e96d337e2320810f9 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:15.887988 6308 catalog_manager.cc:5484] T 0f53daddf5fd44a3a9d8febcf0792ce8 P 99ead02915954b4fab23b033ad6d546b reported cstate change: term changed from 0 to 1, leader changed from <none> to 99ead02915954b4fab23b033ad6d546b (127.0.99.134). New cstate: current_term: 1 leader_uuid: "99ead02915954b4fab23b033ad6d546b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:15.888336 6582 raft_consensus.cc:2454] T 4966bb684477438e96d337e2320810f9 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:15.888675 6393 leader_election.cc:302] T 4966bb684477438e96d337e2320810f9 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:15.888953 13829 raft_consensus.cc:2781] T 4966bb684477438e96d337e2320810f9 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:15.889047 13829 raft_consensus.cc:686] T 4966bb684477438e96d337e2320810f9 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Becoming Leader. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Running, Role: LEADER
I20240503 02:00:15.889191 13829 consensus_queue.cc:237] T 4966bb684477438e96d337e2320810f9 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:15.891600 6308 catalog_manager.cc:5484] T 4966bb684477438e96d337e2320810f9 P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: term changed from 0 to 1, leader changed from <none> to dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129). New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:15.907866 13975 raft_consensus.cc:482] T 350fc92a3e034564800e8fbd8f6914a5 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:15.908020 13975 raft_consensus.cc:504] T 350fc92a3e034564800e8fbd8f6914a5 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:15.908634 13975 leader_election.cc:288] T 350fc92a3e034564800e8fbd8f6914a5 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:15.909093 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "350fc92a3e034564800e8fbd8f6914a5" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:15.909132 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "350fc92a3e034564800e8fbd8f6914a5" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 02:00:15.909245 9178 raft_consensus.cc:2454] T 350fc92a3e034564800e8fbd8f6914a5 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:15.909263 6849 raft_consensus.cc:2454] T 350fc92a3e034564800e8fbd8f6914a5 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:15.909526 6656 leader_election.cc:302] T 350fc92a3e034564800e8fbd8f6914a5 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:15.909718 13975 raft_consensus.cc:2781] T 350fc92a3e034564800e8fbd8f6914a5 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:15.909776 13975 raft_consensus.cc:482] T 350fc92a3e034564800e8fbd8f6914a5 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:15.909812 13975 raft_consensus.cc:3037] T 350fc92a3e034564800e8fbd8f6914a5 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:15.910676 13975 raft_consensus.cc:504] T 350fc92a3e034564800e8fbd8f6914a5 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:15.911024 13975 leader_election.cc:288] T 350fc92a3e034564800e8fbd8f6914a5 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Requested vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:15.911419 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "350fc92a3e034564800e8fbd8f6914a5" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:15.911419 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "350fc92a3e034564800e8fbd8f6914a5" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 02:00:15.911551 6849 raft_consensus.cc:3037] T 350fc92a3e034564800e8fbd8f6914a5 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:15.911551 9178 raft_consensus.cc:3037] T 350fc92a3e034564800e8fbd8f6914a5 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:15.912601 9178 raft_consensus.cc:2454] T 350fc92a3e034564800e8fbd8f6914a5 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:15.912601 6849 raft_consensus.cc:2454] T 350fc92a3e034564800e8fbd8f6914a5 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:15.912891 6653 leader_election.cc:302] T 350fc92a3e034564800e8fbd8f6914a5 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:15.913081 13975 raft_consensus.cc:2781] T 350fc92a3e034564800e8fbd8f6914a5 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:15.913195 13975 raft_consensus.cc:686] T 350fc92a3e034564800e8fbd8f6914a5 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Becoming Leader. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Running, Role: LEADER
I20240503 02:00:15.913359 13975 consensus_queue.cc:237] T 350fc92a3e034564800e8fbd8f6914a5 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:15.915706 6308 catalog_manager.cc:5484] T 350fc92a3e034564800e8fbd8f6914a5 P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:15.959856 6308 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:15.965605 14001 tablet.cc:1722] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:15.965946 14001 tablet.cc:1620] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b: MemRowSet was empty: no flush needed.
I20240503 02:00:15.966488 14042 tablet.cc:1722] T 61ea3cc712c54205955c7e585d02025f P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:15.966832 14042 tablet.cc:1620] T 61ea3cc712c54205955c7e585d02025f P 492448570cc94ed1879feb2ec3b9051d: MemRowSet was empty: no flush needed.
W20240503 02:00:15.969913 6654 consensus_peers.cc:485] T ba17d4d3e0754b2798eb95484d37229a P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:00:15.970688 14043 tablet.cc:1722] T 61ea3cc712c54205955c7e585d02025f P c2ac2eb7cc264047b9ab32fe3a4fc025: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:15.970999 14043 tablet.cc:1620] T 61ea3cc712c54205955c7e585d02025f P c2ac2eb7cc264047b9ab32fe3a4fc025: MemRowSet was empty: no flush needed.
I20240503 02:00:15.973464 6309 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=73b1d9c7686d4af099d8797f3009a419] alter complete (version 2)
W20240503 02:00:16.014598 6523 consensus_peers.cc:485] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:16.028046 6654 consensus_peers.cc:485] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:00:16.077383 6309 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
W20240503 02:00:16.081918 6523 consensus_peers.cc:485] T d0d5dd42db3e4883af7fd16d8813b550 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:16.081991 8851 consensus_peers.cc:485] T 067f54011a37432d99157884e70fc601 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:00:16.083861 14001 tablet.cc:1722] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
W20240503 02:00:16.084163 6785 consensus_peers.cc:485] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:00:16.084722 14043 tablet.cc:1722] T 61ea3cc712c54205955c7e585d02025f P c2ac2eb7cc264047b9ab32fe3a4fc025: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:16.085592 14042 tablet.cc:1722] T 61ea3cc712c54205955c7e585d02025f P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:16.089983 6309 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=73b1d9c7686d4af099d8797f3009a419] alter complete (version 3)
W20240503 02:00:16.115957 9113 consensus_peers.cc:485] T f44b59277c3b47adbd4dd6aff9350af1 P 2a01ace80baf4057b6bf1995cf88468b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:00:16.192467 6313 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:16.192616 6313 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:16.194583 6313 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet 61ea3cc712c54205955c7e585d02025f
I20240503 02:00:16.195214 6699 tablet_service.cc:1514] Processing DeleteTablet for tablet 61ea3cc712c54205955c7e585d02025f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:16 UTC) from {username='slave'} at 127.0.0.1:40336
I20240503 02:00:16.195272 6829 tablet_service.cc:1514] Processing DeleteTablet for tablet 61ea3cc712c54205955c7e585d02025f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:16 UTC) from {username='slave'} at 127.0.0.1:52812
I20240503 02:00:16.195437 9025 tablet_service.cc:1514] Processing DeleteTablet for tablet 61ea3cc712c54205955c7e585d02025f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:16 UTC) from {username='slave'} at 127.0.0.1:52584
I20240503 02:00:16.195562 14049 tablet_replica.cc:330] T 61ea3cc712c54205955c7e585d02025f P c2ac2eb7cc264047b9ab32fe3a4fc025: stopping tablet replica
I20240503 02:00:16.195583 14050 tablet_replica.cc:330] T 61ea3cc712c54205955c7e585d02025f P 492448570cc94ed1879feb2ec3b9051d: stopping tablet replica
I20240503 02:00:16.195716 14050 raft_consensus.cc:2229] T 61ea3cc712c54205955c7e585d02025f P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:16.195721 14049 raft_consensus.cc:2229] T 61ea3cc712c54205955c7e585d02025f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:16.195740 14051 tablet_replica.cc:330] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b: stopping tablet replica
I20240503 02:00:16.195845 14049 raft_consensus.cc:2258] T 61ea3cc712c54205955c7e585d02025f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:16.195868 14050 raft_consensus.cc:2258] T 61ea3cc712c54205955c7e585d02025f P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:16.195863 14051 raft_consensus.cc:2229] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:16.196115 14051 raft_consensus.cc:2258] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:16.196612 14049 ts_tablet_manager.cc:1875] T 61ea3cc712c54205955c7e585d02025f P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:16.196691 14051 ts_tablet_manager.cc:1875] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:16.196691 14050 ts_tablet_manager.cc:1875] T 61ea3cc712c54205955c7e585d02025f P 492448570cc94ed1879feb2ec3b9051d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:16.198971 14051 ts_tablet_manager.cc:1888] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:16.198977 14050 ts_tablet_manager.cc:1888] T 61ea3cc712c54205955c7e585d02025f P 492448570cc94ed1879feb2ec3b9051d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:16.199065 14050 log.cc:1196] T 61ea3cc712c54205955c7e585d02025f P 492448570cc94ed1879feb2ec3b9051d: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/61ea3cc712c54205955c7e585d02025f
I20240503 02:00:16.199090 14051 log.cc:1196] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/61ea3cc712c54205955c7e585d02025f
I20240503 02:00:16.199234 14049 ts_tablet_manager.cc:1888] T 61ea3cc712c54205955c7e585d02025f P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:16.199339 14049 log.cc:1196] T 61ea3cc712c54205955c7e585d02025f P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/61ea3cc712c54205955c7e585d02025f
I20240503 02:00:16.199355 14051 ts_tablet_manager.cc:1909] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b: Deleting consensus metadata
I20240503 02:00:16.199373 14050 ts_tablet_manager.cc:1909] T 61ea3cc712c54205955c7e585d02025f P 492448570cc94ed1879feb2ec3b9051d: Deleting consensus metadata
I20240503 02:00:16.199663 14049 ts_tablet_manager.cc:1909] T 61ea3cc712c54205955c7e585d02025f P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting consensus metadata
I20240503 02:00:16.200430 6303 catalog_manager.cc:4830] TS c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655): tablet 61ea3cc712c54205955c7e585d02025f (table rebalancer_dynamic_table_new_name [id=73b1d9c7686d4af099d8797f3009a419]) successfully deleted
I20240503 02:00:16.200553 6301 catalog_manager.cc:4830] TS 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157): tablet 61ea3cc712c54205955c7e585d02025f (table rebalancer_dynamic_table_new_name [id=73b1d9c7686d4af099d8797f3009a419]) successfully deleted
I20240503 02:00:16.200835 6303 catalog_manager.cc:4830] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719): tablet 61ea3cc712c54205955c7e585d02025f (table rebalancer_dynamic_table_new_name [id=73b1d9c7686d4af099d8797f3009a419]) successfully deleted
W20240503 02:00:16.222460 6785 consensus_peers.cc:485] T 46ad540deb624a25b97d3b967fe1f6f4 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:16.224550 8851 consensus_peers.cc:485] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:16.259510 6523 consensus_peers.cc:485] T 925612b660ec471da5b3c548802ede6e P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:00:16.279611 14052 consensus_queue.cc:1035] T 4966bb684477438e96d337e2320810f9 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:16.283103 13991 consensus_queue.cc:1035] T 4966bb684477438e96d337e2320810f9 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:00:16.289844 6785 consensus_peers.cc:485] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:00:16.296200 6309 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:00:16.301681 9025 tablet_service.cc:1467] Processing CreateTablet for tablet 90a0a91005714af39c8f0e28b4446e01 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7d9cea0b1a7a4cddabd18ed44a365181]), partition=
I20240503 02:00:16.301697 8897 tablet_service.cc:1467] Processing CreateTablet for tablet 90a0a91005714af39c8f0e28b4446e01 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7d9cea0b1a7a4cddabd18ed44a365181]), partition=
I20240503 02:00:16.301967 8897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90a0a91005714af39c8f0e28b4446e01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:16.301972 9025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90a0a91005714af39c8f0e28b4446e01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:16.304126 14016 tablet_bootstrap.cc:492] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:16.304729 9158 tablet_service.cc:1467] Processing CreateTablet for tablet 90a0a91005714af39c8f0e28b4446e01 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7d9cea0b1a7a4cddabd18ed44a365181]), partition=
I20240503 02:00:16.304884 14059 tablet_bootstrap.cc:492] T 90a0a91005714af39c8f0e28b4446e01 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:16.304934 9158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90a0a91005714af39c8f0e28b4446e01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:16.305155 14016 tablet_bootstrap.cc:654] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:16.305774 14059 tablet_bootstrap.cc:654] T 90a0a91005714af39c8f0e28b4446e01 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:16.306864 14059 tablet_bootstrap.cc:492] T 90a0a91005714af39c8f0e28b4446e01 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:16.306960 14059 ts_tablet_manager.cc:1367] T 90a0a91005714af39c8f0e28b4446e01 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:16.307030 14027 tablet_bootstrap.cc:492] T 90a0a91005714af39c8f0e28b4446e01 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:16.307379 14016 tablet_bootstrap.cc:492] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:16.307344 14059 raft_consensus.cc:348] T 90a0a91005714af39c8f0e28b4446e01 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:16.307431 14059 raft_consensus.cc:374] T 90a0a91005714af39c8f0e28b4446e01 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:16.307451 14059 raft_consensus.cc:729] T 90a0a91005714af39c8f0e28b4446e01 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:16.307466 14016 ts_tablet_manager.cc:1367] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:16.307535 14059 consensus_queue.cc:260] T 90a0a91005714af39c8f0e28b4446e01 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:16.307787 14059 ts_tablet_manager.cc:1398] T 90a0a91005714af39c8f0e28b4446e01 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:16.307950 14027 tablet_bootstrap.cc:654] T 90a0a91005714af39c8f0e28b4446e01 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:16.307910 14016 raft_consensus.cc:348] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:16.308013 14016 raft_consensus.cc:374] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:16.308041 14016 raft_consensus.cc:729] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:16.308177 14016 consensus_queue.cc:260] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:16.308487 14016 ts_tablet_manager.cc:1398] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:16.321663 13900 consensus_queue.cc:1035] T 0f53daddf5fd44a3a9d8febcf0792ce8 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:16.325084 13943 consensus_queue.cc:1035] T 0f53daddf5fd44a3a9d8febcf0792ce8 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:16.340945 14027 tablet_bootstrap.cc:492] T 90a0a91005714af39c8f0e28b4446e01 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:16.341091 14027 ts_tablet_manager.cc:1367] T 90a0a91005714af39c8f0e28b4446e01 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.034s user 0.002s sys 0.000s
I20240503 02:00:16.341665 14027 raft_consensus.cc:348] T 90a0a91005714af39c8f0e28b4446e01 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:16.341787 14027 raft_consensus.cc:374] T 90a0a91005714af39c8f0e28b4446e01 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:16.341817 14027 raft_consensus.cc:729] T 90a0a91005714af39c8f0e28b4446e01 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:16.341949 14027 consensus_queue.cc:260] T 90a0a91005714af39c8f0e28b4446e01 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:16.342260 14027 ts_tablet_manager.cc:1398] T 90a0a91005714af39c8f0e28b4446e01 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20240503 02:00:16.353497 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
W20240503 02:00:16.361248 6785 consensus_peers.cc:485] T 8246488dec204bf6b09e868d7951ac5a P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:00:16.375476 13975 consensus_queue.cc:1035] T 350fc92a3e034564800e8fbd8f6914a5 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:16.379936 14066 consensus_queue.cc:1035] T 350fc92a3e034564800e8fbd8f6914a5 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:16.404758 13943 raft_consensus.cc:482] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:16.404911 13943 raft_consensus.cc:504] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:16.405419 13943 leader_election.cc:288] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:16.405875 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90a0a91005714af39c8f0e28b4446e01" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:16.405934 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90a0a91005714af39c8f0e28b4446e01" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:16.406049 8916 raft_consensus.cc:2454] T 90a0a91005714af39c8f0e28b4446e01 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:16.406085 9178 raft_consensus.cc:2454] T 90a0a91005714af39c8f0e28b4446e01 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:16.406494 8981 leader_election.cc:302] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:16.406734 13943 raft_consensus.cc:2781] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:16.406781 13943 raft_consensus.cc:482] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:16.406813 13943 raft_consensus.cc:3037] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:16.407789 13943 raft_consensus.cc:504] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:16.408157 13943 leader_election.cc:288] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Requested vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:16.408597 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90a0a91005714af39c8f0e28b4446e01" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:16.408764 8916 raft_consensus.cc:3037] T 90a0a91005714af39c8f0e28b4446e01 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:16.409608 8916 raft_consensus.cc:2454] T 90a0a91005714af39c8f0e28b4446e01 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:16.409873 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90a0a91005714af39c8f0e28b4446e01" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:16.409914 8984 leader_election.cc:302] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9942306855404cfd891adf227dd65eb2, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:16.410008 9178 raft_consensus.cc:3037] T 90a0a91005714af39c8f0e28b4446e01 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:16.410080 13943 raft_consensus.cc:2781] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:16.410163 13943 raft_consensus.cc:686] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Becoming Leader. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Running, Role: LEADER
I20240503 02:00:16.410312 13943 consensus_queue.cc:237] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:16.410985 9178 raft_consensus.cc:2454] T 90a0a91005714af39c8f0e28b4446e01 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:16.413386 6312 catalog_manager.cc:5484] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b reported cstate change: term changed from 0 to 1, leader changed from <none> to 99ead02915954b4fab23b033ad6d546b (127.0.99.134). New cstate: current_term: 1 leader_uuid: "99ead02915954b4fab23b033ad6d546b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } }
W20240503 02:00:16.432929 6785 consensus_peers.cc:485] T d897c21e630d4770832bb10f43778e82 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:00:16.438104 6312 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_extra_table_42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:16.444370 6829 tablet_service.cc:1467] Processing CreateTablet for tablet 2c14900a27ce4c7eb9c0ad0f0c34488d (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=eb0be093febd40a99ecffa1fbe86b63e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:16.444759 6829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c14900a27ce4c7eb9c0ad0f0c34488d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:16.444957 6436 tablet_service.cc:1467] Processing CreateTablet for tablet 2c14900a27ce4c7eb9c0ad0f0c34488d (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=eb0be093febd40a99ecffa1fbe86b63e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:16.445223 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c14900a27ce4c7eb9c0ad0f0c34488d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:16.445915 9028 tablet_service.cc:1467] Processing CreateTablet for tablet f93281ad992049c38b8d446b09f821c0 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=eb0be093febd40a99ecffa1fbe86b63e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:16.446221 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f93281ad992049c38b8d446b09f821c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:16.446944 6569 tablet_service.cc:1467] Processing CreateTablet for tablet c280f4c27f17437fa7012deca745307b (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=eb0be093febd40a99ecffa1fbe86b63e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:16.447201 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c280f4c27f17437fa7012deca745307b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:16.447306 6831 tablet_service.cc:1467] Processing CreateTablet for tablet f93281ad992049c38b8d446b09f821c0 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=eb0be093febd40a99ecffa1fbe86b63e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:16.447542 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f93281ad992049c38b8d446b09f821c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:16.448184 6699 tablet_service.cc:1467] Processing CreateTablet for tablet c280f4c27f17437fa7012deca745307b (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=eb0be093febd40a99ecffa1fbe86b63e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:16.448483 6699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c280f4c27f17437fa7012deca745307b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:16.448503 14016 tablet_bootstrap.cc:492] T f93281ad992049c38b8d446b09f821c0 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:16.449504 14016 tablet_bootstrap.cc:654] T f93281ad992049c38b8d446b09f821c0 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:16.449986 9158 tablet_service.cc:1467] Processing CreateTablet for tablet f93281ad992049c38b8d446b09f821c0 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=eb0be093febd40a99ecffa1fbe86b63e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:16.450213 14070 tablet_bootstrap.cc:492] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:16.450274 9158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f93281ad992049c38b8d446b09f821c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:16.450580 14072 tablet_bootstrap.cc:492] T c280f4c27f17437fa7012deca745307b P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:16.451589 14072 tablet_bootstrap.cc:654] T c280f4c27f17437fa7012deca745307b P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:16.451790 14070 tablet_bootstrap.cc:654] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:16.452173 14071 tablet_bootstrap.cc:492] T 2c14900a27ce4c7eb9c0ad0f0c34488d P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:16.452256 8897 tablet_service.cc:1467] Processing CreateTablet for tablet c280f4c27f17437fa7012deca745307b (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=eb0be093febd40a99ecffa1fbe86b63e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:16.452520 8897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c280f4c27f17437fa7012deca745307b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:16.452688 9025 tablet_service.cc:1467] Processing CreateTablet for tablet 2c14900a27ce4c7eb9c0ad0f0c34488d (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=eb0be093febd40a99ecffa1fbe86b63e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:16.452966 9025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c14900a27ce4c7eb9c0ad0f0c34488d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:16.453562 14074 tablet_bootstrap.cc:492] T c280f4c27f17437fa7012deca745307b P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:16.454530 14074 tablet_bootstrap.cc:654] T c280f4c27f17437fa7012deca745307b P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:16.454995 14059 tablet_bootstrap.cc:492] T c280f4c27f17437fa7012deca745307b P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:16.455931 14059 tablet_bootstrap.cc:654] T c280f4c27f17437fa7012deca745307b P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:16.456012 14072 tablet_bootstrap.cc:492] T c280f4c27f17437fa7012deca745307b P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:16.456105 14072 ts_tablet_manager.cc:1367] T c280f4c27f17437fa7012deca745307b P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:00:16.456326 14027 tablet_bootstrap.cc:492] T f93281ad992049c38b8d446b09f821c0 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:16.456631 14072 raft_consensus.cc:348] T c280f4c27f17437fa7012deca745307b P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:16.456749 14072 raft_consensus.cc:374] T c280f4c27f17437fa7012deca745307b P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:16.456784 14072 raft_consensus.cc:729] T c280f4c27f17437fa7012deca745307b P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:16.456904 14072 consensus_queue.cc:260] T c280f4c27f17437fa7012deca745307b P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:16.457180 14072 ts_tablet_manager.cc:1398] T c280f4c27f17437fa7012deca745307b P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:16.457345 14027 tablet_bootstrap.cc:654] T f93281ad992049c38b8d446b09f821c0 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:16.459208 14071 tablet_bootstrap.cc:654] T 2c14900a27ce4c7eb9c0ad0f0c34488d P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:16.460144 14016 tablet_bootstrap.cc:492] T f93281ad992049c38b8d446b09f821c0 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:16.460242 14016 ts_tablet_manager.cc:1367] T f93281ad992049c38b8d446b09f821c0 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 02:00:16.460215 14070 tablet_bootstrap.cc:492] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:16.460333 14070 ts_tablet_manager.cc:1367] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 02:00:16.460743 14016 raft_consensus.cc:348] T f93281ad992049c38b8d446b09f821c0 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:16.460863 14016 raft_consensus.cc:374] T f93281ad992049c38b8d446b09f821c0 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:16.460894 14016 raft_consensus.cc:729] T f93281ad992049c38b8d446b09f821c0 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:16.460892 14070 raft_consensus.cc:348] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:16.461007 14070 raft_consensus.cc:374] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:16.461000 14016 consensus_queue.cc:260] T f93281ad992049c38b8d446b09f821c0 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:16.461061 14070 raft_consensus.cc:729] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:16.461180 14070 consensus_queue.cc:260] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:16.461290 14016 ts_tablet_manager.cc:1398] T f93281ad992049c38b8d446b09f821c0 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:16.461464 14070 ts_tablet_manager.cc:1398] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:16.461697 14016 tablet_bootstrap.cc:492] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:16.461891 14070 tablet_bootstrap.cc:492] T f93281ad992049c38b8d446b09f821c0 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:16.462497 14027 tablet_bootstrap.cc:492] T f93281ad992049c38b8d446b09f821c0 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:16.462604 14027 ts_tablet_manager.cc:1367] T f93281ad992049c38b8d446b09f821c0 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:00:16.462672 14016 tablet_bootstrap.cc:654] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:16.462699 14071 tablet_bootstrap.cc:492] T 2c14900a27ce4c7eb9c0ad0f0c34488d P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:16.462790 14071 ts_tablet_manager.cc:1367] T 2c14900a27ce4c7eb9c0ad0f0c34488d P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 02:00:16.462857 14070 tablet_bootstrap.cc:654] T f93281ad992049c38b8d446b09f821c0 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:16.463094 14027 raft_consensus.cc:348] T f93281ad992049c38b8d446b09f821c0 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:16.463215 14027 raft_consensus.cc:374] T f93281ad992049c38b8d446b09f821c0 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:16.463248 14027 raft_consensus.cc:729] T f93281ad992049c38b8d446b09f821c0 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:16.463297 14071 raft_consensus.cc:348] T 2c14900a27ce4c7eb9c0ad0f0c34488d P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:16.463366 14027 consensus_queue.cc:260] T f93281ad992049c38b8d446b09f821c0 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:16.463407 14071 raft_consensus.cc:374] T 2c14900a27ce4c7eb9c0ad0f0c34488d P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:16.463439 14071 raft_consensus.cc:729] T 2c14900a27ce4c7eb9c0ad0f0c34488d P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:16.463546 14071 consensus_queue.cc:260] T 2c14900a27ce4c7eb9c0ad0f0c34488d P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:16.463636 14027 ts_tablet_manager.cc:1398] T f93281ad992049c38b8d446b09f821c0 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:16.463881 14071 ts_tablet_manager.cc:1398] T 2c14900a27ce4c7eb9c0ad0f0c34488d P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:16.464818 14074 tablet_bootstrap.cc:492] T c280f4c27f17437fa7012deca745307b P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:16.465175 14074 ts_tablet_manager.cc:1367] T c280f4c27f17437fa7012deca745307b P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.001s
I20240503 02:00:16.465725 14074 raft_consensus.cc:348] T c280f4c27f17437fa7012deca745307b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:16.465921 14074 raft_consensus.cc:374] T c280f4c27f17437fa7012deca745307b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:16.466001 14074 raft_consensus.cc:729] T c280f4c27f17437fa7012deca745307b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:16.466483 14074 consensus_queue.cc:260] T c280f4c27f17437fa7012deca745307b P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:16.467864 14074 ts_tablet_manager.cc:1398] T c280f4c27f17437fa7012deca745307b P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 02:00:16.470026 14016 tablet_bootstrap.cc:492] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:16.470214 14016 ts_tablet_manager.cc:1367] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.001s
I20240503 02:00:16.473268 14016 raft_consensus.cc:348] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:16.473433 14070 tablet_bootstrap.cc:492] T f93281ad992049c38b8d446b09f821c0 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:16.473500 14016 raft_consensus.cc:374] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:16.473544 14016 raft_consensus.cc:729] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:16.473567 14070 ts_tablet_manager.cc:1367] T f93281ad992049c38b8d446b09f821c0 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 02:00:16.473668 14016 consensus_queue.cc:260] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:16.473953 14016 ts_tablet_manager.cc:1398] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.004s user 0.000s sys 0.000s
I20240503 02:00:16.474074 14070 raft_consensus.cc:348] T f93281ad992049c38b8d446b09f821c0 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:16.474195 14070 raft_consensus.cc:374] T f93281ad992049c38b8d446b09f821c0 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:16.474225 14070 raft_consensus.cc:729] T f93281ad992049c38b8d446b09f821c0 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:16.475705 14070 consensus_queue.cc:260] T f93281ad992049c38b8d446b09f821c0 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:16.476464 14070 ts_tablet_manager.cc:1398] T f93281ad992049c38b8d446b09f821c0 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:00:16.476909 14059 tablet_bootstrap.cc:492] T c280f4c27f17437fa7012deca745307b P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:16.477017 14059 ts_tablet_manager.cc:1367] T c280f4c27f17437fa7012deca745307b P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.001s
I20240503 02:00:16.477658 14059 raft_consensus.cc:348] T c280f4c27f17437fa7012deca745307b P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:16.477800 14059 raft_consensus.cc:374] T c280f4c27f17437fa7012deca745307b P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:16.477833 14059 raft_consensus.cc:729] T c280f4c27f17437fa7012deca745307b P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:16.477963 14059 consensus_queue.cc:260] T c280f4c27f17437fa7012deca745307b P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:16.478272 14059 ts_tablet_manager.cc:1398] T c280f4c27f17437fa7012deca745307b P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20240503 02:00:16.494074 8982 consensus_peers.cc:485] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:00:16.515985 14053 raft_consensus.cc:482] T f93281ad992049c38b8d446b09f821c0 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:16.516132 14053 raft_consensus.cc:504] T f93281ad992049c38b8d446b09f821c0 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:16.516682 14053 leader_election.cc:288] T f93281ad992049c38b8d446b09f821c0 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:16.517277 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f93281ad992049c38b8d446b09f821c0" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:16.517266 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f93281ad992049c38b8d446b09f821c0" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 02:00:16.517406 6851 raft_consensus.cc:2454] T f93281ad992049c38b8d446b09f821c0 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:16.517411 9048 raft_consensus.cc:2454] T f93281ad992049c38b8d446b09f821c0 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:16.517674 9115 leader_election.cc:302] T f93281ad992049c38b8d446b09f821c0 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 492448570cc94ed1879feb2ec3b9051d; no voters:
I20240503 02:00:16.517838 14053 raft_consensus.cc:2781] T f93281ad992049c38b8d446b09f821c0 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:16.517899 14053 raft_consensus.cc:482] T f93281ad992049c38b8d446b09f821c0 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:16.517938 14053 raft_consensus.cc:3037] T f93281ad992049c38b8d446b09f821c0 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:16.518962 14053 raft_consensus.cc:504] T f93281ad992049c38b8d446b09f821c0 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:16.519342 14053 leader_election.cc:288] T f93281ad992049c38b8d446b09f821c0 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Requested vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:16.519713 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f93281ad992049c38b8d446b09f821c0" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 02:00:16.519732 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f93281ad992049c38b8d446b09f821c0" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:16.519826 6849 raft_consensus.cc:3037] T f93281ad992049c38b8d446b09f821c0 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:16.519834 9048 raft_consensus.cc:3037] T f93281ad992049c38b8d446b09f821c0 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:16.520694 6849 raft_consensus.cc:2454] T f93281ad992049c38b8d446b09f821c0 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:16.520730 9048 raft_consensus.cc:2454] T f93281ad992049c38b8d446b09f821c0 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:16.521020 9115 leader_election.cc:302] T f93281ad992049c38b8d446b09f821c0 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 492448570cc94ed1879feb2ec3b9051d; no voters:
I20240503 02:00:16.521214 14053 raft_consensus.cc:2781] T f93281ad992049c38b8d446b09f821c0 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:16.521319 14053 raft_consensus.cc:686] T f93281ad992049c38b8d446b09f821c0 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Becoming Leader. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Running, Role: LEADER
I20240503 02:00:16.521481 14053 consensus_queue.cc:237] T f93281ad992049c38b8d446b09f821c0 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
W20240503 02:00:16.522233 6767 tablet.cc:2369] T c280f4c27f17437fa7012deca745307b P c2ac2eb7cc264047b9ab32fe3a4fc025: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:00:16.523916 6309 catalog_manager.cc:5484] T f93281ad992049c38b8d446b09f821c0 P 2a01ace80baf4057b6bf1995cf88468b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135). New cstate: current_term: 1 leader_uuid: "2a01ace80baf4057b6bf1995cf88468b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: HEALTHY } } }
W20240503 02:00:16.526990 6523 consensus_peers.cc:485] T 4385d51d11574192883d01376e2d5620 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:00:16.533735 6309 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:16.537921 13969 raft_consensus.cc:482] T c280f4c27f17437fa7012deca745307b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:16.538056 13969 raft_consensus.cc:504] T c280f4c27f17437fa7012deca745307b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:16.538650 9178 raft_consensus.cc:1261] T 90a0a91005714af39c8f0e28b4446e01 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer 99ead02915954b4fab23b033ad6d546b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:16.538749 13969 leader_election.cc:288] T c280f4c27f17437fa7012deca745307b P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:16.538841 8916 raft_consensus.cc:1261] T 90a0a91005714af39c8f0e28b4446e01 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Refusing update from remote peer 99ead02915954b4fab23b033ad6d546b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:16.539157 13900 consensus_queue.cc:1035] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:16.539255 6582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c280f4c27f17437fa7012deca745307b" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 02:00:16.539325 13956 consensus_queue.cc:1035] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:16.539388 6582 raft_consensus.cc:2454] T c280f4c27f17437fa7012deca745307b P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:16.539763 6655 leader_election.cc:302] T c280f4c27f17437fa7012deca745307b P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:16.540009 13969 raft_consensus.cc:2781] T c280f4c27f17437fa7012deca745307b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:16.540217 8917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c280f4c27f17437fa7012deca745307b" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:16.540428 8917 raft_consensus.cc:2454] T c280f4c27f17437fa7012deca745307b P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:16.540695 13969 raft_consensus.cc:482] T c280f4c27f17437fa7012deca745307b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:16.540805 13969 raft_consensus.cc:3037] T c280f4c27f17437fa7012deca745307b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:16.542454 13969 raft_consensus.cc:504] T c280f4c27f17437fa7012deca745307b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:16.543447 8917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c280f4c27f17437fa7012deca745307b" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:16.543589 8917 raft_consensus.cc:3037] T c280f4c27f17437fa7012deca745307b P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:16.544198 14001 tablet.cc:1722] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:16.544556 14001 tablet.cc:1620] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b: MemRowSet was empty: no flush needed.
I20240503 02:00:16.544673 8917 raft_consensus.cc:2454] T c280f4c27f17437fa7012deca745307b P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:16.545131 6656 leader_election.cc:302] T c280f4c27f17437fa7012deca745307b P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9942306855404cfd891adf227dd65eb2, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:16.546456 6582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c280f4c27f17437fa7012deca745307b" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 02:00:16.546612 6582 raft_consensus.cc:3037] T c280f4c27f17437fa7012deca745307b P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:16.547000 14066 raft_consensus.cc:2781] T c280f4c27f17437fa7012deca745307b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:16.547086 14066 raft_consensus.cc:686] T c280f4c27f17437fa7012deca745307b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Becoming Leader. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Running, Role: LEADER
I20240503 02:00:16.547238 14066 consensus_queue.cc:237] T c280f4c27f17437fa7012deca745307b P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:16.547729 6582 raft_consensus.cc:2454] T c280f4c27f17437fa7012deca745307b P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:16.549646 13900 raft_consensus.cc:482] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:16.549742 13900 raft_consensus.cc:504] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:16.550093 13900 leader_election.cc:288] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 02:00:16.550515 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c14900a27ce4c7eb9c0ad0f0c34488d" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 02:00:16.550473 6312 catalog_manager.cc:5484] T c280f4c27f17437fa7012deca745307b P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:16.550637 6458 raft_consensus.cc:2454] T 2c14900a27ce4c7eb9c0ad0f0c34488d P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:16.550916 8981 leader_election.cc:302] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99ead02915954b4fab23b033ad6d546b, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:16.551065 13900 raft_consensus.cc:2781] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:16.551101 13900 raft_consensus.cc:482] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:16.551131 13900 raft_consensus.cc:3037] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:16.552156 13900 raft_consensus.cc:504] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:16.552592 13900 leader_election.cc:288] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Requested vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 02:00:16.554039 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c14900a27ce4c7eb9c0ad0f0c34488d" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 02:00:16.554188 6458 raft_consensus.cc:3037] T 2c14900a27ce4c7eb9c0ad0f0c34488d P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:16.554275 13969 leader_election.cc:288] T c280f4c27f17437fa7012deca745307b P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Requested vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:16.555240 6458 raft_consensus.cc:2454] T 2c14900a27ce4c7eb9c0ad0f0c34488d P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:16.555580 8981 leader_election.cc:302] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99ead02915954b4fab23b033ad6d546b, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:16.556318 13956 raft_consensus.cc:2781] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:16.556542 13956 raft_consensus.cc:686] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Becoming Leader. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Running, Role: LEADER
I20240503 02:00:16.556684 13956 consensus_queue.cc:237] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 99ead02915954b4fab23b033ad6d546b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:16.557754 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c14900a27ce4c7eb9c0ad0f0c34488d" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 02:00:16.557878 6851 raft_consensus.cc:3037] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:16.558828 6851 raft_consensus.cc:2454] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:16.557770 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c14900a27ce4c7eb9c0ad0f0c34488d" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 02:00:16.559094 6849 raft_consensus.cc:2362] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1. Re-sending same reply.
W20240503 02:00:16.560447 8851 consensus_peers.cc:485] T 5f51fe5f799f4e479b62f5917a2c2606 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:00:16.561012 14090 tablet.cc:1722] T 90a0a91005714af39c8f0e28b4446e01 P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:16.561389 14090 tablet.cc:1620] T 90a0a91005714af39c8f0e28b4446e01 P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
I20240503 02:00:16.561591 6308 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=7d9cea0b1a7a4cddabd18ed44a365181] alter complete (version 1)
I20240503 02:00:16.564208 6309 catalog_manager.cc:5576] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211) does not have the latest schema for tablet 90a0a91005714af39c8f0e28b4446e01 (table rebalancer_dynamic_table [id=7d9cea0b1a7a4cddabd18ed44a365181]). Expected version 1 got 0
I20240503 02:00:16.564904 6308 catalog_manager.cc:5484] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 99ead02915954b4fab23b033ad6d546b reported cstate change: term changed from 0 to 1, leader changed from <none> to 99ead02915954b4fab23b033ad6d546b (127.0.99.134). New cstate: current_term: 1 leader_uuid: "99ead02915954b4fab23b033ad6d546b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:16.569403 6312 catalog_manager.cc:5576] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211) does not have the latest schema for tablet 90a0a91005714af39c8f0e28b4446e01 (table rebalancer_dynamic_table [id=7d9cea0b1a7a4cddabd18ed44a365181]). Expected version 1 got 0
I20240503 02:00:16.573462 6312 catalog_manager.cc:5576] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079) does not have the latest schema for tablet 90a0a91005714af39c8f0e28b4446e01 (table rebalancer_dynamic_table [id=7d9cea0b1a7a4cddabd18ed44a365181]). Expected version 1 got 0
I20240503 02:00:16.576496 14091 tablet.cc:1722] T 90a0a91005714af39c8f0e28b4446e01 P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:16.576871 14091 tablet.cc:1620] T 90a0a91005714af39c8f0e28b4446e01 P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
W20240503 02:00:16.586419 6636 tablet.cc:2369] T c280f4c27f17437fa7012deca745307b P 80fd18ef4d604c86b909d5c008a46b4f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 02:00:16.638167 9113 consensus_peers.cc:485] T b1740b877c1e4372a0337d99f5bcab0e P 2a01ace80baf4057b6bf1995cf88468b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:16.656208 6505 tablet.cc:2369] T 2c14900a27ce4c7eb9c0ad0f0c34488d P dec3e4a04d604f7eaaa3f077d081218a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:00:16.676721 6309 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
W20240503 02:00:16.677179 8982 consensus_peers.cc:485] T 13756a6e44814fe4b09a6b0f087546c3 P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:00:16.682971 14001 tablet.cc:1722] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:16.683363 14001 tablet.cc:1620] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b: MemRowSet was empty: no flush needed.
I20240503 02:00:16.683799 14091 tablet.cc:1722] T 90a0a91005714af39c8f0e28b4446e01 P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:16.684198 14091 tablet.cc:1620] T 90a0a91005714af39c8f0e28b4446e01 P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
W20240503 02:00:16.687206 6898 tablet.cc:2369] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 492448570cc94ed1879feb2ec3b9051d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:00:16.687014 14090 tablet.cc:1722] T 90a0a91005714af39c8f0e28b4446e01 P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:16.689318 14090 tablet.cc:1620] T 90a0a91005714af39c8f0e28b4446e01 P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
I20240503 02:00:16.690321 6309 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=7d9cea0b1a7a4cddabd18ed44a365181] alter complete (version 2)
W20240503 02:00:16.718788 6523 consensus_peers.cc:485] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:16.772617 6523 consensus_peers.cc:485] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:16.789618 8851 consensus_peers.cc:485] T f28586e70bd9473fbdc6b9126b666fa9 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:00:16.793658 6309 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:16.799192 14001 tablet.cc:1722] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:16.799604 14091 tablet.cc:1722] T 90a0a91005714af39c8f0e28b4446e01 P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:16.799928 14090 tablet.cc:1722] T 90a0a91005714af39c8f0e28b4446e01 P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:16.805409 6309 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=7d9cea0b1a7a4cddabd18ed44a365181] alter complete (version 3)
W20240503 02:00:16.821132 8851 consensus_peers.cc:485] T 7da8f614a70e4177b70e7c2f31d22a10 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:00:16.908059 6308 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:16.908246 6308 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:16.910475 6308 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet 90a0a91005714af39c8f0e28b4446e01
I20240503 02:00:16.911142 9025 tablet_service.cc:1514] Processing DeleteTablet for tablet 90a0a91005714af39c8f0e28b4446e01 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:16 UTC) from {username='slave'} at 127.0.0.1:52584
I20240503 02:00:16.911199 8897 tablet_service.cc:1514] Processing DeleteTablet for tablet 90a0a91005714af39c8f0e28b4446e01 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:16 UTC) from {username='slave'} at 127.0.0.1:40296
I20240503 02:00:16.911561 14092 tablet_replica.cc:330] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b: stopping tablet replica
I20240503 02:00:16.911561 14093 tablet_replica.cc:330] T 90a0a91005714af39c8f0e28b4446e01 P 9942306855404cfd891adf227dd65eb2: stopping tablet replica
I20240503 02:00:16.911710 14093 raft_consensus.cc:2229] T 90a0a91005714af39c8f0e28b4446e01 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:16.911710 14092 raft_consensus.cc:2229] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:16.911885 9158 tablet_service.cc:1514] Processing DeleteTablet for tablet 90a0a91005714af39c8f0e28b4446e01 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:16 UTC) from {username='slave'} at 127.0.0.1:51940
I20240503 02:00:16.911938 14092 raft_consensus.cc:2258] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:16.912226 14093 raft_consensus.cc:2258] T 90a0a91005714af39c8f0e28b4446e01 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:16.912362 14094 tablet_replica.cc:330] T 90a0a91005714af39c8f0e28b4446e01 P 2a01ace80baf4057b6bf1995cf88468b: stopping tablet replica
I20240503 02:00:16.912564 14094 raft_consensus.cc:2229] T 90a0a91005714af39c8f0e28b4446e01 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:16.912664 14092 ts_tablet_manager.cc:1875] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:16.912698 14094 raft_consensus.cc:2258] T 90a0a91005714af39c8f0e28b4446e01 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:16.912881 14093 ts_tablet_manager.cc:1875] T 90a0a91005714af39c8f0e28b4446e01 P 9942306855404cfd891adf227dd65eb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:16.913209 14094 ts_tablet_manager.cc:1875] T 90a0a91005714af39c8f0e28b4446e01 P 2a01ace80baf4057b6bf1995cf88468b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:16.915357 14094 ts_tablet_manager.cc:1888] T 90a0a91005714af39c8f0e28b4446e01 P 2a01ace80baf4057b6bf1995cf88468b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:16.915391 14092 ts_tablet_manager.cc:1888] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:16.915400 14093 ts_tablet_manager.cc:1888] T 90a0a91005714af39c8f0e28b4446e01 P 9942306855404cfd891adf227dd65eb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:16.915441 14094 log.cc:1196] T 90a0a91005714af39c8f0e28b4446e01 P 2a01ace80baf4057b6bf1995cf88468b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/90a0a91005714af39c8f0e28b4446e01
I20240503 02:00:16.915488 14092 log.cc:1196] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/90a0a91005714af39c8f0e28b4446e01
I20240503 02:00:16.915488 14093 log.cc:1196] T 90a0a91005714af39c8f0e28b4446e01 P 9942306855404cfd891adf227dd65eb2: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/90a0a91005714af39c8f0e28b4446e01
I20240503 02:00:16.915745 14094 ts_tablet_manager.cc:1909] T 90a0a91005714af39c8f0e28b4446e01 P 2a01ace80baf4057b6bf1995cf88468b: Deleting consensus metadata
I20240503 02:00:16.915803 14092 ts_tablet_manager.cc:1909] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b: Deleting consensus metadata
I20240503 02:00:16.915803 14093 ts_tablet_manager.cc:1909] T 90a0a91005714af39c8f0e28b4446e01 P 9942306855404cfd891adf227dd65eb2: Deleting consensus metadata
I20240503 02:00:16.916632 6301 catalog_manager.cc:4830] TS 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157): tablet 90a0a91005714af39c8f0e28b4446e01 (table rebalancer_dynamic_table_new_name [id=7d9cea0b1a7a4cddabd18ed44a365181]) successfully deleted
I20240503 02:00:16.916639 6300 catalog_manager.cc:4830] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079): tablet 90a0a91005714af39c8f0e28b4446e01 (table rebalancer_dynamic_table_new_name [id=7d9cea0b1a7a4cddabd18ed44a365181]) successfully deleted
I20240503 02:00:16.916898 6303 catalog_manager.cc:4830] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211): tablet 90a0a91005714af39c8f0e28b4446e01 (table rebalancer_dynamic_table_new_name [id=7d9cea0b1a7a4cddabd18ed44a365181]) successfully deleted
I20240503 02:00:16.929770 13975 consensus_queue.cc:1035] T c280f4c27f17437fa7012deca745307b P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:16.933979 13975 consensus_queue.cc:1035] T c280f4c27f17437fa7012deca745307b P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:00:16.935741 6654 consensus_peers.cc:485] T cea1ea50fa434d31b42451f84f26bc76 P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:16.968093 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:16.974040 13956 consensus_queue.cc:1035] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:16.978891 13943 consensus_queue.cc:1035] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:16.991117 14053 consensus_queue.cc:1035] T f93281ad992049c38b8d446b09f821c0 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:16.997408 13968 consensus_queue.cc:1035] T f93281ad992049c38b8d446b09f821c0 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:00:17.009279 8982 consensus_peers.cc:485] T 1a7ee79312ea4bad816264375ca5bb2d P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:00:17.013185 6312 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:00:17.018968 9158 tablet_service.cc:1467] Processing CreateTablet for tablet e90a9c9fbf844f5b8ff12e0973ab4ebc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=db4fa84dbe7b41dfbee6df24fc2fcabc]), partition=
I20240503 02:00:17.019287 9158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e90a9c9fbf844f5b8ff12e0973ab4ebc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:17.019475 8897 tablet_service.cc:1467] Processing CreateTablet for tablet e90a9c9fbf844f5b8ff12e0973ab4ebc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=db4fa84dbe7b41dfbee6df24fc2fcabc]), partition=
I20240503 02:00:17.019721 8897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e90a9c9fbf844f5b8ff12e0973ab4ebc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:17.021378 14107 tablet_bootstrap.cc:492] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:17.022663 14107 tablet_bootstrap.cc:654] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:17.024735 14107 tablet_bootstrap.cc:492] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:17.024976 14107 ts_tablet_manager.cc:1367] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:17.025429 6569 tablet_service.cc:1467] Processing CreateTablet for tablet e90a9c9fbf844f5b8ff12e0973ab4ebc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=db4fa84dbe7b41dfbee6df24fc2fcabc]), partition=
I20240503 02:00:17.025544 14107 raft_consensus.cc:348] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.025660 14107 raft_consensus.cc:374] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:17.025688 14107 raft_consensus.cc:729] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:17.025700 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e90a9c9fbf844f5b8ff12e0973ab4ebc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:17.025826 14107 consensus_queue.cc:260] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.027223 14107 ts_tablet_manager.cc:1398] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:17.028754 14111 tablet_bootstrap.cc:492] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:17.029875 14111 tablet_bootstrap.cc:654] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:17.032233 14111 tablet_bootstrap.cc:492] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:17.032450 14111 ts_tablet_manager.cc:1367] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:17.033006 14111 raft_consensus.cc:348] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.033217 14111 raft_consensus.cc:374] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:17.033296 14111 raft_consensus.cc:729] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:17.033478 14111 consensus_queue.cc:260] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.034010 14111 ts_tablet_manager.cc:1398] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:17.038473 14108 tablet_bootstrap.cc:492] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:17.039407 14108 tablet_bootstrap.cc:654] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:17.040511 14108 tablet_bootstrap.cc:492] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:17.040611 14108 ts_tablet_manager.cc:1367] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:17.041110 14108 raft_consensus.cc:348] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.041273 14108 raft_consensus.cc:374] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:17.041340 14108 raft_consensus.cc:729] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:17.041515 14108 consensus_queue.cc:260] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.043776 14108 ts_tablet_manager.cc:1398] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
W20240503 02:00:17.056495 8982 consensus_peers.cc:485] T beeea71f02ae4c89a1af13d3584ef70a P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:00:17.091604 13968 raft_consensus.cc:482] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:17.091769 13968 raft_consensus.cc:504] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.092278 13968 leader_election.cc:288] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:17.092630 6582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e90a9c9fbf844f5b8ff12e0973ab4ebc" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 02:00:17.092682 8917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e90a9c9fbf844f5b8ff12e0973ab4ebc" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:17.092767 6582 raft_consensus.cc:2454] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:17.092845 8917 raft_consensus.cc:2454] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:17.093055 9114 leader_election.cc:302] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 80fd18ef4d604c86b909d5c008a46b4f; no voters:
I20240503 02:00:17.093277 13968 raft_consensus.cc:2781] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:17.093322 13968 raft_consensus.cc:482] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:17.093358 13968 raft_consensus.cc:3037] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:17.094273 13968 raft_consensus.cc:504] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.094606 13968 leader_election.cc:288] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Requested vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:17.094946 8917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e90a9c9fbf844f5b8ff12e0973ab4ebc" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:17.094969 6582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e90a9c9fbf844f5b8ff12e0973ab4ebc" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 02:00:17.095080 8917 raft_consensus.cc:3037] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:17.095091 6582 raft_consensus.cc:3037] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:17.095917 6312 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_extra_table_44"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:17.096017 8917 raft_consensus.cc:2454] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:17.096021 6582 raft_consensus.cc:2454] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:17.096297 9115 leader_election.cc:302] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:17.096798 13968 raft_consensus.cc:2781] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:17.096977 13968 raft_consensus.cc:686] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Becoming Leader. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Running, Role: LEADER
I20240503 02:00:17.097126 13968 consensus_queue.cc:237] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.099771 6308 catalog_manager.cc:5484] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135). New cstate: current_term: 1 leader_uuid: "2a01ace80baf4057b6bf1995cf88468b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:17.104457 9158 tablet_service.cc:1467] Processing CreateTablet for tablet 83f5452445214999951ede92cea0d569 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=21134e2d123046cd9d1503cb044e8384]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:17.104692 6569 tablet_service.cc:1467] Processing CreateTablet for tablet b40e5d2709bf488c98f0f65c13cfce03 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=21134e2d123046cd9d1503cb044e8384]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:17.104801 9158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83f5452445214999951ede92cea0d569. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:17.104964 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b40e5d2709bf488c98f0f65c13cfce03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:17.106768 14107 tablet_bootstrap.cc:492] T 83f5452445214999951ede92cea0d569 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:17.107028 9025 tablet_service.cc:1467] Processing CreateTablet for tablet 2cf429a418a6493588685b18d864d6b1 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=21134e2d123046cd9d1503cb044e8384]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:17.107306 9025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cf429a418a6493588685b18d864d6b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:17.107252 6436 tablet_service.cc:1467] Processing CreateTablet for tablet 83f5452445214999951ede92cea0d569 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=21134e2d123046cd9d1503cb044e8384]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:17.107354 6438 tablet_service.cc:1467] Processing CreateTablet for tablet 2cf429a418a6493588685b18d864d6b1 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=21134e2d123046cd9d1503cb044e8384]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:17.107486 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83f5452445214999951ede92cea0d569. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:17.107595 6438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cf429a418a6493588685b18d864d6b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:17.108247 8897 tablet_service.cc:1467] Processing CreateTablet for tablet 83f5452445214999951ede92cea0d569 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=21134e2d123046cd9d1503cb044e8384]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:17.108533 8897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83f5452445214999951ede92cea0d569. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:17.108656 8880 tablet_service.cc:1467] Processing CreateTablet for tablet b40e5d2709bf488c98f0f65c13cfce03 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=21134e2d123046cd9d1503cb044e8384]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:17.108886 8880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b40e5d2709bf488c98f0f65c13cfce03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:17.109874 6831 tablet_service.cc:1467] Processing CreateTablet for tablet b40e5d2709bf488c98f0f65c13cfce03 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=21134e2d123046cd9d1503cb044e8384]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:17.109897 14116 tablet_bootstrap.cc:492] T 2cf429a418a6493588685b18d864d6b1 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:17.110147 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b40e5d2709bf488c98f0f65c13cfce03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:17.110958 8887 tablet_service.cc:1467] Processing CreateTablet for tablet 2cf429a418a6493588685b18d864d6b1 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=21134e2d123046cd9d1503cb044e8384]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:17.111187 8887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cf429a418a6493588685b18d864d6b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:17.112504 14108 tablet_bootstrap.cc:492] T 83f5452445214999951ede92cea0d569 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:17.112686 14116 tablet_bootstrap.cc:654] T 2cf429a418a6493588685b18d864d6b1 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:17.113490 14108 tablet_bootstrap.cc:654] T 83f5452445214999951ede92cea0d569 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:17.113720 14111 tablet_bootstrap.cc:492] T b40e5d2709bf488c98f0f65c13cfce03 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:17.114739 14111 tablet_bootstrap.cc:654] T b40e5d2709bf488c98f0f65c13cfce03 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:17.115569 14108 tablet_bootstrap.cc:492] T 83f5452445214999951ede92cea0d569 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:17.115887 14108 ts_tablet_manager.cc:1367] T 83f5452445214999951ede92cea0d569 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:17.117913 14107 tablet_bootstrap.cc:654] T 83f5452445214999951ede92cea0d569 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:17.117404 14108 raft_consensus.cc:348] T 83f5452445214999951ede92cea0d569 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.119376 14108 raft_consensus.cc:374] T 83f5452445214999951ede92cea0d569 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:17.119647 14108 raft_consensus.cc:729] T 83f5452445214999951ede92cea0d569 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:17.120332 14115 tablet_bootstrap.cc:492] T 2cf429a418a6493588685b18d864d6b1 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:17.120332 14108 consensus_queue.cc:260] T 83f5452445214999951ede92cea0d569 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.124405 14116 tablet_bootstrap.cc:492] T 2cf429a418a6493588685b18d864d6b1 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:17.124490 14115 tablet_bootstrap.cc:654] T 2cf429a418a6493588685b18d864d6b1 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:17.124531 14116 ts_tablet_manager.cc:1367] T 2cf429a418a6493588685b18d864d6b1 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240503 02:00:17.125101 14116 raft_consensus.cc:348] T 2cf429a418a6493588685b18d864d6b1 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.125229 14116 raft_consensus.cc:374] T 2cf429a418a6493588685b18d864d6b1 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:17.125259 14116 raft_consensus.cc:729] T 2cf429a418a6493588685b18d864d6b1 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:17.125383 14116 consensus_queue.cc:260] T 2cf429a418a6493588685b18d864d6b1 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.125684 14115 tablet_bootstrap.cc:492] T 2cf429a418a6493588685b18d864d6b1 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:17.125690 14116 ts_tablet_manager.cc:1398] T 2cf429a418a6493588685b18d864d6b1 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:17.125773 14115 ts_tablet_manager.cc:1367] T 2cf429a418a6493588685b18d864d6b1 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20240503 02:00:17.125772 14107 tablet_bootstrap.cc:492] T 83f5452445214999951ede92cea0d569 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:17.125854 14107 ts_tablet_manager.cc:1367] T 83f5452445214999951ede92cea0d569 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.004s
I20240503 02:00:17.126267 14115 raft_consensus.cc:348] T 2cf429a418a6493588685b18d864d6b1 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.126309 14107 raft_consensus.cc:348] T 83f5452445214999951ede92cea0d569 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.126374 14115 raft_consensus.cc:374] T 2cf429a418a6493588685b18d864d6b1 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:17.126406 14115 raft_consensus.cc:729] T 2cf429a418a6493588685b18d864d6b1 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:17.126410 14107 raft_consensus.cc:374] T 83f5452445214999951ede92cea0d569 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:17.126438 14107 raft_consensus.cc:729] T 83f5452445214999951ede92cea0d569 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:17.126523 14115 consensus_queue.cc:260] T 2cf429a418a6493588685b18d864d6b1 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.126559 14107 consensus_queue.cc:260] T 83f5452445214999951ede92cea0d569 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.126763 14116 tablet_bootstrap.cc:492] T 83f5452445214999951ede92cea0d569 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:17.126821 14115 ts_tablet_manager.cc:1398] T 2cf429a418a6493588685b18d864d6b1 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:17.126845 14107 ts_tablet_manager.cc:1398] T 83f5452445214999951ede92cea0d569 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 02:00:17.128144 14116 tablet_bootstrap.cc:654] T 83f5452445214999951ede92cea0d569 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:17.128686 14108 ts_tablet_manager.cc:1398] T 83f5452445214999951ede92cea0d569 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.013s user 0.000s sys 0.003s
I20240503 02:00:17.128926 14108 tablet_bootstrap.cc:492] T b40e5d2709bf488c98f0f65c13cfce03 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:17.131194 14108 tablet_bootstrap.cc:654] T b40e5d2709bf488c98f0f65c13cfce03 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:17.132444 14117 tablet_bootstrap.cc:492] T b40e5d2709bf488c98f0f65c13cfce03 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:17.133796 14117 tablet_bootstrap.cc:654] T b40e5d2709bf488c98f0f65c13cfce03 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:17.135171 14116 tablet_bootstrap.cc:492] T 83f5452445214999951ede92cea0d569 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:17.135283 14116 ts_tablet_manager.cc:1367] T 83f5452445214999951ede92cea0d569 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.001s
I20240503 02:00:17.135941 14116 raft_consensus.cc:348] T 83f5452445214999951ede92cea0d569 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.136056 14116 raft_consensus.cc:374] T 83f5452445214999951ede92cea0d569 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:17.136086 14116 raft_consensus.cc:729] T 83f5452445214999951ede92cea0d569 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:17.136260 14116 consensus_queue.cc:260] T 83f5452445214999951ede92cea0d569 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.136554 14116 ts_tablet_manager.cc:1398] T 83f5452445214999951ede92cea0d569 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:17.136768 14108 tablet_bootstrap.cc:492] T b40e5d2709bf488c98f0f65c13cfce03 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:17.136893 14108 ts_tablet_manager.cc:1367] T b40e5d2709bf488c98f0f65c13cfce03 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.001s
I20240503 02:00:17.137308 14108 raft_consensus.cc:348] T b40e5d2709bf488c98f0f65c13cfce03 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.137405 14108 raft_consensus.cc:374] T b40e5d2709bf488c98f0f65c13cfce03 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:17.137428 14108 raft_consensus.cc:729] T b40e5d2709bf488c98f0f65c13cfce03 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:17.137548 14108 consensus_queue.cc:260] T b40e5d2709bf488c98f0f65c13cfce03 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.137814 14108 ts_tablet_manager.cc:1398] T b40e5d2709bf488c98f0f65c13cfce03 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:17.138073 14108 tablet_bootstrap.cc:492] T 2cf429a418a6493588685b18d864d6b1 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:17.139264 14108 tablet_bootstrap.cc:654] T 2cf429a418a6493588685b18d864d6b1 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:17.140630 14111 tablet_bootstrap.cc:492] T b40e5d2709bf488c98f0f65c13cfce03 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:17.140802 14111 ts_tablet_manager.cc:1367] T b40e5d2709bf488c98f0f65c13cfce03 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.000s
I20240503 02:00:17.140913 14108 tablet_bootstrap.cc:492] T 2cf429a418a6493588685b18d864d6b1 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:17.141047 14108 ts_tablet_manager.cc:1367] T 2cf429a418a6493588685b18d864d6b1 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:00:17.141489 14111 raft_consensus.cc:348] T b40e5d2709bf488c98f0f65c13cfce03 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.141613 14111 raft_consensus.cc:374] T b40e5d2709bf488c98f0f65c13cfce03 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:17.141640 14111 raft_consensus.cc:729] T b40e5d2709bf488c98f0f65c13cfce03 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:17.141628 14108 raft_consensus.cc:348] T 2cf429a418a6493588685b18d864d6b1 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.141748 14108 raft_consensus.cc:374] T 2cf429a418a6493588685b18d864d6b1 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:17.141779 14108 raft_consensus.cc:729] T 2cf429a418a6493588685b18d864d6b1 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:17.141796 14111 consensus_queue.cc:260] T b40e5d2709bf488c98f0f65c13cfce03 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.141902 14108 consensus_queue.cc:260] T 2cf429a418a6493588685b18d864d6b1 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.142149 14111 ts_tablet_manager.cc:1398] T b40e5d2709bf488c98f0f65c13cfce03 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:17.142308 14108 ts_tablet_manager.cc:1398] T 2cf429a418a6493588685b18d864d6b1 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:17.143401 14117 tablet_bootstrap.cc:492] T b40e5d2709bf488c98f0f65c13cfce03 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:17.143507 14117 ts_tablet_manager.cc:1367] T b40e5d2709bf488c98f0f65c13cfce03 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 02:00:17.144001 14117 raft_consensus.cc:348] T b40e5d2709bf488c98f0f65c13cfce03 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.144698 14117 raft_consensus.cc:374] T b40e5d2709bf488c98f0f65c13cfce03 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:17.144795 14117 raft_consensus.cc:729] T b40e5d2709bf488c98f0f65c13cfce03 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:17.145737 14117 consensus_queue.cc:260] T b40e5d2709bf488c98f0f65c13cfce03 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.146136 14117 ts_tablet_manager.cc:1398] T b40e5d2709bf488c98f0f65c13cfce03 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:00:17.216567 6309 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:17.221279 6582 raft_consensus.cc:1261] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer 2a01ace80baf4057b6bf1995cf88468b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:17.221279 8917 raft_consensus.cc:1261] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Refusing update from remote peer 2a01ace80baf4057b6bf1995cf88468b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:17.221663 13968 consensus_queue.cc:1035] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:17.221794 14053 consensus_queue.cc:1035] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:17.225106 6309 catalog_manager.cc:5576] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211) does not have the latest schema for tablet e90a9c9fbf844f5b8ff12e0973ab4ebc (table rebalancer_dynamic_table [id=db4fa84dbe7b41dfbee6df24fc2fcabc]). Expected version 1 got 0
I20240503 02:00:17.225052 14090 tablet.cc:1722] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:17.225456 14090 tablet.cc:1620] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
I20240503 02:00:17.225499 6312 catalog_manager.cc:5576] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863) does not have the latest schema for tablet e90a9c9fbf844f5b8ff12e0973ab4ebc (table rebalancer_dynamic_table [id=db4fa84dbe7b41dfbee6df24fc2fcabc]). Expected version 1 got 0
I20240503 02:00:17.225979 14047 raft_consensus.cc:482] T b40e5d2709bf488c98f0f65c13cfce03 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:17.226136 14047 raft_consensus.cc:504] T b40e5d2709bf488c98f0f65c13cfce03 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.226640 14047 leader_election.cc:288] T b40e5d2709bf488c98f0f65c13cfce03 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:17.227066 6582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b40e5d2709bf488c98f0f65c13cfce03" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 02:00:17.227210 6582 raft_consensus.cc:2454] T b40e5d2709bf488c98f0f65c13cfce03 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 02:00:17.227389 14091 tablet.cc:1722] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:17.227743 14091 tablet.cc:1620] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
I20240503 02:00:17.228282 6786 leader_election.cc:302] T b40e5d2709bf488c98f0f65c13cfce03 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 80fd18ef4d604c86b909d5c008a46b4f; no voters:
I20240503 02:00:17.228521 14047 raft_consensus.cc:2781] T b40e5d2709bf488c98f0f65c13cfce03 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:17.228577 14047 raft_consensus.cc:482] T b40e5d2709bf488c98f0f65c13cfce03 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:17.228610 14047 raft_consensus.cc:3037] T b40e5d2709bf488c98f0f65c13cfce03 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:17.229415 8917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b40e5d2709bf488c98f0f65c13cfce03" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:17.229566 8917 raft_consensus.cc:2454] T b40e5d2709bf488c98f0f65c13cfce03 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 02:00:17.229717 14047 raft_consensus.cc:504] T b40e5d2709bf488c98f0f65c13cfce03 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.230094 14047 leader_election.cc:288] T b40e5d2709bf488c98f0f65c13cfce03 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Requested vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:17.230615 6582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b40e5d2709bf488c98f0f65c13cfce03" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 02:00:17.230753 6582 raft_consensus.cc:3037] T b40e5d2709bf488c98f0f65c13cfce03 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:17.231398 8917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b40e5d2709bf488c98f0f65c13cfce03" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:17.231519 8917 raft_consensus.cc:3037] T b40e5d2709bf488c98f0f65c13cfce03 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:17.231886 6582 raft_consensus.cc:2454] T b40e5d2709bf488c98f0f65c13cfce03 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 02:00:17.232297 6786 leader_election.cc:302] T b40e5d2709bf488c98f0f65c13cfce03 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 80fd18ef4d604c86b909d5c008a46b4f; no voters:
I20240503 02:00:17.232511 8917 raft_consensus.cc:2454] T b40e5d2709bf488c98f0f65c13cfce03 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 02:00:17.233364 14047 raft_consensus.cc:2781] T b40e5d2709bf488c98f0f65c13cfce03 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:17.233454 14047 raft_consensus.cc:686] T b40e5d2709bf488c98f0f65c13cfce03 P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Becoming Leader. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Running, Role: LEADER
I20240503 02:00:17.233597 14047 consensus_queue.cc:237] T b40e5d2709bf488c98f0f65c13cfce03 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.236629 14053 raft_consensus.cc:482] T 83f5452445214999951ede92cea0d569 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:17.236754 14053 raft_consensus.cc:504] T 83f5452445214999951ede92cea0d569 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.237089 14098 mvcc.cc:205] Tried to move back new op lower bound from 7023417824134352896 to 7023417823631486976. Current Snapshot: MvccSnapshot[applied={T|T < 7023417824134352896}]
I20240503 02:00:17.237207 14053 leader_election.cc:288] T 83f5452445214999951ede92cea0d569 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:17.237214 14132 tablet.cc:1722] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:17.237545 14132 tablet.cc:1620] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 80fd18ef4d604c86b909d5c008a46b4f: MemRowSet was empty: no flush needed.
I20240503 02:00:17.238103 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f5452445214999951ede92cea0d569" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 02:00:17.238252 6458 raft_consensus.cc:2454] T 83f5452445214999951ede92cea0d569 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:17.238636 9112 leader_election.cc:302] T 83f5452445214999951ede92cea0d569 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:17.238663 8917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f5452445214999951ede92cea0d569" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:17.238797 8917 raft_consensus.cc:2454] T 83f5452445214999951ede92cea0d569 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:17.238852 14053 raft_consensus.cc:2781] T 83f5452445214999951ede92cea0d569 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:17.238903 14053 raft_consensus.cc:482] T 83f5452445214999951ede92cea0d569 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:17.238934 14053 raft_consensus.cc:3037] T 83f5452445214999951ede92cea0d569 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:17.240021 14053 raft_consensus.cc:504] T 83f5452445214999951ede92cea0d569 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.241626 6308 catalog_manager.cc:5484] T b40e5d2709bf488c98f0f65c13cfce03 P 492448570cc94ed1879feb2ec3b9051d reported cstate change: term changed from 0 to 1, leader changed from <none> to 492448570cc94ed1879feb2ec3b9051d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:17.242256 6309 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=db4fa84dbe7b41dfbee6df24fc2fcabc] alter complete (version 1)
I20240503 02:00:17.242563 14053 leader_election.cc:288] T 83f5452445214999951ede92cea0d569 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Requested vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:17.242321 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f5452445214999951ede92cea0d569" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 02:00:17.242674 6458 raft_consensus.cc:3037] T 83f5452445214999951ede92cea0d569 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:17.243767 6458 raft_consensus.cc:2454] T 83f5452445214999951ede92cea0d569 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:17.244021 8917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f5452445214999951ede92cea0d569" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:17.244102 9112 leader_election.cc:302] T 83f5452445214999951ede92cea0d569 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:17.244156 8917 raft_consensus.cc:3037] T 83f5452445214999951ede92cea0d569 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:17.244364 14053 raft_consensus.cc:2781] T 83f5452445214999951ede92cea0d569 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:17.244482 14053 raft_consensus.cc:686] T 83f5452445214999951ede92cea0d569 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Becoming Leader. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Running, Role: LEADER
I20240503 02:00:17.244616 14053 consensus_queue.cc:237] T 83f5452445214999951ede92cea0d569 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.245194 8917 raft_consensus.cc:2454] T 83f5452445214999951ede92cea0d569 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:17.249415 6308 catalog_manager.cc:5484] T 83f5452445214999951ede92cea0d569 P 2a01ace80baf4057b6bf1995cf88468b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135). New cstate: current_term: 1 leader_uuid: "2a01ace80baf4057b6bf1995cf88468b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:17.274847 14056 raft_consensus.cc:482] T 2cf429a418a6493588685b18d864d6b1 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:17.274973 14056 raft_consensus.cc:504] T 2cf429a418a6493588685b18d864d6b1 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.275424 14056 leader_election.cc:288] T 2cf429a418a6493588685b18d864d6b1 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:17.275876 9039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cf429a418a6493588685b18d864d6b1" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:17.276052 9039 raft_consensus.cc:2454] T 2cf429a418a6493588685b18d864d6b1 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:17.276430 6392 leader_election.cc:302] T 2cf429a418a6493588685b18d864d6b1 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99ead02915954b4fab23b033ad6d546b, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:17.276446 8917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cf429a418a6493588685b18d864d6b1" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:17.276600 8917 raft_consensus.cc:2454] T 2cf429a418a6493588685b18d864d6b1 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:17.276727 14056 raft_consensus.cc:2781] T 2cf429a418a6493588685b18d864d6b1 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:17.276778 14056 raft_consensus.cc:482] T 2cf429a418a6493588685b18d864d6b1 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:17.276813 14056 raft_consensus.cc:3037] T 2cf429a418a6493588685b18d864d6b1 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:17.277935 14056 raft_consensus.cc:504] T 2cf429a418a6493588685b18d864d6b1 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.278306 14056 leader_election.cc:288] T 2cf429a418a6493588685b18d864d6b1 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Requested vote from peers 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:17.278945 9047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cf429a418a6493588685b18d864d6b1" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:17.279058 9047 raft_consensus.cc:3037] T 2cf429a418a6493588685b18d864d6b1 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:17.280094 9047 raft_consensus.cc:2454] T 2cf429a418a6493588685b18d864d6b1 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:17.280464 6392 leader_election.cc:302] T 2cf429a418a6493588685b18d864d6b1 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99ead02915954b4fab23b033ad6d546b, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:17.280894 8917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cf429a418a6493588685b18d864d6b1" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:17.281029 8917 raft_consensus.cc:3037] T 2cf429a418a6493588685b18d864d6b1 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:17.281788 8917 raft_consensus.cc:2454] T 2cf429a418a6493588685b18d864d6b1 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:17.281994 14056 raft_consensus.cc:2781] T 2cf429a418a6493588685b18d864d6b1 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:17.282172 14056 raft_consensus.cc:686] T 2cf429a418a6493588685b18d864d6b1 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Becoming Leader. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Running, Role: LEADER
I20240503 02:00:17.282341 14056 consensus_queue.cc:237] T 2cf429a418a6493588685b18d864d6b1 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.285866 6308 catalog_manager.cc:5484] T 2cf429a418a6493588685b18d864d6b1 P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: term changed from 0 to 1, leader changed from <none> to dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129). New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:17.347667 6308 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:17.353247 14090 tablet.cc:1722] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:17.353619 14090 tablet.cc:1620] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
I20240503 02:00:17.353634 14132 tablet.cc:1722] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:17.353969 14132 tablet.cc:1620] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 80fd18ef4d604c86b909d5c008a46b4f: MemRowSet was empty: no flush needed.
I20240503 02:00:17.356916 14091 tablet.cc:1722] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:17.357283 14091 tablet.cc:1620] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
I20240503 02:00:17.359094 6313 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=db4fa84dbe7b41dfbee6df24fc2fcabc] alter complete (version 2)
W20240503 02:00:17.394699 6301 catalog_manager.cc:4573] TS e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3d3463f1ff2b41a396bb1b039ddf1946: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:17.462828 6312 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:17.468237 14090 tablet.cc:1722] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:17.468791 14091 tablet.cc:1722] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:17.471163 14132 tablet.cc:1722] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:17.474341 6312 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=db4fa84dbe7b41dfbee6df24fc2fcabc] alter complete (version 3)
W20240503 02:00:17.539021 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:17.578639 6309 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:17.578817 6309 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:17.580955 6309 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet e90a9c9fbf844f5b8ff12e0973ab4ebc
I20240503 02:00:17.581604 9158 tablet_service.cc:1514] Processing DeleteTablet for tablet e90a9c9fbf844f5b8ff12e0973ab4ebc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:17 UTC) from {username='slave'} at 127.0.0.1:51940
I20240503 02:00:17.581696 8887 tablet_service.cc:1514] Processing DeleteTablet for tablet e90a9c9fbf844f5b8ff12e0973ab4ebc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:17 UTC) from {username='slave'} at 127.0.0.1:40296
I20240503 02:00:17.581966 14139 tablet_replica.cc:330] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b: stopping tablet replica
I20240503 02:00:17.582106 14139 raft_consensus.cc:2229] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:17.582103 6569 tablet_service.cc:1514] Processing DeleteTablet for tablet e90a9c9fbf844f5b8ff12e0973ab4ebc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:17 UTC) from {username='slave'} at 127.0.0.1:59846
I20240503 02:00:17.582336 14139 raft_consensus.cc:2258] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:17.582381 14141 tablet_replica.cc:330] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 02:00:17.582556 14141 raft_consensus.cc:2229] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:17.582690 14141 raft_consensus.cc:2258] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:17.582407 14140 tablet_replica.cc:330] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 9942306855404cfd891adf227dd65eb2: stopping tablet replica
I20240503 02:00:17.582861 14140 raft_consensus.cc:2229] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:17.583081 14139 ts_tablet_manager.cc:1875] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:17.583424 14141 ts_tablet_manager.cc:1875] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:17.584542 14140 raft_consensus.cc:2258] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:17.585150 14140 ts_tablet_manager.cc:1875] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 9942306855404cfd891adf227dd65eb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:17.585644 14141 ts_tablet_manager.cc:1888] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:17.585651 14139 ts_tablet_manager.cc:1888] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:17.585724 14139 log.cc:1196] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/e90a9c9fbf844f5b8ff12e0973ab4ebc
I20240503 02:00:17.585726 14141 log.cc:1196] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/e90a9c9fbf844f5b8ff12e0973ab4ebc
I20240503 02:00:17.586004 14139 ts_tablet_manager.cc:1909] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b: Deleting consensus metadata
I20240503 02:00:17.586042 14141 ts_tablet_manager.cc:1909] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 80fd18ef4d604c86b909d5c008a46b4f: Deleting consensus metadata
I20240503 02:00:17.586822 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet e90a9c9fbf844f5b8ff12e0973ab4ebc (table rebalancer_dynamic_table_new_name [id=db4fa84dbe7b41dfbee6df24fc2fcabc]) successfully deleted
I20240503 02:00:17.586822 6300 catalog_manager.cc:4830] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079): tablet e90a9c9fbf844f5b8ff12e0973ab4ebc (table rebalancer_dynamic_table_new_name [id=db4fa84dbe7b41dfbee6df24fc2fcabc]) successfully deleted
I20240503 02:00:17.587491 14140 ts_tablet_manager.cc:1888] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 9942306855404cfd891adf227dd65eb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:17.587566 14140 log.cc:1196] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 9942306855404cfd891adf227dd65eb2: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/e90a9c9fbf844f5b8ff12e0973ab4ebc
I20240503 02:00:17.587775 14140 ts_tablet_manager.cc:1909] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 9942306855404cfd891adf227dd65eb2: Deleting consensus metadata
I20240503 02:00:17.588786 6303 catalog_manager.cc:4830] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211): tablet e90a9c9fbf844f5b8ff12e0973ab4ebc (table rebalancer_dynamic_table_new_name [id=db4fa84dbe7b41dfbee6df24fc2fcabc]) successfully deleted
I20240503 02:00:17.682777 6309 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:00:17.688422 6699 tablet_service.cc:1467] Processing CreateTablet for tablet 89aa8aa378d540a5aec5f61b21b4695c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0d3383ad1add444f973b21aa7d708ad4]), partition=
I20240503 02:00:17.688422 6436 tablet_service.cc:1467] Processing CreateTablet for tablet 89aa8aa378d540a5aec5f61b21b4695c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0d3383ad1add444f973b21aa7d708ad4]), partition=
I20240503 02:00:17.688422 9025 tablet_service.cc:1467] Processing CreateTablet for tablet 89aa8aa378d540a5aec5f61b21b4695c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0d3383ad1add444f973b21aa7d708ad4]), partition=
I20240503 02:00:17.688719 6699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89aa8aa378d540a5aec5f61b21b4695c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:17.688727 9025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89aa8aa378d540a5aec5f61b21b4695c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:17.688727 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89aa8aa378d540a5aec5f61b21b4695c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:17.691222 14144 tablet_bootstrap.cc:492] T 89aa8aa378d540a5aec5f61b21b4695c P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:17.691373 14142 tablet_bootstrap.cc:492] T 89aa8aa378d540a5aec5f61b21b4695c P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:17.692447 14142 tablet_bootstrap.cc:654] T 89aa8aa378d540a5aec5f61b21b4695c P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:17.692476 14144 tablet_bootstrap.cc:654] T 89aa8aa378d540a5aec5f61b21b4695c P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:17.693153 14143 tablet_bootstrap.cc:492] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:17.694137 14143 tablet_bootstrap.cc:654] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:17.694638 14144 tablet_bootstrap.cc:492] T 89aa8aa378d540a5aec5f61b21b4695c P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:17.694636 14142 tablet_bootstrap.cc:492] T 89aa8aa378d540a5aec5f61b21b4695c P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:17.694742 14142 ts_tablet_manager.cc:1367] T 89aa8aa378d540a5aec5f61b21b4695c P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:00:17.694772 14144 ts_tablet_manager.cc:1367] T 89aa8aa378d540a5aec5f61b21b4695c P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 02:00:17.695176 14144 raft_consensus.cc:348] T 89aa8aa378d540a5aec5f61b21b4695c P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:17.695284 14144 raft_consensus.cc:374] T 89aa8aa378d540a5aec5f61b21b4695c P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:17.695255 14142 raft_consensus.cc:348] T 89aa8aa378d540a5aec5f61b21b4695c P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:17.695315 14144 raft_consensus.cc:729] T 89aa8aa378d540a5aec5f61b21b4695c P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:17.695366 14142 raft_consensus.cc:374] T 89aa8aa378d540a5aec5f61b21b4695c P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:17.695405 14142 raft_consensus.cc:729] T 89aa8aa378d540a5aec5f61b21b4695c P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:17.695466 14144 consensus_queue.cc:260] T 89aa8aa378d540a5aec5f61b21b4695c P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:17.695581 14142 consensus_queue.cc:260] T 89aa8aa378d540a5aec5f61b21b4695c P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:17.695760 14144 ts_tablet_manager.cc:1398] T 89aa8aa378d540a5aec5f61b21b4695c P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:17.695892 14142 ts_tablet_manager.cc:1398] T 89aa8aa378d540a5aec5f61b21b4695c P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:17.697423 14143 tablet_bootstrap.cc:492] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:17.697535 14143 ts_tablet_manager.cc:1367] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 02:00:17.698035 14143 raft_consensus.cc:348] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:17.698158 14143 raft_consensus.cc:374] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:17.698187 14143 raft_consensus.cc:729] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:17.699101 14143 consensus_queue.cc:260] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:17.699553 14143 ts_tablet_manager.cc:1398] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:00:17.733987 13968 consensus_queue.cc:1035] T 83f5452445214999951ede92cea0d569 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:17.737313 14135 consensus_queue.cc:1035] T 83f5452445214999951ede92cea0d569 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:17.776531 14056 consensus_queue.cc:1035] T 2cf429a418a6493588685b18d864d6b1 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:17.779151 13991 consensus_queue.cc:1035] T 2cf429a418a6493588685b18d864d6b1 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:17.815986 14106 consensus_queue.cc:1035] T b40e5d2709bf488c98f0f65c13cfce03 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:17.817005 6309 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_extra_table_46"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:17.820365 14106 consensus_queue.cc:1035] T b40e5d2709bf488c98f0f65c13cfce03 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:17.825361 13991 raft_consensus.cc:482] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:17.825495 13991 raft_consensus.cc:504] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:17.825980 13991 leader_election.cc:288] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:17.826565 6720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89aa8aa378d540a5aec5f61b21b4695c" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 02:00:17.826536 9047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89aa8aa378d540a5aec5f61b21b4695c" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:17.826694 9047 raft_consensus.cc:2454] T 89aa8aa378d540a5aec5f61b21b4695c P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:17.826694 6720 raft_consensus.cc:2454] T 89aa8aa378d540a5aec5f61b21b4695c P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:17.826989 6394 leader_election.cc:302] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2ac2eb7cc264047b9ab32fe3a4fc025, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:17.827232 13991 raft_consensus.cc:2781] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:17.827289 13991 raft_consensus.cc:482] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:17.827332 13991 raft_consensus.cc:3037] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:17.828332 6569 tablet_service.cc:1467] Processing CreateTablet for tablet 53b328f875b94dc8906adfdf763821a8 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=b8371b74307b4001945492dae50b8bf4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:17.828666 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53b328f875b94dc8906adfdf763821a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:17.828837 6699 tablet_service.cc:1467] Processing CreateTablet for tablet 0d03b39b002d44e38df27382b107cbbf (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=b8371b74307b4001945492dae50b8bf4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:17.829102 6699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d03b39b002d44e38df27382b107cbbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:17.829067 6567 tablet_service.cc:1467] Processing CreateTablet for tablet c6690a8e5aaa4d83b47899cc8eb96d3d (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=b8371b74307b4001945492dae50b8bf4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:17.829161 9158 tablet_service.cc:1467] Processing CreateTablet for tablet 0d03b39b002d44e38df27382b107cbbf (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=b8371b74307b4001945492dae50b8bf4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:17.829279 6567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6690a8e5aaa4d83b47899cc8eb96d3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:17.829406 9158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d03b39b002d44e38df27382b107cbbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:17.831387 14142 tablet_bootstrap.cc:492] T 0d03b39b002d44e38df27382b107cbbf P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:17.831962 8887 tablet_service.cc:1467] Processing CreateTablet for tablet 0d03b39b002d44e38df27382b107cbbf (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=b8371b74307b4001945492dae50b8bf4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:17.832234 8887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d03b39b002d44e38df27382b107cbbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:17.832408 14142 tablet_bootstrap.cc:654] T 0d03b39b002d44e38df27382b107cbbf P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:17.833235 9025 tablet_service.cc:1467] Processing CreateTablet for tablet 53b328f875b94dc8906adfdf763821a8 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=b8371b74307b4001945492dae50b8bf4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:17.833499 9025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53b328f875b94dc8906adfdf763821a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:17.833659 13991 raft_consensus.cc:504] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:17.834122 13991 leader_election.cc:288] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Requested vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:17.834501 6720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89aa8aa378d540a5aec5f61b21b4695c" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 02:00:17.834616 6720 raft_consensus.cc:3037] T 89aa8aa378d540a5aec5f61b21b4695c P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:17.834723 8880 tablet_service.cc:1467] Processing CreateTablet for tablet c6690a8e5aaa4d83b47899cc8eb96d3d (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=b8371b74307b4001945492dae50b8bf4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:17.834950 8880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6690a8e5aaa4d83b47899cc8eb96d3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:17.835487 6720 raft_consensus.cc:2454] T 89aa8aa378d540a5aec5f61b21b4695c P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:17.835620 14144 tablet_bootstrap.cc:492] T 53b328f875b94dc8906adfdf763821a8 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:17.835839 6831 tablet_service.cc:1467] Processing CreateTablet for tablet 53b328f875b94dc8906adfdf763821a8 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=b8371b74307b4001945492dae50b8bf4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:17.836212 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53b328f875b94dc8906adfdf763821a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:17.836225 6394 leader_election.cc:302] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2ac2eb7cc264047b9ab32fe3a4fc025, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:17.836444 13991 raft_consensus.cc:2781] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:17.836520 13991 raft_consensus.cc:686] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Becoming Leader. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Running, Role: LEADER
I20240503 02:00:17.836663 13991 consensus_queue.cc:237] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:17.836748 14144 tablet_bootstrap.cc:654] T 53b328f875b94dc8906adfdf763821a8 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:17.838084 14159 tablet_bootstrap.cc:492] T 53b328f875b94dc8906adfdf763821a8 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:17.838773 6829 tablet_service.cc:1467] Processing CreateTablet for tablet c6690a8e5aaa4d83b47899cc8eb96d3d (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=b8371b74307b4001945492dae50b8bf4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:17.839015 6829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6690a8e5aaa4d83b47899cc8eb96d3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:17.839118 14159 tablet_bootstrap.cc:654] T 53b328f875b94dc8906adfdf763821a8 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:17.839692 6312 catalog_manager.cc:5484] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: term changed from 0 to 1, leader changed from <none> to dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129). New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:17.841305 9047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89aa8aa378d540a5aec5f61b21b4695c" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:17.841452 9047 raft_consensus.cc:3037] T 89aa8aa378d540a5aec5f61b21b4695c P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:17.842187 14165 tablet_bootstrap.cc:492] T 53b328f875b94dc8906adfdf763821a8 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:17.842468 9047 raft_consensus.cc:2454] T 89aa8aa378d540a5aec5f61b21b4695c P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:17.843277 14162 tablet_bootstrap.cc:492] T 0d03b39b002d44e38df27382b107cbbf P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:17.843418 14165 tablet_bootstrap.cc:654] T 53b328f875b94dc8906adfdf763821a8 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:17.844957 14162 tablet_bootstrap.cc:654] T 0d03b39b002d44e38df27382b107cbbf P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:17.845995 14144 tablet_bootstrap.cc:492] T 53b328f875b94dc8906adfdf763821a8 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:17.846101 14144 ts_tablet_manager.cc:1367] T 53b328f875b94dc8906adfdf763821a8 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.001s
I20240503 02:00:17.846438 14162 tablet_bootstrap.cc:492] T 0d03b39b002d44e38df27382b107cbbf P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:17.846521 14144 raft_consensus.cc:348] T 53b328f875b94dc8906adfdf763821a8 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:17.846607 14162 ts_tablet_manager.cc:1367] T 0d03b39b002d44e38df27382b107cbbf P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:17.846645 14144 raft_consensus.cc:374] T 53b328f875b94dc8906adfdf763821a8 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:17.846679 14144 raft_consensus.cc:729] T 53b328f875b94dc8906adfdf763821a8 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:17.846745 14160 tablet_bootstrap.cc:492] T 0d03b39b002d44e38df27382b107cbbf P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:17.846799 14144 consensus_queue.cc:260] T 53b328f875b94dc8906adfdf763821a8 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:17.847080 14144 ts_tablet_manager.cc:1398] T 53b328f875b94dc8906adfdf763821a8 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:17.847977 14165 tablet_bootstrap.cc:492] T 53b328f875b94dc8906adfdf763821a8 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:17.848109 14165 ts_tablet_manager.cc:1367] T 53b328f875b94dc8906adfdf763821a8 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:00:17.848629 14165 raft_consensus.cc:348] T 53b328f875b94dc8906adfdf763821a8 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:17.848771 14165 raft_consensus.cc:374] T 53b328f875b94dc8906adfdf763821a8 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:17.848814 14165 raft_consensus.cc:729] T 53b328f875b94dc8906adfdf763821a8 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:17.848940 14165 consensus_queue.cc:260] T 53b328f875b94dc8906adfdf763821a8 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:17.849246 14165 ts_tablet_manager.cc:1398] T 53b328f875b94dc8906adfdf763821a8 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:17.849426 14165 tablet_bootstrap.cc:492] T c6690a8e5aaa4d83b47899cc8eb96d3d P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:17.850785 14165 tablet_bootstrap.cc:654] T c6690a8e5aaa4d83b47899cc8eb96d3d P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:17.850845 14160 tablet_bootstrap.cc:654] T 0d03b39b002d44e38df27382b107cbbf P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:17.851148 14142 tablet_bootstrap.cc:492] T 0d03b39b002d44e38df27382b107cbbf P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:17.851341 14142 ts_tablet_manager.cc:1367] T 0d03b39b002d44e38df27382b107cbbf P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.001s
I20240503 02:00:17.851941 14142 raft_consensus.cc:348] T 0d03b39b002d44e38df27382b107cbbf P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.852058 14142 raft_consensus.cc:374] T 0d03b39b002d44e38df27382b107cbbf P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:17.852059 14160 tablet_bootstrap.cc:492] T 0d03b39b002d44e38df27382b107cbbf P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:17.852090 14142 raft_consensus.cc:729] T 0d03b39b002d44e38df27382b107cbbf P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:17.852097 14159 tablet_bootstrap.cc:492] T 53b328f875b94dc8906adfdf763821a8 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:17.852149 14160 ts_tablet_manager.cc:1367] T 0d03b39b002d44e38df27382b107cbbf P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:00:17.852185 14159 ts_tablet_manager.cc:1367] T 53b328f875b94dc8906adfdf763821a8 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 02:00:17.852208 14142 consensus_queue.cc:260] T 0d03b39b002d44e38df27382b107cbbf P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.852550 14142 ts_tablet_manager.cc:1398] T 0d03b39b002d44e38df27382b107cbbf P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:17.852634 14160 raft_consensus.cc:348] T 0d03b39b002d44e38df27382b107cbbf P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.852671 14159 raft_consensus.cc:348] T 53b328f875b94dc8906adfdf763821a8 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:17.852746 14160 raft_consensus.cc:374] T 0d03b39b002d44e38df27382b107cbbf P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:17.852770 14159 raft_consensus.cc:374] T 53b328f875b94dc8906adfdf763821a8 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:17.852782 14160 raft_consensus.cc:729] T 0d03b39b002d44e38df27382b107cbbf P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:17.852808 14159 raft_consensus.cc:729] T 53b328f875b94dc8906adfdf763821a8 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:17.852905 14160 consensus_queue.cc:260] T 0d03b39b002d44e38df27382b107cbbf P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.852918 14159 consensus_queue.cc:260] T 53b328f875b94dc8906adfdf763821a8 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:17.853191 14159 ts_tablet_manager.cc:1398] T 53b328f875b94dc8906adfdf763821a8 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:17.853195 14160 ts_tablet_manager.cc:1398] T 0d03b39b002d44e38df27382b107cbbf P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:17.853364 14159 tablet_bootstrap.cc:492] T c6690a8e5aaa4d83b47899cc8eb96d3d P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:17.854362 14159 tablet_bootstrap.cc:654] T c6690a8e5aaa4d83b47899cc8eb96d3d P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:17.858774 14162 raft_consensus.cc:348] T 0d03b39b002d44e38df27382b107cbbf P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.859136 14162 raft_consensus.cc:374] T 0d03b39b002d44e38df27382b107cbbf P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:17.859472 14162 raft_consensus.cc:729] T 0d03b39b002d44e38df27382b107cbbf P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:17.859660 14162 consensus_queue.cc:260] T 0d03b39b002d44e38df27382b107cbbf P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.859987 14162 ts_tablet_manager.cc:1398] T 0d03b39b002d44e38df27382b107cbbf P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.013s user 0.001s sys 0.000s
I20240503 02:00:17.860118 14159 tablet_bootstrap.cc:492] T c6690a8e5aaa4d83b47899cc8eb96d3d P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:17.860153 14162 tablet_bootstrap.cc:492] T c6690a8e5aaa4d83b47899cc8eb96d3d P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:17.860226 14159 ts_tablet_manager.cc:1367] T c6690a8e5aaa4d83b47899cc8eb96d3d P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:00:17.860960 14159 raft_consensus.cc:348] T c6690a8e5aaa4d83b47899cc8eb96d3d P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:17.861194 14159 raft_consensus.cc:374] T c6690a8e5aaa4d83b47899cc8eb96d3d P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:17.861219 14162 tablet_bootstrap.cc:654] T c6690a8e5aaa4d83b47899cc8eb96d3d P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:17.861250 14165 tablet_bootstrap.cc:492] T c6690a8e5aaa4d83b47899cc8eb96d3d P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:17.861378 14159 raft_consensus.cc:729] T c6690a8e5aaa4d83b47899cc8eb96d3d P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:17.861467 14165 ts_tablet_manager.cc:1367] T c6690a8e5aaa4d83b47899cc8eb96d3d P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 02:00:17.861617 14159 consensus_queue.cc:260] T c6690a8e5aaa4d83b47899cc8eb96d3d P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:17.862381 14162 tablet_bootstrap.cc:492] T c6690a8e5aaa4d83b47899cc8eb96d3d P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:17.862470 14162 ts_tablet_manager.cc:1367] T c6690a8e5aaa4d83b47899cc8eb96d3d P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:17.862740 14159 ts_tablet_manager.cc:1398] T c6690a8e5aaa4d83b47899cc8eb96d3d P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:17.862917 14162 raft_consensus.cc:348] T c6690a8e5aaa4d83b47899cc8eb96d3d P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:17.863018 14162 raft_consensus.cc:374] T c6690a8e5aaa4d83b47899cc8eb96d3d P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:17.863049 14162 raft_consensus.cc:729] T c6690a8e5aaa4d83b47899cc8eb96d3d P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:17.863109 14165 raft_consensus.cc:348] T c6690a8e5aaa4d83b47899cc8eb96d3d P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:17.863174 14162 consensus_queue.cc:260] T c6690a8e5aaa4d83b47899cc8eb96d3d P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:17.863221 14165 raft_consensus.cc:374] T c6690a8e5aaa4d83b47899cc8eb96d3d P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:17.863449 14162 ts_tablet_manager.cc:1398] T c6690a8e5aaa4d83b47899cc8eb96d3d P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:17.863255 14165 raft_consensus.cc:729] T c6690a8e5aaa4d83b47899cc8eb96d3d P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:17.864279 14165 consensus_queue.cc:260] T c6690a8e5aaa4d83b47899cc8eb96d3d P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:17.864665 14165 ts_tablet_manager.cc:1398] T c6690a8e5aaa4d83b47899cc8eb96d3d P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:00:17.896737 13954 raft_consensus.cc:482] T c6690a8e5aaa4d83b47899cc8eb96d3d P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:17.896862 13954 raft_consensus.cc:504] T c6690a8e5aaa4d83b47899cc8eb96d3d P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:17.897339 13954 leader_election.cc:288] T c6690a8e5aaa4d83b47899cc8eb96d3d P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:17.897706 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6690a8e5aaa4d83b47899cc8eb96d3d" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 02:00:17.897759 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6690a8e5aaa4d83b47899cc8eb96d3d" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:17.897836 6849 raft_consensus.cc:2454] T c6690a8e5aaa4d83b47899cc8eb96d3d P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:17.897907 8916 raft_consensus.cc:2454] T c6690a8e5aaa4d83b47899cc8eb96d3d P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:17.898133 6525 leader_election.cc:302] T c6690a8e5aaa4d83b47899cc8eb96d3d P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 80fd18ef4d604c86b909d5c008a46b4f; no voters:
I20240503 02:00:17.898293 13954 raft_consensus.cc:2781] T c6690a8e5aaa4d83b47899cc8eb96d3d P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:17.898347 13954 raft_consensus.cc:482] T c6690a8e5aaa4d83b47899cc8eb96d3d P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:17.898391 13954 raft_consensus.cc:3037] T c6690a8e5aaa4d83b47899cc8eb96d3d P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:17.899238 13954 raft_consensus.cc:504] T c6690a8e5aaa4d83b47899cc8eb96d3d P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:17.899562 13954 leader_election.cc:288] T c6690a8e5aaa4d83b47899cc8eb96d3d P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Requested vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:17.899972 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6690a8e5aaa4d83b47899cc8eb96d3d" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 02:00:17.899972 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6690a8e5aaa4d83b47899cc8eb96d3d" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:17.900104 6849 raft_consensus.cc:3037] T c6690a8e5aaa4d83b47899cc8eb96d3d P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:17.900104 8916 raft_consensus.cc:3037] T c6690a8e5aaa4d83b47899cc8eb96d3d P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:17.901118 8916 raft_consensus.cc:2454] T c6690a8e5aaa4d83b47899cc8eb96d3d P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:17.901144 6849 raft_consensus.cc:2454] T c6690a8e5aaa4d83b47899cc8eb96d3d P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:17.901413 6525 leader_election.cc:302] T c6690a8e5aaa4d83b47899cc8eb96d3d P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:17.901672 13954 raft_consensus.cc:2781] T c6690a8e5aaa4d83b47899cc8eb96d3d P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:17.901758 13954 raft_consensus.cc:686] T c6690a8e5aaa4d83b47899cc8eb96d3d P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Becoming Leader. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Running, Role: LEADER
I20240503 02:00:17.901894 13954 consensus_queue.cc:237] T c6690a8e5aaa4d83b47899cc8eb96d3d P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:17.904089 6309 catalog_manager.cc:5484] T c6690a8e5aaa4d83b47899cc8eb96d3d P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:17.907795 14138 raft_consensus.cc:482] T 0d03b39b002d44e38df27382b107cbbf P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:17.907930 14138 raft_consensus.cc:504] T 0d03b39b002d44e38df27382b107cbbf P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.908365 14138 leader_election.cc:288] T 0d03b39b002d44e38df27382b107cbbf P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:17.908773 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d03b39b002d44e38df27382b107cbbf" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:17.908916 9178 raft_consensus.cc:2454] T 0d03b39b002d44e38df27382b107cbbf P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:17.908900 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d03b39b002d44e38df27382b107cbbf" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:17.909020 8916 raft_consensus.cc:2454] T 0d03b39b002d44e38df27382b107cbbf P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:17.909343 6656 leader_election.cc:302] T 0d03b39b002d44e38df27382b107cbbf P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9942306855404cfd891adf227dd65eb2, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:17.909509 14138 raft_consensus.cc:2781] T 0d03b39b002d44e38df27382b107cbbf P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:17.909560 14138 raft_consensus.cc:482] T 0d03b39b002d44e38df27382b107cbbf P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:17.909595 14138 raft_consensus.cc:3037] T 0d03b39b002d44e38df27382b107cbbf P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:17.910559 14138 raft_consensus.cc:504] T 0d03b39b002d44e38df27382b107cbbf P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.910881 14138 leader_election.cc:288] T 0d03b39b002d44e38df27382b107cbbf P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Requested vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:17.911237 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d03b39b002d44e38df27382b107cbbf" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:17.911237 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d03b39b002d44e38df27382b107cbbf" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:17.911365 9178 raft_consensus.cc:3037] T 0d03b39b002d44e38df27382b107cbbf P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:17.911365 8916 raft_consensus.cc:3037] T 0d03b39b002d44e38df27382b107cbbf P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:17.912351 8916 raft_consensus.cc:2454] T 0d03b39b002d44e38df27382b107cbbf P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:17.912351 9178 raft_consensus.cc:2454] T 0d03b39b002d44e38df27382b107cbbf P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:17.912694 6653 leader_election.cc:302] T 0d03b39b002d44e38df27382b107cbbf P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:17.912851 14138 raft_consensus.cc:2781] T 0d03b39b002d44e38df27382b107cbbf P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:17.912928 14138 raft_consensus.cc:686] T 0d03b39b002d44e38df27382b107cbbf P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Becoming Leader. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Running, Role: LEADER
I20240503 02:00:17.913089 14138 consensus_queue.cc:237] T 0d03b39b002d44e38df27382b107cbbf P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.915343 6309 catalog_manager.cc:5484] T 0d03b39b002d44e38df27382b107cbbf P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } }
W20240503 02:00:17.949244 6785 consensus_peers.cc:485] T c3485701bf3b48f5ae2e1568afeed133 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:00:17.952591 6309 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:17.956645 9048 raft_consensus.cc:1261] T 89aa8aa378d540a5aec5f61b21b4695c P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:17.956959 6720 raft_consensus.cc:1261] T 89aa8aa378d540a5aec5f61b21b4695c P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:17.957034 13991 consensus_queue.cc:1035] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:17.957520 13991 consensus_queue.cc:1035] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:17.961328 14179 tablet.cc:1722] T 89aa8aa378d540a5aec5f61b21b4695c P c2ac2eb7cc264047b9ab32fe3a4fc025: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:17.961566 6308 catalog_manager.cc:5576] TS c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655) does not have the latest schema for tablet 89aa8aa378d540a5aec5f61b21b4695c (table rebalancer_dynamic_table [id=0d3383ad1add444f973b21aa7d708ad4]). Expected version 1 got 0
I20240503 02:00:17.962558 6312 catalog_manager.cc:5576] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093) does not have the latest schema for tablet 89aa8aa378d540a5aec5f61b21b4695c (table rebalancer_dynamic_table [id=0d3383ad1add444f973b21aa7d708ad4]). Expected version 1 got 0
I20240503 02:00:17.963254 14179 tablet.cc:1620] T 89aa8aa378d540a5aec5f61b21b4695c P c2ac2eb7cc264047b9ab32fe3a4fc025: MemRowSet was empty: no flush needed.
I20240503 02:00:17.963521 14180 tablet.cc:1722] T 89aa8aa378d540a5aec5f61b21b4695c P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:17.963893 14180 tablet.cc:1620] T 89aa8aa378d540a5aec5f61b21b4695c P 99ead02915954b4fab23b033ad6d546b: MemRowSet was empty: no flush needed.
I20240503 02:00:17.966733 6312 catalog_manager.cc:5576] TS 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157) does not have the latest schema for tablet 89aa8aa378d540a5aec5f61b21b4695c (table rebalancer_dynamic_table [id=0d3383ad1add444f973b21aa7d708ad4]). Expected version 1 got 0
I20240503 02:00:17.969127 6309 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=0d3383ad1add444f973b21aa7d708ad4] alter complete (version 1)
I20240503 02:00:17.969586 14178 tablet.cc:1722] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:17.969928 14178 tablet.cc:1620] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a: MemRowSet was empty: no flush needed.
I20240503 02:00:17.970834 13900 raft_consensus.cc:482] T 53b328f875b94dc8906adfdf763821a8 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:17.970934 13900 raft_consensus.cc:504] T 53b328f875b94dc8906adfdf763821a8 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:17.971393 13900 leader_election.cc:288] T 53b328f875b94dc8906adfdf763821a8 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:17.971997 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b328f875b94dc8906adfdf763821a8" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 02:00:17.972134 6849 raft_consensus.cc:2454] T 53b328f875b94dc8906adfdf763821a8 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:17.972457 8984 leader_election.cc:302] T 53b328f875b94dc8906adfdf763821a8 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:17.972548 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b328f875b94dc8906adfdf763821a8" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 02:00:17.972615 13900 raft_consensus.cc:2781] T 53b328f875b94dc8906adfdf763821a8 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:17.972659 13900 raft_consensus.cc:482] T 53b328f875b94dc8906adfdf763821a8 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:17.972674 6589 raft_consensus.cc:2454] T 53b328f875b94dc8906adfdf763821a8 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:17.972693 13900 raft_consensus.cc:3037] T 53b328f875b94dc8906adfdf763821a8 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:17.973798 13900 raft_consensus.cc:504] T 53b328f875b94dc8906adfdf763821a8 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:17.974205 13900 leader_election.cc:288] T 53b328f875b94dc8906adfdf763821a8 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Requested vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:17.974480 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b328f875b94dc8906adfdf763821a8" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 02:00:17.974596 6589 raft_consensus.cc:3037] T 53b328f875b94dc8906adfdf763821a8 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:17.974807 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b328f875b94dc8906adfdf763821a8" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 02:00:17.974925 6849 raft_consensus.cc:3037] T 53b328f875b94dc8906adfdf763821a8 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:17.975697 6589 raft_consensus.cc:2454] T 53b328f875b94dc8906adfdf763821a8 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:17.976028 6849 raft_consensus.cc:2454] T 53b328f875b94dc8906adfdf763821a8 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:17.976143 8983 leader_election.cc:302] T 53b328f875b94dc8906adfdf763821a8 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:17.976470 13900 raft_consensus.cc:2781] T 53b328f875b94dc8906adfdf763821a8 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:17.976572 13900 raft_consensus.cc:686] T 53b328f875b94dc8906adfdf763821a8 P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Becoming Leader. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Running, Role: LEADER
I20240503 02:00:17.976706 13900 consensus_queue.cc:237] T 53b328f875b94dc8906adfdf763821a8 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:17.979898 6312 catalog_manager.cc:5484] T 53b328f875b94dc8906adfdf763821a8 P 99ead02915954b4fab23b033ad6d546b reported cstate change: term changed from 0 to 1, leader changed from <none> to 99ead02915954b4fab23b033ad6d546b (127.0.99.134). New cstate: current_term: 1 leader_uuid: "99ead02915954b4fab23b033ad6d546b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } }
W20240503 02:00:18.040329 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:18.075299 6312 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:18.080706 14178 tablet.cc:1722] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:18.081166 14180 tablet.cc:1722] T 89aa8aa378d540a5aec5f61b21b4695c P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:18.081252 14178 tablet.cc:1620] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a: MemRowSet was empty: no flush needed.
I20240503 02:00:18.081250 14179 tablet.cc:1722] T 89aa8aa378d540a5aec5f61b21b4695c P c2ac2eb7cc264047b9ab32fe3a4fc025: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:18.081518 14179 tablet.cc:1620] T 89aa8aa378d540a5aec5f61b21b4695c P c2ac2eb7cc264047b9ab32fe3a4fc025: MemRowSet was empty: no flush needed.
I20240503 02:00:18.081512 14180 tablet.cc:1620] T 89aa8aa378d540a5aec5f61b21b4695c P 99ead02915954b4fab23b033ad6d546b: MemRowSet was empty: no flush needed.
I20240503 02:00:18.087538 6308 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=0d3383ad1add444f973b21aa7d708ad4] alter complete (version 2)
W20240503 02:00:18.144089 8982 consensus_peers.cc:485] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:00:18.190490 6312 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:18.196534 14178 tablet.cc:1722] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:18.197516 14179 tablet.cc:1722] T 89aa8aa378d540a5aec5f61b21b4695c P c2ac2eb7cc264047b9ab32fe3a4fc025: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:18.200986 14180 tablet.cc:1722] T 89aa8aa378d540a5aec5f61b21b4695c P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:18.202361 6312 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=0d3383ad1add444f973b21aa7d708ad4] alter complete (version 3)
I20240503 02:00:18.305567 6312 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:18.305724 6312 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:18.307997 6312 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet 89aa8aa378d540a5aec5f61b21b4695c
I20240503 02:00:18.308635 9025 tablet_service.cc:1514] Processing DeleteTablet for tablet 89aa8aa378d540a5aec5f61b21b4695c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:18 UTC) from {username='slave'} at 127.0.0.1:52584
I20240503 02:00:18.308701 6436 tablet_service.cc:1514] Processing DeleteTablet for tablet 89aa8aa378d540a5aec5f61b21b4695c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:18 UTC) from {username='slave'} at 127.0.0.1:51338
I20240503 02:00:18.309037 14185 tablet_replica.cc:330] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a: stopping tablet replica
I20240503 02:00:18.309053 14186 tablet_replica.cc:330] T 89aa8aa378d540a5aec5f61b21b4695c P 99ead02915954b4fab23b033ad6d546b: stopping tablet replica
I20240503 02:00:18.309196 14186 raft_consensus.cc:2229] T 89aa8aa378d540a5aec5f61b21b4695c P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:18.309211 14185 raft_consensus.cc:2229] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:18.309177 6699 tablet_service.cc:1514] Processing DeleteTablet for tablet 89aa8aa378d540a5aec5f61b21b4695c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:18 UTC) from {username='slave'} at 127.0.0.1:40336
I20240503 02:00:18.309340 14186 raft_consensus.cc:2258] T 89aa8aa378d540a5aec5f61b21b4695c P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:18.309427 14185 raft_consensus.cc:2258] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:18.309515 14187 tablet_replica.cc:330] T 89aa8aa378d540a5aec5f61b21b4695c P c2ac2eb7cc264047b9ab32fe3a4fc025: stopping tablet replica
I20240503 02:00:18.309641 14187 raft_consensus.cc:2229] T 89aa8aa378d540a5aec5f61b21b4695c P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:18.309760 14187 raft_consensus.cc:2258] T 89aa8aa378d540a5aec5f61b21b4695c P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:18.310186 14186 ts_tablet_manager.cc:1875] T 89aa8aa378d540a5aec5f61b21b4695c P 99ead02915954b4fab23b033ad6d546b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:18.310307 14187 ts_tablet_manager.cc:1875] T 89aa8aa378d540a5aec5f61b21b4695c P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:18.310308 14185 ts_tablet_manager.cc:1875] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:18.312835 14185 ts_tablet_manager.cc:1888] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:18.312846 14186 ts_tablet_manager.cc:1888] T 89aa8aa378d540a5aec5f61b21b4695c P 99ead02915954b4fab23b033ad6d546b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:18.312862 14187 ts_tablet_manager.cc:1888] T 89aa8aa378d540a5aec5f61b21b4695c P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:18.312933 14187 log.cc:1196] T 89aa8aa378d540a5aec5f61b21b4695c P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/89aa8aa378d540a5aec5f61b21b4695c
I20240503 02:00:18.312935 14186 log.cc:1196] T 89aa8aa378d540a5aec5f61b21b4695c P 99ead02915954b4fab23b033ad6d546b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/89aa8aa378d540a5aec5f61b21b4695c
I20240503 02:00:18.312999 14185 log.cc:1196] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/89aa8aa378d540a5aec5f61b21b4695c
I20240503 02:00:18.313336 14185 ts_tablet_manager.cc:1909] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a: Deleting consensus metadata
I20240503 02:00:18.313336 14187 ts_tablet_manager.cc:1909] T 89aa8aa378d540a5aec5f61b21b4695c P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting consensus metadata
I20240503 02:00:18.313576 14186 ts_tablet_manager.cc:1909] T 89aa8aa378d540a5aec5f61b21b4695c P 99ead02915954b4fab23b033ad6d546b: Deleting consensus metadata
I20240503 02:00:18.314338 6300 catalog_manager.cc:4830] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): tablet 89aa8aa378d540a5aec5f61b21b4695c (table rebalancer_dynamic_table_new_name [id=0d3383ad1add444f973b21aa7d708ad4]) successfully deleted
I20240503 02:00:18.314426 6303 catalog_manager.cc:4830] TS c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655): tablet 89aa8aa378d540a5aec5f61b21b4695c (table rebalancer_dynamic_table_new_name [id=0d3383ad1add444f973b21aa7d708ad4]) successfully deleted
I20240503 02:00:18.314361 6301 catalog_manager.cc:4830] TS 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157): tablet 89aa8aa378d540a5aec5f61b21b4695c (table rebalancer_dynamic_table_new_name [id=0d3383ad1add444f973b21aa7d708ad4]) successfully deleted
I20240503 02:00:18.379634 14138 consensus_queue.cc:1035] T 0d03b39b002d44e38df27382b107cbbf P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:00:18.382259 6654 consensus_peers.cc:485] T ba17d4d3e0754b2798eb95484d37229a P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:00:18.382709 13969 consensus_queue.cc:1035] T 0d03b39b002d44e38df27382b107cbbf P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:18.400043 13900 consensus_queue.cc:1035] T 53b328f875b94dc8906adfdf763821a8 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:18.407176 14195 consensus_queue.cc:1035] T 53b328f875b94dc8906adfdf763821a8 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:18.410064 6308 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
W20240503 02:00:18.411206 6523 consensus_peers.cc:485] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:00:18.416102 6567 tablet_service.cc:1467] Processing CreateTablet for tablet 4f191e65b61c499fb0a821ec9d5488a1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f2514b38a37b435aaaba4bc751e7ee6e]), partition=
I20240503 02:00:18.416102 6436 tablet_service.cc:1467] Processing CreateTablet for tablet 4f191e65b61c499fb0a821ec9d5488a1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f2514b38a37b435aaaba4bc751e7ee6e]), partition=
I20240503 02:00:18.416431 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f191e65b61c499fb0a821ec9d5488a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:18.416431 6567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f191e65b61c499fb0a821ec9d5488a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:18.416895 9158 tablet_service.cc:1467] Processing CreateTablet for tablet 4f191e65b61c499fb0a821ec9d5488a1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f2514b38a37b435aaaba4bc751e7ee6e]), partition=
I20240503 02:00:18.417155 9158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f191e65b61c499fb0a821ec9d5488a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:18.419922 14201 tablet_bootstrap.cc:492] T 4f191e65b61c499fb0a821ec9d5488a1 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:18.419880 14199 tablet_bootstrap.cc:492] T 4f191e65b61c499fb0a821ec9d5488a1 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:18.421083 14201 tablet_bootstrap.cc:654] T 4f191e65b61c499fb0a821ec9d5488a1 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:18.422739 14199 tablet_bootstrap.cc:654] T 4f191e65b61c499fb0a821ec9d5488a1 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:18.422868 14201 tablet_bootstrap.cc:492] T 4f191e65b61c499fb0a821ec9d5488a1 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:18.422968 14201 ts_tablet_manager.cc:1367] T 4f191e65b61c499fb0a821ec9d5488a1 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:00:18.423463 14200 tablet_bootstrap.cc:492] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:18.423457 14201 raft_consensus.cc:348] T 4f191e65b61c499fb0a821ec9d5488a1 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:18.423606 14201 raft_consensus.cc:374] T 4f191e65b61c499fb0a821ec9d5488a1 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:18.423655 14201 raft_consensus.cc:729] T 4f191e65b61c499fb0a821ec9d5488a1 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:18.423883 14199 tablet_bootstrap.cc:492] T 4f191e65b61c499fb0a821ec9d5488a1 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:18.423862 14201 consensus_queue.cc:260] T 4f191e65b61c499fb0a821ec9d5488a1 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:18.423979 14199 ts_tablet_manager.cc:1367] T 4f191e65b61c499fb0a821ec9d5488a1 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:18.424427 14200 tablet_bootstrap.cc:654] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:18.424484 14199 raft_consensus.cc:348] T 4f191e65b61c499fb0a821ec9d5488a1 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:18.424602 14199 raft_consensus.cc:374] T 4f191e65b61c499fb0a821ec9d5488a1 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:18.424666 14199 raft_consensus.cc:729] T 4f191e65b61c499fb0a821ec9d5488a1 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:18.424800 14199 consensus_queue.cc:260] T 4f191e65b61c499fb0a821ec9d5488a1 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:18.425172 14199 ts_tablet_manager.cc:1398] T 4f191e65b61c499fb0a821ec9d5488a1 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:18.425910 14201 ts_tablet_manager.cc:1398] T 4f191e65b61c499fb0a821ec9d5488a1 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:00:18.430229 14200 tablet_bootstrap.cc:492] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:18.430457 14200 ts_tablet_manager.cc:1367] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240503 02:00:18.431257 14200 raft_consensus.cc:348] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:18.431416 14200 raft_consensus.cc:374] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:18.431460 14200 raft_consensus.cc:729] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:18.431584 14200 consensus_queue.cc:260] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:18.431892 14200 ts_tablet_manager.cc:1398] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:18.435761 14202 consensus_queue.cc:1035] T c6690a8e5aaa4d83b47899cc8eb96d3d P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:00:18.437105 6785 consensus_peers.cc:485] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:00:18.439603 14202 consensus_queue.cc:1035] T c6690a8e5aaa4d83b47899cc8eb96d3d P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:00:18.442914 6301 catalog_manager.cc:4573] TS e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): CreateTablet RPC failed for tablet 56824b553788471c9a2fcd6ec82d507e: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:18.496364 6314 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_extra_table_48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:18.503767 6700 tablet_service.cc:1467] Processing CreateTablet for tablet fa74c52d8f6f43d7ab61ae500fec62d4 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=52f31af1c95b4104b394062d67c75c08]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:18.503809 8880 tablet_service.cc:1467] Processing CreateTablet for tablet 1029229ea6924a31b637639a95a04f9e (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=52f31af1c95b4104b394062d67c75c08]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:18.503767 6699 tablet_service.cc:1467] Processing CreateTablet for tablet 6338547c9784461d80cee2c69aaf905b (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=52f31af1c95b4104b394062d67c75c08]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:18.504154 8880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1029229ea6924a31b637639a95a04f9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:18.504160 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa74c52d8f6f43d7ab61ae500fec62d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:18.504760 9025 tablet_service.cc:1467] Processing CreateTablet for tablet 6338547c9784461d80cee2c69aaf905b (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=52f31af1c95b4104b394062d67c75c08]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:18.504837 9028 tablet_service.cc:1467] Processing CreateTablet for tablet fa74c52d8f6f43d7ab61ae500fec62d4 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=52f31af1c95b4104b394062d67c75c08]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:18.505079 9025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6338547c9784461d80cee2c69aaf905b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:18.505472 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa74c52d8f6f43d7ab61ae500fec62d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:18.506799 9027 tablet_service.cc:1467] Processing CreateTablet for tablet 1029229ea6924a31b637639a95a04f9e (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=52f31af1c95b4104b394062d67c75c08]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:18.507349 9027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1029229ea6924a31b637639a95a04f9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:18.507690 9158 tablet_service.cc:1467] Processing CreateTablet for tablet 6338547c9784461d80cee2c69aaf905b (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=52f31af1c95b4104b394062d67c75c08]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:18.507740 9157 tablet_service.cc:1467] Processing CreateTablet for tablet fa74c52d8f6f43d7ab61ae500fec62d4 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=52f31af1c95b4104b394062d67c75c08]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:18.507953 9158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6338547c9784461d80cee2c69aaf905b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:18.507951 6699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6338547c9784461d80cee2c69aaf905b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:18.508229 9157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa74c52d8f6f43d7ab61ae500fec62d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:18.510056 14201 tablet_bootstrap.cc:492] T 6338547c9784461d80cee2c69aaf905b P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:18.511224 14210 tablet_bootstrap.cc:492] T fa74c52d8f6f43d7ab61ae500fec62d4 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:18.511279 14201 tablet_bootstrap.cc:654] T 6338547c9784461d80cee2c69aaf905b P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:18.511343 6436 tablet_service.cc:1467] Processing CreateTablet for tablet 1029229ea6924a31b637639a95a04f9e (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=52f31af1c95b4104b394062d67c75c08]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:18.511643 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1029229ea6924a31b637639a95a04f9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:18.512513 14210 tablet_bootstrap.cc:654] T fa74c52d8f6f43d7ab61ae500fec62d4 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:18.513289 14209 tablet_bootstrap.cc:492] T 6338547c9784461d80cee2c69aaf905b P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:18.513456 14201 tablet_bootstrap.cc:492] T 6338547c9784461d80cee2c69aaf905b P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:18.513561 14201 ts_tablet_manager.cc:1367] T 6338547c9784461d80cee2c69aaf905b P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 02:00:18.514039 14201 raft_consensus.cc:348] T 6338547c9784461d80cee2c69aaf905b P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:18.514147 14201 raft_consensus.cc:374] T 6338547c9784461d80cee2c69aaf905b P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:18.514178 14201 raft_consensus.cc:729] T 6338547c9784461d80cee2c69aaf905b P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:18.514294 14201 consensus_queue.cc:260] T 6338547c9784461d80cee2c69aaf905b P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:18.514433 14199 tablet_bootstrap.cc:492] T 1029229ea6924a31b637639a95a04f9e P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:18.514587 14201 ts_tablet_manager.cc:1398] T 6338547c9784461d80cee2c69aaf905b P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:18.514767 14201 tablet_bootstrap.cc:492] T fa74c52d8f6f43d7ab61ae500fec62d4 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:18.514936 14209 tablet_bootstrap.cc:654] T 6338547c9784461d80cee2c69aaf905b P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:18.515663 14201 tablet_bootstrap.cc:654] T fa74c52d8f6f43d7ab61ae500fec62d4 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:18.515899 14199 tablet_bootstrap.cc:654] T 1029229ea6924a31b637639a95a04f9e P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:18.516520 14208 tablet_bootstrap.cc:492] T 1029229ea6924a31b637639a95a04f9e P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:18.517072 14209 tablet_bootstrap.cc:492] T 6338547c9784461d80cee2c69aaf905b P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:18.517176 14209 ts_tablet_manager.cc:1367] T 6338547c9784461d80cee2c69aaf905b P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:18.517642 14209 raft_consensus.cc:348] T 6338547c9784461d80cee2c69aaf905b P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:18.517762 14208 tablet_bootstrap.cc:654] T 1029229ea6924a31b637639a95a04f9e P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:18.517771 14209 raft_consensus.cc:374] T 6338547c9784461d80cee2c69aaf905b P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:18.517800 14209 raft_consensus.cc:729] T 6338547c9784461d80cee2c69aaf905b P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:18.517911 14209 consensus_queue.cc:260] T 6338547c9784461d80cee2c69aaf905b P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:18.518236 14209 ts_tablet_manager.cc:1398] T 6338547c9784461d80cee2c69aaf905b P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:18.518417 14209 tablet_bootstrap.cc:492] T fa74c52d8f6f43d7ab61ae500fec62d4 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:18.518967 14201 tablet_bootstrap.cc:492] T fa74c52d8f6f43d7ab61ae500fec62d4 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:18.519044 14201 ts_tablet_manager.cc:1367] T fa74c52d8f6f43d7ab61ae500fec62d4 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240503 02:00:18.519098 14210 tablet_bootstrap.cc:492] T fa74c52d8f6f43d7ab61ae500fec62d4 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:18.519253 14210 ts_tablet_manager.cc:1367] T fa74c52d8f6f43d7ab61ae500fec62d4 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20240503 02:00:18.519368 14209 tablet_bootstrap.cc:654] T fa74c52d8f6f43d7ab61ae500fec62d4 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:18.519480 14201 raft_consensus.cc:348] T fa74c52d8f6f43d7ab61ae500fec62d4 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:18.519596 14201 raft_consensus.cc:374] T fa74c52d8f6f43d7ab61ae500fec62d4 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:18.519629 14201 raft_consensus.cc:729] T fa74c52d8f6f43d7ab61ae500fec62d4 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:18.519739 14201 consensus_queue.cc:260] T fa74c52d8f6f43d7ab61ae500fec62d4 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:18.520006 14201 ts_tablet_manager.cc:1398] T fa74c52d8f6f43d7ab61ae500fec62d4 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:18.519950 14210 raft_consensus.cc:348] T fa74c52d8f6f43d7ab61ae500fec62d4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:18.520058 14210 raft_consensus.cc:374] T fa74c52d8f6f43d7ab61ae500fec62d4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:18.520093 14210 raft_consensus.cc:729] T fa74c52d8f6f43d7ab61ae500fec62d4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:18.520226 14210 consensus_queue.cc:260] T fa74c52d8f6f43d7ab61ae500fec62d4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:18.520857 14210 ts_tablet_manager.cc:1398] T fa74c52d8f6f43d7ab61ae500fec62d4 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:18.520954 14209 tablet_bootstrap.cc:492] T fa74c52d8f6f43d7ab61ae500fec62d4 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:18.521029 14209 ts_tablet_manager.cc:1367] T fa74c52d8f6f43d7ab61ae500fec62d4 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:00:18.521044 14210 tablet_bootstrap.cc:492] T 6338547c9784461d80cee2c69aaf905b P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:18.521454 14209 raft_consensus.cc:348] T fa74c52d8f6f43d7ab61ae500fec62d4 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:18.521603 14209 raft_consensus.cc:374] T fa74c52d8f6f43d7ab61ae500fec62d4 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:18.521632 14209 raft_consensus.cc:729] T fa74c52d8f6f43d7ab61ae500fec62d4 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:18.521734 14209 consensus_queue.cc:260] T fa74c52d8f6f43d7ab61ae500fec62d4 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:18.521992 14209 ts_tablet_manager.cc:1398] T fa74c52d8f6f43d7ab61ae500fec62d4 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:18.522166 14209 tablet_bootstrap.cc:492] T 1029229ea6924a31b637639a95a04f9e P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:18.522198 14210 tablet_bootstrap.cc:654] T 6338547c9784461d80cee2c69aaf905b P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:18.523059 14209 tablet_bootstrap.cc:654] T 1029229ea6924a31b637639a95a04f9e P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:18.524114 14209 tablet_bootstrap.cc:492] T 1029229ea6924a31b637639a95a04f9e P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:18.524235 14209 ts_tablet_manager.cc:1367] T 1029229ea6924a31b637639a95a04f9e P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:00:18.524869 14209 raft_consensus.cc:348] T 1029229ea6924a31b637639a95a04f9e P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:18.525010 14209 raft_consensus.cc:374] T 1029229ea6924a31b637639a95a04f9e P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:18.525045 14209 raft_consensus.cc:729] T 1029229ea6924a31b637639a95a04f9e P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:18.525185 14209 consensus_queue.cc:260] T 1029229ea6924a31b637639a95a04f9e P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:18.525565 14209 ts_tablet_manager.cc:1398] T 1029229ea6924a31b637639a95a04f9e P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:18.528671 14199 tablet_bootstrap.cc:492] T 1029229ea6924a31b637639a95a04f9e P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:18.528800 14199 ts_tablet_manager.cc:1367] T 1029229ea6924a31b637639a95a04f9e P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 02:00:18.529284 14199 raft_consensus.cc:348] T 1029229ea6924a31b637639a95a04f9e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:18.529420 14199 raft_consensus.cc:374] T 1029229ea6924a31b637639a95a04f9e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:18.529454 14199 raft_consensus.cc:729] T 1029229ea6924a31b637639a95a04f9e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:18.529587 14199 consensus_queue.cc:260] T 1029229ea6924a31b637639a95a04f9e P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:18.530620 14199 ts_tablet_manager.cc:1398] T 1029229ea6924a31b637639a95a04f9e P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:18.531240 14208 tablet_bootstrap.cc:492] T 1029229ea6924a31b637639a95a04f9e P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:18.531360 14208 ts_tablet_manager.cc:1367] T 1029229ea6924a31b637639a95a04f9e P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240503 02:00:18.532029 14208 raft_consensus.cc:348] T 1029229ea6924a31b637639a95a04f9e P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:18.532153 14208 raft_consensus.cc:374] T 1029229ea6924a31b637639a95a04f9e P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:18.532186 14208 raft_consensus.cc:729] T 1029229ea6924a31b637639a95a04f9e P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:18.532307 14208 consensus_queue.cc:260] T 1029229ea6924a31b637639a95a04f9e P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:18.532641 14208 ts_tablet_manager.cc:1398] T 1029229ea6924a31b637639a95a04f9e P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:18.537848 14210 tablet_bootstrap.cc:492] T 6338547c9784461d80cee2c69aaf905b P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:18.537983 14210 ts_tablet_manager.cc:1367] T 6338547c9784461d80cee2c69aaf905b P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.001s
I20240503 02:00:18.538460 14210 raft_consensus.cc:348] T 6338547c9784461d80cee2c69aaf905b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:18.538566 14210 raft_consensus.cc:374] T 6338547c9784461d80cee2c69aaf905b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:18.538587 14210 raft_consensus.cc:729] T 6338547c9784461d80cee2c69aaf905b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:18.538710 14210 consensus_queue.cc:260] T 6338547c9784461d80cee2c69aaf905b P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:18.539829 14210 ts_tablet_manager.cc:1398] T 6338547c9784461d80cee2c69aaf905b P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:00:18.574126 14190 raft_consensus.cc:482] T fa74c52d8f6f43d7ab61ae500fec62d4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:18.574271 14190 raft_consensus.cc:504] T fa74c52d8f6f43d7ab61ae500fec62d4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:18.574790 14190 leader_election.cc:288] T fa74c52d8f6f43d7ab61ae500fec62d4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:18.575147 9047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa74c52d8f6f43d7ab61ae500fec62d4" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:18.575219 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa74c52d8f6f43d7ab61ae500fec62d4" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:18.575292 9047 raft_consensus.cc:2454] T fa74c52d8f6f43d7ab61ae500fec62d4 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:18.575371 9178 raft_consensus.cc:2454] T fa74c52d8f6f43d7ab61ae500fec62d4 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:18.575618 6654 leader_election.cc:302] T fa74c52d8f6f43d7ab61ae500fec62d4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99ead02915954b4fab23b033ad6d546b, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:18.575794 14190 raft_consensus.cc:2781] T fa74c52d8f6f43d7ab61ae500fec62d4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:18.575860 14190 raft_consensus.cc:482] T fa74c52d8f6f43d7ab61ae500fec62d4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:18.575903 14190 raft_consensus.cc:3037] T fa74c52d8f6f43d7ab61ae500fec62d4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:18.576906 14190 raft_consensus.cc:504] T fa74c52d8f6f43d7ab61ae500fec62d4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:18.577268 14190 leader_election.cc:288] T fa74c52d8f6f43d7ab61ae500fec62d4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Requested vote from peers 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:18.577675 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa74c52d8f6f43d7ab61ae500fec62d4" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:18.577831 9178 raft_consensus.cc:3037] T fa74c52d8f6f43d7ab61ae500fec62d4 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:18.577836 9047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa74c52d8f6f43d7ab61ae500fec62d4" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:18.577975 9047 raft_consensus.cc:3037] T fa74c52d8f6f43d7ab61ae500fec62d4 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:18.578902 9047 raft_consensus.cc:2454] T fa74c52d8f6f43d7ab61ae500fec62d4 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:18.578902 9178 raft_consensus.cc:2454] T fa74c52d8f6f43d7ab61ae500fec62d4 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:18.579252 6653 leader_election.cc:302] T fa74c52d8f6f43d7ab61ae500fec62d4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:18.579574 14190 raft_consensus.cc:2781] T fa74c52d8f6f43d7ab61ae500fec62d4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:18.579712 14190 raft_consensus.cc:686] T fa74c52d8f6f43d7ab61ae500fec62d4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Becoming Leader. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Running, Role: LEADER
I20240503 02:00:18.579886 14190 consensus_queue.cc:237] T fa74c52d8f6f43d7ab61ae500fec62d4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:18.582201 6312 catalog_manager.cc:5484] T fa74c52d8f6f43d7ab61ae500fec62d4 P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } }
W20240503 02:00:18.608253 6654 consensus_peers.cc:485] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:18.614535 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
W20240503 02:00:18.617149 6785 consensus_peers.cc:485] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:00:18.638736 14190 raft_consensus.cc:482] T 6338547c9784461d80cee2c69aaf905b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:18.638857 14190 raft_consensus.cc:504] T 6338547c9784461d80cee2c69aaf905b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:18.639264 14190 leader_election.cc:288] T 6338547c9784461d80cee2c69aaf905b P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:18.639680 9047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6338547c9784461d80cee2c69aaf905b" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:18.639683 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6338547c9784461d80cee2c69aaf905b" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:18.639822 9179 raft_consensus.cc:2454] T 6338547c9784461d80cee2c69aaf905b P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:18.639822 9047 raft_consensus.cc:2454] T 6338547c9784461d80cee2c69aaf905b P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:18.640201 6654 leader_election.cc:302] T 6338547c9784461d80cee2c69aaf905b P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99ead02915954b4fab23b033ad6d546b, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:18.640439 14190 raft_consensus.cc:2781] T 6338547c9784461d80cee2c69aaf905b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:18.640512 14190 raft_consensus.cc:482] T 6338547c9784461d80cee2c69aaf905b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:18.640559 14190 raft_consensus.cc:3037] T 6338547c9784461d80cee2c69aaf905b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:18.642122 14190 raft_consensus.cc:504] T 6338547c9784461d80cee2c69aaf905b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:18.642649 14190 leader_election.cc:288] T 6338547c9784461d80cee2c69aaf905b P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Requested vote from peers 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
W20240503 02:00:18.642692 6523 consensus_peers.cc:485] T d0d5dd42db3e4883af7fd16d8813b550 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:00:18.643152 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6338547c9784461d80cee2c69aaf905b" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:18.643291 9047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6338547c9784461d80cee2c69aaf905b" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:18.643291 9179 raft_consensus.cc:3037] T 6338547c9784461d80cee2c69aaf905b P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:18.643397 9047 raft_consensus.cc:3037] T 6338547c9784461d80cee2c69aaf905b P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:18.644204 9047 raft_consensus.cc:2454] T 6338547c9784461d80cee2c69aaf905b P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:18.644203 9179 raft_consensus.cc:2454] T 6338547c9784461d80cee2c69aaf905b P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:18.644635 6654 leader_election.cc:302] T 6338547c9784461d80cee2c69aaf905b P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99ead02915954b4fab23b033ad6d546b, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:18.644841 14190 raft_consensus.cc:2781] T 6338547c9784461d80cee2c69aaf905b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:18.644915 14190 raft_consensus.cc:686] T 6338547c9784461d80cee2c69aaf905b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Becoming Leader. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Running, Role: LEADER
I20240503 02:00:18.645083 14190 consensus_queue.cc:237] T 6338547c9784461d80cee2c69aaf905b P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:18.647436 6312 catalog_manager.cc:5484] T 6338547c9784461d80cee2c69aaf905b P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:18.662559 14202 raft_consensus.cc:482] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:18.662711 14202 raft_consensus.cc:504] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:18.663384 14202 leader_election.cc:288] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
W20240503 02:00:18.663486 8851 consensus_peers.cc:485] T 067f54011a37432d99157884e70fc601 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:00:18.663758 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f191e65b61c499fb0a821ec9d5488a1" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 02:00:18.663879 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f191e65b61c499fb0a821ec9d5488a1" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:18.663913 6458 raft_consensus.cc:2454] T 4f191e65b61c499fb0a821ec9d5488a1 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:18.664016 9179 raft_consensus.cc:2454] T 4f191e65b61c499fb0a821ec9d5488a1 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:18.664312 6522 leader_election.cc:302] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:18.664511 14202 raft_consensus.cc:2781] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:18.664568 14202 raft_consensus.cc:482] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:18.664610 14202 raft_consensus.cc:3037] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:18.665505 14202 raft_consensus.cc:504] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:18.665865 14202 leader_election.cc:288] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Requested vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:18.666237 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f191e65b61c499fb0a821ec9d5488a1" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 02:00:18.666319 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f191e65b61c499fb0a821ec9d5488a1" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:18.666373 6458 raft_consensus.cc:3037] T 4f191e65b61c499fb0a821ec9d5488a1 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:18.666450 9179 raft_consensus.cc:3037] T 4f191e65b61c499fb0a821ec9d5488a1 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:18.667414 9179 raft_consensus.cc:2454] T 4f191e65b61c499fb0a821ec9d5488a1 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:18.667414 6458 raft_consensus.cc:2454] T 4f191e65b61c499fb0a821ec9d5488a1 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:18.667867 6522 leader_election.cc:302] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 80fd18ef4d604c86b909d5c008a46b4f; no voters:
I20240503 02:00:18.668176 14202 raft_consensus.cc:2781] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:18.668336 14202 raft_consensus.cc:686] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Becoming Leader. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Running, Role: LEADER
I20240503 02:00:18.668527 14202 consensus_queue.cc:237] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:18.671768 6312 catalog_manager.cc:5484] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } }
W20240503 02:00:18.686398 9113 consensus_peers.cc:485] T f44b59277c3b47adbd4dd6aff9350af1 P 2a01ace80baf4057b6bf1995cf88468b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:00:18.767609 14195 raft_consensus.cc:482] T 1029229ea6924a31b637639a95a04f9e P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:18.767753 14195 raft_consensus.cc:504] T 1029229ea6924a31b637639a95a04f9e P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:18.768173 14195 leader_election.cc:288] T 1029229ea6924a31b637639a95a04f9e P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 02:00:18.768561 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1029229ea6924a31b637639a95a04f9e" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 02:00:18.768709 6458 raft_consensus.cc:2454] T 1029229ea6924a31b637639a95a04f9e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:18.768733 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1029229ea6924a31b637639a95a04f9e" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:18.768847 8916 raft_consensus.cc:2454] T 1029229ea6924a31b637639a95a04f9e P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:18.769045 8981 leader_election.cc:302] T 1029229ea6924a31b637639a95a04f9e P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99ead02915954b4fab23b033ad6d546b, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:18.769284 14195 raft_consensus.cc:2781] T 1029229ea6924a31b637639a95a04f9e P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:18.769332 14195 raft_consensus.cc:482] T 1029229ea6924a31b637639a95a04f9e P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:18.769356 14195 raft_consensus.cc:3037] T 1029229ea6924a31b637639a95a04f9e P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:18.770248 14195 raft_consensus.cc:504] T 1029229ea6924a31b637639a95a04f9e P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:18.770633 14195 leader_election.cc:288] T 1029229ea6924a31b637639a95a04f9e P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Requested vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 02:00:18.770973 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1029229ea6924a31b637639a95a04f9e" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:18.771034 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1029229ea6924a31b637639a95a04f9e" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 02:00:18.771109 8916 raft_consensus.cc:3037] T 1029229ea6924a31b637639a95a04f9e P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:18.771157 6458 raft_consensus.cc:3037] T 1029229ea6924a31b637639a95a04f9e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:18.772141 8916 raft_consensus.cc:2454] T 1029229ea6924a31b637639a95a04f9e P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:18.772141 6458 raft_consensus.cc:2454] T 1029229ea6924a31b637639a95a04f9e P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:18.772636 8984 leader_election.cc:302] T 1029229ea6924a31b637639a95a04f9e P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9942306855404cfd891adf227dd65eb2, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:18.772833 14195 raft_consensus.cc:2781] T 1029229ea6924a31b637639a95a04f9e P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:18.772902 14195 raft_consensus.cc:686] T 1029229ea6924a31b637639a95a04f9e P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Becoming Leader. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Running, Role: LEADER
I20240503 02:00:18.773051 14195 consensus_queue.cc:237] T 1029229ea6924a31b637639a95a04f9e P 99ead02915954b4fab23b033ad6d546b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:18.778419 6312 catalog_manager.cc:5484] T 1029229ea6924a31b637639a95a04f9e P 99ead02915954b4fab23b033ad6d546b reported cstate change: term changed from 0 to 1, leader changed from <none> to 99ead02915954b4fab23b033ad6d546b (127.0.99.134). New cstate: current_term: 1 leader_uuid: "99ead02915954b4fab23b033ad6d546b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:18.820266 6312 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:18.824810 9179 raft_consensus.cc:1261] T 4f191e65b61c499fb0a821ec9d5488a1 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:18.825474 6456 raft_consensus.cc:1261] T 4f191e65b61c499fb0a821ec9d5488a1 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:18.825428 13954 consensus_queue.cc:1035] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:18.826009 13954 consensus_queue.cc:1035] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:18.828787 14227 tablet.cc:1722] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:18.829227 6308 catalog_manager.cc:5576] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079) does not have the latest schema for tablet 4f191e65b61c499fb0a821ec9d5488a1 (table rebalancer_dynamic_table [id=f2514b38a37b435aaaba4bc751e7ee6e]). Expected version 1 got 0
I20240503 02:00:18.830174 14193 mvcc.cc:205] Tried to move back new op lower bound from 7023417830702645248 to 7023417830068908032. Current Snapshot: MvccSnapshot[applied={T|T < 7023417830702645248}]
I20240503 02:00:18.830931 14227 tablet.cc:1620] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f: MemRowSet was empty: no flush needed.
I20240503 02:00:18.831017 6308 catalog_manager.cc:5576] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093) does not have the latest schema for tablet 4f191e65b61c499fb0a821ec9d5488a1 (table rebalancer_dynamic_table [id=f2514b38a37b435aaaba4bc751e7ee6e]). Expected version 1 got 0
I20240503 02:00:18.831921 6312 catalog_manager.cc:5576] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863) does not have the latest schema for tablet 4f191e65b61c499fb0a821ec9d5488a1 (table rebalancer_dynamic_table [id=f2514b38a37b435aaaba4bc751e7ee6e]). Expected version 1 got 0
I20240503 02:00:18.832237 6314 catalog_manager.cc:5576] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079) does not have the latest schema for tablet 4f191e65b61c499fb0a821ec9d5488a1 (table rebalancer_dynamic_table [id=f2514b38a37b435aaaba4bc751e7ee6e]). Expected version 1 got 0
I20240503 02:00:18.830353 14229 tablet.cc:1722] T 4f191e65b61c499fb0a821ec9d5488a1 P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:18.832607 14229 tablet.cc:1620] T 4f191e65b61c499fb0a821ec9d5488a1 P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
I20240503 02:00:18.833527 14226 mvcc.cc:205] Tried to move back new op lower bound from 7023417830702645248 to 7023417830068908032. Current Snapshot: MvccSnapshot[applied={T|T < 7023417830702645248}]
W20240503 02:00:18.835520 8851 consensus_peers.cc:485] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:00:18.837497 6312 catalog_manager.cc:5576] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093) does not have the latest schema for tablet 4f191e65b61c499fb0a821ec9d5488a1 (table rebalancer_dynamic_table [id=f2514b38a37b435aaaba4bc751e7ee6e]). Expected version 1 got 0
I20240503 02:00:18.837746 6314 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=f2514b38a37b435aaaba4bc751e7ee6e] alter complete (version 1)
W20240503 02:00:18.838284 6523 consensus_peers.cc:485] T 4385d51d11574192883d01376e2d5620 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:00:18.839368 14231 tablet.cc:1722] T 4f191e65b61c499fb0a821ec9d5488a1 P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:18.839756 14231 tablet.cc:1620] T 4f191e65b61c499fb0a821ec9d5488a1 P dec3e4a04d604f7eaaa3f077d081218a: MemRowSet was empty: no flush needed.
W20240503 02:00:18.908777 8982 consensus_peers.cc:485] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:18.933183 6523 consensus_peers.cc:485] T 925612b660ec471da5b3c548802ede6e P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:00:18.941829 6314 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:18.948406 14227 tablet.cc:1722] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:18.948787 14227 tablet.cc:1620] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f: MemRowSet was empty: no flush needed.
I20240503 02:00:18.949031 14229 tablet.cc:1722] T 4f191e65b61c499fb0a821ec9d5488a1 P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:18.949334 14229 tablet.cc:1620] T 4f191e65b61c499fb0a821ec9d5488a1 P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
W20240503 02:00:18.951365 6785 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240503 02:00:18.951395 8851 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111) [suppressed 48 similar messages]
I20240503 02:00:18.951395 14231 tablet.cc:1722] T 4f191e65b61c499fb0a821ec9d5488a1 P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:18.951746 14231 tablet.cc:1620] T 4f191e65b61c499fb0a821ec9d5488a1 P dec3e4a04d604f7eaaa3f077d081218a: MemRowSet was empty: no flush needed.
I20240503 02:00:18.955374 6314 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=f2514b38a37b435aaaba4bc751e7ee6e] alter complete (version 2)
W20240503 02:00:18.958595 6785 consensus_peers.cc:485] T 8246488dec204bf6b09e868d7951ac5a P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:00:18.962817 13969 consensus_queue.cc:1035] T fa74c52d8f6f43d7ab61ae500fec62d4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:18.973165 13969 consensus_queue.cc:1035] T fa74c52d8f6f43d7ab61ae500fec62d4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:00:18.999888 6785 consensus_peers.cc:485] T 46ad540deb624a25b97d3b967fe1f6f4 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:19.015749 8982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 02:00:19.016360 8982 consensus_peers.cc:485] T 13756a6e44814fe4b09a6b0f087546c3 P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:19.024590 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
W20240503 02:00:19.025542 6654 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 02:00:19.026162 6654 consensus_peers.cc:485] T cea1ea50fa434d31b42451f84f26bc76 P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:19.048748 6785 consensus_peers.cc:485] T d897c21e630d4770832bb10f43778e82 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:00:19.057284 6308 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:19.062883 14227 tablet.cc:1722] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:19.063702 14229 tablet.cc:1722] T 4f191e65b61c499fb0a821ec9d5488a1 P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:19.064074 14231 tablet.cc:1722] T 4f191e65b61c499fb0a821ec9d5488a1 P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:19.069939 6314 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=f2514b38a37b435aaaba4bc751e7ee6e] alter complete (version 3)
I20240503 02:00:19.073679 14190 consensus_queue.cc:1035] T 6338547c9784461d80cee2c69aaf905b P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:19.077131 14190 consensus_queue.cc:1035] T 6338547c9784461d80cee2c69aaf905b P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:00:19.081373 6523 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240503 02:00:19.166998 13943 consensus_queue.cc:1035] T 1029229ea6924a31b637639a95a04f9e P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:19.170635 13900 consensus_queue.cc:1035] T 1029229ea6924a31b637639a95a04f9e P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:19.173913 6308 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:19.174085 6308 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
W20240503 02:00:19.174924 8851 consensus_peers.cc:485] T 5f51fe5f799f4e479b62f5917a2c2606 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:00:19.177659 6308 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet 4f191e65b61c499fb0a821ec9d5488a1
I20240503 02:00:19.178231 6436 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f191e65b61c499fb0a821ec9d5488a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:19 UTC) from {username='slave'} at 127.0.0.1:51338
I20240503 02:00:19.178422 6567 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f191e65b61c499fb0a821ec9d5488a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:19 UTC) from {username='slave'} at 127.0.0.1:59846
I20240503 02:00:19.178611 9157 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f191e65b61c499fb0a821ec9d5488a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:19 UTC) from {username='slave'} at 127.0.0.1:51940
I20240503 02:00:19.179023 14244 tablet_replica.cc:330] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 02:00:19.179211 14244 raft_consensus.cc:2229] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:19.179479 14244 raft_consensus.cc:2258] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:19.180162 14245 tablet_replica.cc:330] T 4f191e65b61c499fb0a821ec9d5488a1 P 2a01ace80baf4057b6bf1995cf88468b: stopping tablet replica
I20240503 02:00:19.180562 14245 raft_consensus.cc:2229] T 4f191e65b61c499fb0a821ec9d5488a1 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:19.180693 14244 ts_tablet_manager.cc:1875] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:19.180717 14245 raft_consensus.cc:2258] T 4f191e65b61c499fb0a821ec9d5488a1 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:19.181284 14245 ts_tablet_manager.cc:1875] T 4f191e65b61c499fb0a821ec9d5488a1 P 2a01ace80baf4057b6bf1995cf88468b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:19.183243 14244 ts_tablet_manager.cc:1888] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:19.183391 14244 log.cc:1196] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/4f191e65b61c499fb0a821ec9d5488a1
I20240503 02:00:19.183859 14244 ts_tablet_manager.cc:1909] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting consensus metadata
I20240503 02:00:19.184068 14245 ts_tablet_manager.cc:1888] T 4f191e65b61c499fb0a821ec9d5488a1 P 2a01ace80baf4057b6bf1995cf88468b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:19.184238 14245 log.cc:1196] T 4f191e65b61c499fb0a821ec9d5488a1 P 2a01ace80baf4057b6bf1995cf88468b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/4f191e65b61c499fb0a821ec9d5488a1
I20240503 02:00:19.184818 14245 ts_tablet_manager.cc:1909] T 4f191e65b61c499fb0a821ec9d5488a1 P 2a01ace80baf4057b6bf1995cf88468b: Deleting consensus metadata
I20240503 02:00:19.184932 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet 4f191e65b61c499fb0a821ec9d5488a1 (table rebalancer_dynamic_table_new_name [id=f2514b38a37b435aaaba4bc751e7ee6e]) successfully deleted
I20240503 02:00:19.185681 6300 catalog_manager.cc:4830] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079): tablet 4f191e65b61c499fb0a821ec9d5488a1 (table rebalancer_dynamic_table_new_name [id=f2514b38a37b435aaaba4bc751e7ee6e]) successfully deleted
I20240503 02:00:19.188797 14243 tablet_replica.cc:330] T 4f191e65b61c499fb0a821ec9d5488a1 P dec3e4a04d604f7eaaa3f077d081218a: stopping tablet replica
I20240503 02:00:19.188964 14243 raft_consensus.cc:2229] T 4f191e65b61c499fb0a821ec9d5488a1 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:19.189095 14243 raft_consensus.cc:2258] T 4f191e65b61c499fb0a821ec9d5488a1 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:19.189783 14243 ts_tablet_manager.cc:1875] T 4f191e65b61c499fb0a821ec9d5488a1 P dec3e4a04d604f7eaaa3f077d081218a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:19.192313 14243 ts_tablet_manager.cc:1888] T 4f191e65b61c499fb0a821ec9d5488a1 P dec3e4a04d604f7eaaa3f077d081218a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:19.192427 14243 log.cc:1196] T 4f191e65b61c499fb0a821ec9d5488a1 P dec3e4a04d604f7eaaa3f077d081218a: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/4f191e65b61c499fb0a821ec9d5488a1
I20240503 02:00:19.192723 14243 ts_tablet_manager.cc:1909] T 4f191e65b61c499fb0a821ec9d5488a1 P dec3e4a04d604f7eaaa3f077d081218a: Deleting consensus metadata
I20240503 02:00:19.193389 6300 catalog_manager.cc:4830] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): tablet 4f191e65b61c499fb0a821ec9d5488a1 (table rebalancer_dynamic_table_new_name [id=f2514b38a37b435aaaba4bc751e7ee6e]) successfully deleted
W20240503 02:00:19.269558 6523 consensus_peers.cc:485] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:00:19.280041 6312 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
W20240503 02:00:19.281942 9113 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240503 02:00:19.285883 6829 tablet_service.cc:1467] Processing CreateTablet for tablet 32dd15923a55417c97d54822f930bbdd (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d6007f227ea147b58a002001972f41cf]), partition=
I20240503 02:00:19.286185 6829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32dd15923a55417c97d54822f930bbdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:19.286437 8880 tablet_service.cc:1467] Processing CreateTablet for tablet 32dd15923a55417c97d54822f930bbdd (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d6007f227ea147b58a002001972f41cf]), partition=
I20240503 02:00:19.286655 8880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32dd15923a55417c97d54822f930bbdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:19.288781 14247 tablet_bootstrap.cc:492] T 32dd15923a55417c97d54822f930bbdd P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:19.289300 6436 tablet_service.cc:1467] Processing CreateTablet for tablet 32dd15923a55417c97d54822f930bbdd (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d6007f227ea147b58a002001972f41cf]), partition=
I20240503 02:00:19.289487 14246 tablet_bootstrap.cc:492] T 32dd15923a55417c97d54822f930bbdd P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:19.289620 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32dd15923a55417c97d54822f930bbdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:19.290423 14246 tablet_bootstrap.cc:654] T 32dd15923a55417c97d54822f930bbdd P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:19.291646 14246 tablet_bootstrap.cc:492] T 32dd15923a55417c97d54822f930bbdd P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:19.291738 14246 ts_tablet_manager.cc:1367] T 32dd15923a55417c97d54822f930bbdd P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:19.292147 14249 tablet_bootstrap.cc:492] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:19.293124 14249 tablet_bootstrap.cc:654] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
W20240503 02:00:19.293432 8851 consensus_peers.cc:485] T 7da8f614a70e4177b70e7c2f31d22a10 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:00:19.294420 14246 raft_consensus.cc:348] T 32dd15923a55417c97d54822f930bbdd P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:19.294574 14246 raft_consensus.cc:374] T 32dd15923a55417c97d54822f930bbdd P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:19.294586 14247 tablet_bootstrap.cc:654] T 32dd15923a55417c97d54822f930bbdd P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:19.294623 14246 raft_consensus.cc:729] T 32dd15923a55417c97d54822f930bbdd P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:19.295286 14246 consensus_queue.cc:260] T 32dd15923a55417c97d54822f930bbdd P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:19.295661 14246 ts_tablet_manager.cc:1398] T 32dd15923a55417c97d54822f930bbdd P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:19.295980 14247 tablet_bootstrap.cc:492] T 32dd15923a55417c97d54822f930bbdd P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:19.296092 14247 ts_tablet_manager.cc:1367] T 32dd15923a55417c97d54822f930bbdd P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:00:19.296911 14247 raft_consensus.cc:348] T 32dd15923a55417c97d54822f930bbdd P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:19.297039 14247 raft_consensus.cc:374] T 32dd15923a55417c97d54822f930bbdd P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:19.297080 14247 raft_consensus.cc:729] T 32dd15923a55417c97d54822f930bbdd P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:19.298473 14249 tablet_bootstrap.cc:492] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:19.298583 14249 ts_tablet_manager.cc:1367] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:00:19.298561 14247 consensus_queue.cc:260] T 32dd15923a55417c97d54822f930bbdd P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:19.298878 14247 ts_tablet_manager.cc:1398] T 32dd15923a55417c97d54822f930bbdd P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:00:19.299067 14249 raft_consensus.cc:348] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:19.299163 14249 raft_consensus.cc:374] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:19.299192 14249 raft_consensus.cc:729] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:19.299309 14249 consensus_queue.cc:260] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:19.299537 14249 ts_tablet_manager.cc:1398] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:19.317831 6308 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_extra_table_50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:19.325495 6699 tablet_service.cc:1467] Processing CreateTablet for tablet 2b7a65bae2c5459c9a378b80f2255ed3 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=2490acf99a114d9296316c67e2418726]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:19.325495 6436 tablet_service.cc:1467] Processing CreateTablet for tablet 2b7a65bae2c5459c9a378b80f2255ed3 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=2490acf99a114d9296316c67e2418726]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:19.325908 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b7a65bae2c5459c9a378b80f2255ed3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:19.325908 6699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b7a65bae2c5459c9a378b80f2255ed3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:19.326144 6829 tablet_service.cc:1467] Processing CreateTablet for tablet 5d8e8972c10241108d716d92a38f4c17 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=2490acf99a114d9296316c67e2418726]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:19.326144 6831 tablet_service.cc:1467] Processing CreateTablet for tablet 72aea5be835849d9a392fbab35357b7b (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=2490acf99a114d9296316c67e2418726]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:19.326442 6829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d8e8972c10241108d716d92a38f4c17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:19.326911 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72aea5be835849d9a392fbab35357b7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:19.328522 6700 tablet_service.cc:1467] Processing CreateTablet for tablet 5d8e8972c10241108d716d92a38f4c17 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=2490acf99a114d9296316c67e2418726]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:19.328783 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d8e8972c10241108d716d92a38f4c17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:19.328806 14249 tablet_bootstrap.cc:492] T 2b7a65bae2c5459c9a378b80f2255ed3 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:19.329066 14246 tablet_bootstrap.cc:492] T 5d8e8972c10241108d716d92a38f4c17 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:19.329788 14249 tablet_bootstrap.cc:654] T 2b7a65bae2c5459c9a378b80f2255ed3 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:19.330045 14246 tablet_bootstrap.cc:654] T 5d8e8972c10241108d716d92a38f4c17 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:19.331267 9027 tablet_service.cc:1467] Processing CreateTablet for tablet 2b7a65bae2c5459c9a378b80f2255ed3 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=2490acf99a114d9296316c67e2418726]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:19.331564 9027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b7a65bae2c5459c9a378b80f2255ed3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:19.331585 14246 tablet_bootstrap.cc:492] T 5d8e8972c10241108d716d92a38f4c17 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:19.331606 14252 tablet_bootstrap.cc:492] T 2b7a65bae2c5459c9a378b80f2255ed3 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:19.331662 14246 ts_tablet_manager.cc:1367] T 5d8e8972c10241108d716d92a38f4c17 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:19.332152 14246 raft_consensus.cc:348] T 5d8e8972c10241108d716d92a38f4c17 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:19.332259 14246 raft_consensus.cc:374] T 5d8e8972c10241108d716d92a38f4c17 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:19.332285 14246 raft_consensus.cc:729] T 5d8e8972c10241108d716d92a38f4c17 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:19.332422 14246 consensus_queue.cc:260] T 5d8e8972c10241108d716d92a38f4c17 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:19.332460 14252 tablet_bootstrap.cc:654] T 2b7a65bae2c5459c9a378b80f2255ed3 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:19.333187 14249 tablet_bootstrap.cc:492] T 2b7a65bae2c5459c9a378b80f2255ed3 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:19.333283 14249 ts_tablet_manager.cc:1367] T 2b7a65bae2c5459c9a378b80f2255ed3 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:00:19.333577 14252 tablet_bootstrap.cc:492] T 2b7a65bae2c5459c9a378b80f2255ed3 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:19.333657 14252 ts_tablet_manager.cc:1367] T 2b7a65bae2c5459c9a378b80f2255ed3 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:00:19.333679 14246 ts_tablet_manager.cc:1398] T 5d8e8972c10241108d716d92a38f4c17 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:19.333768 14249 raft_consensus.cc:348] T 2b7a65bae2c5459c9a378b80f2255ed3 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:19.333873 14249 raft_consensus.cc:374] T 2b7a65bae2c5459c9a378b80f2255ed3 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:19.333884 14246 tablet_bootstrap.cc:492] T 72aea5be835849d9a392fbab35357b7b P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:19.333904 14249 raft_consensus.cc:729] T 2b7a65bae2c5459c9a378b80f2255ed3 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:19.334038 14249 consensus_queue.cc:260] T 2b7a65bae2c5459c9a378b80f2255ed3 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:19.334082 14252 raft_consensus.cc:348] T 2b7a65bae2c5459c9a378b80f2255ed3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:19.334169 14252 raft_consensus.cc:374] T 2b7a65bae2c5459c9a378b80f2255ed3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:19.334195 14252 raft_consensus.cc:729] T 2b7a65bae2c5459c9a378b80f2255ed3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:19.334323 14249 ts_tablet_manager.cc:1398] T 2b7a65bae2c5459c9a378b80f2255ed3 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:19.334300 14252 consensus_queue.cc:260] T 2b7a65bae2c5459c9a378b80f2255ed3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:19.334544 14252 ts_tablet_manager.cc:1398] T 2b7a65bae2c5459c9a378b80f2255ed3 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:19.334698 14252 tablet_bootstrap.cc:492] T 5d8e8972c10241108d716d92a38f4c17 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:19.334900 9157 tablet_service.cc:1467] Processing CreateTablet for tablet 72aea5be835849d9a392fbab35357b7b (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=2490acf99a114d9296316c67e2418726]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:19.335175 9157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72aea5be835849d9a392fbab35357b7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:19.335111 6567 tablet_service.cc:1467] Processing CreateTablet for tablet 5d8e8972c10241108d716d92a38f4c17 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=2490acf99a114d9296316c67e2418726]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:19.335407 6567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d8e8972c10241108d716d92a38f4c17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:19.335568 14246 tablet_bootstrap.cc:654] T 72aea5be835849d9a392fbab35357b7b P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:19.335574 14252 tablet_bootstrap.cc:654] T 5d8e8972c10241108d716d92a38f4c17 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:19.336524 14256 tablet_bootstrap.cc:492] T 2b7a65bae2c5459c9a378b80f2255ed3 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:19.337108 14246 tablet_bootstrap.cc:492] T 72aea5be835849d9a392fbab35357b7b P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:19.337266 14246 ts_tablet_manager.cc:1367] T 72aea5be835849d9a392fbab35357b7b P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:19.337795 14246 raft_consensus.cc:348] T 72aea5be835849d9a392fbab35357b7b P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:19.337936 14246 raft_consensus.cc:374] T 72aea5be835849d9a392fbab35357b7b P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:19.337944 14260 tablet_bootstrap.cc:492] T 5d8e8972c10241108d716d92a38f4c17 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:19.337973 14246 raft_consensus.cc:729] T 72aea5be835849d9a392fbab35357b7b P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:19.338162 14246 consensus_queue.cc:260] T 72aea5be835849d9a392fbab35357b7b P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:19.338452 14246 ts_tablet_manager.cc:1398] T 72aea5be835849d9a392fbab35357b7b P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:19.338666 14259 tablet_bootstrap.cc:492] T 72aea5be835849d9a392fbab35357b7b P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:19.339803 14256 tablet_bootstrap.cc:654] T 2b7a65bae2c5459c9a378b80f2255ed3 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:19.339812 6700 tablet_service.cc:1467] Processing CreateTablet for tablet 72aea5be835849d9a392fbab35357b7b (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=2490acf99a114d9296316c67e2418726]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:19.340036 14259 tablet_bootstrap.cc:654] T 72aea5be835849d9a392fbab35357b7b P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:19.340085 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72aea5be835849d9a392fbab35357b7b. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:00:19.342213 9113 consensus_peers.cc:485] T b1740b877c1e4372a0337d99f5bcab0e P 2a01ace80baf4057b6bf1995cf88468b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:00:19.342859 14252 tablet_bootstrap.cc:492] T 5d8e8972c10241108d716d92a38f4c17 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:19.342952 14252 ts_tablet_manager.cc:1367] T 5d8e8972c10241108d716d92a38f4c17 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.001s
I20240503 02:00:19.343437 14252 raft_consensus.cc:348] T 5d8e8972c10241108d716d92a38f4c17 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:19.343562 14252 raft_consensus.cc:374] T 5d8e8972c10241108d716d92a38f4c17 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:19.343606 14252 raft_consensus.cc:729] T 5d8e8972c10241108d716d92a38f4c17 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:19.343732 14252 consensus_queue.cc:260] T 5d8e8972c10241108d716d92a38f4c17 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:19.343995 14252 ts_tablet_manager.cc:1398] T 5d8e8972c10241108d716d92a38f4c17 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:19.344139 14252 tablet_bootstrap.cc:492] T 72aea5be835849d9a392fbab35357b7b P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:19.344528 14256 tablet_bootstrap.cc:492] T 2b7a65bae2c5459c9a378b80f2255ed3 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:19.344544 14056 raft_consensus.cc:482] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:19.344642 14056 raft_consensus.cc:504] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:19.344774 14256 ts_tablet_manager.cc:1367] T 2b7a65bae2c5459c9a378b80f2255ed3 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 02:00:19.345108 14252 tablet_bootstrap.cc:654] T 72aea5be835849d9a392fbab35357b7b P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:19.345160 14056 leader_election.cc:288] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:19.345634 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32dd15923a55417c97d54822f930bbdd" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 02:00:19.345759 6849 raft_consensus.cc:2454] T 32dd15923a55417c97d54822f930bbdd P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:19.346074 6394 leader_election.cc:302] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:19.346295 8917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32dd15923a55417c97d54822f930bbdd" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:19.346355 14056 raft_consensus.cc:2781] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:19.346410 14056 raft_consensus.cc:482] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:19.346444 14056 raft_consensus.cc:3037] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:19.346427 8917 raft_consensus.cc:2454] T 32dd15923a55417c97d54822f930bbdd P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:19.346839 14260 tablet_bootstrap.cc:654] T 5d8e8972c10241108d716d92a38f4c17 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:19.347543 14056 raft_consensus.cc:504] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:19.347904 14260 tablet_bootstrap.cc:492] T 5d8e8972c10241108d716d92a38f4c17 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:19.347906 14056 leader_election.cc:288] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Requested vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:19.347996 14260 ts_tablet_manager.cc:1367] T 5d8e8972c10241108d716d92a38f4c17 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20240503 02:00:19.348526 14252 tablet_bootstrap.cc:492] T 72aea5be835849d9a392fbab35357b7b P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:19.348489 8917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32dd15923a55417c97d54822f930bbdd" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:19.348481 14260 raft_consensus.cc:348] T 5d8e8972c10241108d716d92a38f4c17 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:19.348583 14260 raft_consensus.cc:374] T 5d8e8972c10241108d716d92a38f4c17 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:19.348613 14260 raft_consensus.cc:729] T 5d8e8972c10241108d716d92a38f4c17 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:19.348614 14252 ts_tablet_manager.cc:1367] T 72aea5be835849d9a392fbab35357b7b P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:00:19.348608 8917 raft_consensus.cc:3037] T 32dd15923a55417c97d54822f930bbdd P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:19.348733 14260 consensus_queue.cc:260] T 5d8e8972c10241108d716d92a38f4c17 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:19.348737 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32dd15923a55417c97d54822f930bbdd" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 02:00:19.348861 6849 raft_consensus.cc:3037] T 32dd15923a55417c97d54822f930bbdd P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:19.349002 14260 ts_tablet_manager.cc:1398] T 5d8e8972c10241108d716d92a38f4c17 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:19.349157 14252 raft_consensus.cc:348] T 72aea5be835849d9a392fbab35357b7b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:19.349264 14252 raft_consensus.cc:374] T 72aea5be835849d9a392fbab35357b7b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:19.349290 14252 raft_consensus.cc:729] T 72aea5be835849d9a392fbab35357b7b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:19.349401 14252 consensus_queue.cc:260] T 72aea5be835849d9a392fbab35357b7b P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:19.349618 8917 raft_consensus.cc:2454] T 32dd15923a55417c97d54822f930bbdd P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:19.349664 14252 ts_tablet_manager.cc:1398] T 72aea5be835849d9a392fbab35357b7b P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:19.349859 6849 raft_consensus.cc:2454] T 32dd15923a55417c97d54822f930bbdd P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:19.350246 14259 tablet_bootstrap.cc:492] T 72aea5be835849d9a392fbab35357b7b P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:19.350270 6394 leader_election.cc:302] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:19.350335 14259 ts_tablet_manager.cc:1367] T 72aea5be835849d9a392fbab35357b7b P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 02:00:19.350944 14259 raft_consensus.cc:348] T 72aea5be835849d9a392fbab35357b7b P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:19.351058 14259 raft_consensus.cc:374] T 72aea5be835849d9a392fbab35357b7b P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:19.351123 14259 raft_consensus.cc:729] T 72aea5be835849d9a392fbab35357b7b P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:19.351171 14056 raft_consensus.cc:2781] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:19.351272 14056 raft_consensus.cc:686] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Becoming Leader. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Running, Role: LEADER
I20240503 02:00:19.351311 14259 consensus_queue.cc:260] T 72aea5be835849d9a392fbab35357b7b P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:19.351409 14056 consensus_queue.cc:237] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:19.351742 14259 ts_tablet_manager.cc:1398] T 72aea5be835849d9a392fbab35357b7b P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 02:00:19.345325 14256 raft_consensus.cc:348] T 2b7a65bae2c5459c9a378b80f2255ed3 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:19.353992 14256 raft_consensus.cc:374] T 2b7a65bae2c5459c9a378b80f2255ed3 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:19.354110 14256 raft_consensus.cc:729] T 2b7a65bae2c5459c9a378b80f2255ed3 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:19.354321 14256 consensus_queue.cc:260] T 2b7a65bae2c5459c9a378b80f2255ed3 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:19.354540 6309 catalog_manager.cc:5484] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: term changed from 0 to 1, leader changed from <none> to dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129). New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:19.357532 14256 ts_tablet_manager.cc:1398] T 2b7a65bae2c5459c9a378b80f2255ed3 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.013s user 0.001s sys 0.000s
I20240503 02:00:19.360335 13954 raft_consensus.cc:482] T 5d8e8972c10241108d716d92a38f4c17 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:19.360497 13954 raft_consensus.cc:504] T 5d8e8972c10241108d716d92a38f4c17 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:19.360942 13954 leader_election.cc:288] T 5d8e8972c10241108d716d92a38f4c17 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:19.361353 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d8e8972c10241108d716d92a38f4c17" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 02:00:19.361403 6720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d8e8972c10241108d716d92a38f4c17" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 02:00:19.361503 6849 raft_consensus.cc:2454] T 5d8e8972c10241108d716d92a38f4c17 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:19.361567 6720 raft_consensus.cc:2454] T 5d8e8972c10241108d716d92a38f4c17 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:19.361788 6525 leader_election.cc:302] T 5d8e8972c10241108d716d92a38f4c17 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 80fd18ef4d604c86b909d5c008a46b4f; no voters:
I20240503 02:00:19.361963 13954 raft_consensus.cc:2781] T 5d8e8972c10241108d716d92a38f4c17 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:19.362023 13954 raft_consensus.cc:482] T 5d8e8972c10241108d716d92a38f4c17 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:19.362057 13954 raft_consensus.cc:3037] T 5d8e8972c10241108d716d92a38f4c17 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:19.362979 13954 raft_consensus.cc:504] T 5d8e8972c10241108d716d92a38f4c17 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:19.363327 13954 leader_election.cc:288] T 5d8e8972c10241108d716d92a38f4c17 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Requested vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:19.363795 6720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d8e8972c10241108d716d92a38f4c17" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 02:00:19.363770 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d8e8972c10241108d716d92a38f4c17" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 02:00:19.363915 6720 raft_consensus.cc:3037] T 5d8e8972c10241108d716d92a38f4c17 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:19.363924 6849 raft_consensus.cc:3037] T 5d8e8972c10241108d716d92a38f4c17 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:19.364768 6720 raft_consensus.cc:2454] T 5d8e8972c10241108d716d92a38f4c17 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:19.364768 6849 raft_consensus.cc:2454] T 5d8e8972c10241108d716d92a38f4c17 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:19.365231 6525 leader_election.cc:302] T 5d8e8972c10241108d716d92a38f4c17 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 80fd18ef4d604c86b909d5c008a46b4f; no voters:
I20240503 02:00:19.365620 13954 raft_consensus.cc:2781] T 5d8e8972c10241108d716d92a38f4c17 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:19.365731 13954 raft_consensus.cc:686] T 5d8e8972c10241108d716d92a38f4c17 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Becoming Leader. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Running, Role: LEADER
I20240503 02:00:19.365906 13954 consensus_queue.cc:237] T 5d8e8972c10241108d716d92a38f4c17 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:19.368669 6312 catalog_manager.cc:5484] T 5d8e8972c10241108d716d92a38f4c17 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } }
W20240503 02:00:19.400913 6301 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20240503 02:00:19.421371 8982 consensus_peers.cc:485] T beeea71f02ae4c89a1af13d3584ef70a P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:00:19.464830 13969 raft_consensus.cc:482] T 2b7a65bae2c5459c9a378b80f2255ed3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:19.464984 13969 raft_consensus.cc:504] T 2b7a65bae2c5459c9a378b80f2255ed3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:19.465502 13969 leader_election.cc:288] T 2b7a65bae2c5459c9a378b80f2255ed3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:19.465941 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b7a65bae2c5459c9a378b80f2255ed3" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 02:00:19.465958 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b7a65bae2c5459c9a378b80f2255ed3" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:19.466092 9048 raft_consensus.cc:2454] T 2b7a65bae2c5459c9a378b80f2255ed3 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:19.466104 6456 raft_consensus.cc:2454] T 2b7a65bae2c5459c9a378b80f2255ed3 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:19.466413 6653 leader_election.cc:302] T 2b7a65bae2c5459c9a378b80f2255ed3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2ac2eb7cc264047b9ab32fe3a4fc025, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:19.466588 13969 raft_consensus.cc:2781] T 2b7a65bae2c5459c9a378b80f2255ed3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:19.466668 13969 raft_consensus.cc:482] T 2b7a65bae2c5459c9a378b80f2255ed3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:19.466701 13969 raft_consensus.cc:3037] T 2b7a65bae2c5459c9a378b80f2255ed3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:19.467762 13969 raft_consensus.cc:504] T 2b7a65bae2c5459c9a378b80f2255ed3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:19.469170 6309 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:19.469205 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b7a65bae2c5459c9a378b80f2255ed3" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 02:00:19.469398 6456 raft_consensus.cc:3037] T 2b7a65bae2c5459c9a378b80f2255ed3 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:19.470297 6456 raft_consensus.cc:2454] T 2b7a65bae2c5459c9a378b80f2255ed3 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:19.468771 13969 leader_election.cc:288] T 2b7a65bae2c5459c9a378b80f2255ed3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Requested vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:19.470662 6653 leader_election.cc:302] T 2b7a65bae2c5459c9a378b80f2255ed3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2ac2eb7cc264047b9ab32fe3a4fc025, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:19.470815 13969 raft_consensus.cc:2781] T 2b7a65bae2c5459c9a378b80f2255ed3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:19.470940 13969 raft_consensus.cc:686] T 2b7a65bae2c5459c9a378b80f2255ed3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Becoming Leader. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Running, Role: LEADER
I20240503 02:00:19.471052 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b7a65bae2c5459c9a378b80f2255ed3" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:19.471172 13969 consensus_queue.cc:237] T 2b7a65bae2c5459c9a378b80f2255ed3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:19.471266 9048 raft_consensus.cc:3037] T 2b7a65bae2c5459c9a378b80f2255ed3 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:19.472852 9048 raft_consensus.cc:2454] T 2b7a65bae2c5459c9a378b80f2255ed3 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:19.473603 6849 raft_consensus.cc:1261] T 32dd15923a55417c97d54822f930bbdd P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:19.474016 8917 raft_consensus.cc:1261] T 32dd15923a55417c97d54822f930bbdd P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:19.474133 14056 consensus_queue.cc:1035] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:19.474762 14153 consensus_queue.cc:1035] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:19.475209 6309 catalog_manager.cc:5484] T 2b7a65bae2c5459c9a378b80f2255ed3 P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:19.478020 14231 tablet.cc:1722] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:19.478397 14231 tablet.cc:1620] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a: MemRowSet was empty: no flush needed.
I20240503 02:00:19.479257 6312 catalog_manager.cc:5576] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211) does not have the latest schema for tablet 32dd15923a55417c97d54822f930bbdd (table rebalancer_dynamic_table [id=d6007f227ea147b58a002001972f41cf]). Expected version 1 got 0
I20240503 02:00:19.480370 6314 catalog_manager.cc:5576] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719) does not have the latest schema for tablet 32dd15923a55417c97d54822f930bbdd (table rebalancer_dynamic_table [id=d6007f227ea147b58a002001972f41cf]). Expected version 1 got 0
I20240503 02:00:19.480705 14269 mvcc.cc:205] Tried to move back new op lower bound from 7023417833358508032 to 7023417832868155392. Current Snapshot: MvccSnapshot[applied={T|T < 7023417833358508032}]
I20240503 02:00:19.480943 14272 tablet.cc:1722] T 32dd15923a55417c97d54822f930bbdd P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:19.481245 14272 tablet.cc:1620] T 32dd15923a55417c97d54822f930bbdd P 492448570cc94ed1879feb2ec3b9051d: MemRowSet was empty: no flush needed.
I20240503 02:00:19.481793 14274 tablet.cc:1722] T 32dd15923a55417c97d54822f930bbdd P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:19.483021 14274 tablet.cc:1620] T 32dd15923a55417c97d54822f930bbdd P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
I20240503 02:00:19.486246 6314 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=d6007f227ea147b58a002001972f41cf] alter complete (version 1)
W20240503 02:00:19.526684 6523 consensus_peers.cc:485] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:19.538025 8851 consensus_peers.cc:485] T f28586e70bd9473fbdc6b9126b666fa9 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:19.542378 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:19.591318 6314 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:19.596755 14231 tablet.cc:1722] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:19.597190 14231 tablet.cc:1620] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a: MemRowSet was empty: no flush needed.
I20240503 02:00:19.600607 14272 tablet.cc:1722] T 32dd15923a55417c97d54822f930bbdd P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:19.600965 14272 tablet.cc:1620] T 32dd15923a55417c97d54822f930bbdd P 492448570cc94ed1879feb2ec3b9051d: MemRowSet was empty: no flush needed.
I20240503 02:00:19.603163 14274 tablet.cc:1722] T 32dd15923a55417c97d54822f930bbdd P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:19.603480 14274 tablet.cc:1620] T 32dd15923a55417c97d54822f930bbdd P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
I20240503 02:00:19.604059 6314 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=d6007f227ea147b58a002001972f41cf] alter complete (version 2)
W20240503 02:00:19.609004 8982 consensus_peers.cc:485] T 1a7ee79312ea4bad816264375ca5bb2d P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:00:19.613512 14106 raft_consensus.cc:482] T 72aea5be835849d9a392fbab35357b7b P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:19.613667 14106 raft_consensus.cc:504] T 72aea5be835849d9a392fbab35357b7b P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:19.614194 14106 leader_election.cc:288] T 72aea5be835849d9a392fbab35357b7b P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:19.614650 6720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72aea5be835849d9a392fbab35357b7b" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 02:00:19.614794 6720 raft_consensus.cc:2454] T 72aea5be835849d9a392fbab35357b7b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 02:00:19.614765 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72aea5be835849d9a392fbab35357b7b" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:19.614894 9178 raft_consensus.cc:2454] T 72aea5be835849d9a392fbab35357b7b P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 02:00:19.615077 6787 leader_election.cc:302] T 72aea5be835849d9a392fbab35357b7b P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:19.615397 14106 raft_consensus.cc:2781] T 72aea5be835849d9a392fbab35357b7b P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:19.615461 14106 raft_consensus.cc:482] T 72aea5be835849d9a392fbab35357b7b P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:19.615489 14106 raft_consensus.cc:3037] T 72aea5be835849d9a392fbab35357b7b P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:19.616408 14106 raft_consensus.cc:504] T 72aea5be835849d9a392fbab35357b7b P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:19.616772 14106 leader_election.cc:288] T 72aea5be835849d9a392fbab35357b7b P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Requested vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:19.617137 6720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72aea5be835849d9a392fbab35357b7b" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 02:00:19.617146 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72aea5be835849d9a392fbab35357b7b" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:19.617353 6720 raft_consensus.cc:3037] T 72aea5be835849d9a392fbab35357b7b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:19.617362 9178 raft_consensus.cc:3037] T 72aea5be835849d9a392fbab35357b7b P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:19.618392 9178 raft_consensus.cc:2454] T 72aea5be835849d9a392fbab35357b7b P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 02:00:19.618425 6720 raft_consensus.cc:2454] T 72aea5be835849d9a392fbab35357b7b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 02:00:19.618734 6787 leader_election.cc:302] T 72aea5be835849d9a392fbab35357b7b P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:19.618947 14106 raft_consensus.cc:2781] T 72aea5be835849d9a392fbab35357b7b P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:19.619140 14106 raft_consensus.cc:686] T 72aea5be835849d9a392fbab35357b7b P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Becoming Leader. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Running, Role: LEADER
I20240503 02:00:19.619290 14106 consensus_queue.cc:237] T 72aea5be835849d9a392fbab35357b7b P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:19.621780 6308 catalog_manager.cc:5484] T 72aea5be835849d9a392fbab35357b7b P 492448570cc94ed1879feb2ec3b9051d reported cstate change: term changed from 0 to 1, leader changed from <none> to 492448570cc94ed1879feb2ec3b9051d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:19.708534 6308 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:19.714170 14231 tablet.cc:1722] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:19.714623 14274 tablet.cc:1722] T 32dd15923a55417c97d54822f930bbdd P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:19.714658 14272 tablet.cc:1722] T 32dd15923a55417c97d54822f930bbdd P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:19.720556 6308 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=d6007f227ea147b58a002001972f41cf] alter complete (version 3)
I20240503 02:00:19.824115 6314 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:19.824278 6314 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:19.826346 6314 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet 32dd15923a55417c97d54822f930bbdd
I20240503 02:00:19.826916 8880 tablet_service.cc:1514] Processing DeleteTablet for tablet 32dd15923a55417c97d54822f930bbdd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:19 UTC) from {username='slave'} at 127.0.0.1:40296
I20240503 02:00:19.827025 6436 tablet_service.cc:1514] Processing DeleteTablet for tablet 32dd15923a55417c97d54822f930bbdd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:19 UTC) from {username='slave'} at 127.0.0.1:51338
I20240503 02:00:19.827304 14278 tablet_replica.cc:330] T 32dd15923a55417c97d54822f930bbdd P 9942306855404cfd891adf227dd65eb2: stopping tablet replica
I20240503 02:00:19.827278 6831 tablet_service.cc:1514] Processing DeleteTablet for tablet 32dd15923a55417c97d54822f930bbdd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:19 UTC) from {username='slave'} at 127.0.0.1:52812
I20240503 02:00:19.827358 14279 tablet_replica.cc:330] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a: stopping tablet replica
I20240503 02:00:19.827466 14278 raft_consensus.cc:2229] T 32dd15923a55417c97d54822f930bbdd P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:19.827491 14279 raft_consensus.cc:2229] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:19.827653 14280 tablet_replica.cc:330] T 32dd15923a55417c97d54822f930bbdd P 492448570cc94ed1879feb2ec3b9051d: stopping tablet replica
I20240503 02:00:19.827677 14278 raft_consensus.cc:2258] T 32dd15923a55417c97d54822f930bbdd P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:19.827787 14279 raft_consensus.cc:2258] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:19.827832 14280 raft_consensus.cc:2229] T 32dd15923a55417c97d54822f930bbdd P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:19.827963 14280 raft_consensus.cc:2258] T 32dd15923a55417c97d54822f930bbdd P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:19.828290 14278 ts_tablet_manager.cc:1875] T 32dd15923a55417c97d54822f930bbdd P 9942306855404cfd891adf227dd65eb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:19.828474 14279 ts_tablet_manager.cc:1875] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:19.828598 14280 ts_tablet_manager.cc:1875] T 32dd15923a55417c97d54822f930bbdd P 492448570cc94ed1879feb2ec3b9051d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:19.830646 14279 ts_tablet_manager.cc:1888] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:19.830734 14279 log.cc:1196] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/32dd15923a55417c97d54822f930bbdd
I20240503 02:00:19.830783 14280 ts_tablet_manager.cc:1888] T 32dd15923a55417c97d54822f930bbdd P 492448570cc94ed1879feb2ec3b9051d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:19.830828 14278 ts_tablet_manager.cc:1888] T 32dd15923a55417c97d54822f930bbdd P 9942306855404cfd891adf227dd65eb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:19.830896 14280 log.cc:1196] T 32dd15923a55417c97d54822f930bbdd P 492448570cc94ed1879feb2ec3b9051d: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/32dd15923a55417c97d54822f930bbdd
I20240503 02:00:19.830889 14278 log.cc:1196] T 32dd15923a55417c97d54822f930bbdd P 9942306855404cfd891adf227dd65eb2: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/32dd15923a55417c97d54822f930bbdd
I20240503 02:00:19.831140 14279 ts_tablet_manager.cc:1909] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a: Deleting consensus metadata
I20240503 02:00:19.831259 14280 ts_tablet_manager.cc:1909] T 32dd15923a55417c97d54822f930bbdd P 492448570cc94ed1879feb2ec3b9051d: Deleting consensus metadata
I20240503 02:00:19.831288 14278 ts_tablet_manager.cc:1909] T 32dd15923a55417c97d54822f930bbdd P 9942306855404cfd891adf227dd65eb2: Deleting consensus metadata
I20240503 02:00:19.832036 6303 catalog_manager.cc:4830] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719): tablet 32dd15923a55417c97d54822f930bbdd (table rebalancer_dynamic_table_new_name [id=d6007f227ea147b58a002001972f41cf]) successfully deleted
I20240503 02:00:19.832036 6300 catalog_manager.cc:4830] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): tablet 32dd15923a55417c97d54822f930bbdd (table rebalancer_dynamic_table_new_name [id=d6007f227ea147b58a002001972f41cf]) successfully deleted
I20240503 02:00:19.832249 6303 catalog_manager.cc:4830] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211): tablet 32dd15923a55417c97d54822f930bbdd (table rebalancer_dynamic_table_new_name [id=d6007f227ea147b58a002001972f41cf]) successfully deleted
I20240503 02:00:19.892845 14190 consensus_queue.cc:1035] T 2b7a65bae2c5459c9a378b80f2255ed3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:19.896586 13969 consensus_queue.cc:1035] T 2b7a65bae2c5459c9a378b80f2255ed3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:19.898124 14202 consensus_queue.cc:1035] T 5d8e8972c10241108d716d92a38f4c17 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:19.905500 14287 consensus_queue.cc:1035] T 5d8e8972c10241108d716d92a38f4c17 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:19.928005 6312 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:00:19.934177 9157 tablet_service.cc:1467] Processing CreateTablet for tablet 2ed12d2ca4c74194a99bd6e4f9c28834 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=78ad5e8d3227453fb447fb66f8d78a05]), partition=
I20240503 02:00:19.934177 6567 tablet_service.cc:1467] Processing CreateTablet for tablet 2ed12d2ca4c74194a99bd6e4f9c28834 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=78ad5e8d3227453fb447fb66f8d78a05]), partition=
I20240503 02:00:19.934473 6567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ed12d2ca4c74194a99bd6e4f9c28834. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:19.934473 9157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ed12d2ca4c74194a99bd6e4f9c28834. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:19.937086 6831 tablet_service.cc:1467] Processing CreateTablet for tablet 2ed12d2ca4c74194a99bd6e4f9c28834 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=78ad5e8d3227453fb447fb66f8d78a05]), partition=
I20240503 02:00:19.937325 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ed12d2ca4c74194a99bd6e4f9c28834. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:19.939164 14293 tablet_bootstrap.cc:492] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:19.939265 14291 tablet_bootstrap.cc:492] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:19.940114 14291 tablet_bootstrap.cc:654] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:19.940153 14293 tablet_bootstrap.cc:654] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:19.940867 14292 tablet_bootstrap.cc:492] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:19.941670 14293 tablet_bootstrap.cc:492] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:19.941834 14293 ts_tablet_manager.cc:1367] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:19.941895 14292 tablet_bootstrap.cc:654] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:19.942322 14293 raft_consensus.cc:348] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:19.942459 14293 raft_consensus.cc:374] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:19.942504 14293 raft_consensus.cc:729] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:19.942662 14293 consensus_queue.cc:260] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:19.943045 14291 tablet_bootstrap.cc:492] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:19.943204 14291 ts_tablet_manager.cc:1367] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:19.943325 14293 ts_tablet_manager.cc:1398] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:19.943688 14291 raft_consensus.cc:348] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:19.944362 14292 tablet_bootstrap.cc:492] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:19.944487 14292 ts_tablet_manager.cc:1367] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240503 02:00:19.944749 14291 raft_consensus.cc:374] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:19.945221 14292 raft_consensus.cc:348] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:19.945405 14291 raft_consensus.cc:729] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:19.945566 14291 consensus_queue.cc:260] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:19.945971 14291 ts_tablet_manager.cc:1398] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:00:19.946063 14292 raft_consensus.cc:374] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:19.946153 14292 raft_consensus.cc:729] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:19.946451 14292 consensus_queue.cc:260] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
W20240503 02:00:19.946756 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:19.946907 14292 ts_tablet_manager.cc:1398] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:00:19.998762 14276 consensus_queue.cc:1035] T 72aea5be835849d9a392fbab35357b7b P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:20.002166 14276 consensus_queue.cc:1035] T 72aea5be835849d9a392fbab35357b7b P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:20.134070 14106 raft_consensus.cc:482] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:20.134438 14106 raft_consensus.cc:504] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:20.135380 14106 leader_election.cc:288] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:20.135352 6582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ed12d2ca4c74194a99bd6e4f9c28834" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 02:00:20.135478 6582 raft_consensus.cc:2454] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 02:00:20.135746 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ed12d2ca4c74194a99bd6e4f9c28834" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:20.135809 6786 leader_election.cc:302] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 80fd18ef4d604c86b909d5c008a46b4f; no voters:
I20240503 02:00:20.135870 9178 raft_consensus.cc:2454] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 02:00:20.136047 14106 raft_consensus.cc:2781] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:20.136103 14106 raft_consensus.cc:482] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:20.136140 14106 raft_consensus.cc:3037] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:20.137259 14106 raft_consensus.cc:504] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:20.137585 14106 leader_election.cc:288] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Requested vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:20.137878 6582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ed12d2ca4c74194a99bd6e4f9c28834" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 02:00:20.137881 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ed12d2ca4c74194a99bd6e4f9c28834" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:20.137990 9178 raft_consensus.cc:3037] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:20.137989 6582 raft_consensus.cc:3037] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:20.138788 9178 raft_consensus.cc:2454] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 02:00:20.138862 6582 raft_consensus.cc:2454] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 02:00:20.139081 6784 leader_election.cc:302] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 492448570cc94ed1879feb2ec3b9051d; no voters:
I20240503 02:00:20.139250 14106 raft_consensus.cc:2781] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:20.139408 14106 raft_consensus.cc:686] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Becoming Leader. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Running, Role: LEADER
I20240503 02:00:20.139554 14106 consensus_queue.cc:237] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:20.140059 6314 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_extra_table_52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:20.141767 6308 catalog_manager.cc:5484] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d reported cstate change: term changed from 0 to 1, leader changed from <none> to 492448570cc94ed1879feb2ec3b9051d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:20.146596 9157 tablet_service.cc:1467] Processing CreateTablet for tablet 0433ba47dea84e2eb869df726191f722 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=63d2db6cf1d6486481601028d594114d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:20.146732 6700 tablet_service.cc:1467] Processing CreateTablet for tablet 0433ba47dea84e2eb869df726191f722 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=63d2db6cf1d6486481601028d594114d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:20.146934 9157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0433ba47dea84e2eb869df726191f722. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:20.146978 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0433ba47dea84e2eb869df726191f722. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:20.147053 8880 tablet_service.cc:1467] Processing CreateTablet for tablet 0433ba47dea84e2eb869df726191f722 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=63d2db6cf1d6486481601028d594114d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:20.147397 8880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0433ba47dea84e2eb869df726191f722. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:20.149415 14301 tablet_bootstrap.cc:492] T 0433ba47dea84e2eb869df726191f722 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:20.149538 9027 tablet_service.cc:1467] Processing CreateTablet for tablet db02d12f83a841f09588e7d6ce0a8398 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=63d2db6cf1d6486481601028d594114d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:20.149827 9027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db02d12f83a841f09588e7d6ce0a8398. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:20.150174 6831 tablet_service.cc:1467] Processing CreateTablet for tablet ae19b035d4594a29a770205dad7ac6ba (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=63d2db6cf1d6486481601028d594114d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:20.150413 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae19b035d4594a29a770205dad7ac6ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:20.150583 14301 tablet_bootstrap.cc:654] T 0433ba47dea84e2eb869df726191f722 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:20.152114 14291 tablet_bootstrap.cc:492] T 0433ba47dea84e2eb869df726191f722 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:20.152120 14304 tablet_bootstrap.cc:492] T db02d12f83a841f09588e7d6ce0a8398 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:20.152665 14293 tablet_bootstrap.cc:492] T ae19b035d4594a29a770205dad7ac6ba P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:20.153134 14291 tablet_bootstrap.cc:654] T 0433ba47dea84e2eb869df726191f722 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:20.153142 14304 tablet_bootstrap.cc:654] T db02d12f83a841f09588e7d6ce0a8398 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:20.154191 14293 tablet_bootstrap.cc:654] T ae19b035d4594a29a770205dad7ac6ba P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:20.154527 6438 tablet_service.cc:1467] Processing CreateTablet for tablet db02d12f83a841f09588e7d6ce0a8398 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=63d2db6cf1d6486481601028d594114d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:20.154798 6438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db02d12f83a841f09588e7d6ce0a8398. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:20.154789 6569 tablet_service.cc:1467] Processing CreateTablet for tablet db02d12f83a841f09588e7d6ce0a8398 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=63d2db6cf1d6486481601028d594114d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:20.154881 14302 tablet_bootstrap.cc:492] T 0433ba47dea84e2eb869df726191f722 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:20.154958 6436 tablet_service.cc:1467] Processing CreateTablet for tablet ae19b035d4594a29a770205dad7ac6ba (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=63d2db6cf1d6486481601028d594114d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:20.155050 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db02d12f83a841f09588e7d6ce0a8398. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:20.155179 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae19b035d4594a29a770205dad7ac6ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:20.155854 14302 tablet_bootstrap.cc:654] T 0433ba47dea84e2eb869df726191f722 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:20.157168 14293 tablet_bootstrap.cc:492] T ae19b035d4594a29a770205dad7ac6ba P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:20.157269 14293 ts_tablet_manager.cc:1367] T ae19b035d4594a29a770205dad7ac6ba P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:00:20.157433 14309 tablet_bootstrap.cc:492] T db02d12f83a841f09588e7d6ce0a8398 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:20.157750 14293 raft_consensus.cc:348] T ae19b035d4594a29a770205dad7ac6ba P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:20.157884 14293 raft_consensus.cc:374] T ae19b035d4594a29a770205dad7ac6ba P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:20.157914 14293 raft_consensus.cc:729] T ae19b035d4594a29a770205dad7ac6ba P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:20.157928 6567 tablet_service.cc:1467] Processing CreateTablet for tablet ae19b035d4594a29a770205dad7ac6ba (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=63d2db6cf1d6486481601028d594114d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:20.158025 14293 consensus_queue.cc:260] T ae19b035d4594a29a770205dad7ac6ba P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:20.158180 6567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae19b035d4594a29a770205dad7ac6ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:20.158288 14293 ts_tablet_manager.cc:1398] T ae19b035d4594a29a770205dad7ac6ba P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:20.158485 14309 tablet_bootstrap.cc:654] T db02d12f83a841f09588e7d6ce0a8398 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:20.160319 14292 tablet_bootstrap.cc:492] T db02d12f83a841f09588e7d6ce0a8398 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:20.161289 14292 tablet_bootstrap.cc:654] T db02d12f83a841f09588e7d6ce0a8398 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:20.165889 14292 tablet_bootstrap.cc:492] T db02d12f83a841f09588e7d6ce0a8398 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:20.166013 14292 ts_tablet_manager.cc:1367] T db02d12f83a841f09588e7d6ce0a8398 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20240503 02:00:20.166476 14292 raft_consensus.cc:348] T db02d12f83a841f09588e7d6ce0a8398 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:20.166597 14292 raft_consensus.cc:374] T db02d12f83a841f09588e7d6ce0a8398 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:20.166631 14292 raft_consensus.cc:729] T db02d12f83a841f09588e7d6ce0a8398 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:20.166747 14292 consensus_queue.cc:260] T db02d12f83a841f09588e7d6ce0a8398 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:20.167055 14292 ts_tablet_manager.cc:1398] T db02d12f83a841f09588e7d6ce0a8398 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:20.167227 14292 tablet_bootstrap.cc:492] T ae19b035d4594a29a770205dad7ac6ba P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:20.167310 14301 tablet_bootstrap.cc:492] T 0433ba47dea84e2eb869df726191f722 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:20.167410 14301 ts_tablet_manager.cc:1367] T 0433ba47dea84e2eb869df726191f722 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20240503 02:00:20.167857 14301 raft_consensus.cc:348] T 0433ba47dea84e2eb869df726191f722 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:20.167963 14301 raft_consensus.cc:374] T 0433ba47dea84e2eb869df726191f722 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:20.167991 14301 raft_consensus.cc:729] T 0433ba47dea84e2eb869df726191f722 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:20.168336 14301 consensus_queue.cc:260] T 0433ba47dea84e2eb869df726191f722 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:20.168558 14304 tablet_bootstrap.cc:492] T db02d12f83a841f09588e7d6ce0a8398 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:20.168653 14304 ts_tablet_manager.cc:1367] T db02d12f83a841f09588e7d6ce0a8398 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240503 02:00:20.168769 14301 ts_tablet_manager.cc:1398] T 0433ba47dea84e2eb869df726191f722 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:20.169116 14304 raft_consensus.cc:348] T db02d12f83a841f09588e7d6ce0a8398 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:20.169217 14304 raft_consensus.cc:374] T db02d12f83a841f09588e7d6ce0a8398 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:20.169248 14304 raft_consensus.cc:729] T db02d12f83a841f09588e7d6ce0a8398 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:20.169384 14304 consensus_queue.cc:260] T db02d12f83a841f09588e7d6ce0a8398 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:20.169596 14292 tablet_bootstrap.cc:654] T ae19b035d4594a29a770205dad7ac6ba P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:20.169679 14304 ts_tablet_manager.cc:1398] T db02d12f83a841f09588e7d6ce0a8398 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:20.176472 14302 tablet_bootstrap.cc:492] T 0433ba47dea84e2eb869df726191f722 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:20.176601 14302 ts_tablet_manager.cc:1367] T 0433ba47dea84e2eb869df726191f722 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20240503 02:00:20.176709 14291 tablet_bootstrap.cc:492] T 0433ba47dea84e2eb869df726191f722 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:20.176820 14291 ts_tablet_manager.cc:1367] T 0433ba47dea84e2eb869df726191f722 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.000s
I20240503 02:00:20.177042 14302 raft_consensus.cc:348] T 0433ba47dea84e2eb869df726191f722 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:20.177135 14302 raft_consensus.cc:374] T 0433ba47dea84e2eb869df726191f722 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:20.177155 14302 raft_consensus.cc:729] T 0433ba47dea84e2eb869df726191f722 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:20.177263 14302 consensus_queue.cc:260] T 0433ba47dea84e2eb869df726191f722 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:20.177259 14291 raft_consensus.cc:348] T 0433ba47dea84e2eb869df726191f722 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:20.177363 14291 raft_consensus.cc:374] T 0433ba47dea84e2eb869df726191f722 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:20.177388 14291 raft_consensus.cc:729] T 0433ba47dea84e2eb869df726191f722 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:20.177492 14291 consensus_queue.cc:260] T 0433ba47dea84e2eb869df726191f722 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:20.177556 14302 ts_tablet_manager.cc:1398] T 0433ba47dea84e2eb869df726191f722 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:20.177742 14291 ts_tablet_manager.cc:1398] T 0433ba47dea84e2eb869df726191f722 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:20.178704 14292 tablet_bootstrap.cc:492] T ae19b035d4594a29a770205dad7ac6ba P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:20.178807 14292 ts_tablet_manager.cc:1367] T ae19b035d4594a29a770205dad7ac6ba P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.001s
I20240503 02:00:20.179302 14292 raft_consensus.cc:348] T ae19b035d4594a29a770205dad7ac6ba P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:20.179416 14292 raft_consensus.cc:374] T ae19b035d4594a29a770205dad7ac6ba P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:20.179446 14292 raft_consensus.cc:729] T ae19b035d4594a29a770205dad7ac6ba P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:20.179564 14292 consensus_queue.cc:260] T ae19b035d4594a29a770205dad7ac6ba P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:20.179776 14309 tablet_bootstrap.cc:492] T db02d12f83a841f09588e7d6ce0a8398 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:20.179865 14309 ts_tablet_manager.cc:1367] T db02d12f83a841f09588e7d6ce0a8398 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20240503 02:00:20.179876 14292 ts_tablet_manager.cc:1398] T ae19b035d4594a29a770205dad7ac6ba P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:20.182152 14309 raft_consensus.cc:348] T db02d12f83a841f09588e7d6ce0a8398 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:20.182416 14309 raft_consensus.cc:374] T db02d12f83a841f09588e7d6ce0a8398 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:20.182510 14309 raft_consensus.cc:729] T db02d12f83a841f09588e7d6ce0a8398 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:20.182829 14309 consensus_queue.cc:260] T db02d12f83a841f09588e7d6ce0a8398 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:20.183231 14309 ts_tablet_manager.cc:1398] T db02d12f83a841f09588e7d6ce0a8398 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:20.184587 14309 tablet_bootstrap.cc:492] T ae19b035d4594a29a770205dad7ac6ba P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:20.185834 14309 tablet_bootstrap.cc:654] T ae19b035d4594a29a770205dad7ac6ba P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:20.187248 14309 tablet_bootstrap.cc:492] T ae19b035d4594a29a770205dad7ac6ba P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:20.187355 14309 ts_tablet_manager.cc:1367] T ae19b035d4594a29a770205dad7ac6ba P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:20.187734 14309 raft_consensus.cc:348] T ae19b035d4594a29a770205dad7ac6ba P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:20.187819 14309 raft_consensus.cc:374] T ae19b035d4594a29a770205dad7ac6ba P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:20.187846 14309 raft_consensus.cc:729] T ae19b035d4594a29a770205dad7ac6ba P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:20.187951 14309 consensus_queue.cc:260] T ae19b035d4594a29a770205dad7ac6ba P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:20.188200 14309 ts_tablet_manager.cc:1398] T ae19b035d4594a29a770205dad7ac6ba P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:20.193431 14106 raft_consensus.cc:482] T ae19b035d4594a29a770205dad7ac6ba P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:20.193557 14106 raft_consensus.cc:504] T ae19b035d4594a29a770205dad7ac6ba P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:20.194015 14106 leader_election.cc:288] T ae19b035d4594a29a770205dad7ac6ba P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 02:00:20.194375 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae19b035d4594a29a770205dad7ac6ba" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 02:00:20.194453 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae19b035d4594a29a770205dad7ac6ba" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 02:00:20.194535 6589 raft_consensus.cc:2454] T ae19b035d4594a29a770205dad7ac6ba P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 02:00:20.194592 6458 raft_consensus.cc:2454] T ae19b035d4594a29a770205dad7ac6ba P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 02:00:20.194856 6786 leader_election.cc:302] T ae19b035d4594a29a770205dad7ac6ba P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 80fd18ef4d604c86b909d5c008a46b4f; no voters:
I20240503 02:00:20.195081 14106 raft_consensus.cc:2781] T ae19b035d4594a29a770205dad7ac6ba P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:20.195128 14106 raft_consensus.cc:482] T ae19b035d4594a29a770205dad7ac6ba P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:20.195166 14106 raft_consensus.cc:3037] T ae19b035d4594a29a770205dad7ac6ba P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:20.196022 14106 raft_consensus.cc:504] T ae19b035d4594a29a770205dad7ac6ba P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:20.196411 14106 leader_election.cc:288] T ae19b035d4594a29a770205dad7ac6ba P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Requested vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 02:00:20.196740 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae19b035d4594a29a770205dad7ac6ba" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 02:00:20.196858 6458 raft_consensus.cc:3037] T ae19b035d4594a29a770205dad7ac6ba P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:20.197630 6458 raft_consensus.cc:2454] T ae19b035d4594a29a770205dad7ac6ba P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 02:00:20.197909 6784 leader_election.cc:302] T ae19b035d4594a29a770205dad7ac6ba P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:20.198089 14106 raft_consensus.cc:2781] T ae19b035d4594a29a770205dad7ac6ba P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:20.198170 14106 raft_consensus.cc:686] T ae19b035d4594a29a770205dad7ac6ba P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Becoming Leader. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Running, Role: LEADER
I20240503 02:00:20.198309 14106 consensus_queue.cc:237] T ae19b035d4594a29a770205dad7ac6ba P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:20.204865 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae19b035d4594a29a770205dad7ac6ba" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 02:00:20.205024 6589 raft_consensus.cc:3037] T ae19b035d4594a29a770205dad7ac6ba P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:20.206087 6589 raft_consensus.cc:2454] T ae19b035d4594a29a770205dad7ac6ba P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 02:00:20.206938 6309 catalog_manager.cc:5484] T ae19b035d4594a29a770205dad7ac6ba P 492448570cc94ed1879feb2ec3b9051d reported cstate change: term changed from 0 to 1, leader changed from <none> to 492448570cc94ed1879feb2ec3b9051d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:20.244220 14056 raft_consensus.cc:482] T db02d12f83a841f09588e7d6ce0a8398 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:20.244417 14056 raft_consensus.cc:504] T db02d12f83a841f09588e7d6ce0a8398 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:20.244928 14056 leader_election.cc:288] T db02d12f83a841f09588e7d6ce0a8398 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:20.245410 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db02d12f83a841f09588e7d6ce0a8398" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 02:00:20.245429 9047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db02d12f83a841f09588e7d6ce0a8398" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:20.245559 9047 raft_consensus.cc:2454] T db02d12f83a841f09588e7d6ce0a8398 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:20.245559 6589 raft_consensus.cc:2454] T db02d12f83a841f09588e7d6ce0a8398 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:20.245877 6393 leader_election.cc:302] T db02d12f83a841f09588e7d6ce0a8398 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:20.246059 14056 raft_consensus.cc:2781] T db02d12f83a841f09588e7d6ce0a8398 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:20.246114 14056 raft_consensus.cc:482] T db02d12f83a841f09588e7d6ce0a8398 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:20.246157 14056 raft_consensus.cc:3037] T db02d12f83a841f09588e7d6ce0a8398 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:20.247100 14056 raft_consensus.cc:504] T db02d12f83a841f09588e7d6ce0a8398 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:20.247437 14056 leader_election.cc:288] T db02d12f83a841f09588e7d6ce0a8398 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Requested vote from peers 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:20.247756 9047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db02d12f83a841f09588e7d6ce0a8398" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:20.247782 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db02d12f83a841f09588e7d6ce0a8398" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 02:00:20.247908 6589 raft_consensus.cc:3037] T db02d12f83a841f09588e7d6ce0a8398 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:20.247926 9047 raft_consensus.cc:3037] T db02d12f83a841f09588e7d6ce0a8398 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:20.249003 9047 raft_consensus.cc:2454] T db02d12f83a841f09588e7d6ce0a8398 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:20.249158 6589 raft_consensus.cc:2454] T db02d12f83a841f09588e7d6ce0a8398 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:20.249328 6392 leader_election.cc:302] T db02d12f83a841f09588e7d6ce0a8398 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99ead02915954b4fab23b033ad6d546b, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:20.249506 14056 raft_consensus.cc:2781] T db02d12f83a841f09588e7d6ce0a8398 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:20.249608 14056 raft_consensus.cc:686] T db02d12f83a841f09588e7d6ce0a8398 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Becoming Leader. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Running, Role: LEADER
I20240503 02:00:20.249748 14056 consensus_queue.cc:237] T db02d12f83a841f09588e7d6ce0a8398 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:20.252213 6314 catalog_manager.cc:5484] T db02d12f83a841f09588e7d6ce0a8398 P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: term changed from 0 to 1, leader changed from <none> to dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129). New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:20.275350 6314 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:20.279667 6589 raft_consensus.cc:1261] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:20.279721 9178 raft_consensus.cc:1261] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:20.280102 14276 consensus_queue.cc:1035] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:20.280232 14106 consensus_queue.cc:1035] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:20.283452 6314 catalog_manager.cc:5576] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863) does not have the latest schema for tablet 2ed12d2ca4c74194a99bd6e4f9c28834 (table rebalancer_dynamic_table [id=78ad5e8d3227453fb447fb66f8d78a05]). Expected version 1 got 0
I20240503 02:00:20.284984 6309 catalog_manager.cc:5576] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719) does not have the latest schema for tablet 2ed12d2ca4c74194a99bd6e4f9c28834 (table rebalancer_dynamic_table [id=78ad5e8d3227453fb447fb66f8d78a05]). Expected version 1 got 0
I20240503 02:00:20.285471 14318 tablet.cc:1722] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:20.285526 6314 catalog_manager.cc:5576] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079) does not have the latest schema for tablet 2ed12d2ca4c74194a99bd6e4f9c28834 (table rebalancer_dynamic_table [id=78ad5e8d3227453fb447fb66f8d78a05]). Expected version 1 got 0
I20240503 02:00:20.285840 14318 tablet.cc:1620] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d: MemRowSet was empty: no flush needed.
I20240503 02:00:20.284355 14319 tablet.cc:1722] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:20.286468 14319 tablet.cc:1620] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 80fd18ef4d604c86b909d5c008a46b4f: MemRowSet was empty: no flush needed.
I20240503 02:00:20.286868 6314 catalog_manager.cc:5576] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863) does not have the latest schema for tablet 2ed12d2ca4c74194a99bd6e4f9c28834 (table rebalancer_dynamic_table [id=78ad5e8d3227453fb447fb66f8d78a05]). Expected version 1 got 0
I20240503 02:00:20.287791 14320 tablet.cc:1722] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:20.288008 6309 catalog_manager.cc:5576] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079) does not have the latest schema for tablet 2ed12d2ca4c74194a99bd6e4f9c28834 (table rebalancer_dynamic_table [id=78ad5e8d3227453fb447fb66f8d78a05]). Expected version 1 got 0
I20240503 02:00:20.288134 14320 tablet.cc:1620] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
I20240503 02:00:20.291407 6308 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=78ad5e8d3227453fb447fb66f8d78a05] alter complete (version 1)
W20240503 02:00:20.354885 8982 consensus_peers.cc:485] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:00:20.362969 14284 raft_consensus.cc:482] T 0433ba47dea84e2eb869df726191f722 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:20.363109 14284 raft_consensus.cc:504] T 0433ba47dea84e2eb869df726191f722 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:20.363505 14284 leader_election.cc:288] T 0433ba47dea84e2eb869df726191f722 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:20.363809 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0433ba47dea84e2eb869df726191f722" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:20.363869 8917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0433ba47dea84e2eb869df726191f722" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:20.363952 9178 raft_consensus.cc:2454] T 0433ba47dea84e2eb869df726191f722 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:20.364015 8917 raft_consensus.cc:2454] T 0433ba47dea84e2eb869df726191f722 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:20.364257 6653 leader_election.cc:302] T 0433ba47dea84e2eb869df726191f722 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:20.364463 14284 raft_consensus.cc:2781] T 0433ba47dea84e2eb869df726191f722 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:20.364523 14284 raft_consensus.cc:482] T 0433ba47dea84e2eb869df726191f722 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:20.364552 14284 raft_consensus.cc:3037] T 0433ba47dea84e2eb869df726191f722 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:20.365538 14284 raft_consensus.cc:504] T 0433ba47dea84e2eb869df726191f722 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:20.365907 14284 leader_election.cc:288] T 0433ba47dea84e2eb869df726191f722 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Requested vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:20.366267 8917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0433ba47dea84e2eb869df726191f722" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:20.366405 8917 raft_consensus.cc:3037] T 0433ba47dea84e2eb869df726191f722 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:20.366380 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0433ba47dea84e2eb869df726191f722" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:20.366508 9178 raft_consensus.cc:3037] T 0433ba47dea84e2eb869df726191f722 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:20.367367 8917 raft_consensus.cc:2454] T 0433ba47dea84e2eb869df726191f722 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:20.367458 9178 raft_consensus.cc:2454] T 0433ba47dea84e2eb869df726191f722 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:20.367666 6656 leader_election.cc:302] T 0433ba47dea84e2eb869df726191f722 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9942306855404cfd891adf227dd65eb2, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:20.367832 14284 raft_consensus.cc:2781] T 0433ba47dea84e2eb869df726191f722 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:20.367925 14284 raft_consensus.cc:686] T 0433ba47dea84e2eb869df726191f722 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Becoming Leader. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Running, Role: LEADER
I20240503 02:00:20.368064 14284 consensus_queue.cc:237] T 0433ba47dea84e2eb869df726191f722 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:20.370608 6309 catalog_manager.cc:5484] T 0433ba47dea84e2eb869df726191f722 P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:20.403846 6309 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:20.413297 14318 tablet.cc:1722] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:20.413650 14318 tablet.cc:1620] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d: MemRowSet was empty: no flush needed.
I20240503 02:00:20.413921 14320 tablet.cc:1722] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:20.413657 14319 tablet.cc:1722] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:20.416023 14319 tablet.cc:1620] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 80fd18ef4d604c86b909d5c008a46b4f: MemRowSet was empty: no flush needed.
W20240503 02:00:20.416414 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:20.416642 14320 tablet.cc:1620] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
I20240503 02:00:20.419878 6309 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=78ad5e8d3227453fb447fb66f8d78a05] alter complete (version 2)
W20240503 02:00:20.454217 6785 consensus_peers.cc:485] T c3485701bf3b48f5ae2e1568afeed133 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:00:20.522709 6308 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:20.528911 14318 tablet.cc:1722] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:20.529829 14320 tablet.cc:1722] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:20.531289 14319 tablet.cc:1722] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:20.536793 6308 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=78ad5e8d3227453fb447fb66f8d78a05] alter complete (version 3)
I20240503 02:00:20.589056 14322 consensus_queue.cc:1035] T ae19b035d4594a29a770205dad7ac6ba P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:20.592350 14324 consensus_queue.cc:1035] T ae19b035d4594a29a770205dad7ac6ba P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:20.641884 6308 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:20.642060 6308 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:20.644143 6308 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet 2ed12d2ca4c74194a99bd6e4f9c28834
I20240503 02:00:20.644927 6831 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ed12d2ca4c74194a99bd6e4f9c28834 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:20 UTC) from {username='slave'} at 127.0.0.1:52812
I20240503 02:00:20.645022 6567 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ed12d2ca4c74194a99bd6e4f9c28834 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:20 UTC) from {username='slave'} at 127.0.0.1:59846
I20240503 02:00:20.645356 14328 tablet_replica.cc:330] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d: stopping tablet replica
I20240503 02:00:20.645378 14329 tablet_replica.cc:330] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 02:00:20.645524 14329 raft_consensus.cc:2229] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:20.645531 14328 raft_consensus.cc:2229] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:20.645640 9157 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ed12d2ca4c74194a99bd6e4f9c28834 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:20 UTC) from {username='slave'} at 127.0.0.1:51940
I20240503 02:00:20.645825 14329 raft_consensus.cc:2258] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:20.646144 14330 tablet_replica.cc:330] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 2a01ace80baf4057b6bf1995cf88468b: stopping tablet replica
I20240503 02:00:20.646320 14330 raft_consensus.cc:2229] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:20.646334 14328 raft_consensus.cc:2258] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:20.646472 14330 raft_consensus.cc:2258] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:20.646612 14329 ts_tablet_manager.cc:1875] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:20.646916 14328 ts_tablet_manager.cc:1875] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:20.647152 14330 ts_tablet_manager.cc:1875] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 2a01ace80baf4057b6bf1995cf88468b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:20.649324 14329 ts_tablet_manager.cc:1888] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:20.649423 14328 ts_tablet_manager.cc:1888] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:20.649451 14329 log.cc:1196] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/2ed12d2ca4c74194a99bd6e4f9c28834
I20240503 02:00:20.649463 14330 ts_tablet_manager.cc:1888] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 2a01ace80baf4057b6bf1995cf88468b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:20.649519 14330 log.cc:1196] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 2a01ace80baf4057b6bf1995cf88468b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/2ed12d2ca4c74194a99bd6e4f9c28834
I20240503 02:00:20.649539 14328 log.cc:1196] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/2ed12d2ca4c74194a99bd6e4f9c28834
I20240503 02:00:20.649780 14329 ts_tablet_manager.cc:1909] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting consensus metadata
I20240503 02:00:20.649780 14330 ts_tablet_manager.cc:1909] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 2a01ace80baf4057b6bf1995cf88468b: Deleting consensus metadata
I20240503 02:00:20.649894 14328 ts_tablet_manager.cc:1909] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d: Deleting consensus metadata
I20240503 02:00:20.650579 6303 catalog_manager.cc:4830] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719): tablet 2ed12d2ca4c74194a99bd6e4f9c28834 (table rebalancer_dynamic_table_new_name [id=78ad5e8d3227453fb447fb66f8d78a05]) successfully deleted
I20240503 02:00:20.650792 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet 2ed12d2ca4c74194a99bd6e4f9c28834 (table rebalancer_dynamic_table_new_name [id=78ad5e8d3227453fb447fb66f8d78a05]) successfully deleted
I20240503 02:00:20.651173 6300 catalog_manager.cc:4830] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079): tablet 2ed12d2ca4c74194a99bd6e4f9c28834 (table rebalancer_dynamic_table_new_name [id=78ad5e8d3227453fb447fb66f8d78a05]) successfully deleted
I20240503 02:00:20.677201 14153 consensus_queue.cc:1035] T db02d12f83a841f09588e7d6ce0a8398 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:20.680784 14153 consensus_queue.cc:1035] T db02d12f83a841f09588e7d6ce0a8398 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:20.746182 6308 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:00:20.752024 6436 tablet_service.cc:1467] Processing CreateTablet for tablet 451cd82331904b84884f7fdaf2b7dec9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=364d2beeb13a4e77a5977e4c7d9786dc]), partition=
I20240503 02:00:20.752131 6567 tablet_service.cc:1467] Processing CreateTablet for tablet 451cd82331904b84884f7fdaf2b7dec9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=364d2beeb13a4e77a5977e4c7d9786dc]), partition=
I20240503 02:00:20.752301 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 451cd82331904b84884f7fdaf2b7dec9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:20.752422 6567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 451cd82331904b84884f7fdaf2b7dec9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:20.752357 8880 tablet_service.cc:1467] Processing CreateTablet for tablet 451cd82331904b84884f7fdaf2b7dec9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=364d2beeb13a4e77a5977e4c7d9786dc]), partition=
I20240503 02:00:20.752605 8880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 451cd82331904b84884f7fdaf2b7dec9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:20.754515 14334 tablet_bootstrap.cc:492] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:20.755326 14336 tablet_bootstrap.cc:492] T 451cd82331904b84884f7fdaf2b7dec9 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:20.755573 14335 tablet_bootstrap.cc:492] T 451cd82331904b84884f7fdaf2b7dec9 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:20.756290 14336 tablet_bootstrap.cc:654] T 451cd82331904b84884f7fdaf2b7dec9 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:20.756292 14334 tablet_bootstrap.cc:654] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:20.756479 14335 tablet_bootstrap.cc:654] T 451cd82331904b84884f7fdaf2b7dec9 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:20.757726 14336 tablet_bootstrap.cc:492] T 451cd82331904b84884f7fdaf2b7dec9 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:20.757827 14336 ts_tablet_manager.cc:1367] T 451cd82331904b84884f7fdaf2b7dec9 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:20.758085 14334 tablet_bootstrap.cc:492] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:20.758185 14334 ts_tablet_manager.cc:1367] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:20.758260 14336 raft_consensus.cc:348] T 451cd82331904b84884f7fdaf2b7dec9 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:20.758373 14336 raft_consensus.cc:374] T 451cd82331904b84884f7fdaf2b7dec9 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:20.758412 14336 raft_consensus.cc:729] T 451cd82331904b84884f7fdaf2b7dec9 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:20.758546 14336 consensus_queue.cc:260] T 451cd82331904b84884f7fdaf2b7dec9 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:20.758836 14334 raft_consensus.cc:348] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:20.758864 14336 ts_tablet_manager.cc:1398] T 451cd82331904b84884f7fdaf2b7dec9 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:20.758965 14334 raft_consensus.cc:374] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:20.758998 14334 raft_consensus.cc:729] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:20.759188 14334 consensus_queue.cc:260] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:20.759436 14335 tablet_bootstrap.cc:492] T 451cd82331904b84884f7fdaf2b7dec9 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:20.759538 14335 ts_tablet_manager.cc:1367] T 451cd82331904b84884f7fdaf2b7dec9 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240503 02:00:20.760027 14335 raft_consensus.cc:348] T 451cd82331904b84884f7fdaf2b7dec9 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:20.760141 14335 raft_consensus.cc:374] T 451cd82331904b84884f7fdaf2b7dec9 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:20.760176 14335 raft_consensus.cc:729] T 451cd82331904b84884f7fdaf2b7dec9 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:20.760300 14335 consensus_queue.cc:260] T 451cd82331904b84884f7fdaf2b7dec9 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:20.760603 14335 ts_tablet_manager.cc:1398] T 451cd82331904b84884f7fdaf2b7dec9 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:20.763206 14056 raft_consensus.cc:482] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:20.763317 14056 raft_consensus.cc:504] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:20.763695 14056 leader_election.cc:288] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:20.764240 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "451cd82331904b84884f7fdaf2b7dec9" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 02:00:20.764310 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "451cd82331904b84884f7fdaf2b7dec9" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:20.764370 6589 raft_consensus.cc:2454] T 451cd82331904b84884f7fdaf2b7dec9 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:20.764467 8916 raft_consensus.cc:2454] T 451cd82331904b84884f7fdaf2b7dec9 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
W20240503 02:00:20.764719 6785 consensus_peers.cc:485] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:00:20.765015 6393 leader_election.cc:302] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:20.765995 14153 raft_consensus.cc:2781] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:20.766057 14153 raft_consensus.cc:482] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:20.766094 14153 raft_consensus.cc:3037] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:20.767045 14153 raft_consensus.cc:504] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:20.767371 14153 leader_election.cc:288] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Requested vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:20.767765 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "451cd82331904b84884f7fdaf2b7dec9" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:20.767915 8916 raft_consensus.cc:3037] T 451cd82331904b84884f7fdaf2b7dec9 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:20.767875 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "451cd82331904b84884f7fdaf2b7dec9" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 02:00:20.767987 6589 raft_consensus.cc:3037] T 451cd82331904b84884f7fdaf2b7dec9 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:20.768838 6589 raft_consensus.cc:2454] T 451cd82331904b84884f7fdaf2b7dec9 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:20.768931 8916 raft_consensus.cc:2454] T 451cd82331904b84884f7fdaf2b7dec9 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:20.769176 6393 leader_election.cc:302] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:20.769361 14153 raft_consensus.cc:2781] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:20.769440 14153 raft_consensus.cc:686] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Becoming Leader. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Running, Role: LEADER
I20240503 02:00:20.769552 14153 consensus_queue.cc:237] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:20.777469 14334 ts_tablet_manager.cc:1398] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.019s user 0.001s sys 0.000s
I20240503 02:00:20.778122 6314 catalog_manager.cc:5484] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: term changed from 0 to 1, leader changed from <none> to dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129). New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:20.788797 14190 consensus_queue.cc:1035] T 0433ba47dea84e2eb869df726191f722 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:20.794026 14190 consensus_queue.cc:1035] T 0433ba47dea84e2eb869df726191f722 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:00:20.846863 6654 consensus_peers.cc:485] T ba17d4d3e0754b2798eb95484d37229a P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:00:20.889060 6314 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:20.893214 8916 raft_consensus.cc:1261] T 451cd82331904b84884f7fdaf2b7dec9 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:20.893754 6589 raft_consensus.cc:1261] T 451cd82331904b84884f7fdaf2b7dec9 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:20.893963 14153 consensus_queue.cc:1035] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:20.894388 14056 consensus_queue.cc:1035] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:20.894362 6314 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_extra_table_54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:20.897919 6308 catalog_manager.cc:5576] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863) does not have the latest schema for tablet 451cd82331904b84884f7fdaf2b7dec9 (table rebalancer_dynamic_table [id=364d2beeb13a4e77a5977e4c7d9786dc]). Expected version 1 got 0
I20240503 02:00:20.898679 6309 catalog_manager.cc:5576] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211) does not have the latest schema for tablet 451cd82331904b84884f7fdaf2b7dec9 (table rebalancer_dynamic_table [id=364d2beeb13a4e77a5977e4c7d9786dc]). Expected version 1 got 0
I20240503 02:00:20.899163 14319 tablet.cc:1722] T 451cd82331904b84884f7fdaf2b7dec9 P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:20.899524 14319 tablet.cc:1620] T 451cd82331904b84884f7fdaf2b7dec9 P 80fd18ef4d604c86b909d5c008a46b4f: MemRowSet was empty: no flush needed.
I20240503 02:00:20.900132 14348 tablet.cc:1722] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:20.900606 14348 tablet.cc:1620] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a: MemRowSet was empty: no flush needed.
I20240503 02:00:20.903079 6308 catalog_manager.cc:5576] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093) does not have the latest schema for tablet 451cd82331904b84884f7fdaf2b7dec9 (table rebalancer_dynamic_table [id=364d2beeb13a4e77a5977e4c7d9786dc]). Expected version 1 got 0
I20240503 02:00:20.903825 9027 tablet_service.cc:1467] Processing CreateTablet for tablet 3af89caa97634222ab834577a9217ce9 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=0293809e35df48c6bbb4a371a24b2499]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:20.903846 9157 tablet_service.cc:1467] Processing CreateTablet for tablet ca907fb5b04f4d2492e7f4cea7114c95 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=0293809e35df48c6bbb4a371a24b2499]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:20.903836 9028 tablet_service.cc:1467] Processing CreateTablet for tablet 87c4b630a58749e6be9f26cceda27573 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=0293809e35df48c6bbb4a371a24b2499]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:20.904131 9027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3af89caa97634222ab834577a9217ce9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:20.904146 9157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca907fb5b04f4d2492e7f4cea7114c95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:20.904714 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87c4b630a58749e6be9f26cceda27573. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:20.906142 14350 tablet_bootstrap.cc:492] T 3af89caa97634222ab834577a9217ce9 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:20.906368 9158 tablet_service.cc:1467] Processing CreateTablet for tablet 87c4b630a58749e6be9f26cceda27573 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=0293809e35df48c6bbb4a371a24b2499]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:20.906608 9158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87c4b630a58749e6be9f26cceda27573. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:20.907003 14351 tablet_bootstrap.cc:492] T ca907fb5b04f4d2492e7f4cea7114c95 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:20.907191 14350 tablet_bootstrap.cc:654] T 3af89caa97634222ab834577a9217ce9 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:20.907447 6309 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=364d2beeb13a4e77a5977e4c7d9786dc] alter complete (version 1)
I20240503 02:00:20.908399 14351 tablet_bootstrap.cc:654] T ca907fb5b04f4d2492e7f4cea7114c95 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:20.910374 6700 tablet_service.cc:1467] Processing CreateTablet for tablet ca907fb5b04f4d2492e7f4cea7114c95 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=0293809e35df48c6bbb4a371a24b2499]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:20.910674 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca907fb5b04f4d2492e7f4cea7114c95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:20.911375 6309 catalog_manager.cc:5576] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211) does not have the latest schema for tablet 451cd82331904b84884f7fdaf2b7dec9 (table rebalancer_dynamic_table [id=364d2beeb13a4e77a5977e4c7d9786dc]). Expected version 1 got 0
I20240503 02:00:20.912678 8880 tablet_service.cc:1467] Processing CreateTablet for tablet ca907fb5b04f4d2492e7f4cea7114c95 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=0293809e35df48c6bbb4a371a24b2499]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:20.912967 8880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca907fb5b04f4d2492e7f4cea7114c95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:20.914105 14343 mvcc.cc:205] Tried to move back new op lower bound from 7023417839174885376 to 7023417838673567744. Current Snapshot: MvccSnapshot[applied={T|T < 7023417839174885376}]
I20240503 02:00:20.914330 14351 tablet_bootstrap.cc:492] T ca907fb5b04f4d2492e7f4cea7114c95 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:20.914438 14351 ts_tablet_manager.cc:1367] T ca907fb5b04f4d2492e7f4cea7114c95 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 02:00:20.914495 14354 tablet.cc:1722] T 451cd82331904b84884f7fdaf2b7dec9 P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:20.914783 14354 tablet.cc:1620] T 451cd82331904b84884f7fdaf2b7dec9 P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
I20240503 02:00:20.914919 14351 raft_consensus.cc:348] T ca907fb5b04f4d2492e7f4cea7114c95 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:20.915014 14336 tablet_bootstrap.cc:492] T ca907fb5b04f4d2492e7f4cea7114c95 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:20.915035 14351 raft_consensus.cc:374] T ca907fb5b04f4d2492e7f4cea7114c95 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:20.915066 14351 raft_consensus.cc:729] T ca907fb5b04f4d2492e7f4cea7114c95 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:20.915185 14351 consensus_queue.cc:260] T ca907fb5b04f4d2492e7f4cea7114c95 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:20.915232 8887 tablet_service.cc:1467] Processing CreateTablet for tablet 3af89caa97634222ab834577a9217ce9 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=0293809e35df48c6bbb4a371a24b2499]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:20.915465 8887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3af89caa97634222ab834577a9217ce9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:20.915489 14351 ts_tablet_manager.cc:1398] T ca907fb5b04f4d2492e7f4cea7114c95 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:20.915761 14351 tablet_bootstrap.cc:492] T 87c4b630a58749e6be9f26cceda27573 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:20.915987 14336 tablet_bootstrap.cc:654] T ca907fb5b04f4d2492e7f4cea7114c95 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:20.916671 14351 tablet_bootstrap.cc:654] T 87c4b630a58749e6be9f26cceda27573 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:20.917021 14350 tablet_bootstrap.cc:492] T 3af89caa97634222ab834577a9217ce9 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:20.917117 14350 ts_tablet_manager.cc:1367] T 3af89caa97634222ab834577a9217ce9 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 02:00:20.917589 14350 raft_consensus.cc:348] T 3af89caa97634222ab834577a9217ce9 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:20.917672 14351 tablet_bootstrap.cc:492] T 87c4b630a58749e6be9f26cceda27573 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:20.917699 14350 raft_consensus.cc:374] T 3af89caa97634222ab834577a9217ce9 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:20.917733 14350 raft_consensus.cc:729] T 3af89caa97634222ab834577a9217ce9 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:20.917745 14351 ts_tablet_manager.cc:1367] T 87c4b630a58749e6be9f26cceda27573 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:20.917865 14350 consensus_queue.cc:260] T 3af89caa97634222ab834577a9217ce9 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:20.917981 6831 tablet_service.cc:1467] Processing CreateTablet for tablet 3af89caa97634222ab834577a9217ce9 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=0293809e35df48c6bbb4a371a24b2499]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:20.918180 14350 ts_tablet_manager.cc:1398] T 3af89caa97634222ab834577a9217ce9 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:20.918283 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3af89caa97634222ab834577a9217ce9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:20.918345 14350 tablet_bootstrap.cc:492] T 87c4b630a58749e6be9f26cceda27573 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:20.918309 14351 raft_consensus.cc:348] T 87c4b630a58749e6be9f26cceda27573 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:20.918406 14351 raft_consensus.cc:374] T 87c4b630a58749e6be9f26cceda27573 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:20.918433 14351 raft_consensus.cc:729] T 87c4b630a58749e6be9f26cceda27573 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:20.918545 14351 consensus_queue.cc:260] T 87c4b630a58749e6be9f26cceda27573 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:20.918918 14351 ts_tablet_manager.cc:1398] T 87c4b630a58749e6be9f26cceda27573 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:20.917984 6829 tablet_service.cc:1467] Processing CreateTablet for tablet 87c4b630a58749e6be9f26cceda27573 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=0293809e35df48c6bbb4a371a24b2499]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:20.919268 6829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87c4b630a58749e6be9f26cceda27573. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:20.919279 14350 tablet_bootstrap.cc:654] T 87c4b630a58749e6be9f26cceda27573 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:20.921962 14336 tablet_bootstrap.cc:492] T ca907fb5b04f4d2492e7f4cea7114c95 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:20.922073 14336 ts_tablet_manager.cc:1367] T ca907fb5b04f4d2492e7f4cea7114c95 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:00:20.922456 14357 tablet_bootstrap.cc:492] T 3af89caa97634222ab834577a9217ce9 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:20.922593 14336 raft_consensus.cc:348] T ca907fb5b04f4d2492e7f4cea7114c95 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:20.922708 14336 raft_consensus.cc:374] T ca907fb5b04f4d2492e7f4cea7114c95 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:20.922735 14336 raft_consensus.cc:729] T ca907fb5b04f4d2492e7f4cea7114c95 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:20.922868 14336 consensus_queue.cc:260] T ca907fb5b04f4d2492e7f4cea7114c95 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:20.923171 14336 ts_tablet_manager.cc:1398] T ca907fb5b04f4d2492e7f4cea7114c95 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:20.923609 14357 tablet_bootstrap.cc:654] T 3af89caa97634222ab834577a9217ce9 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:20.925228 14357 tablet_bootstrap.cc:492] T 3af89caa97634222ab834577a9217ce9 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:20.925323 14357 ts_tablet_manager.cc:1367] T 3af89caa97634222ab834577a9217ce9 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:20.925772 14357 raft_consensus.cc:348] T 3af89caa97634222ab834577a9217ce9 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:20.925884 14357 raft_consensus.cc:374] T 3af89caa97634222ab834577a9217ce9 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:20.925912 14357 raft_consensus.cc:729] T 3af89caa97634222ab834577a9217ce9 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:20.926028 14357 consensus_queue.cc:260] T 3af89caa97634222ab834577a9217ce9 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:20.923350 14336 tablet_bootstrap.cc:492] T 3af89caa97634222ab834577a9217ce9 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:20.927053 14357 ts_tablet_manager.cc:1398] T 3af89caa97634222ab834577a9217ce9 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:00:20.927229 14357 tablet_bootstrap.cc:492] T 87c4b630a58749e6be9f26cceda27573 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:20.928198 14357 tablet_bootstrap.cc:654] T 87c4b630a58749e6be9f26cceda27573 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:20.928512 14336 tablet_bootstrap.cc:654] T 3af89caa97634222ab834577a9217ce9 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:20.929657 14359 tablet_bootstrap.cc:492] T ca907fb5b04f4d2492e7f4cea7114c95 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:20.930253 14350 tablet_bootstrap.cc:492] T 87c4b630a58749e6be9f26cceda27573 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:20.930354 14350 ts_tablet_manager.cc:1367] T 87c4b630a58749e6be9f26cceda27573 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20240503 02:00:20.930482 14336 tablet_bootstrap.cc:492] T 3af89caa97634222ab834577a9217ce9 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:20.930550 14336 ts_tablet_manager.cc:1367] T 3af89caa97634222ab834577a9217ce9 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20240503 02:00:20.931159 14350 raft_consensus.cc:348] T 87c4b630a58749e6be9f26cceda27573 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:20.931277 14350 raft_consensus.cc:374] T 87c4b630a58749e6be9f26cceda27573 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:20.931308 14350 raft_consensus.cc:729] T 87c4b630a58749e6be9f26cceda27573 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:20.931425 14350 consensus_queue.cc:260] T 87c4b630a58749e6be9f26cceda27573 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:20.931727 14350 ts_tablet_manager.cc:1398] T 87c4b630a58749e6be9f26cceda27573 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:20.932001 14357 tablet_bootstrap.cc:492] T 87c4b630a58749e6be9f26cceda27573 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:20.932338 14357 ts_tablet_manager.cc:1367] T 87c4b630a58749e6be9f26cceda27573 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240503 02:00:20.932875 14357 raft_consensus.cc:348] T 87c4b630a58749e6be9f26cceda27573 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:20.933063 14357 raft_consensus.cc:374] T 87c4b630a58749e6be9f26cceda27573 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:20.933115 14357 raft_consensus.cc:729] T 87c4b630a58749e6be9f26cceda27573 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:20.933211 14336 raft_consensus.cc:348] T 3af89caa97634222ab834577a9217ce9 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:20.933266 14357 consensus_queue.cc:260] T 87c4b630a58749e6be9f26cceda27573 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:20.933342 14336 raft_consensus.cc:374] T 3af89caa97634222ab834577a9217ce9 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:20.933408 14336 raft_consensus.cc:729] T 3af89caa97634222ab834577a9217ce9 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:20.933547 14336 consensus_queue.cc:260] T 3af89caa97634222ab834577a9217ce9 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:20.933615 14357 ts_tablet_manager.cc:1398] T 87c4b630a58749e6be9f26cceda27573 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:20.933846 14336 ts_tablet_manager.cc:1398] T 3af89caa97634222ab834577a9217ce9 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 02:00:20.934881 14359 tablet_bootstrap.cc:654] T ca907fb5b04f4d2492e7f4cea7114c95 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
W20240503 02:00:20.936086 6654 consensus_peers.cc:485] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:00:20.936097 14359 tablet_bootstrap.cc:492] T ca907fb5b04f4d2492e7f4cea7114c95 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:20.936389 14359 ts_tablet_manager.cc:1367] T ca907fb5b04f4d2492e7f4cea7114c95 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:00:20.936885 14359 raft_consensus.cc:348] T ca907fb5b04f4d2492e7f4cea7114c95 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:20.936976 14359 raft_consensus.cc:374] T ca907fb5b04f4d2492e7f4cea7114c95 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:20.936995 14359 raft_consensus.cc:729] T ca907fb5b04f4d2492e7f4cea7114c95 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:20.937095 14359 consensus_queue.cc:260] T ca907fb5b04f4d2492e7f4cea7114c95 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:20.937373 14359 ts_tablet_manager.cc:1398] T ca907fb5b04f4d2492e7f4cea7114c95 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:20.972299 14220 raft_consensus.cc:482] T ca907fb5b04f4d2492e7f4cea7114c95 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:20.972476 14220 raft_consensus.cc:504] T ca907fb5b04f4d2492e7f4cea7114c95 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:20.973002 14220 leader_election.cc:288] T ca907fb5b04f4d2492e7f4cea7114c95 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:20.973428 6719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca907fb5b04f4d2492e7f4cea7114c95" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 02:00:20.973589 6719 raft_consensus.cc:2454] T ca907fb5b04f4d2492e7f4cea7114c95 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 0.
I20240503 02:00:20.973587 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca907fb5b04f4d2492e7f4cea7114c95" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:20.973703 9179 raft_consensus.cc:2454] T ca907fb5b04f4d2492e7f4cea7114c95 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 0.
I20240503 02:00:20.973958 8853 leader_election.cc:302] T ca907fb5b04f4d2492e7f4cea7114c95 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9942306855404cfd891adf227dd65eb2, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:20.974124 14220 raft_consensus.cc:2781] T ca907fb5b04f4d2492e7f4cea7114c95 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:20.974182 14220 raft_consensus.cc:482] T ca907fb5b04f4d2492e7f4cea7114c95 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:20.974215 14220 raft_consensus.cc:3037] T ca907fb5b04f4d2492e7f4cea7114c95 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:20.975301 14220 raft_consensus.cc:504] T ca907fb5b04f4d2492e7f4cea7114c95 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:20.975672 14220 leader_election.cc:288] T ca907fb5b04f4d2492e7f4cea7114c95 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 election: Requested vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:20.976017 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca907fb5b04f4d2492e7f4cea7114c95" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:20.976140 9179 raft_consensus.cc:3037] T ca907fb5b04f4d2492e7f4cea7114c95 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:20.976045 6719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca907fb5b04f4d2492e7f4cea7114c95" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 02:00:20.976254 6719 raft_consensus.cc:3037] T ca907fb5b04f4d2492e7f4cea7114c95 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:20.977144 9179 raft_consensus.cc:2454] T ca907fb5b04f4d2492e7f4cea7114c95 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 1.
I20240503 02:00:20.977315 6719 raft_consensus.cc:2454] T ca907fb5b04f4d2492e7f4cea7114c95 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 1.
I20240503 02:00:20.977479 8850 leader_election.cc:302] T ca907fb5b04f4d2492e7f4cea7114c95 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:20.977644 14220 raft_consensus.cc:2781] T ca907fb5b04f4d2492e7f4cea7114c95 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:20.977861 14220 raft_consensus.cc:686] T ca907fb5b04f4d2492e7f4cea7114c95 P 9942306855404cfd891adf227dd65eb2 [term 1 LEADER]: Becoming Leader. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Running, Role: LEADER
I20240503 02:00:20.978013 14220 consensus_queue.cc:237] T ca907fb5b04f4d2492e7f4cea7114c95 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:20.980813 6314 catalog_manager.cc:5484] T ca907fb5b04f4d2492e7f4cea7114c95 P 9942306855404cfd891adf227dd65eb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9942306855404cfd891adf227dd65eb2 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "9942306855404cfd891adf227dd65eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: HEALTHY } } }
W20240503 02:00:20.987742 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:21.011898 13968 raft_consensus.cc:482] T 87c4b630a58749e6be9f26cceda27573 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:21.012059 13968 raft_consensus.cc:504] T 87c4b630a58749e6be9f26cceda27573 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:21.012598 13968 leader_election.cc:288] T 87c4b630a58749e6be9f26cceda27573 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:21.012908 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87c4b630a58749e6be9f26cceda27573" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 02:00:21.013052 6851 raft_consensus.cc:2454] T 87c4b630a58749e6be9f26cceda27573 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:21.013129 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87c4b630a58749e6be9f26cceda27573" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:21.013280 9048 raft_consensus.cc:2454] T 87c4b630a58749e6be9f26cceda27573 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:21.013388 9115 leader_election.cc:302] T 87c4b630a58749e6be9f26cceda27573 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 492448570cc94ed1879feb2ec3b9051d; no voters:
I20240503 02:00:21.013586 13968 raft_consensus.cc:2781] T 87c4b630a58749e6be9f26cceda27573 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:21.013643 13968 raft_consensus.cc:482] T 87c4b630a58749e6be9f26cceda27573 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:21.013674 13968 raft_consensus.cc:3037] T 87c4b630a58749e6be9f26cceda27573 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:21.014564 13968 raft_consensus.cc:504] T 87c4b630a58749e6be9f26cceda27573 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:21.014973 13968 leader_election.cc:288] T 87c4b630a58749e6be9f26cceda27573 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Requested vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:21.015368 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87c4b630a58749e6be9f26cceda27573" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 02:00:21.015372 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87c4b630a58749e6be9f26cceda27573" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:21.015511 6851 raft_consensus.cc:3037] T 87c4b630a58749e6be9f26cceda27573 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:21.015511 9048 raft_consensus.cc:3037] T 87c4b630a58749e6be9f26cceda27573 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:21.016580 6851 raft_consensus.cc:2454] T 87c4b630a58749e6be9f26cceda27573 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:21.016580 9048 raft_consensus.cc:2454] T 87c4b630a58749e6be9f26cceda27573 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:21.016903 9115 leader_election.cc:302] T 87c4b630a58749e6be9f26cceda27573 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 492448570cc94ed1879feb2ec3b9051d; no voters:
I20240503 02:00:21.017648 13968 raft_consensus.cc:2781] T 87c4b630a58749e6be9f26cceda27573 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:21.017796 13968 raft_consensus.cc:686] T 87c4b630a58749e6be9f26cceda27573 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Becoming Leader. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Running, Role: LEADER
I20240503 02:00:21.018183 13968 consensus_queue.cc:237] T 87c4b630a58749e6be9f26cceda27573 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:21.020735 6314 catalog_manager.cc:5484] T 87c4b630a58749e6be9f26cceda27573 P 2a01ace80baf4057b6bf1995cf88468b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135). New cstate: current_term: 1 leader_uuid: "2a01ace80baf4057b6bf1995cf88468b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:21.022698 6309 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:21.029265 14348 tablet.cc:1722] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:21.029645 14348 tablet.cc:1620] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a: MemRowSet was empty: no flush needed.
I20240503 02:00:21.029887 14319 tablet.cc:1722] T 451cd82331904b84884f7fdaf2b7dec9 P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:21.030205 14319 tablet.cc:1620] T 451cd82331904b84884f7fdaf2b7dec9 P 80fd18ef4d604c86b909d5c008a46b4f: MemRowSet was empty: no flush needed.
I20240503 02:00:21.036324 6314 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=364d2beeb13a4e77a5977e4c7d9786dc] alter complete (version 2)
I20240503 02:00:21.036685 14354 tablet.cc:1722] T 451cd82331904b84884f7fdaf2b7dec9 P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:21.037016 14354 tablet.cc:1620] T 451cd82331904b84884f7fdaf2b7dec9 P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
I20240503 02:00:21.104014 13900 raft_consensus.cc:482] T 3af89caa97634222ab834577a9217ce9 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:21.104142 13900 raft_consensus.cc:504] T 3af89caa97634222ab834577a9217ce9 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:21.104566 13900 leader_election.cc:288] T 3af89caa97634222ab834577a9217ce9 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:21.104874 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3af89caa97634222ab834577a9217ce9" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:21.104948 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3af89caa97634222ab834577a9217ce9" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 02:00:21.105038 8916 raft_consensus.cc:2454] T 3af89caa97634222ab834577a9217ce9 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:21.105074 6851 raft_consensus.cc:2454] T 3af89caa97634222ab834577a9217ce9 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:21.105324 8984 leader_election.cc:302] T 3af89caa97634222ab834577a9217ce9 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:21.105480 13900 raft_consensus.cc:2781] T 3af89caa97634222ab834577a9217ce9 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:21.105540 13900 raft_consensus.cc:482] T 3af89caa97634222ab834577a9217ce9 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:21.105587 13900 raft_consensus.cc:3037] T 3af89caa97634222ab834577a9217ce9 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:21.106415 13900 raft_consensus.cc:504] T 3af89caa97634222ab834577a9217ce9 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:21.106762 13900 leader_election.cc:288] T 3af89caa97634222ab834577a9217ce9 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Requested vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:21.107164 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3af89caa97634222ab834577a9217ce9" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:21.107164 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3af89caa97634222ab834577a9217ce9" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 02:00:21.107303 6851 raft_consensus.cc:3037] T 3af89caa97634222ab834577a9217ce9 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:21.107299 8916 raft_consensus.cc:3037] T 3af89caa97634222ab834577a9217ce9 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:21.108299 6851 raft_consensus.cc:2454] T 3af89caa97634222ab834577a9217ce9 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:21.108299 8916 raft_consensus.cc:2454] T 3af89caa97634222ab834577a9217ce9 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:21.108700 8984 leader_election.cc:302] T 3af89caa97634222ab834577a9217ce9 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9942306855404cfd891adf227dd65eb2, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:21.108886 13900 raft_consensus.cc:2781] T 3af89caa97634222ab834577a9217ce9 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:21.108969 13900 raft_consensus.cc:686] T 3af89caa97634222ab834577a9217ce9 P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Becoming Leader. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Running, Role: LEADER
I20240503 02:00:21.109098 13900 consensus_queue.cc:237] T 3af89caa97634222ab834577a9217ce9 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
W20240503 02:00:21.112915 6523 consensus_peers.cc:485] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:00:21.113802 6309 catalog_manager.cc:5484] T 3af89caa97634222ab834577a9217ce9 P 99ead02915954b4fab23b033ad6d546b reported cstate change: term changed from 0 to 1, leader changed from <none> to 99ead02915954b4fab23b033ad6d546b (127.0.99.134). New cstate: current_term: 1 leader_uuid: "99ead02915954b4fab23b033ad6d546b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } }
W20240503 02:00:21.138303 6785 consensus_peers.cc:485] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:00:21.141315 6309 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:21.146302 14348 tablet.cc:1722] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:21.146957 14354 tablet.cc:1722] T 451cd82331904b84884f7fdaf2b7dec9 P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:21.148751 14319 tablet.cc:1722] T 451cd82331904b84884f7fdaf2b7dec9 P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
W20240503 02:00:21.150354 6523 consensus_peers.cc:485] T d0d5dd42db3e4883af7fd16d8813b550 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:00:21.152753 6309 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=364d2beeb13a4e77a5977e4c7d9786dc] alter complete (version 3)
W20240503 02:00:21.182763 6523 consensus_peers.cc:485] T 925612b660ec471da5b3c548802ede6e P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:21.255148 8982 consensus_peers.cc:485] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:00:21.256206 6308 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:21.256341 6308 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:21.258425 6308 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet 451cd82331904b84884f7fdaf2b7dec9
I20240503 02:00:21.259097 6436 tablet_service.cc:1514] Processing DeleteTablet for tablet 451cd82331904b84884f7fdaf2b7dec9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:21 UTC) from {username='slave'} at 127.0.0.1:51338
I20240503 02:00:21.259395 6567 tablet_service.cc:1514] Processing DeleteTablet for tablet 451cd82331904b84884f7fdaf2b7dec9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:21 UTC) from {username='slave'} at 127.0.0.1:59846
I20240503 02:00:21.259438 14368 tablet_replica.cc:330] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a: stopping tablet replica
I20240503 02:00:21.259580 14368 raft_consensus.cc:2229] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:21.259769 14369 tablet_replica.cc:330] T 451cd82331904b84884f7fdaf2b7dec9 P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 02:00:21.259904 14368 raft_consensus.cc:2258] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:21.260051 14369 raft_consensus.cc:2229] T 451cd82331904b84884f7fdaf2b7dec9 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:21.260421 8887 tablet_service.cc:1514] Processing DeleteTablet for tablet 451cd82331904b84884f7fdaf2b7dec9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:21 UTC) from {username='slave'} at 127.0.0.1:40296
I20240503 02:00:21.260658 14368 ts_tablet_manager.cc:1875] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:21.260666 14369 raft_consensus.cc:2258] T 451cd82331904b84884f7fdaf2b7dec9 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:21.261742 14370 tablet_replica.cc:330] T 451cd82331904b84884f7fdaf2b7dec9 P 9942306855404cfd891adf227dd65eb2: stopping tablet replica
I20240503 02:00:21.261953 14370 raft_consensus.cc:2229] T 451cd82331904b84884f7fdaf2b7dec9 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:21.262107 14370 raft_consensus.cc:2258] T 451cd82331904b84884f7fdaf2b7dec9 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:21.262440 14369 ts_tablet_manager.cc:1875] T 451cd82331904b84884f7fdaf2b7dec9 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:21.262735 14370 ts_tablet_manager.cc:1875] T 451cd82331904b84884f7fdaf2b7dec9 P 9942306855404cfd891adf227dd65eb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:21.264307 14368 ts_tablet_manager.cc:1888] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:21.264478 14368 log.cc:1196] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/451cd82331904b84884f7fdaf2b7dec9
I20240503 02:00:21.264814 14368 ts_tablet_manager.cc:1909] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a: Deleting consensus metadata
I20240503 02:00:21.265072 14369 ts_tablet_manager.cc:1888] T 451cd82331904b84884f7fdaf2b7dec9 P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:21.265151 14369 log.cc:1196] T 451cd82331904b84884f7fdaf2b7dec9 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/451cd82331904b84884f7fdaf2b7dec9
I20240503 02:00:21.265509 14369 ts_tablet_manager.cc:1909] T 451cd82331904b84884f7fdaf2b7dec9 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting consensus metadata
I20240503 02:00:21.265604 6300 catalog_manager.cc:4830] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): tablet 451cd82331904b84884f7fdaf2b7dec9 (table rebalancer_dynamic_table_new_name [id=364d2beeb13a4e77a5977e4c7d9786dc]) successfully deleted
I20240503 02:00:21.265666 14370 ts_tablet_manager.cc:1888] T 451cd82331904b84884f7fdaf2b7dec9 P 9942306855404cfd891adf227dd65eb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:21.265727 14370 log.cc:1196] T 451cd82331904b84884f7fdaf2b7dec9 P 9942306855404cfd891adf227dd65eb2: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/451cd82331904b84884f7fdaf2b7dec9
I20240503 02:00:21.266012 14370 ts_tablet_manager.cc:1909] T 451cd82331904b84884f7fdaf2b7dec9 P 9942306855404cfd891adf227dd65eb2: Deleting consensus metadata
I20240503 02:00:21.266448 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet 451cd82331904b84884f7fdaf2b7dec9 (table rebalancer_dynamic_table_new_name [id=364d2beeb13a4e77a5977e4c7d9786dc]) successfully deleted
I20240503 02:00:21.266707 6303 catalog_manager.cc:4830] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211): tablet 451cd82331904b84884f7fdaf2b7dec9 (table rebalancer_dynamic_table_new_name [id=364d2beeb13a4e77a5977e4c7d9786dc]) successfully deleted
W20240503 02:00:21.286124 9113 consensus_peers.cc:485] T f44b59277c3b47adbd4dd6aff9350af1 P 2a01ace80baf4057b6bf1995cf88468b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:21.297313 8851 consensus_peers.cc:485] T 067f54011a37432d99157884e70fc601 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:00:21.361188 6308 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
W20240503 02:00:21.365478 6785 consensus_peers.cc:485] T 46ad540deb624a25b97d3b967fe1f6f4 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:00:21.368053 6436 tablet_service.cc:1467] Processing CreateTablet for tablet d22b0105836a43e09a8de16843474dfc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1fe7d1ed4aee47f1ae55ec3c13859000]), partition=
I20240503 02:00:21.368243 9158 tablet_service.cc:1467] Processing CreateTablet for tablet d22b0105836a43e09a8de16843474dfc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1fe7d1ed4aee47f1ae55ec3c13859000]), partition=
I20240503 02:00:21.368324 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d22b0105836a43e09a8de16843474dfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:21.368542 9158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d22b0105836a43e09a8de16843474dfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:21.368649 6700 tablet_service.cc:1467] Processing CreateTablet for tablet d22b0105836a43e09a8de16843474dfc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1fe7d1ed4aee47f1ae55ec3c13859000]), partition=
I20240503 02:00:21.368901 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d22b0105836a43e09a8de16843474dfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:21.370748 14351 tablet_bootstrap.cc:492] T d22b0105836a43e09a8de16843474dfc P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:21.371016 14359 tablet_bootstrap.cc:492] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:21.371796 14351 tablet_bootstrap.cc:654] T d22b0105836a43e09a8de16843474dfc P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:21.371855 14359 tablet_bootstrap.cc:654] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:21.373106 14359 tablet_bootstrap.cc:492] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:21.373194 14359 ts_tablet_manager.cc:1367] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:21.373374 14351 tablet_bootstrap.cc:492] T d22b0105836a43e09a8de16843474dfc P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:21.373456 14351 ts_tablet_manager.cc:1367] T d22b0105836a43e09a8de16843474dfc P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:21.373632 14359 raft_consensus.cc:348] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:21.373733 14359 raft_consensus.cc:374] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:21.373762 14359 raft_consensus.cc:729] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:21.373876 14359 consensus_queue.cc:260] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:21.373905 14351 raft_consensus.cc:348] T d22b0105836a43e09a8de16843474dfc P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:21.374022 14351 raft_consensus.cc:374] T d22b0105836a43e09a8de16843474dfc P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:21.374051 14351 raft_consensus.cc:729] T d22b0105836a43e09a8de16843474dfc P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:21.374178 14351 consensus_queue.cc:260] T d22b0105836a43e09a8de16843474dfc P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:21.374251 14359 ts_tablet_manager.cc:1398] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:21.374467 14351 ts_tablet_manager.cc:1398] T d22b0105836a43e09a8de16843474dfc P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:21.375849 14374 tablet_bootstrap.cc:492] T d22b0105836a43e09a8de16843474dfc P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:21.376953 14374 tablet_bootstrap.cc:654] T d22b0105836a43e09a8de16843474dfc P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:21.380280 14374 tablet_bootstrap.cc:492] T d22b0105836a43e09a8de16843474dfc P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:21.380506 14374 ts_tablet_manager.cc:1367] T d22b0105836a43e09a8de16843474dfc P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:00:21.381341 14374 raft_consensus.cc:348] T d22b0105836a43e09a8de16843474dfc P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:21.381561 14374 raft_consensus.cc:374] T d22b0105836a43e09a8de16843474dfc P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:21.381644 14374 raft_consensus.cc:729] T d22b0105836a43e09a8de16843474dfc P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:21.381793 14374 consensus_queue.cc:260] T d22b0105836a43e09a8de16843474dfc P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:21.382915 14374 ts_tablet_manager.cc:1398] T d22b0105836a43e09a8de16843474dfc P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:21.388321 14371 raft_consensus.cc:482] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:21.388478 14371 raft_consensus.cc:504] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:21.389142 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d22b0105836a43e09a8de16843474dfc" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 02:00:21.389286 6456 raft_consensus.cc:2454] T d22b0105836a43e09a8de16843474dfc P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:21.389418 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d22b0105836a43e09a8de16843474dfc" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:21.389549 9179 raft_consensus.cc:2454] T d22b0105836a43e09a8de16843474dfc P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:21.389567 6653 leader_election.cc:302] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2ac2eb7cc264047b9ab32fe3a4fc025, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:21.389725 14284 raft_consensus.cc:2781] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:21.389743 14371 leader_election.cc:288] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:21.389814 14284 raft_consensus.cc:482] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:21.389890 14284 raft_consensus.cc:3037] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:21.390794 14284 raft_consensus.cc:504] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:21.391152 14284 leader_election.cc:288] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Requested vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:21.391534 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d22b0105836a43e09a8de16843474dfc" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 02:00:21.391657 6456 raft_consensus.cc:3037] T d22b0105836a43e09a8de16843474dfc P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:21.391646 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d22b0105836a43e09a8de16843474dfc" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:21.391758 9179 raft_consensus.cc:3037] T d22b0105836a43e09a8de16843474dfc P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:21.392642 6456 raft_consensus.cc:2454] T d22b0105836a43e09a8de16843474dfc P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:21.392747 9179 raft_consensus.cc:2454] T d22b0105836a43e09a8de16843474dfc P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:21.393132 6653 leader_election.cc:302] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:21.393349 14284 raft_consensus.cc:2781] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:21.393435 14284 raft_consensus.cc:686] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Becoming Leader. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Running, Role: LEADER
I20240503 02:00:21.393559 14284 consensus_queue.cc:237] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
W20240503 02:00:21.394825 6785 consensus_peers.cc:485] T d897c21e630d4770832bb10f43778e82 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:00:21.396116 6309 catalog_manager.cc:5484] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } }
W20240503 02:00:21.431757 6785 consensus_peers.cc:485] T 8246488dec204bf6b09e868d7951ac5a P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:21.436466 6654 consensus_peers.cc:485] T cea1ea50fa434d31b42451f84f26bc76 P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:21.449671 6523 consensus_peers.cc:485] T 4385d51d11574192883d01376e2d5620 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:21.458770 8851 consensus_peers.cc:485] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:00:21.468163 14220 consensus_queue.cc:1035] T ca907fb5b04f4d2492e7f4cea7114c95 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:21.469563 14220 consensus_queue.cc:1035] T ca907fb5b04f4d2492e7f4cea7114c95 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:21.500659 6309 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:21.505206 9179 raft_consensus.cc:1261] T d22b0105836a43e09a8de16843474dfc P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:21.505661 14284 consensus_queue.cc:1035] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:21.505792 6456 raft_consensus.cc:1261] T d22b0105836a43e09a8de16843474dfc P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:21.506172 14284 consensus_queue.cc:1035] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:00:21.508189 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:21.510370 6308 catalog_manager.cc:5576] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093) does not have the latest schema for tablet d22b0105836a43e09a8de16843474dfc (table rebalancer_dynamic_table [id=1fe7d1ed4aee47f1ae55ec3c13859000]). Expected version 1 got 0
I20240503 02:00:21.510360 14348 tablet.cc:1722] T d22b0105836a43e09a8de16843474dfc P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:21.510720 14348 tablet.cc:1620] T d22b0105836a43e09a8de16843474dfc P dec3e4a04d604f7eaaa3f077d081218a: MemRowSet was empty: no flush needed.
I20240503 02:00:21.512770 6309 catalog_manager.cc:5576] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079) does not have the latest schema for tablet d22b0105836a43e09a8de16843474dfc (table rebalancer_dynamic_table [id=1fe7d1ed4aee47f1ae55ec3c13859000]). Expected version 1 got 0
I20240503 02:00:21.514603 13968 consensus_queue.cc:1035] T 87c4b630a58749e6be9f26cceda27573 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:21.515468 6308 catalog_manager.cc:5576] TS c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655) does not have the latest schema for tablet d22b0105836a43e09a8de16843474dfc (table rebalancer_dynamic_table [id=1fe7d1ed4aee47f1ae55ec3c13859000]). Expected version 1 got 0
I20240503 02:00:21.517808 14384 tablet.cc:1722] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:21.518448 14384 tablet.cc:1620] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025: MemRowSet was empty: no flush needed.
I20240503 02:00:21.519250 13968 consensus_queue.cc:1035] T 87c4b630a58749e6be9f26cceda27573 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:21.520107 6313 catalog_manager.cc:5576] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079) does not have the latest schema for tablet d22b0105836a43e09a8de16843474dfc (table rebalancer_dynamic_table [id=1fe7d1ed4aee47f1ae55ec3c13859000]). Expected version 1 got 0
I20240503 02:00:21.520311 6314 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=1fe7d1ed4aee47f1ae55ec3c13859000] alter complete (version 1)
I20240503 02:00:21.525693 14385 tablet.cc:1722] T d22b0105836a43e09a8de16843474dfc P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:21.526095 14385 tablet.cc:1620] T d22b0105836a43e09a8de16843474dfc P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
W20240503 02:00:21.592730 8851 consensus_peers.cc:485] T f28586e70bd9473fbdc6b9126b666fa9 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:21.614953 9113 consensus_peers.cc:485] T b1740b877c1e4372a0337d99f5bcab0e P 2a01ace80baf4057b6bf1995cf88468b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:00:21.623046 6313 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_extra_table_56"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:21.624662 6309 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:21.630523 6567 tablet_service.cc:1467] Processing CreateTablet for tablet ea801f686a0a4efa9da66b24bdaa10cc (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=f0c22c72d4c14f279e3366a16e75aae5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:21.630878 6567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea801f686a0a4efa9da66b24bdaa10cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:21.630955 9028 tablet_service.cc:1467] Processing CreateTablet for tablet ea801f686a0a4efa9da66b24bdaa10cc (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=f0c22c72d4c14f279e3366a16e75aae5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:21.631186 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea801f686a0a4efa9da66b24bdaa10cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:21.632283 6569 tablet_service.cc:1467] Processing CreateTablet for tablet 98b3ee95a66e43788e5e2a428d9fa9c6 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=f0c22c72d4c14f279e3366a16e75aae5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:21.632560 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98b3ee95a66e43788e5e2a428d9fa9c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:21.632534 9027 tablet_service.cc:1467] Processing CreateTablet for tablet 8a8b7b032eea4726b24b805236601450 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=f0c22c72d4c14f279e3366a16e75aae5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:21.632797 9027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a8b7b032eea4726b24b805236601450. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:21.633651 6568 tablet_service.cc:1467] Processing CreateTablet for tablet 8a8b7b032eea4726b24b805236601450 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=f0c22c72d4c14f279e3366a16e75aae5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:21.633862 6568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a8b7b032eea4726b24b805236601450. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:21.634783 14392 tablet_bootstrap.cc:492] T ea801f686a0a4efa9da66b24bdaa10cc P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:21.634894 8897 tablet_service.cc:1467] Processing CreateTablet for tablet 8a8b7b032eea4726b24b805236601450 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=f0c22c72d4c14f279e3366a16e75aae5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:21.635157 8897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a8b7b032eea4726b24b805236601450. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:21.635911 8887 tablet_service.cc:1467] Processing CreateTablet for tablet ea801f686a0a4efa9da66b24bdaa10cc (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=f0c22c72d4c14f279e3366a16e75aae5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:21.636142 8887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea801f686a0a4efa9da66b24bdaa10cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:21.636322 14393 tablet_bootstrap.cc:492] T ea801f686a0a4efa9da66b24bdaa10cc P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:21.636713 14392 tablet_bootstrap.cc:654] T ea801f686a0a4efa9da66b24bdaa10cc P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:21.634902 8880 tablet_service.cc:1467] Processing CreateTablet for tablet 98b3ee95a66e43788e5e2a428d9fa9c6 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=f0c22c72d4c14f279e3366a16e75aae5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:21.637583 8880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98b3ee95a66e43788e5e2a428d9fa9c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:21.638765 6436 tablet_service.cc:1467] Processing CreateTablet for tablet 98b3ee95a66e43788e5e2a428d9fa9c6 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=f0c22c72d4c14f279e3366a16e75aae5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:21.639084 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98b3ee95a66e43788e5e2a428d9fa9c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:21.639767 14394 tablet_bootstrap.cc:492] T 8a8b7b032eea4726b24b805236601450 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:21.640694 14348 tablet.cc:1722] T d22b0105836a43e09a8de16843474dfc P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:21.640964 14394 tablet_bootstrap.cc:654] T 8a8b7b032eea4726b24b805236601450 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:21.641011 14348 tablet.cc:1620] T d22b0105836a43e09a8de16843474dfc P dec3e4a04d604f7eaaa3f077d081218a: MemRowSet was empty: no flush needed.
I20240503 02:00:21.642570 14394 tablet_bootstrap.cc:492] T 8a8b7b032eea4726b24b805236601450 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:21.642714 14394 ts_tablet_manager.cc:1367] T 8a8b7b032eea4726b24b805236601450 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:00:21.642678 14385 tablet.cc:1722] T d22b0105836a43e09a8de16843474dfc P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:21.643018 14385 tablet.cc:1620] T d22b0105836a43e09a8de16843474dfc P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
I20240503 02:00:21.644688 14384 tablet.cc:1722] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:21.645021 14384 tablet.cc:1620] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025: MemRowSet was empty: no flush needed.
I20240503 02:00:21.645748 14394 raft_consensus.cc:348] T 8a8b7b032eea4726b24b805236601450 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:21.645956 14394 raft_consensus.cc:374] T 8a8b7b032eea4726b24b805236601450 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:21.646013 14394 raft_consensus.cc:729] T 8a8b7b032eea4726b24b805236601450 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:21.646159 14394 consensus_queue.cc:260] T 8a8b7b032eea4726b24b805236601450 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:21.646669 14374 tablet_bootstrap.cc:492] T 98b3ee95a66e43788e5e2a428d9fa9c6 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:21.646695 14394 ts_tablet_manager.cc:1398] T 8a8b7b032eea4726b24b805236601450 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240503 02:00:21.647049 14394 tablet_bootstrap.cc:492] T ea801f686a0a4efa9da66b24bdaa10cc P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:21.648200 14374 tablet_bootstrap.cc:654] T 98b3ee95a66e43788e5e2a428d9fa9c6 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:21.652940 14374 tablet_bootstrap.cc:492] T 98b3ee95a66e43788e5e2a428d9fa9c6 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:21.653065 14374 ts_tablet_manager.cc:1367] T 98b3ee95a66e43788e5e2a428d9fa9c6 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:00:21.653553 14374 raft_consensus.cc:348] T 98b3ee95a66e43788e5e2a428d9fa9c6 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:21.653687 14374 raft_consensus.cc:374] T 98b3ee95a66e43788e5e2a428d9fa9c6 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:21.653719 14374 raft_consensus.cc:729] T 98b3ee95a66e43788e5e2a428d9fa9c6 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:21.653486 6314 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=1fe7d1ed4aee47f1ae55ec3c13859000] alter complete (version 2)
I20240503 02:00:21.653836 14374 consensus_queue.cc:260] T 98b3ee95a66e43788e5e2a428d9fa9c6 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:21.654136 14374 ts_tablet_manager.cc:1398] T 98b3ee95a66e43788e5e2a428d9fa9c6 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:21.654800 14394 tablet_bootstrap.cc:654] T ea801f686a0a4efa9da66b24bdaa10cc P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:21.656265 13900 consensus_queue.cc:1035] T 3af89caa97634222ab834577a9217ce9 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:21.658380 14393 tablet_bootstrap.cc:654] T ea801f686a0a4efa9da66b24bdaa10cc P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:21.659833 14392 tablet_bootstrap.cc:492] T ea801f686a0a4efa9da66b24bdaa10cc P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:21.659955 14392 ts_tablet_manager.cc:1367] T ea801f686a0a4efa9da66b24bdaa10cc P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.000s
I20240503 02:00:21.660526 14392 raft_consensus.cc:348] T ea801f686a0a4efa9da66b24bdaa10cc P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:21.660665 14392 raft_consensus.cc:374] T ea801f686a0a4efa9da66b24bdaa10cc P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:21.660699 14392 raft_consensus.cc:729] T ea801f686a0a4efa9da66b24bdaa10cc P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:21.660813 14392 consensus_queue.cc:260] T ea801f686a0a4efa9da66b24bdaa10cc P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:21.660953 14393 tablet_bootstrap.cc:492] T ea801f686a0a4efa9da66b24bdaa10cc P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:21.661053 14393 ts_tablet_manager.cc:1367] T ea801f686a0a4efa9da66b24bdaa10cc P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.000s
I20240503 02:00:21.661113 14392 ts_tablet_manager.cc:1398] T ea801f686a0a4efa9da66b24bdaa10cc P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:21.661288 14392 tablet_bootstrap.cc:492] T 8a8b7b032eea4726b24b805236601450 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:21.661518 14393 raft_consensus.cc:348] T ea801f686a0a4efa9da66b24bdaa10cc P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:21.661614 14393 raft_consensus.cc:374] T ea801f686a0a4efa9da66b24bdaa10cc P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:21.661643 14393 raft_consensus.cc:729] T ea801f686a0a4efa9da66b24bdaa10cc P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:21.661769 14393 consensus_queue.cc:260] T ea801f686a0a4efa9da66b24bdaa10cc P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:21.662027 14393 ts_tablet_manager.cc:1398] T ea801f686a0a4efa9da66b24bdaa10cc P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:21.662303 14392 tablet_bootstrap.cc:654] T 8a8b7b032eea4726b24b805236601450 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:21.664170 14393 tablet_bootstrap.cc:492] T 98b3ee95a66e43788e5e2a428d9fa9c6 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:21.664839 14195 consensus_queue.cc:1035] T 3af89caa97634222ab834577a9217ce9 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:21.665263 14393 tablet_bootstrap.cc:654] T 98b3ee95a66e43788e5e2a428d9fa9c6 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:21.668949 14392 tablet_bootstrap.cc:492] T 8a8b7b032eea4726b24b805236601450 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:21.669055 14392 ts_tablet_manager.cc:1367] T 8a8b7b032eea4726b24b805236601450 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 02:00:21.670464 14392 raft_consensus.cc:348] T 8a8b7b032eea4726b24b805236601450 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:21.670715 14392 raft_consensus.cc:374] T 8a8b7b032eea4726b24b805236601450 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:21.670804 14392 raft_consensus.cc:729] T 8a8b7b032eea4726b24b805236601450 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:21.671263 14392 consensus_queue.cc:260] T 8a8b7b032eea4726b24b805236601450 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:21.672621 14392 ts_tablet_manager.cc:1398] T 8a8b7b032eea4726b24b805236601450 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:21.673786 14394 tablet_bootstrap.cc:492] T ea801f686a0a4efa9da66b24bdaa10cc P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:21.673897 14394 ts_tablet_manager.cc:1367] T ea801f686a0a4efa9da66b24bdaa10cc P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.027s user 0.000s sys 0.003s
I20240503 02:00:21.674424 14394 raft_consensus.cc:348] T ea801f686a0a4efa9da66b24bdaa10cc P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:21.674659 14394 raft_consensus.cc:374] T ea801f686a0a4efa9da66b24bdaa10cc P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:21.674710 14394 raft_consensus.cc:729] T ea801f686a0a4efa9da66b24bdaa10cc P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:21.674822 14394 consensus_queue.cc:260] T ea801f686a0a4efa9da66b24bdaa10cc P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:21.675112 14394 ts_tablet_manager.cc:1398] T ea801f686a0a4efa9da66b24bdaa10cc P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:21.675383 14394 tablet_bootstrap.cc:492] T 98b3ee95a66e43788e5e2a428d9fa9c6 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:21.676635 14394 tablet_bootstrap.cc:654] T 98b3ee95a66e43788e5e2a428d9fa9c6 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:21.676753 14393 tablet_bootstrap.cc:492] T 98b3ee95a66e43788e5e2a428d9fa9c6 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:21.676857 14393 ts_tablet_manager.cc:1367] T 98b3ee95a66e43788e5e2a428d9fa9c6 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 02:00:21.677352 14393 raft_consensus.cc:348] T 98b3ee95a66e43788e5e2a428d9fa9c6 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:21.677477 14393 raft_consensus.cc:374] T 98b3ee95a66e43788e5e2a428d9fa9c6 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:21.677507 14393 raft_consensus.cc:729] T 98b3ee95a66e43788e5e2a428d9fa9c6 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:21.677632 14393 consensus_queue.cc:260] T 98b3ee95a66e43788e5e2a428d9fa9c6 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:21.677896 14393 ts_tablet_manager.cc:1398] T 98b3ee95a66e43788e5e2a428d9fa9c6 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:21.678067 14393 tablet_bootstrap.cc:492] T 8a8b7b032eea4726b24b805236601450 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:21.678112 14394 tablet_bootstrap.cc:492] T 98b3ee95a66e43788e5e2a428d9fa9c6 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:21.678198 14394 ts_tablet_manager.cc:1367] T 98b3ee95a66e43788e5e2a428d9fa9c6 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:21.678634 14394 raft_consensus.cc:348] T 98b3ee95a66e43788e5e2a428d9fa9c6 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:21.678753 14394 raft_consensus.cc:374] T 98b3ee95a66e43788e5e2a428d9fa9c6 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:21.678784 14394 raft_consensus.cc:729] T 98b3ee95a66e43788e5e2a428d9fa9c6 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:21.678941 14394 consensus_queue.cc:260] T 98b3ee95a66e43788e5e2a428d9fa9c6 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:21.679380 14394 ts_tablet_manager.cc:1398] T 98b3ee95a66e43788e5e2a428d9fa9c6 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:21.680756 14393 tablet_bootstrap.cc:654] T 8a8b7b032eea4726b24b805236601450 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:21.682283 14393 tablet_bootstrap.cc:492] T 8a8b7b032eea4726b24b805236601450 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:21.682408 14393 ts_tablet_manager.cc:1367] T 8a8b7b032eea4726b24b805236601450 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:00:21.682907 14393 raft_consensus.cc:348] T 8a8b7b032eea4726b24b805236601450 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:21.683035 14393 raft_consensus.cc:374] T 8a8b7b032eea4726b24b805236601450 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:21.683068 14393 raft_consensus.cc:729] T 8a8b7b032eea4726b24b805236601450 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:21.683197 14393 consensus_queue.cc:260] T 8a8b7b032eea4726b24b805236601450 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:21.683486 14393 ts_tablet_manager.cc:1398] T 8a8b7b032eea4726b24b805236601450 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240503 02:00:21.691885 8851 consensus_peers.cc:485] T 5f51fe5f799f4e479b62f5917a2c2606 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:21.744040 8982 consensus_peers.cc:485] T 13756a6e44814fe4b09a6b0f087546c3 P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:00:21.751487 14323 raft_consensus.cc:482] T ea801f686a0a4efa9da66b24bdaa10cc P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:21.751601 14323 raft_consensus.cc:504] T ea801f686a0a4efa9da66b24bdaa10cc P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:21.752008 14323 leader_election.cc:288] T ea801f686a0a4efa9da66b24bdaa10cc P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:21.752394 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea801f686a0a4efa9da66b24bdaa10cc" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:21.752463 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea801f686a0a4efa9da66b24bdaa10cc" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:21.752544 8916 raft_consensus.cc:2454] T ea801f686a0a4efa9da66b24bdaa10cc P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:21.752586 9048 raft_consensus.cc:2454] T ea801f686a0a4efa9da66b24bdaa10cc P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:21.752877 6525 leader_election.cc:302] T ea801f686a0a4efa9da66b24bdaa10cc P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:21.753043 14323 raft_consensus.cc:2781] T ea801f686a0a4efa9da66b24bdaa10cc P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:21.753087 14323 raft_consensus.cc:482] T ea801f686a0a4efa9da66b24bdaa10cc P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:21.753121 14323 raft_consensus.cc:3037] T ea801f686a0a4efa9da66b24bdaa10cc P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:21.754398 14323 raft_consensus.cc:504] T ea801f686a0a4efa9da66b24bdaa10cc P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:21.754741 14323 leader_election.cc:288] T ea801f686a0a4efa9da66b24bdaa10cc P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Requested vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:21.755156 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea801f686a0a4efa9da66b24bdaa10cc" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:21.755275 9048 raft_consensus.cc:3037] T ea801f686a0a4efa9da66b24bdaa10cc P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:21.756227 9048 raft_consensus.cc:2454] T ea801f686a0a4efa9da66b24bdaa10cc P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:21.756513 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea801f686a0a4efa9da66b24bdaa10cc" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:21.756635 8916 raft_consensus.cc:3037] T ea801f686a0a4efa9da66b24bdaa10cc P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:21.757608 6523 leader_election.cc:302] T ea801f686a0a4efa9da66b24bdaa10cc P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:21.757619 8916 raft_consensus.cc:2454] T ea801f686a0a4efa9da66b24bdaa10cc P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:21.757786 14323 raft_consensus.cc:2781] T ea801f686a0a4efa9da66b24bdaa10cc P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:21.757987 14323 raft_consensus.cc:686] T ea801f686a0a4efa9da66b24bdaa10cc P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Becoming Leader. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Running, Role: LEADER
I20240503 02:00:21.758141 14323 consensus_queue.cc:237] T ea801f686a0a4efa9da66b24bdaa10cc P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:21.759351 6314 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:21.760640 6309 catalog_manager.cc:5484] T ea801f686a0a4efa9da66b24bdaa10cc P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:21.765852 14384 tablet.cc:1722] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:21.766635 14348 tablet.cc:1722] T d22b0105836a43e09a8de16843474dfc P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:21.770128 14385 tablet.cc:1722] T d22b0105836a43e09a8de16843474dfc P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:21.772284 14323 raft_consensus.cc:482] T 8a8b7b032eea4726b24b805236601450 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:21.772418 14323 raft_consensus.cc:504] T 8a8b7b032eea4726b24b805236601450 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:21.772812 14323 leader_election.cc:288] T 8a8b7b032eea4726b24b805236601450 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:21.773130 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8b7b032eea4726b24b805236601450" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:21.773262 8916 raft_consensus.cc:2454] T 8a8b7b032eea4726b24b805236601450 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:21.773579 6525 leader_election.cc:302] T 8a8b7b032eea4726b24b805236601450 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:21.773747 14323 raft_consensus.cc:2781] T 8a8b7b032eea4726b24b805236601450 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:21.773793 14323 raft_consensus.cc:482] T 8a8b7b032eea4726b24b805236601450 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:21.773859 14323 raft_consensus.cc:3037] T 8a8b7b032eea4726b24b805236601450 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:21.774812 14323 raft_consensus.cc:504] T 8a8b7b032eea4726b24b805236601450 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:21.775182 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8b7b032eea4726b24b805236601450" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:21.775417 9048 raft_consensus.cc:2454] T 8a8b7b032eea4726b24b805236601450 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:21.775537 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8b7b032eea4726b24b805236601450" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:21.775656 8916 raft_consensus.cc:3037] T 8a8b7b032eea4726b24b805236601450 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:21.776762 8916 raft_consensus.cc:2454] T 8a8b7b032eea4726b24b805236601450 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:21.777174 6525 leader_election.cc:302] T 8a8b7b032eea4726b24b805236601450 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:21.777441 14323 leader_election.cc:288] T 8a8b7b032eea4726b24b805236601450 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Requested vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:21.777562 14323 raft_consensus.cc:2781] T 8a8b7b032eea4726b24b805236601450 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:21.777632 14323 raft_consensus.cc:686] T 8a8b7b032eea4726b24b805236601450 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Becoming Leader. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Running, Role: LEADER
I20240503 02:00:21.777774 14323 consensus_queue.cc:237] T 8a8b7b032eea4726b24b805236601450 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:21.777824 9047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8b7b032eea4726b24b805236601450" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:21.777922 9047 raft_consensus.cc:3037] T 8a8b7b032eea4726b24b805236601450 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:21.778028 6314 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=1fe7d1ed4aee47f1ae55ec3c13859000] alter complete (version 3)
I20240503 02:00:21.778952 9047 raft_consensus.cc:2454] T 8a8b7b032eea4726b24b805236601450 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:21.781087 6313 catalog_manager.cc:5484] T 8a8b7b032eea4726b24b805236601450 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } }
W20240503 02:00:21.818346 8851 consensus_peers.cc:485] T 7da8f614a70e4177b70e7c2f31d22a10 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:00:21.821183 14153 raft_consensus.cc:482] T 98b3ee95a66e43788e5e2a428d9fa9c6 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:21.821336 14153 raft_consensus.cc:504] T 98b3ee95a66e43788e5e2a428d9fa9c6 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:21.821748 14153 leader_election.cc:288] T 98b3ee95a66e43788e5e2a428d9fa9c6 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:21.822206 6587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98b3ee95a66e43788e5e2a428d9fa9c6" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 02:00:21.822202 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98b3ee95a66e43788e5e2a428d9fa9c6" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:21.822337 6587 raft_consensus.cc:2454] T 98b3ee95a66e43788e5e2a428d9fa9c6 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:21.822356 8916 raft_consensus.cc:2454] T 98b3ee95a66e43788e5e2a428d9fa9c6 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:21.822675 6393 leader_election.cc:302] T 98b3ee95a66e43788e5e2a428d9fa9c6 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:21.822851 14153 raft_consensus.cc:2781] T 98b3ee95a66e43788e5e2a428d9fa9c6 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:21.822921 14153 raft_consensus.cc:482] T 98b3ee95a66e43788e5e2a428d9fa9c6 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:21.822952 14153 raft_consensus.cc:3037] T 98b3ee95a66e43788e5e2a428d9fa9c6 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:21.823853 14153 raft_consensus.cc:504] T 98b3ee95a66e43788e5e2a428d9fa9c6 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:21.824236 14153 leader_election.cc:288] T 98b3ee95a66e43788e5e2a428d9fa9c6 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Requested vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:21.824640 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98b3ee95a66e43788e5e2a428d9fa9c6" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:21.824643 6587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98b3ee95a66e43788e5e2a428d9fa9c6" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 02:00:21.824784 8916 raft_consensus.cc:3037] T 98b3ee95a66e43788e5e2a428d9fa9c6 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:21.824797 6587 raft_consensus.cc:3037] T 98b3ee95a66e43788e5e2a428d9fa9c6 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:21.825796 8916 raft_consensus.cc:2454] T 98b3ee95a66e43788e5e2a428d9fa9c6 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:21.825805 6587 raft_consensus.cc:2454] T 98b3ee95a66e43788e5e2a428d9fa9c6 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:21.826107 6393 leader_election.cc:302] T 98b3ee95a66e43788e5e2a428d9fa9c6 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:21.826306 14153 raft_consensus.cc:2781] T 98b3ee95a66e43788e5e2a428d9fa9c6 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:21.826407 14153 raft_consensus.cc:686] T 98b3ee95a66e43788e5e2a428d9fa9c6 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Becoming Leader. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Running, Role: LEADER
I20240503 02:00:21.826563 14153 consensus_queue.cc:237] T 98b3ee95a66e43788e5e2a428d9fa9c6 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:21.829187 6313 catalog_manager.cc:5484] T 98b3ee95a66e43788e5e2a428d9fa9c6 P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: term changed from 0 to 1, leader changed from <none> to dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129). New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } }
W20240503 02:00:21.864775 6523 consensus_peers.cc:485] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:21.866426 6523 consensus_peers.cc:485] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:21.879271 8982 consensus_peers.cc:485] T 1a7ee79312ea4bad816264375ca5bb2d P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:21.882658 8982 consensus_peers.cc:485] T beeea71f02ae4c89a1af13d3584ef70a P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:00:21.883751 6313 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:21.883920 6313 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:21.886737 6313 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet d22b0105836a43e09a8de16843474dfc
I20240503 02:00:21.887353 6436 tablet_service.cc:1514] Processing DeleteTablet for tablet d22b0105836a43e09a8de16843474dfc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:21 UTC) from {username='slave'} at 127.0.0.1:51338
I20240503 02:00:21.887411 9158 tablet_service.cc:1514] Processing DeleteTablet for tablet d22b0105836a43e09a8de16843474dfc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:21 UTC) from {username='slave'} at 127.0.0.1:51940
I20240503 02:00:21.887691 6700 tablet_service.cc:1514] Processing DeleteTablet for tablet d22b0105836a43e09a8de16843474dfc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:21 UTC) from {username='slave'} at 127.0.0.1:40336
I20240503 02:00:21.887740 14411 tablet_replica.cc:330] T d22b0105836a43e09a8de16843474dfc P 2a01ace80baf4057b6bf1995cf88468b: stopping tablet replica
I20240503 02:00:21.887691 14410 tablet_replica.cc:330] T d22b0105836a43e09a8de16843474dfc P dec3e4a04d604f7eaaa3f077d081218a: stopping tablet replica
I20240503 02:00:21.887877 14411 raft_consensus.cc:2229] T d22b0105836a43e09a8de16843474dfc P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:21.887935 14410 raft_consensus.cc:2229] T d22b0105836a43e09a8de16843474dfc P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:21.887991 14412 tablet_replica.cc:330] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025: stopping tablet replica
I20240503 02:00:21.888013 14411 raft_consensus.cc:2258] T d22b0105836a43e09a8de16843474dfc P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:21.888084 14410 raft_consensus.cc:2258] T d22b0105836a43e09a8de16843474dfc P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:21.888653 14411 ts_tablet_manager.cc:1875] T d22b0105836a43e09a8de16843474dfc P 2a01ace80baf4057b6bf1995cf88468b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:21.889735 14412 raft_consensus.cc:2229] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:21.889941 14410 ts_tablet_manager.cc:1875] T d22b0105836a43e09a8de16843474dfc P dec3e4a04d604f7eaaa3f077d081218a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:21.889999 14412 raft_consensus.cc:2258] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:21.890507 14412 ts_tablet_manager.cc:1875] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:21.890687 14411 ts_tablet_manager.cc:1888] T d22b0105836a43e09a8de16843474dfc P 2a01ace80baf4057b6bf1995cf88468b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:21.890800 14411 log.cc:1196] T d22b0105836a43e09a8de16843474dfc P 2a01ace80baf4057b6bf1995cf88468b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/d22b0105836a43e09a8de16843474dfc
I20240503 02:00:21.891059 14411 ts_tablet_manager.cc:1909] T d22b0105836a43e09a8de16843474dfc P 2a01ace80baf4057b6bf1995cf88468b: Deleting consensus metadata
I20240503 02:00:21.891878 6300 catalog_manager.cc:4830] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079): tablet d22b0105836a43e09a8de16843474dfc (table rebalancer_dynamic_table_new_name [id=1fe7d1ed4aee47f1ae55ec3c13859000]) successfully deleted
I20240503 02:00:21.892616 14410 ts_tablet_manager.cc:1888] T d22b0105836a43e09a8de16843474dfc P dec3e4a04d604f7eaaa3f077d081218a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:21.892704 14410 log.cc:1196] T d22b0105836a43e09a8de16843474dfc P dec3e4a04d604f7eaaa3f077d081218a: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/d22b0105836a43e09a8de16843474dfc
I20240503 02:00:21.892741 14412 ts_tablet_manager.cc:1888] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:21.892812 14412 log.cc:1196] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/d22b0105836a43e09a8de16843474dfc
I20240503 02:00:21.893128 14412 ts_tablet_manager.cc:1909] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting consensus metadata
I20240503 02:00:21.893311 14410 ts_tablet_manager.cc:1909] T d22b0105836a43e09a8de16843474dfc P dec3e4a04d604f7eaaa3f077d081218a: Deleting consensus metadata
I20240503 02:00:21.893915 6303 catalog_manager.cc:4830] TS c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655): tablet d22b0105836a43e09a8de16843474dfc (table rebalancer_dynamic_table_new_name [id=1fe7d1ed4aee47f1ae55ec3c13859000]) successfully deleted
I20240503 02:00:21.894116 6300 catalog_manager.cc:4830] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): tablet d22b0105836a43e09a8de16843474dfc (table rebalancer_dynamic_table_new_name [id=1fe7d1ed4aee47f1ae55ec3c13859000]) successfully deleted
I20240503 02:00:21.988433 6313 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:00:21.993745 9158 tablet_service.cc:1467] Processing CreateTablet for tablet 00367fac98d94b809e5f788fbb49d4db (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6f878f01c35f4ae08eb12b7b12e4b350]), partition=
I20240503 02:00:21.993788 6829 tablet_service.cc:1467] Processing CreateTablet for tablet 00367fac98d94b809e5f788fbb49d4db (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6f878f01c35f4ae08eb12b7b12e4b350]), partition=
I20240503 02:00:21.993997 6436 tablet_service.cc:1467] Processing CreateTablet for tablet 00367fac98d94b809e5f788fbb49d4db (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6f878f01c35f4ae08eb12b7b12e4b350]), partition=
I20240503 02:00:21.994051 9158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00367fac98d94b809e5f788fbb49d4db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:21.994089 6829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00367fac98d94b809e5f788fbb49d4db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:21.994235 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00367fac98d94b809e5f788fbb49d4db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:21.996644 14374 tablet_bootstrap.cc:492] T 00367fac98d94b809e5f788fbb49d4db P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:21.997570 14374 tablet_bootstrap.cc:654] T 00367fac98d94b809e5f788fbb49d4db P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:21.997367 14415 tablet_bootstrap.cc:492] T 00367fac98d94b809e5f788fbb49d4db P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:21.998245 14414 tablet_bootstrap.cc:492] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:21.998982 14415 tablet_bootstrap.cc:654] T 00367fac98d94b809e5f788fbb49d4db P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:21.999210 14414 tablet_bootstrap.cc:654] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:22.000123 14374 tablet_bootstrap.cc:492] T 00367fac98d94b809e5f788fbb49d4db P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:22.000231 14374 ts_tablet_manager.cc:1367] T 00367fac98d94b809e5f788fbb49d4db P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:22.000905 14415 tablet_bootstrap.cc:492] T 00367fac98d94b809e5f788fbb49d4db P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:22.001008 14415 ts_tablet_manager.cc:1367] T 00367fac98d94b809e5f788fbb49d4db P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 02:00:22.001313 14374 raft_consensus.cc:348] T 00367fac98d94b809e5f788fbb49d4db P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:22.001380 14415 raft_consensus.cc:348] T 00367fac98d94b809e5f788fbb49d4db P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:22.001448 14374 raft_consensus.cc:374] T 00367fac98d94b809e5f788fbb49d4db P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:22.001472 14415 raft_consensus.cc:374] T 00367fac98d94b809e5f788fbb49d4db P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:22.001488 14374 raft_consensus.cc:729] T 00367fac98d94b809e5f788fbb49d4db P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:22.001500 14415 raft_consensus.cc:729] T 00367fac98d94b809e5f788fbb49d4db P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:22.001605 14374 consensus_queue.cc:260] T 00367fac98d94b809e5f788fbb49d4db P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:22.001605 14415 consensus_queue.cc:260] T 00367fac98d94b809e5f788fbb49d4db P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:22.001879 14415 ts_tablet_manager.cc:1398] T 00367fac98d94b809e5f788fbb49d4db P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:22.001890 14374 ts_tablet_manager.cc:1398] T 00367fac98d94b809e5f788fbb49d4db P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:00:22.002411 14414 tablet_bootstrap.cc:492] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:22.002504 14414 ts_tablet_manager.cc:1367] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:22.002993 14414 raft_consensus.cc:348] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:22.003093 14414 raft_consensus.cc:374] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:22.003120 14414 raft_consensus.cc:729] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:22.003276 14414 consensus_queue.cc:260] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:22.003569 14414 ts_tablet_manager.cc:1398] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:22.011693 13968 raft_consensus.cc:482] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:22.011806 13968 raft_consensus.cc:504] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:22.012257 13968 leader_election.cc:288] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 02:00:22.012698 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00367fac98d94b809e5f788fbb49d4db" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 02:00:22.012866 6849 raft_consensus.cc:2454] T 00367fac98d94b809e5f788fbb49d4db P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:22.012837 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00367fac98d94b809e5f788fbb49d4db" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 02:00:22.012961 6456 raft_consensus.cc:2454] T 00367fac98d94b809e5f788fbb49d4db P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:22.013255 9115 leader_election.cc:302] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 492448570cc94ed1879feb2ec3b9051d; no voters:
I20240503 02:00:22.013689 13968 raft_consensus.cc:2781] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:22.013746 13968 raft_consensus.cc:482] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:22.013790 13968 raft_consensus.cc:3037] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:22.014837 13968 raft_consensus.cc:504] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:22.015197 13968 leader_election.cc:288] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Requested vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 02:00:22.015566 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00367fac98d94b809e5f788fbb49d4db" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 02:00:22.015764 6456 raft_consensus.cc:3037] T 00367fac98d94b809e5f788fbb49d4db P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:22.016510 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00367fac98d94b809e5f788fbb49d4db" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 02:00:22.016633 6849 raft_consensus.cc:3037] T 00367fac98d94b809e5f788fbb49d4db P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:22.016814 6456 raft_consensus.cc:2454] T 00367fac98d94b809e5f788fbb49d4db P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:22.017591 6849 raft_consensus.cc:2454] T 00367fac98d94b809e5f788fbb49d4db P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:22.017938 9112 leader_election.cc:302] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:22.018115 13968 raft_consensus.cc:2781] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:22.018218 13968 raft_consensus.cc:686] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Becoming Leader. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Running, Role: LEADER
I20240503 02:00:22.018354 13968 consensus_queue.cc:237] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:22.021082 6313 catalog_manager.cc:5484] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135). New cstate: current_term: 1 leader_uuid: "2a01ace80baf4057b6bf1995cf88468b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } }
W20240503 02:00:22.076439 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:22.130957 6313 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:22.135102 6849 raft_consensus.cc:1261] T 00367fac98d94b809e5f788fbb49d4db P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer 2a01ace80baf4057b6bf1995cf88468b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:22.135102 6456 raft_consensus.cc:1261] T 00367fac98d94b809e5f788fbb49d4db P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 2a01ace80baf4057b6bf1995cf88468b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:22.135854 14413 consensus_queue.cc:1035] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:22.136165 14413 consensus_queue.cc:1035] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:22.138798 14385 tablet.cc:1722] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:22.139114 6309 catalog_manager.cc:5576] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719) does not have the latest schema for tablet 00367fac98d94b809e5f788fbb49d4db (table rebalancer_dynamic_table [id=6f878f01c35f4ae08eb12b7b12e4b350]). Expected version 1 got 0
I20240503 02:00:22.139153 14385 tablet.cc:1620] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
I20240503 02:00:22.139520 6308 catalog_manager.cc:5576] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093) does not have the latest schema for tablet 00367fac98d94b809e5f788fbb49d4db (table rebalancer_dynamic_table [id=6f878f01c35f4ae08eb12b7b12e4b350]). Expected version 1 got 0
I20240503 02:00:22.141737 14348 tablet.cc:1722] T 00367fac98d94b809e5f788fbb49d4db P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:22.142107 14348 tablet.cc:1620] T 00367fac98d94b809e5f788fbb49d4db P dec3e4a04d604f7eaaa3f077d081218a: MemRowSet was empty: no flush needed.
I20240503 02:00:22.143407 6309 catalog_manager.cc:5576] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093) does not have the latest schema for tablet 00367fac98d94b809e5f788fbb49d4db (table rebalancer_dynamic_table [id=6f878f01c35f4ae08eb12b7b12e4b350]). Expected version 1 got 0
I20240503 02:00:22.145233 14422 tablet.cc:1722] T 00367fac98d94b809e5f788fbb49d4db P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:22.145464 6308 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=6f878f01c35f4ae08eb12b7b12e4b350] alter complete (version 1)
I20240503 02:00:22.145747 14422 tablet.cc:1620] T 00367fac98d94b809e5f788fbb49d4db P 492448570cc94ed1879feb2ec3b9051d: MemRowSet was empty: no flush needed.
I20240503 02:00:22.146548 6313 catalog_manager.cc:5576] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719) does not have the latest schema for tablet 00367fac98d94b809e5f788fbb49d4db (table rebalancer_dynamic_table [id=6f878f01c35f4ae08eb12b7b12e4b350]). Expected version 1 got 0
I20240503 02:00:22.220109 14349 consensus_queue.cc:1035] T 98b3ee95a66e43788e5e2a428d9fa9c6 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:22.223747 14406 consensus_queue.cc:1035] T ea801f686a0a4efa9da66b24bdaa10cc P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:22.224721 14349 consensus_queue.cc:1035] T 98b3ee95a66e43788e5e2a428d9fa9c6 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:22.229391 14323 consensus_queue.cc:1035] T ea801f686a0a4efa9da66b24bdaa10cc P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:22.251166 6313 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:22.256884 14385 tablet.cc:1722] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:22.257285 14385 tablet.cc:1620] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
I20240503 02:00:22.257476 14422 tablet.cc:1722] T 00367fac98d94b809e5f788fbb49d4db P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:22.257774 14422 tablet.cc:1620] T 00367fac98d94b809e5f788fbb49d4db P 492448570cc94ed1879feb2ec3b9051d: MemRowSet was empty: no flush needed.
I20240503 02:00:22.258503 14348 tablet.cc:1722] T 00367fac98d94b809e5f788fbb49d4db P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:22.259081 14348 tablet.cc:1620] T 00367fac98d94b809e5f788fbb49d4db P dec3e4a04d604f7eaaa3f077d081218a: MemRowSet was empty: no flush needed.
I20240503 02:00:22.264219 6309 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=6f878f01c35f4ae08eb12b7b12e4b350] alter complete (version 2)
I20240503 02:00:22.326375 14323 consensus_queue.cc:1035] T 8a8b7b032eea4726b24b805236601450 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:22.329953 14433 consensus_queue.cc:1035] T 8a8b7b032eea4726b24b805236601450 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:22.339630 6309 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_extra_table_58"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:22.346544 9158 tablet_service.cc:1467] Processing CreateTablet for tablet 703b917cf3e74caaae52326e3604fb6e (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=65ca51846bbe433c9b20578b01d413e2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:22.346879 9158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 703b917cf3e74caaae52326e3604fb6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:22.347050 6829 tablet_service.cc:1467] Processing CreateTablet for tablet dc67e2e6fcdc45f39504cb4c9e0a02e1 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=65ca51846bbe433c9b20578b01d413e2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:22.347060 6569 tablet_service.cc:1467] Processing CreateTablet for tablet dc67e2e6fcdc45f39504cb4c9e0a02e1 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=65ca51846bbe433c9b20578b01d413e2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:22.347291 6829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc67e2e6fcdc45f39504cb4c9e0a02e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:22.347304 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc67e2e6fcdc45f39504cb4c9e0a02e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:22.349148 14415 tablet_bootstrap.cc:492] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:22.349474 6700 tablet_service.cc:1467] Processing CreateTablet for tablet 703b917cf3e74caaae52326e3604fb6e (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=65ca51846bbe433c9b20578b01d413e2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:22.349737 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 703b917cf3e74caaae52326e3604fb6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:22.350126 14415 tablet_bootstrap.cc:654] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:22.350256 14435 tablet_bootstrap.cc:492] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:22.351003 6436 tablet_service.cc:1467] Processing CreateTablet for tablet da5b87a8eaf54dc9b50140491d028cf1 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=65ca51846bbe433c9b20578b01d413e2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:22.351266 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da5b87a8eaf54dc9b50140491d028cf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:22.351282 14435 tablet_bootstrap.cc:654] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:22.351245 6699 tablet_service.cc:1467] Processing CreateTablet for tablet dc67e2e6fcdc45f39504cb4c9e0a02e1 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=65ca51846bbe433c9b20578b01d413e2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:22.351461 6699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc67e2e6fcdc45f39504cb4c9e0a02e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:22.352170 6568 tablet_service.cc:1467] Processing CreateTablet for tablet 703b917cf3e74caaae52326e3604fb6e (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=65ca51846bbe433c9b20578b01d413e2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:22.352423 6568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 703b917cf3e74caaae52326e3604fb6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:22.352749 14414 tablet_bootstrap.cc:492] T 703b917cf3e74caaae52326e3604fb6e P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:22.353520 14374 tablet_bootstrap.cc:492] T da5b87a8eaf54dc9b50140491d028cf1 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:22.353760 14414 tablet_bootstrap.cc:654] T 703b917cf3e74caaae52326e3604fb6e P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:22.354422 14374 tablet_bootstrap.cc:654] T da5b87a8eaf54dc9b50140491d028cf1 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:22.354694 6829 tablet_service.cc:1467] Processing CreateTablet for tablet da5b87a8eaf54dc9b50140491d028cf1 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=65ca51846bbe433c9b20578b01d413e2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:22.354959 6829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da5b87a8eaf54dc9b50140491d028cf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:22.355824 14414 tablet_bootstrap.cc:492] T 703b917cf3e74caaae52326e3604fb6e P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:22.355909 14414 ts_tablet_manager.cc:1367] T 703b917cf3e74caaae52326e3604fb6e P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:22.355947 14374 tablet_bootstrap.cc:492] T da5b87a8eaf54dc9b50140491d028cf1 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:22.355958 14415 tablet_bootstrap.cc:492] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:22.356029 14374 ts_tablet_manager.cc:1367] T da5b87a8eaf54dc9b50140491d028cf1 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:00:22.356035 14415 ts_tablet_manager.cc:1367] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:00:22.356420 14414 raft_consensus.cc:348] T 703b917cf3e74caaae52326e3604fb6e P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:22.356469 14415 raft_consensus.cc:348] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:22.356532 14414 raft_consensus.cc:374] T 703b917cf3e74caaae52326e3604fb6e P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:22.356563 14414 raft_consensus.cc:729] T 703b917cf3e74caaae52326e3604fb6e P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:22.356566 14415 raft_consensus.cc:374] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:22.356505 14374 raft_consensus.cc:348] T da5b87a8eaf54dc9b50140491d028cf1 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:22.356597 14415 raft_consensus.cc:729] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:22.356607 14374 raft_consensus.cc:374] T da5b87a8eaf54dc9b50140491d028cf1 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:22.356637 14374 raft_consensus.cc:729] T da5b87a8eaf54dc9b50140491d028cf1 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:22.356684 14414 consensus_queue.cc:260] T 703b917cf3e74caaae52326e3604fb6e P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:22.356709 14415 consensus_queue.cc:260] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:22.356765 14374 consensus_queue.cc:260] T da5b87a8eaf54dc9b50140491d028cf1 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:22.356983 14415 ts_tablet_manager.cc:1398] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:22.357045 14374 ts_tablet_manager.cc:1398] T da5b87a8eaf54dc9b50140491d028cf1 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:22.357137 14415 tablet_bootstrap.cc:492] T da5b87a8eaf54dc9b50140491d028cf1 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:22.358006 14414 ts_tablet_manager.cc:1398] T 703b917cf3e74caaae52326e3604fb6e P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:00:22.358326 14415 tablet_bootstrap.cc:654] T da5b87a8eaf54dc9b50140491d028cf1 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:22.361593 6699 tablet_service.cc:1467] Processing CreateTablet for tablet da5b87a8eaf54dc9b50140491d028cf1 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=65ca51846bbe433c9b20578b01d413e2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:22.361917 6699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da5b87a8eaf54dc9b50140491d028cf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:22.363004 14438 tablet_bootstrap.cc:492] T 703b917cf3e74caaae52326e3604fb6e P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:22.363950 14438 tablet_bootstrap.cc:654] T 703b917cf3e74caaae52326e3604fb6e P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:22.364939 14415 tablet_bootstrap.cc:492] T da5b87a8eaf54dc9b50140491d028cf1 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:22.365026 14415 ts_tablet_manager.cc:1367] T da5b87a8eaf54dc9b50140491d028cf1 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 02:00:22.365263 14438 tablet_bootstrap.cc:492] T 703b917cf3e74caaae52326e3604fb6e P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:22.365348 14438 ts_tablet_manager.cc:1367] T 703b917cf3e74caaae52326e3604fb6e P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:00:22.365396 14415 raft_consensus.cc:348] T da5b87a8eaf54dc9b50140491d028cf1 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:22.365501 14415 raft_consensus.cc:374] T da5b87a8eaf54dc9b50140491d028cf1 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:22.365532 14415 raft_consensus.cc:729] T da5b87a8eaf54dc9b50140491d028cf1 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:22.365675 14415 consensus_queue.cc:260] T da5b87a8eaf54dc9b50140491d028cf1 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:22.365778 14438 raft_consensus.cc:348] T 703b917cf3e74caaae52326e3604fb6e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:22.365906 14438 raft_consensus.cc:374] T 703b917cf3e74caaae52326e3604fb6e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:22.365937 14438 raft_consensus.cc:729] T 703b917cf3e74caaae52326e3604fb6e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:22.366264 14415 ts_tablet_manager.cc:1398] T da5b87a8eaf54dc9b50140491d028cf1 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:22.367265 14435 tablet_bootstrap.cc:492] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:22.367393 14435 ts_tablet_manager.cc:1367] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240503 02:00:22.366045 14438 consensus_queue.cc:260] T 703b917cf3e74caaae52326e3604fb6e P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:22.367897 14435 raft_consensus.cc:348] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:22.367969 14438 ts_tablet_manager.cc:1398] T 703b917cf3e74caaae52326e3604fb6e P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 02:00:22.368026 14435 raft_consensus.cc:374] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:22.368073 14435 raft_consensus.cc:729] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:22.368134 14438 tablet_bootstrap.cc:492] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:22.368202 14435 consensus_queue.cc:260] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:22.368515 14435 ts_tablet_manager.cc:1398] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:22.369280 6314 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:22.369447 14435 tablet_bootstrap.cc:492] T 703b917cf3e74caaae52326e3604fb6e P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:22.371304 14438 tablet_bootstrap.cc:654] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:22.374513 14435 tablet_bootstrap.cc:654] T 703b917cf3e74caaae52326e3604fb6e P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:22.376799 14435 tablet_bootstrap.cc:492] T 703b917cf3e74caaae52326e3604fb6e P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:22.376901 14435 ts_tablet_manager.cc:1367] T 703b917cf3e74caaae52326e3604fb6e P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:00:22.377386 14435 raft_consensus.cc:348] T 703b917cf3e74caaae52326e3604fb6e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:22.377501 14435 raft_consensus.cc:374] T 703b917cf3e74caaae52326e3604fb6e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:22.377532 14435 raft_consensus.cc:729] T 703b917cf3e74caaae52326e3604fb6e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:22.377638 14435 consensus_queue.cc:260] T 703b917cf3e74caaae52326e3604fb6e P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:22.377943 14435 ts_tablet_manager.cc:1398] T 703b917cf3e74caaae52326e3604fb6e P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:22.380088 14385 tablet.cc:1722] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:22.380546 14422 tablet.cc:1722] T 00367fac98d94b809e5f788fbb49d4db P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:22.380970 14348 tablet.cc:1722] T 00367fac98d94b809e5f788fbb49d4db P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:22.384896 14438 tablet_bootstrap.cc:492] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:22.385026 14438 ts_tablet_manager.cc:1367] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.002s
I20240503 02:00:22.385583 14438 raft_consensus.cc:348] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:22.385789 14438 raft_consensus.cc:374] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:22.385865 14438 raft_consensus.cc:729] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:22.386049 14438 consensus_queue.cc:260] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:22.386476 14438 ts_tablet_manager.cc:1398] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:22.386838 14438 tablet_bootstrap.cc:492] T da5b87a8eaf54dc9b50140491d028cf1 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:22.388350 14438 tablet_bootstrap.cc:654] T da5b87a8eaf54dc9b50140491d028cf1 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:22.391561 6312 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=6f878f01c35f4ae08eb12b7b12e4b350] alter complete (version 3)
I20240503 02:00:22.400965 14438 tablet_bootstrap.cc:492] T da5b87a8eaf54dc9b50140491d028cf1 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:22.401237 14438 ts_tablet_manager.cc:1367] T da5b87a8eaf54dc9b50140491d028cf1 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.003s
I20240503 02:00:22.402272 14438 raft_consensus.cc:348] T da5b87a8eaf54dc9b50140491d028cf1 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:22.402410 14438 raft_consensus.cc:374] T da5b87a8eaf54dc9b50140491d028cf1 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:22.402462 14438 raft_consensus.cc:729] T da5b87a8eaf54dc9b50140491d028cf1 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:22.402613 14438 consensus_queue.cc:260] T da5b87a8eaf54dc9b50140491d028cf1 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:22.402943 14438 ts_tablet_manager.cc:1398] T da5b87a8eaf54dc9b50140491d028cf1 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:22.467536 14386 raft_consensus.cc:482] T 703b917cf3e74caaae52326e3604fb6e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:22.467670 14386 raft_consensus.cc:504] T 703b917cf3e74caaae52326e3604fb6e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:22.468082 14386 leader_election.cc:288] T 703b917cf3e74caaae52326e3604fb6e P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:22.468497 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "703b917cf3e74caaae52326e3604fb6e" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:22.468499 6587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "703b917cf3e74caaae52326e3604fb6e" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 02:00:22.468627 6587 raft_consensus.cc:2454] T 703b917cf3e74caaae52326e3604fb6e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:22.468636 9178 raft_consensus.cc:2454] T 703b917cf3e74caaae52326e3604fb6e P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:22.468921 6655 leader_election.cc:302] T 703b917cf3e74caaae52326e3604fb6e P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:22.469081 14386 raft_consensus.cc:2781] T 703b917cf3e74caaae52326e3604fb6e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:22.469147 14386 raft_consensus.cc:482] T 703b917cf3e74caaae52326e3604fb6e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:22.469215 14386 raft_consensus.cc:3037] T 703b917cf3e74caaae52326e3604fb6e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:22.470139 14386 raft_consensus.cc:504] T 703b917cf3e74caaae52326e3604fb6e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:22.470499 14386 leader_election.cc:288] T 703b917cf3e74caaae52326e3604fb6e P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Requested vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:22.470889 6587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "703b917cf3e74caaae52326e3604fb6e" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 02:00:22.471021 6587 raft_consensus.cc:3037] T 703b917cf3e74caaae52326e3604fb6e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:22.471800 6587 raft_consensus.cc:2454] T 703b917cf3e74caaae52326e3604fb6e P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:22.472136 6655 leader_election.cc:302] T 703b917cf3e74caaae52326e3604fb6e P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:22.472211 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "703b917cf3e74caaae52326e3604fb6e" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:22.472360 14386 raft_consensus.cc:2781] T 703b917cf3e74caaae52326e3604fb6e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:22.472368 9178 raft_consensus.cc:3037] T 703b917cf3e74caaae52326e3604fb6e P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:22.472468 14386 raft_consensus.cc:686] T 703b917cf3e74caaae52326e3604fb6e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Becoming Leader. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Running, Role: LEADER
I20240503 02:00:22.472630 14386 consensus_queue.cc:237] T 703b917cf3e74caaae52326e3604fb6e P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:22.473290 9178 raft_consensus.cc:2454] T 703b917cf3e74caaae52326e3604fb6e P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:22.476938 6312 catalog_manager.cc:5484] T 703b917cf3e74caaae52326e3604fb6e P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:22.500519 6312 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:22.500679 6312 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:22.502696 6312 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet 00367fac98d94b809e5f788fbb49d4db
I20240503 02:00:22.503379 6829 tablet_service.cc:1514] Processing DeleteTablet for tablet 00367fac98d94b809e5f788fbb49d4db with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:22 UTC) from {username='slave'} at 127.0.0.1:52812
I20240503 02:00:22.503378 9158 tablet_service.cc:1514] Processing DeleteTablet for tablet 00367fac98d94b809e5f788fbb49d4db with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:22 UTC) from {username='slave'} at 127.0.0.1:51940
I20240503 02:00:22.503460 6436 tablet_service.cc:1514] Processing DeleteTablet for tablet 00367fac98d94b809e5f788fbb49d4db with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:22 UTC) from {username='slave'} at 127.0.0.1:51338
I20240503 02:00:22.503727 14451 tablet_replica.cc:330] T 00367fac98d94b809e5f788fbb49d4db P 492448570cc94ed1879feb2ec3b9051d: stopping tablet replica
I20240503 02:00:22.503773 14453 tablet_replica.cc:330] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b: stopping tablet replica
I20240503 02:00:22.503882 14451 raft_consensus.cc:2229] T 00367fac98d94b809e5f788fbb49d4db P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:22.503916 14453 raft_consensus.cc:2229] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:22.504029 14451 raft_consensus.cc:2258] T 00367fac98d94b809e5f788fbb49d4db P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:22.504127 14453 raft_consensus.cc:2258] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:22.504405 14452 tablet_replica.cc:330] T 00367fac98d94b809e5f788fbb49d4db P dec3e4a04d604f7eaaa3f077d081218a: stopping tablet replica
I20240503 02:00:22.504552 14452 raft_consensus.cc:2229] T 00367fac98d94b809e5f788fbb49d4db P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:22.504680 14452 raft_consensus.cc:2258] T 00367fac98d94b809e5f788fbb49d4db P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:22.504748 14453 ts_tablet_manager.cc:1875] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:22.504909 14451 ts_tablet_manager.cc:1875] T 00367fac98d94b809e5f788fbb49d4db P 492448570cc94ed1879feb2ec3b9051d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:22.505292 14452 ts_tablet_manager.cc:1875] T 00367fac98d94b809e5f788fbb49d4db P dec3e4a04d604f7eaaa3f077d081218a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:22.507166 14451 ts_tablet_manager.cc:1888] T 00367fac98d94b809e5f788fbb49d4db P 492448570cc94ed1879feb2ec3b9051d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:22.507238 14453 ts_tablet_manager.cc:1888] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:22.507277 14451 log.cc:1196] T 00367fac98d94b809e5f788fbb49d4db P 492448570cc94ed1879feb2ec3b9051d: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/00367fac98d94b809e5f788fbb49d4db
I20240503 02:00:22.507364 14453 log.cc:1196] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/00367fac98d94b809e5f788fbb49d4db
I20240503 02:00:22.507433 14452 ts_tablet_manager.cc:1888] T 00367fac98d94b809e5f788fbb49d4db P dec3e4a04d604f7eaaa3f077d081218a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:22.507539 14452 log.cc:1196] T 00367fac98d94b809e5f788fbb49d4db P dec3e4a04d604f7eaaa3f077d081218a: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/00367fac98d94b809e5f788fbb49d4db
I20240503 02:00:22.507601 14451 ts_tablet_manager.cc:1909] T 00367fac98d94b809e5f788fbb49d4db P 492448570cc94ed1879feb2ec3b9051d: Deleting consensus metadata
I20240503 02:00:22.507644 14453 ts_tablet_manager.cc:1909] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b: Deleting consensus metadata
I20240503 02:00:22.507853 14452 ts_tablet_manager.cc:1909] T 00367fac98d94b809e5f788fbb49d4db P dec3e4a04d604f7eaaa3f077d081218a: Deleting consensus metadata
I20240503 02:00:22.508337 6303 catalog_manager.cc:4830] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719): tablet 00367fac98d94b809e5f788fbb49d4db (table rebalancer_dynamic_table_new_name [id=6f878f01c35f4ae08eb12b7b12e4b350]) successfully deleted
I20240503 02:00:22.508347 6300 catalog_manager.cc:4830] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079): tablet 00367fac98d94b809e5f788fbb49d4db (table rebalancer_dynamic_table_new_name [id=6f878f01c35f4ae08eb12b7b12e4b350]) successfully deleted
I20240503 02:00:22.508718 6300 catalog_manager.cc:4830] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): tablet 00367fac98d94b809e5f788fbb49d4db (table rebalancer_dynamic_table_new_name [id=6f878f01c35f4ae08eb12b7b12e4b350]) successfully deleted
W20240503 02:00:22.540051 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:22.556813 14442 raft_consensus.cc:482] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:22.556965 14442 raft_consensus.cc:504] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:22.557428 14442 leader_election.cc:288] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:22.557865 6587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc67e2e6fcdc45f39504cb4c9e0a02e1" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 02:00:22.558001 6587 raft_consensus.cc:2454] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 02:00:22.558276 6786 leader_election.cc:302] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 80fd18ef4d604c86b909d5c008a46b4f; no voters:
I20240503 02:00:22.558441 14442 raft_consensus.cc:2781] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:22.558490 14442 raft_consensus.cc:482] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:22.558526 14442 raft_consensus.cc:3037] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:22.559592 6720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc67e2e6fcdc45f39504cb4c9e0a02e1" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 02:00:22.559621 14442 raft_consensus.cc:504] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:22.559734 6720 raft_consensus.cc:2454] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 02:00:22.560082 14442 leader_election.cc:288] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Requested vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:22.560590 6587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc67e2e6fcdc45f39504cb4c9e0a02e1" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 02:00:22.560705 6587 raft_consensus.cc:3037] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:22.560720 6720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc67e2e6fcdc45f39504cb4c9e0a02e1" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 02:00:22.560824 6720 raft_consensus.cc:3037] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:22.561704 6720 raft_consensus.cc:2454] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 02:00:22.561695 6587 raft_consensus.cc:2454] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 02:00:22.562004 6787 leader_election.cc:302] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:22.562330 14442 raft_consensus.cc:2781] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:22.562415 14442 raft_consensus.cc:686] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Becoming Leader. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Running, Role: LEADER
I20240503 02:00:22.562546 14442 consensus_queue.cc:237] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:22.565280 6312 catalog_manager.cc:5484] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 492448570cc94ed1879feb2ec3b9051d reported cstate change: term changed from 0 to 1, leader changed from <none> to 492448570cc94ed1879feb2ec3b9051d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:22.604314 6312 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:00:22.609862 6436 tablet_service.cc:1467] Processing CreateTablet for tablet 5976a8a1983f497eb0daa9c4fa9b701e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e890cb2b2c1049fdb59177f7670af7a3]), partition=
I20240503 02:00:22.610028 8880 tablet_service.cc:1467] Processing CreateTablet for tablet 5976a8a1983f497eb0daa9c4fa9b701e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e890cb2b2c1049fdb59177f7670af7a3]), partition=
I20240503 02:00:22.610188 6829 tablet_service.cc:1467] Processing CreateTablet for tablet 5976a8a1983f497eb0daa9c4fa9b701e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e890cb2b2c1049fdb59177f7670af7a3]), partition=
I20240503 02:00:22.610219 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5976a8a1983f497eb0daa9c4fa9b701e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:22.610280 8880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5976a8a1983f497eb0daa9c4fa9b701e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:22.610399 6829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5976a8a1983f497eb0daa9c4fa9b701e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:22.611972 14374 tablet_bootstrap.cc:492] T 5976a8a1983f497eb0daa9c4fa9b701e P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:22.612444 14415 tablet_bootstrap.cc:492] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:22.612941 14374 tablet_bootstrap.cc:654] T 5976a8a1983f497eb0daa9c4fa9b701e P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:22.613296 14415 tablet_bootstrap.cc:654] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:22.614115 14374 tablet_bootstrap.cc:492] T 5976a8a1983f497eb0daa9c4fa9b701e P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:22.614192 14374 ts_tablet_manager.cc:1367] T 5976a8a1983f497eb0daa9c4fa9b701e P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:22.614342 14415 tablet_bootstrap.cc:492] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:22.614424 14415 ts_tablet_manager.cc:1367] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:00:22.614591 14374 raft_consensus.cc:348] T 5976a8a1983f497eb0daa9c4fa9b701e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:22.614704 14374 raft_consensus.cc:374] T 5976a8a1983f497eb0daa9c4fa9b701e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:22.614727 14374 raft_consensus.cc:729] T 5976a8a1983f497eb0daa9c4fa9b701e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:22.614802 14415 raft_consensus.cc:348] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:22.614853 14374 consensus_queue.cc:260] T 5976a8a1983f497eb0daa9c4fa9b701e P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:22.614908 14415 raft_consensus.cc:374] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:22.614949 14415 raft_consensus.cc:729] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:22.615077 14415 consensus_queue.cc:260] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:22.615113 14374 ts_tablet_manager.cc:1398] T 5976a8a1983f497eb0daa9c4fa9b701e P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:22.615365 14415 ts_tablet_manager.cc:1398] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:22.616904 14455 tablet_bootstrap.cc:492] T 5976a8a1983f497eb0daa9c4fa9b701e P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:22.619096 14455 tablet_bootstrap.cc:654] T 5976a8a1983f497eb0daa9c4fa9b701e P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:22.620927 14455 tablet_bootstrap.cc:492] T 5976a8a1983f497eb0daa9c4fa9b701e P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:22.621063 14455 ts_tablet_manager.cc:1367] T 5976a8a1983f497eb0daa9c4fa9b701e P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:22.621505 14455 raft_consensus.cc:348] T 5976a8a1983f497eb0daa9c4fa9b701e P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:22.621594 14455 raft_consensus.cc:374] T 5976a8a1983f497eb0daa9c4fa9b701e P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:22.621614 14455 raft_consensus.cc:729] T 5976a8a1983f497eb0daa9c4fa9b701e P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:22.621707 14455 consensus_queue.cc:260] T 5976a8a1983f497eb0daa9c4fa9b701e P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:22.621982 14455 ts_tablet_manager.cc:1398] T 5976a8a1983f497eb0daa9c4fa9b701e P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:22.745640 14349 raft_consensus.cc:482] T da5b87a8eaf54dc9b50140491d028cf1 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:22.745792 14349 raft_consensus.cc:504] T da5b87a8eaf54dc9b50140491d028cf1 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:22.746253 14349 leader_election.cc:288] T da5b87a8eaf54dc9b50140491d028cf1 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:22.746614 6719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da5b87a8eaf54dc9b50140491d028cf1" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 02:00:22.746652 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da5b87a8eaf54dc9b50140491d028cf1" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 02:00:22.746757 6719 raft_consensus.cc:2454] T da5b87a8eaf54dc9b50140491d028cf1 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:22.746820 6851 raft_consensus.cc:2454] T da5b87a8eaf54dc9b50140491d028cf1 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:22.747016 6394 leader_election.cc:302] T da5b87a8eaf54dc9b50140491d028cf1 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2ac2eb7cc264047b9ab32fe3a4fc025, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:22.747192 14349 raft_consensus.cc:2781] T da5b87a8eaf54dc9b50140491d028cf1 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:22.747262 14349 raft_consensus.cc:482] T da5b87a8eaf54dc9b50140491d028cf1 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:22.747289 14349 raft_consensus.cc:3037] T da5b87a8eaf54dc9b50140491d028cf1 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:22.748224 14349 raft_consensus.cc:504] T da5b87a8eaf54dc9b50140491d028cf1 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:22.748593 14349 leader_election.cc:288] T da5b87a8eaf54dc9b50140491d028cf1 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Requested vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:22.748905 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da5b87a8eaf54dc9b50140491d028cf1" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 02:00:22.748931 6719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da5b87a8eaf54dc9b50140491d028cf1" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 02:00:22.749032 6849 raft_consensus.cc:3037] T da5b87a8eaf54dc9b50140491d028cf1 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:22.749068 6719 raft_consensus.cc:3037] T da5b87a8eaf54dc9b50140491d028cf1 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:22.749943 6849 raft_consensus.cc:2454] T da5b87a8eaf54dc9b50140491d028cf1 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:22.749964 6719 raft_consensus.cc:2454] T da5b87a8eaf54dc9b50140491d028cf1 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:22.750236 6394 leader_election.cc:302] T da5b87a8eaf54dc9b50140491d028cf1 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:22.750407 14349 raft_consensus.cc:2781] T da5b87a8eaf54dc9b50140491d028cf1 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:22.750494 14349 raft_consensus.cc:686] T da5b87a8eaf54dc9b50140491d028cf1 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Becoming Leader. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Running, Role: LEADER
I20240503 02:00:22.750619 14349 consensus_queue.cc:237] T da5b87a8eaf54dc9b50140491d028cf1 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:22.752977 6313 catalog_manager.cc:5484] T da5b87a8eaf54dc9b50140491d028cf1 P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: term changed from 0 to 1, leader changed from <none> to dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129). New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:22.911738 14324 raft_consensus.cc:482] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:22.911873 14324 raft_consensus.cc:504] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
W20240503 02:00:22.912256 8982 consensus_peers.cc:485] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:00:22.912292 14324 leader_election.cc:288] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:22.912669 8917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5976a8a1983f497eb0daa9c4fa9b701e" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:22.912804 8917 raft_consensus.cc:2454] T 5976a8a1983f497eb0daa9c4fa9b701e P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 02:00:22.913054 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5976a8a1983f497eb0daa9c4fa9b701e" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 02:00:22.913128 6787 leader_election.cc:302] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:22.913201 6458 raft_consensus.cc:2454] T 5976a8a1983f497eb0daa9c4fa9b701e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 02:00:22.913393 14324 raft_consensus.cc:2781] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:22.913455 14324 raft_consensus.cc:482] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:22.913504 14324 raft_consensus.cc:3037] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:22.914366 14324 raft_consensus.cc:504] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:22.914702 14324 leader_election.cc:288] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Requested vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:22.915158 8917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5976a8a1983f497eb0daa9c4fa9b701e" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:22.915287 8917 raft_consensus.cc:3037] T 5976a8a1983f497eb0daa9c4fa9b701e P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:22.916222 8917 raft_consensus.cc:2454] T 5976a8a1983f497eb0daa9c4fa9b701e P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 02:00:22.916777 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5976a8a1983f497eb0daa9c4fa9b701e" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 02:00:22.916911 6458 raft_consensus.cc:3037] T 5976a8a1983f497eb0daa9c4fa9b701e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:22.916973 6787 leader_election.cc:302] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:22.917174 14324 raft_consensus.cc:2781] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:22.917362 14324 raft_consensus.cc:686] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Becoming Leader. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Running, Role: LEADER
I20240503 02:00:22.917510 14324 consensus_queue.cc:237] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:22.917987 6458 raft_consensus.cc:2454] T 5976a8a1983f497eb0daa9c4fa9b701e P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 02:00:22.920694 6313 catalog_manager.cc:5484] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d reported cstate change: term changed from 0 to 1, leader changed from <none> to 492448570cc94ed1879feb2ec3b9051d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } }
W20240503 02:00:22.982370 6785 consensus_peers.cc:485] T c3485701bf3b48f5ae2e1568afeed133 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:23.039680 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:23.047782 14386 consensus_queue.cc:1035] T 703b917cf3e74caaae52326e3604fb6e P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:23.053880 14466 consensus_queue.cc:1035] T 703b917cf3e74caaae52326e3604fb6e P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:23.079519 6313 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:23.082118 14324 consensus_queue.cc:1035] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:23.086086 8916 raft_consensus.cc:1261] T 5976a8a1983f497eb0daa9c4fa9b701e P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:23.086541 14442 consensus_queue.cc:1035] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:23.090013 14475 tablet.cc:1722] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:23.090471 14475 tablet.cc:1620] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d: MemRowSet was empty: no flush needed.
I20240503 02:00:23.090801 14476 tablet.cc:1722] T 5976a8a1983f497eb0daa9c4fa9b701e P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:23.090999 6309 catalog_manager.cc:5576] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211) does not have the latest schema for tablet 5976a8a1983f497eb0daa9c4fa9b701e (table rebalancer_dynamic_table [id=e890cb2b2c1049fdb59177f7670af7a3]). Expected version 1 got 0
I20240503 02:00:23.091100 14476 tablet.cc:1620] T 5976a8a1983f497eb0daa9c4fa9b701e P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
I20240503 02:00:23.093607 6458 raft_consensus.cc:1261] T 5976a8a1983f497eb0daa9c4fa9b701e P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:23.094045 14472 consensus_queue.cc:1035] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:23.094404 14472 consensus_queue.cc:1035] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:23.096976 14481 tablet.cc:1722] T 5976a8a1983f497eb0daa9c4fa9b701e P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:23.097414 14481 tablet.cc:1620] T 5976a8a1983f497eb0daa9c4fa9b701e P dec3e4a04d604f7eaaa3f077d081218a: MemRowSet was empty: no flush needed.
I20240503 02:00:23.098665 6314 catalog_manager.cc:5576] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093) does not have the latest schema for tablet 5976a8a1983f497eb0daa9c4fa9b701e (table rebalancer_dynamic_table [id=e890cb2b2c1049fdb59177f7670af7a3]). Expected version 1 got 0
I20240503 02:00:23.099455 6308 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=e890cb2b2c1049fdb59177f7670af7a3] alter complete (version 1)
I20240503 02:00:23.203039 6314 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:23.208948 6314 catalog_manager.cc:5576] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719) does not have the latest schema for tablet 5976a8a1983f497eb0daa9c4fa9b701e (table rebalancer_dynamic_table [id=e890cb2b2c1049fdb59177f7670af7a3]). Expected version 2 got 1
I20240503 02:00:23.208880 14475 tablet.cc:1722] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:23.209586 14475 tablet.cc:1620] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d: MemRowSet was empty: no flush needed.
I20240503 02:00:23.209535 14476 tablet.cc:1722] T 5976a8a1983f497eb0daa9c4fa9b701e P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:23.209841 14476 tablet.cc:1620] T 5976a8a1983f497eb0daa9c4fa9b701e P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
I20240503 02:00:23.213714 14481 tablet.cc:1722] T 5976a8a1983f497eb0daa9c4fa9b701e P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:23.214062 14481 tablet.cc:1620] T 5976a8a1983f497eb0daa9c4fa9b701e P dec3e4a04d604f7eaaa3f077d081218a: MemRowSet was empty: no flush needed.
I20240503 02:00:23.215735 6309 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=e890cb2b2c1049fdb59177f7670af7a3] alter complete (version 2)
I20240503 02:00:23.233090 14349 consensus_queue.cc:1035] T da5b87a8eaf54dc9b50140491d028cf1 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:23.237303 14461 consensus_queue.cc:1035] T da5b87a8eaf54dc9b50140491d028cf1 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:00:23.259267 6785 consensus_peers.cc:485] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:00:23.318502 6309 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
W20240503 02:00:23.321789 6654 consensus_peers.cc:485] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:00:23.325131 14476 tablet.cc:1722] T 5976a8a1983f497eb0daa9c4fa9b701e P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:23.325740 14481 tablet.cc:1722] T 5976a8a1983f497eb0daa9c4fa9b701e P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:23.326153 14475 tablet.cc:1722] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:23.332213 6309 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=e890cb2b2c1049fdb59177f7670af7a3] alter complete (version 3)
I20240503 02:00:23.333287 6308 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_extra_table_60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:23.339680 6699 tablet_service.cc:1467] Processing CreateTablet for tablet 5d237bb82d7c47409a8924c5ebfd194b (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=6a1debe95ca94473b630c72d639b68c5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:23.339917 9027 tablet_service.cc:1467] Processing CreateTablet for tablet 5d237bb82d7c47409a8924c5ebfd194b (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=6a1debe95ca94473b630c72d639b68c5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:23.340041 6699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d237bb82d7c47409a8924c5ebfd194b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.340209 9027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d237bb82d7c47409a8924c5ebfd194b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.340479 9025 tablet_service.cc:1467] Processing CreateTablet for tablet 93d91773da7f403da2eb85808b686354 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=6a1debe95ca94473b630c72d639b68c5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:23.340701 9025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93d91773da7f403da2eb85808b686354. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.340922 9158 tablet_service.cc:1467] Processing CreateTablet for tablet 5d237bb82d7c47409a8924c5ebfd194b (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=6a1debe95ca94473b630c72d639b68c5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:23.341164 9158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d237bb82d7c47409a8924c5ebfd194b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.342845 9159 tablet_service.cc:1467] Processing CreateTablet for tablet 93d91773da7f403da2eb85808b686354 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=6a1debe95ca94473b630c72d639b68c5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:23.342850 9028 tablet_service.cc:1467] Processing CreateTablet for tablet fe9cd379022b49f797122c6727d77ef0 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=6a1debe95ca94473b630c72d639b68c5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:23.343062 9159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93d91773da7f403da2eb85808b686354. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.343087 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe9cd379022b49f797122c6727d77ef0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.343407 14486 tablet_bootstrap.cc:492] T 5d237bb82d7c47409a8924c5ebfd194b P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:23.343578 9157 tablet_service.cc:1467] Processing CreateTablet for tablet fe9cd379022b49f797122c6727d77ef0 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=6a1debe95ca94473b630c72d639b68c5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:23.343783 9157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe9cd379022b49f797122c6727d77ef0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.344470 14486 tablet_bootstrap.cc:654] T 5d237bb82d7c47409a8924c5ebfd194b P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.345908 6436 tablet_service.cc:1467] Processing CreateTablet for tablet 93d91773da7f403da2eb85808b686354 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=6a1debe95ca94473b630c72d639b68c5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:23.346199 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93d91773da7f403da2eb85808b686354. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.346182 6568 tablet_service.cc:1467] Processing CreateTablet for tablet fe9cd379022b49f797122c6727d77ef0 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=6a1debe95ca94473b630c72d639b68c5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:23.346436 6568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe9cd379022b49f797122c6727d77ef0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.347740 14486 tablet_bootstrap.cc:492] T 5d237bb82d7c47409a8924c5ebfd194b P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:23.347833 14486 ts_tablet_manager.cc:1367] T 5d237bb82d7c47409a8924c5ebfd194b P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240503 02:00:23.348371 14486 raft_consensus.cc:348] T 5d237bb82d7c47409a8924c5ebfd194b P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:23.348526 14486 raft_consensus.cc:374] T 5d237bb82d7c47409a8924c5ebfd194b P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.348552 14486 raft_consensus.cc:729] T 5d237bb82d7c47409a8924c5ebfd194b P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.348618 14488 tablet_bootstrap.cc:492] T fe9cd379022b49f797122c6727d77ef0 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:23.348697 14486 consensus_queue.cc:260] T 5d237bb82d7c47409a8924c5ebfd194b P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:23.349010 14486 ts_tablet_manager.cc:1398] T 5d237bb82d7c47409a8924c5ebfd194b P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 02:00:23.349217 14486 tablet_bootstrap.cc:492] T 93d91773da7f403da2eb85808b686354 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:23.349751 14488 tablet_bootstrap.cc:654] T fe9cd379022b49f797122c6727d77ef0 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.350128 14486 tablet_bootstrap.cc:654] T 93d91773da7f403da2eb85808b686354 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.350478 14484 tablet_bootstrap.cc:492] T 5d237bb82d7c47409a8924c5ebfd194b P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:23.351572 14486 tablet_bootstrap.cc:492] T 93d91773da7f403da2eb85808b686354 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:23.351565 14484 tablet_bootstrap.cc:654] T 5d237bb82d7c47409a8924c5ebfd194b P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.351656 14486 ts_tablet_manager.cc:1367] T 93d91773da7f403da2eb85808b686354 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:00:23.352129 14486 raft_consensus.cc:348] T 93d91773da7f403da2eb85808b686354 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:23.352254 14486 raft_consensus.cc:374] T 93d91773da7f403da2eb85808b686354 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.352285 14486 raft_consensus.cc:729] T 93d91773da7f403da2eb85808b686354 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.352742 14486 consensus_queue.cc:260] T 93d91773da7f403da2eb85808b686354 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:23.352819 14484 tablet_bootstrap.cc:492] T 5d237bb82d7c47409a8924c5ebfd194b P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:23.352916 14484 ts_tablet_manager.cc:1367] T 5d237bb82d7c47409a8924c5ebfd194b P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:23.353052 14486 ts_tablet_manager.cc:1398] T 93d91773da7f403da2eb85808b686354 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:23.353201 14486 tablet_bootstrap.cc:492] T fe9cd379022b49f797122c6727d77ef0 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:23.353514 14484 raft_consensus.cc:348] T 5d237bb82d7c47409a8924c5ebfd194b P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:23.353628 14484 raft_consensus.cc:374] T 5d237bb82d7c47409a8924c5ebfd194b P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.353660 14484 raft_consensus.cc:729] T 5d237bb82d7c47409a8924c5ebfd194b P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.353781 14484 consensus_queue.cc:260] T 5d237bb82d7c47409a8924c5ebfd194b P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:23.354072 14484 ts_tablet_manager.cc:1398] T 5d237bb82d7c47409a8924c5ebfd194b P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:23.354140 14489 tablet_bootstrap.cc:492] T 93d91773da7f403da2eb85808b686354 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:23.354250 14484 tablet_bootstrap.cc:492] T 93d91773da7f403da2eb85808b686354 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:23.354336 14486 tablet_bootstrap.cc:654] T fe9cd379022b49f797122c6727d77ef0 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.355206 14484 tablet_bootstrap.cc:654] T 93d91773da7f403da2eb85808b686354 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.355671 14488 tablet_bootstrap.cc:492] T fe9cd379022b49f797122c6727d77ef0 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:23.355756 14488 ts_tablet_manager.cc:1367] T fe9cd379022b49f797122c6727d77ef0 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:00:23.356338 14488 raft_consensus.cc:348] T fe9cd379022b49f797122c6727d77ef0 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:23.356500 14488 raft_consensus.cc:374] T fe9cd379022b49f797122c6727d77ef0 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.356534 14488 raft_consensus.cc:729] T fe9cd379022b49f797122c6727d77ef0 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.356669 14488 consensus_queue.cc:260] T fe9cd379022b49f797122c6727d77ef0 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:23.356961 14488 ts_tablet_manager.cc:1398] T fe9cd379022b49f797122c6727d77ef0 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:23.357304 14484 tablet_bootstrap.cc:492] T 93d91773da7f403da2eb85808b686354 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:23.357461 14484 ts_tablet_manager.cc:1367] T 93d91773da7f403da2eb85808b686354 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:23.357650 14486 tablet_bootstrap.cc:492] T fe9cd379022b49f797122c6727d77ef0 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:23.357761 14486 ts_tablet_manager.cc:1367] T fe9cd379022b49f797122c6727d77ef0 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:00:23.358175 14484 raft_consensus.cc:348] T 93d91773da7f403da2eb85808b686354 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:23.358240 14486 raft_consensus.cc:348] T fe9cd379022b49f797122c6727d77ef0 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:23.358312 14484 raft_consensus.cc:374] T 93d91773da7f403da2eb85808b686354 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.358347 14484 raft_consensus.cc:729] T 93d91773da7f403da2eb85808b686354 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.358377 14486 raft_consensus.cc:374] T fe9cd379022b49f797122c6727d77ef0 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.358425 14486 raft_consensus.cc:729] T fe9cd379022b49f797122c6727d77ef0 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.358484 14484 consensus_queue.cc:260] T 93d91773da7f403da2eb85808b686354 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:23.358587 14486 consensus_queue.cc:260] T fe9cd379022b49f797122c6727d77ef0 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:23.358956 14486 ts_tablet_manager.cc:1398] T fe9cd379022b49f797122c6727d77ef0 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:23.359037 14484 ts_tablet_manager.cc:1398] T 93d91773da7f403da2eb85808b686354 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:23.359241 14484 tablet_bootstrap.cc:492] T fe9cd379022b49f797122c6727d77ef0 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:23.362273 14484 tablet_bootstrap.cc:654] T fe9cd379022b49f797122c6727d77ef0 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.363654 14484 tablet_bootstrap.cc:492] T fe9cd379022b49f797122c6727d77ef0 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:23.363755 14484 ts_tablet_manager.cc:1367] T fe9cd379022b49f797122c6727d77ef0 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:00:23.363996 14489 tablet_bootstrap.cc:654] T 93d91773da7f403da2eb85808b686354 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.364250 14484 raft_consensus.cc:348] T fe9cd379022b49f797122c6727d77ef0 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:23.364491 14484 raft_consensus.cc:374] T fe9cd379022b49f797122c6727d77ef0 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.364573 14484 raft_consensus.cc:729] T fe9cd379022b49f797122c6727d77ef0 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.364786 14484 consensus_queue.cc:260] T fe9cd379022b49f797122c6727d77ef0 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:23.365316 14485 tablet_bootstrap.cc:492] T 5d237bb82d7c47409a8924c5ebfd194b P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:23.366088 14484 ts_tablet_manager.cc:1398] T fe9cd379022b49f797122c6727d77ef0 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:23.366617 14485 tablet_bootstrap.cc:654] T 5d237bb82d7c47409a8924c5ebfd194b P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.368256 14485 tablet_bootstrap.cc:492] T 5d237bb82d7c47409a8924c5ebfd194b P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:23.368541 14485 ts_tablet_manager.cc:1367] T 5d237bb82d7c47409a8924c5ebfd194b P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:23.369058 14485 raft_consensus.cc:348] T 5d237bb82d7c47409a8924c5ebfd194b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:23.369799 14489 tablet_bootstrap.cc:492] T 93d91773da7f403da2eb85808b686354 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:23.369923 14489 ts_tablet_manager.cc:1367] T 93d91773da7f403da2eb85808b686354 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20240503 02:00:23.370432 14489 raft_consensus.cc:348] T 93d91773da7f403da2eb85808b686354 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:23.370565 14489 raft_consensus.cc:374] T 93d91773da7f403da2eb85808b686354 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.370607 14489 raft_consensus.cc:729] T 93d91773da7f403da2eb85808b686354 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.370769 14489 consensus_queue.cc:260] T 93d91773da7f403da2eb85808b686354 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:23.371244 14489 ts_tablet_manager.cc:1398] T 93d91773da7f403da2eb85808b686354 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 02:00:23.369172 14485 raft_consensus.cc:374] T 5d237bb82d7c47409a8924c5ebfd194b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.371398 14485 raft_consensus.cc:729] T 5d237bb82d7c47409a8924c5ebfd194b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.371752 14485 consensus_queue.cc:260] T 5d237bb82d7c47409a8924c5ebfd194b P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:23.372222 14485 ts_tablet_manager.cc:1398] T 5d237bb82d7c47409a8924c5ebfd194b P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 02:00:23.374706 14345 raft_consensus.cc:482] T 5d237bb82d7c47409a8924c5ebfd194b P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:23.374819 14345 raft_consensus.cc:504] T 5d237bb82d7c47409a8924c5ebfd194b P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:23.375289 14345 leader_election.cc:288] T 5d237bb82d7c47409a8924c5ebfd194b P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:23.376149 6719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d237bb82d7c47409a8924c5ebfd194b" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 02:00:23.376276 6719 raft_consensus.cc:2454] T 5d237bb82d7c47409a8924c5ebfd194b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:23.376574 9115 leader_election.cc:302] T 5d237bb82d7c47409a8924c5ebfd194b P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:23.376761 14413 raft_consensus.cc:2781] T 5d237bb82d7c47409a8924c5ebfd194b P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:23.376806 14413 raft_consensus.cc:482] T 5d237bb82d7c47409a8924c5ebfd194b P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:23.376838 14413 raft_consensus.cc:3037] T 5d237bb82d7c47409a8924c5ebfd194b P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:23.378238 14413 raft_consensus.cc:504] T 5d237bb82d7c47409a8924c5ebfd194b P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:23.378715 14413 leader_election.cc:288] T 5d237bb82d7c47409a8924c5ebfd194b P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Requested vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:23.379585 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d237bb82d7c47409a8924c5ebfd194b" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:23.379750 9048 raft_consensus.cc:3037] T 5d237bb82d7c47409a8924c5ebfd194b P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:23.379601 9047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d237bb82d7c47409a8924c5ebfd194b" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:23.380661 6719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d237bb82d7c47409a8924c5ebfd194b" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 02:00:23.380785 6719 raft_consensus.cc:3037] T 5d237bb82d7c47409a8924c5ebfd194b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:23.380895 9048 raft_consensus.cc:2454] T 5d237bb82d7c47409a8924c5ebfd194b P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:23.381425 9113 leader_election.cc:302] T 5d237bb82d7c47409a8924c5ebfd194b P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:23.381664 14413 raft_consensus.cc:2781] T 5d237bb82d7c47409a8924c5ebfd194b P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:23.381686 6719 raft_consensus.cc:2454] T 5d237bb82d7c47409a8924c5ebfd194b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:23.381738 14413 raft_consensus.cc:686] T 5d237bb82d7c47409a8924c5ebfd194b P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Becoming Leader. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Running, Role: LEADER
I20240503 02:00:23.381861 14413 consensus_queue.cc:237] T 5d237bb82d7c47409a8924c5ebfd194b P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:23.384683 6309 catalog_manager.cc:5484] T 5d237bb82d7c47409a8924c5ebfd194b P 2a01ace80baf4057b6bf1995cf88468b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135). New cstate: current_term: 1 leader_uuid: "2a01ace80baf4057b6bf1995cf88468b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:23.434433 6309 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:23.434618 6309 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:23.436852 6309 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet 5976a8a1983f497eb0daa9c4fa9b701e
I20240503 02:00:23.437623 8880 tablet_service.cc:1514] Processing DeleteTablet for tablet 5976a8a1983f497eb0daa9c4fa9b701e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:23 UTC) from {username='slave'} at 127.0.0.1:40296
I20240503 02:00:23.437600 6436 tablet_service.cc:1514] Processing DeleteTablet for tablet 5976a8a1983f497eb0daa9c4fa9b701e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:23 UTC) from {username='slave'} at 127.0.0.1:51338
I20240503 02:00:23.438073 14500 tablet_replica.cc:330] T 5976a8a1983f497eb0daa9c4fa9b701e P dec3e4a04d604f7eaaa3f077d081218a: stopping tablet replica
I20240503 02:00:23.438199 6829 tablet_service.cc:1514] Processing DeleteTablet for tablet 5976a8a1983f497eb0daa9c4fa9b701e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:23 UTC) from {username='slave'} at 127.0.0.1:52812
I20240503 02:00:23.438493 14501 tablet_replica.cc:330] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d: stopping tablet replica
I20240503 02:00:23.438510 14500 raft_consensus.cc:2229] T 5976a8a1983f497eb0daa9c4fa9b701e P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:23.438649 14500 raft_consensus.cc:2258] T 5976a8a1983f497eb0daa9c4fa9b701e P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:23.438663 14501 raft_consensus.cc:2229] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:23.438879 14501 raft_consensus.cc:2258] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:23.439419 14500 ts_tablet_manager.cc:1875] T 5976a8a1983f497eb0daa9c4fa9b701e P dec3e4a04d604f7eaaa3f077d081218a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:23.439425 14501 ts_tablet_manager.cc:1875] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:23.441532 14500 ts_tablet_manager.cc:1888] T 5976a8a1983f497eb0daa9c4fa9b701e P dec3e4a04d604f7eaaa3f077d081218a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:23.441606 14500 log.cc:1196] T 5976a8a1983f497eb0daa9c4fa9b701e P dec3e4a04d604f7eaaa3f077d081218a: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/5976a8a1983f497eb0daa9c4fa9b701e
I20240503 02:00:23.441839 14500 ts_tablet_manager.cc:1909] T 5976a8a1983f497eb0daa9c4fa9b701e P dec3e4a04d604f7eaaa3f077d081218a: Deleting consensus metadata
I20240503 02:00:23.442047 14499 tablet_replica.cc:330] T 5976a8a1983f497eb0daa9c4fa9b701e P 9942306855404cfd891adf227dd65eb2: stopping tablet replica
I20240503 02:00:23.442209 14499 raft_consensus.cc:2229] T 5976a8a1983f497eb0daa9c4fa9b701e P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:23.442296 14501 ts_tablet_manager.cc:1888] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:23.442337 14499 raft_consensus.cc:2258] T 5976a8a1983f497eb0daa9c4fa9b701e P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:23.442379 14501 log.cc:1196] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/5976a8a1983f497eb0daa9c4fa9b701e
I20240503 02:00:23.442692 14501 ts_tablet_manager.cc:1909] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d: Deleting consensus metadata
I20240503 02:00:23.442979 6300 catalog_manager.cc:4830] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): tablet 5976a8a1983f497eb0daa9c4fa9b701e (table rebalancer_dynamic_table_new_name [id=e890cb2b2c1049fdb59177f7670af7a3]) successfully deleted
I20240503 02:00:23.443430 14499 ts_tablet_manager.cc:1875] T 5976a8a1983f497eb0daa9c4fa9b701e P 9942306855404cfd891adf227dd65eb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:23.444291 6303 catalog_manager.cc:4830] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719): tablet 5976a8a1983f497eb0daa9c4fa9b701e (table rebalancer_dynamic_table_new_name [id=e890cb2b2c1049fdb59177f7670af7a3]) successfully deleted
I20240503 02:00:23.445396 14499 ts_tablet_manager.cc:1888] T 5976a8a1983f497eb0daa9c4fa9b701e P 9942306855404cfd891adf227dd65eb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:23.445472 14499 log.cc:1196] T 5976a8a1983f497eb0daa9c4fa9b701e P 9942306855404cfd891adf227dd65eb2: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/5976a8a1983f497eb0daa9c4fa9b701e
I20240503 02:00:23.445650 14499 ts_tablet_manager.cc:1909] T 5976a8a1983f497eb0daa9c4fa9b701e P 9942306855404cfd891adf227dd65eb2: Deleting consensus metadata
I20240503 02:00:23.446475 6303 catalog_manager.cc:4830] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211): tablet 5976a8a1983f497eb0daa9c4fa9b701e (table rebalancer_dynamic_table_new_name [id=e890cb2b2c1049fdb59177f7670af7a3]) successfully deleted
W20240503 02:00:23.476244 6654 consensus_peers.cc:485] T ba17d4d3e0754b2798eb95484d37229a P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:23.486716 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:23.539091 6309 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:00:23.544031 14462 raft_consensus.cc:482] T fe9cd379022b49f797122c6727d77ef0 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:23.544173 14462 raft_consensus.cc:504] T fe9cd379022b49f797122c6727d77ef0 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:23.544749 14462 leader_election.cc:288] T fe9cd379022b49f797122c6727d77ef0 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:23.544801 6436 tablet_service.cc:1467] Processing CreateTablet for tablet 0401cac53a9c44ec83de9643aac3e5dc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9a306ff2611d406e912ebbeffa5b62a4]), partition=
I20240503 02:00:23.544801 6568 tablet_service.cc:1467] Processing CreateTablet for tablet 0401cac53a9c44ec83de9643aac3e5dc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9a306ff2611d406e912ebbeffa5b62a4]), partition=
I20240503 02:00:23.545064 6568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0401cac53a9c44ec83de9643aac3e5dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.545078 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0401cac53a9c44ec83de9643aac3e5dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.545508 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9cd379022b49f797122c6727d77ef0" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:23.545647 9048 raft_consensus.cc:2454] T fe9cd379022b49f797122c6727d77ef0 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:23.546041 6523 leader_election.cc:302] T fe9cd379022b49f797122c6727d77ef0 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:23.546350 14462 raft_consensus.cc:2781] T fe9cd379022b49f797122c6727d77ef0 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:23.546407 14462 raft_consensus.cc:482] T fe9cd379022b49f797122c6727d77ef0 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:23.546442 14462 raft_consensus.cc:3037] T fe9cd379022b49f797122c6727d77ef0 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:23.547252 14488 tablet_bootstrap.cc:492] T 0401cac53a9c44ec83de9643aac3e5dc P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:23.547449 14489 tablet_bootstrap.cc:492] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:23.547405 14462 raft_consensus.cc:504] T fe9cd379022b49f797122c6727d77ef0 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:23.547813 14462 leader_election.cc:288] T fe9cd379022b49f797122c6727d77ef0 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Requested vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:23.547987 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9cd379022b49f797122c6727d77ef0" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:23.548121 9178 raft_consensus.cc:2454] T fe9cd379022b49f797122c6727d77ef0 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:23.548415 14489 tablet_bootstrap.cc:654] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.548414 14488 tablet_bootstrap.cc:654] T 0401cac53a9c44ec83de9643aac3e5dc P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.548460 14431 raft_consensus.cc:482] T 93d91773da7f403da2eb85808b686354 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:23.548558 14431 raft_consensus.cc:504] T 93d91773da7f403da2eb85808b686354 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:23.548763 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9cd379022b49f797122c6727d77ef0" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:23.548856 9178 raft_consensus.cc:3037] T fe9cd379022b49f797122c6727d77ef0 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:23.549084 14431 leader_election.cc:288] T 93d91773da7f403da2eb85808b686354 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 02:00:23.549547 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d91773da7f403da2eb85808b686354" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 02:00:23.549674 6458 raft_consensus.cc:2454] T 93d91773da7f403da2eb85808b686354 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:23.549861 14488 tablet_bootstrap.cc:492] T 0401cac53a9c44ec83de9643aac3e5dc P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:23.549885 9178 raft_consensus.cc:2454] T fe9cd379022b49f797122c6727d77ef0 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:23.549939 14488 ts_tablet_manager.cc:1367] T 0401cac53a9c44ec83de9643aac3e5dc P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:00:23.550065 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d91773da7f403da2eb85808b686354" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:23.550159 9179 raft_consensus.cc:2454] T 93d91773da7f403da2eb85808b686354 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:23.550422 8981 leader_election.cc:302] T 93d91773da7f403da2eb85808b686354 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:23.550397 14488 raft_consensus.cc:348] T 0401cac53a9c44ec83de9643aac3e5dc P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:23.550491 14488 raft_consensus.cc:374] T 0401cac53a9c44ec83de9643aac3e5dc P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.550506 6522 leader_election.cc:302] T fe9cd379022b49f797122c6727d77ef0 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 80fd18ef4d604c86b909d5c008a46b4f; no voters:
I20240503 02:00:23.550518 14488 raft_consensus.cc:729] T 0401cac53a9c44ec83de9643aac3e5dc P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.550674 14488 consensus_queue.cc:260] T 0401cac53a9c44ec83de9643aac3e5dc P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:23.550944 14406 raft_consensus.cc:2781] T fe9cd379022b49f797122c6727d77ef0 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:23.551012 14406 raft_consensus.cc:686] T fe9cd379022b49f797122c6727d77ef0 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Becoming Leader. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Running, Role: LEADER
I20240503 02:00:23.551041 14488 ts_tablet_manager.cc:1398] T 0401cac53a9c44ec83de9643aac3e5dc P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:23.551028 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9cd379022b49f797122c6727d77ef0" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:23.551126 9048 raft_consensus.cc:3037] T fe9cd379022b49f797122c6727d77ef0 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:23.551132 14406 consensus_queue.cc:237] T fe9cd379022b49f797122c6727d77ef0 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:23.552160 9048 raft_consensus.cc:2454] T fe9cd379022b49f797122c6727d77ef0 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:23.552503 14195 raft_consensus.cc:2781] T 93d91773da7f403da2eb85808b686354 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:23.552564 14195 raft_consensus.cc:482] T 93d91773da7f403da2eb85808b686354 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:23.552600 14195 raft_consensus.cc:3037] T 93d91773da7f403da2eb85808b686354 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:23.553265 14489 tablet_bootstrap.cc:492] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:23.553375 14489 ts_tablet_manager.cc:1367] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20240503 02:00:23.553619 14195 raft_consensus.cc:504] T 93d91773da7f403da2eb85808b686354 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:23.553905 14489 raft_consensus.cc:348] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:23.553982 14195 leader_election.cc:288] T 93d91773da7f403da2eb85808b686354 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Requested vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 02:00:23.554010 14489 raft_consensus.cc:374] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.554040 14489 raft_consensus.cc:729] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.554158 14489 consensus_queue.cc:260] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:23.554774 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d91773da7f403da2eb85808b686354" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 02:00:23.554811 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d91773da7f403da2eb85808b686354" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:23.554877 6458 raft_consensus.cc:3037] T 93d91773da7f403da2eb85808b686354 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:23.554773 6309 catalog_manager.cc:5484] T fe9cd379022b49f797122c6727d77ef0 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:23.554924 9179 raft_consensus.cc:3037] T 93d91773da7f403da2eb85808b686354 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:23.555881 6458 raft_consensus.cc:2454] T 93d91773da7f403da2eb85808b686354 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:23.555922 9179 raft_consensus.cc:2454] T 93d91773da7f403da2eb85808b686354 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:23.556036 14489 ts_tablet_manager.cc:1398] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 02:00:23.556397 8981 leader_election.cc:302] T 93d91773da7f403da2eb85808b686354 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:23.557044 14195 raft_consensus.cc:2781] T 93d91773da7f403da2eb85808b686354 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:23.557139 14195 raft_consensus.cc:686] T 93d91773da7f403da2eb85808b686354 P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Becoming Leader. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Running, Role: LEADER
I20240503 02:00:23.557297 14195 consensus_queue.cc:237] T 93d91773da7f403da2eb85808b686354 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:23.559700 8880 tablet_service.cc:1467] Processing CreateTablet for tablet 0401cac53a9c44ec83de9643aac3e5dc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9a306ff2611d406e912ebbeffa5b62a4]), partition=
I20240503 02:00:23.559989 8880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0401cac53a9c44ec83de9643aac3e5dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.561163 6313 catalog_manager.cc:5484] T 93d91773da7f403da2eb85808b686354 P 99ead02915954b4fab23b033ad6d546b reported cstate change: term changed from 0 to 1, leader changed from <none> to 99ead02915954b4fab23b033ad6d546b (127.0.99.134). New cstate: current_term: 1 leader_uuid: "99ead02915954b4fab23b033ad6d546b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:23.562724 14507 tablet_bootstrap.cc:492] T 0401cac53a9c44ec83de9643aac3e5dc P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:23.564548 14507 tablet_bootstrap.cc:654] T 0401cac53a9c44ec83de9643aac3e5dc P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.568220 14507 tablet_bootstrap.cc:492] T 0401cac53a9c44ec83de9643aac3e5dc P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:23.568337 14507 ts_tablet_manager.cc:1367] T 0401cac53a9c44ec83de9643aac3e5dc P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20240503 02:00:23.568982 14507 raft_consensus.cc:348] T 0401cac53a9c44ec83de9643aac3e5dc P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:23.569124 14507 raft_consensus.cc:374] T 0401cac53a9c44ec83de9643aac3e5dc P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.569165 14507 raft_consensus.cc:729] T 0401cac53a9c44ec83de9643aac3e5dc P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.569291 14507 consensus_queue.cc:260] T 0401cac53a9c44ec83de9643aac3e5dc P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:23.569618 14507 ts_tablet_manager.cc:1398] T 0401cac53a9c44ec83de9643aac3e5dc P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240503 02:00:23.619670 9113 consensus_peers.cc:485] T f44b59277c3b47adbd4dd6aff9350af1 P 2a01ace80baf4057b6bf1995cf88468b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:23.630760 6785 consensus_peers.cc:485] T 8246488dec204bf6b09e868d7951ac5a P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:23.675060 6523 consensus_peers.cc:485] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:23.687184 8851 consensus_peers.cc:485] T 067f54011a37432d99157884e70fc601 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:23.709378 6523 consensus_peers.cc:485] T d0d5dd42db3e4883af7fd16d8813b550 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:00:23.789397 14461 raft_consensus.cc:482] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:23.789559 14461 raft_consensus.cc:504] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:23.790028 14461 leader_election.cc:288] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:23.790549 6587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0401cac53a9c44ec83de9643aac3e5dc" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 02:00:23.790549 8917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0401cac53a9c44ec83de9643aac3e5dc" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:23.790701 8917 raft_consensus.cc:2454] T 0401cac53a9c44ec83de9643aac3e5dc P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:23.790802 6587 raft_consensus.cc:2454] T 0401cac53a9c44ec83de9643aac3e5dc P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:23.791139 6393 leader_election.cc:302] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:23.791316 14461 raft_consensus.cc:2781] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:23.791369 14461 raft_consensus.cc:482] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:23.791404 14461 raft_consensus.cc:3037] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:23.792290 14461 raft_consensus.cc:504] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:23.792698 14461 leader_election.cc:288] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Requested vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:23.793038 8917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0401cac53a9c44ec83de9643aac3e5dc" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:23.793083 6587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0401cac53a9c44ec83de9643aac3e5dc" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 02:00:23.793197 6587 raft_consensus.cc:3037] T 0401cac53a9c44ec83de9643aac3e5dc P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:23.793503 8917 raft_consensus.cc:3037] T 0401cac53a9c44ec83de9643aac3e5dc P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:00:23.794085 6785 consensus_peers.cc:485] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:00:23.794266 6587 raft_consensus.cc:2454] T 0401cac53a9c44ec83de9643aac3e5dc P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:23.794610 6393 leader_election.cc:302] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:23.794760 8917 raft_consensus.cc:2454] T 0401cac53a9c44ec83de9643aac3e5dc P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:23.794806 14461 raft_consensus.cc:2781] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:23.794879 14461 raft_consensus.cc:686] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Becoming Leader. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Running, Role: LEADER
I20240503 02:00:23.795030 14461 consensus_queue.cc:237] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:23.797606 6308 catalog_manager.cc:5484] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: term changed from 0 to 1, leader changed from <none> to dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129). New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:23.800735 14448 consensus_queue.cc:1035] T 5d237bb82d7c47409a8924c5ebfd194b P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:23.804241 14448 consensus_queue.cc:1035] T 5d237bb82d7c47409a8924c5ebfd194b P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:00:23.819082 6523 consensus_peers.cc:485] T 925612b660ec471da5b3c548802ede6e P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:23.821225 8982 consensus_peers.cc:485] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:23.838507 6785 consensus_peers.cc:485] T d897c21e630d4770832bb10f43778e82 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:23.870110 8851 consensus_peers.cc:485] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:23.896935 9113 consensus_peers.cc:485] T b1740b877c1e4372a0337d99f5bcab0e P 2a01ace80baf4057b6bf1995cf88468b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:23.900218 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:23.956315 6308 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:23.960568 8917 raft_consensus.cc:1261] T 0401cac53a9c44ec83de9643aac3e5dc P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:23.960568 6587 raft_consensus.cc:1261] T 0401cac53a9c44ec83de9643aac3e5dc P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:23.961046 14515 consensus_queue.cc:1035] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:23.961239 14461 consensus_queue.cc:1035] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:23.964609 6308 catalog_manager.cc:5576] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863) does not have the latest schema for tablet 0401cac53a9c44ec83de9643aac3e5dc (table rebalancer_dynamic_table [id=9a306ff2611d406e912ebbeffa5b62a4]). Expected version 1 got 0
I20240503 02:00:23.965258 6309 catalog_manager.cc:5576] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211) does not have the latest schema for tablet 0401cac53a9c44ec83de9643aac3e5dc (table rebalancer_dynamic_table [id=9a306ff2611d406e912ebbeffa5b62a4]). Expected version 1 got 0
I20240503 02:00:23.967518 6309 catalog_manager.cc:5576] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211) does not have the latest schema for tablet 0401cac53a9c44ec83de9643aac3e5dc (table rebalancer_dynamic_table [id=9a306ff2611d406e912ebbeffa5b62a4]). Expected version 1 got 0
I20240503 02:00:23.969583 14522 tablet.cc:1722] T 0401cac53a9c44ec83de9643aac3e5dc P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:23.969975 14522 tablet.cc:1620] T 0401cac53a9c44ec83de9643aac3e5dc P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
I20240503 02:00:23.970383 14521 tablet.cc:1722] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:23.970609 6308 catalog_manager.cc:5576] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093) does not have the latest schema for tablet 0401cac53a9c44ec83de9643aac3e5dc (table rebalancer_dynamic_table [id=9a306ff2611d406e912ebbeffa5b62a4]). Expected version 1 got 0
I20240503 02:00:23.970736 14521 tablet.cc:1620] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a: MemRowSet was empty: no flush needed.
W20240503 02:00:23.972445 6523 consensus_peers.cc:485] T 4385d51d11574192883d01376e2d5620 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:00:23.977926 6313 catalog_manager.cc:5576] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863) does not have the latest schema for tablet 0401cac53a9c44ec83de9643aac3e5dc (table rebalancer_dynamic_table [id=9a306ff2611d406e912ebbeffa5b62a4]). Expected version 1 got 0
I20240503 02:00:23.978349 6309 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=9a306ff2611d406e912ebbeffa5b62a4] alter complete (version 1)
I20240503 02:00:23.984700 14523 tablet.cc:1722] T 0401cac53a9c44ec83de9643aac3e5dc P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:23.985059 14523 tablet.cc:1620] T 0401cac53a9c44ec83de9643aac3e5dc P 80fd18ef4d604c86b909d5c008a46b4f: MemRowSet was empty: no flush needed.
W20240503 02:00:23.998529 6654 consensus_peers.cc:485] T cea1ea50fa434d31b42451f84f26bc76 P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:24.013710 6785 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111) [suppressed 62 similar messages]
I20240503 02:00:24.016319 14195 consensus_queue.cc:1035] T 93d91773da7f403da2eb85808b686354 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:24.020464 14195 consensus_queue.cc:1035] T 93d91773da7f403da2eb85808b686354 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:00:24.036712 8851 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 02:00:24.037757 8851 consensus_peers.cc:485] T f28586e70bd9473fbdc6b9126b666fa9 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:24.039000 6785 consensus_peers.cc:485] T 46ad540deb624a25b97d3b967fe1f6f4 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:00:24.051651 14462 consensus_queue.cc:1035] T fe9cd379022b49f797122c6727d77ef0 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:24.055063 14504 consensus_queue.cc:1035] T fe9cd379022b49f797122c6727d77ef0 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:24.083456 6309 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:24.088938 14521 tablet.cc:1722] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:24.089311 14521 tablet.cc:1620] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a: MemRowSet was empty: no flush needed.
I20240503 02:00:24.089442 14522 tablet.cc:1722] T 0401cac53a9c44ec83de9643aac3e5dc P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:24.089752 14522 tablet.cc:1620] T 0401cac53a9c44ec83de9643aac3e5dc P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
I20240503 02:00:24.092604 14523 tablet.cc:1722] T 0401cac53a9c44ec83de9643aac3e5dc P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:24.092957 14523 tablet.cc:1620] T 0401cac53a9c44ec83de9643aac3e5dc P 80fd18ef4d604c86b909d5c008a46b4f: MemRowSet was empty: no flush needed.
I20240503 02:00:24.096447 6309 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=9a306ff2611d406e912ebbeffa5b62a4] alter complete (version 2)
I20240503 02:00:24.106464 6313 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_extra_table_62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:24.113404 6699 tablet_service.cc:1467] Processing CreateTablet for tablet 4fb6646299c54e4da1938ea864806be4 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=aa1ce1266eff43659598655cc5166948]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:24.113644 6829 tablet_service.cc:1467] Processing CreateTablet for tablet 4fb6646299c54e4da1938ea864806be4 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=aa1ce1266eff43659598655cc5166948]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:24.113682 6700 tablet_service.cc:1467] Processing CreateTablet for tablet cdb766efec08440aa6a296edb76f84b7 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=aa1ce1266eff43659598655cc5166948]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:24.113744 6699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fb6646299c54e4da1938ea864806be4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.113907 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdb766efec08440aa6a296edb76f84b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.113934 6829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fb6646299c54e4da1938ea864806be4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.116191 14528 tablet_bootstrap.cc:492] T 4fb6646299c54e4da1938ea864806be4 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:24.116449 8880 tablet_service.cc:1467] Processing CreateTablet for tablet cdb766efec08440aa6a296edb76f84b7 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=aa1ce1266eff43659598655cc5166948]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:24.116453 9157 tablet_service.cc:1467] Processing CreateTablet for tablet 4fb6646299c54e4da1938ea864806be4 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=aa1ce1266eff43659598655cc5166948]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:24.116744 9157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fb6646299c54e4da1938ea864806be4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.116744 8880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdb766efec08440aa6a296edb76f84b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.117334 14528 tablet_bootstrap.cc:654] T 4fb6646299c54e4da1938ea864806be4 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.117558 6568 tablet_service.cc:1467] Processing CreateTablet for tablet cdb766efec08440aa6a296edb76f84b7 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=aa1ce1266eff43659598655cc5166948]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:24.117830 6568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdb766efec08440aa6a296edb76f84b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.120173 14531 tablet_bootstrap.cc:492] T cdb766efec08440aa6a296edb76f84b7 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
W20240503 02:00:24.120415 6654 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240503 02:00:24.121366 14531 tablet_bootstrap.cc:654] T cdb766efec08440aa6a296edb76f84b7 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.121922 14528 tablet_bootstrap.cc:492] T 4fb6646299c54e4da1938ea864806be4 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:24.122035 14528 ts_tablet_manager.cc:1367] T 4fb6646299c54e4da1938ea864806be4 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
W20240503 02:00:24.122462 6523 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111) [suppressed 58 similar messages]
I20240503 02:00:24.122737 14528 raft_consensus.cc:348] T 4fb6646299c54e4da1938ea864806be4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:24.122949 14528 raft_consensus.cc:374] T 4fb6646299c54e4da1938ea864806be4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.122988 14528 raft_consensus.cc:729] T 4fb6646299c54e4da1938ea864806be4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.123126 14528 consensus_queue.cc:260] T 4fb6646299c54e4da1938ea864806be4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:24.123423 14528 ts_tablet_manager.cc:1398] T 4fb6646299c54e4da1938ea864806be4 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:24.123591 14528 tablet_bootstrap.cc:492] T cdb766efec08440aa6a296edb76f84b7 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:24.124711 14531 tablet_bootstrap.cc:492] T cdb766efec08440aa6a296edb76f84b7 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:24.124897 14531 ts_tablet_manager.cc:1367] T cdb766efec08440aa6a296edb76f84b7 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20240503 02:00:24.125788 14528 tablet_bootstrap.cc:654] T cdb766efec08440aa6a296edb76f84b7 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.126482 6700 tablet_service.cc:1467] Processing CreateTablet for tablet f55cb3da9c874387a65eff5de0903331 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=aa1ce1266eff43659598655cc5166948]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:24.126756 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f55cb3da9c874387a65eff5de0903331. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.126904 14532 tablet_bootstrap.cc:492] T cdb766efec08440aa6a296edb76f84b7 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:24.127275 14528 tablet_bootstrap.cc:492] T cdb766efec08440aa6a296edb76f84b7 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:24.127344 14528 ts_tablet_manager.cc:1367] T cdb766efec08440aa6a296edb76f84b7 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:24.128365 14528 raft_consensus.cc:348] T cdb766efec08440aa6a296edb76f84b7 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:24.128499 14528 raft_consensus.cc:374] T cdb766efec08440aa6a296edb76f84b7 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.128526 14528 raft_consensus.cc:729] T cdb766efec08440aa6a296edb76f84b7 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.128643 14528 consensus_queue.cc:260] T cdb766efec08440aa6a296edb76f84b7 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:24.128943 14528 ts_tablet_manager.cc:1398] T cdb766efec08440aa6a296edb76f84b7 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:00:24.129088 14528 tablet_bootstrap.cc:492] T f55cb3da9c874387a65eff5de0903331 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:24.129560 8880 tablet_service.cc:1467] Processing CreateTablet for tablet f55cb3da9c874387a65eff5de0903331 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=aa1ce1266eff43659598655cc5166948]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:24.129820 8880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f55cb3da9c874387a65eff5de0903331. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.129990 14528 tablet_bootstrap.cc:654] T f55cb3da9c874387a65eff5de0903331 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.125470 14531 raft_consensus.cc:348] T cdb766efec08440aa6a296edb76f84b7 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:24.130043 14531 raft_consensus.cc:374] T cdb766efec08440aa6a296edb76f84b7 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.130100 14531 raft_consensus.cc:729] T cdb766efec08440aa6a296edb76f84b7 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.130254 14531 consensus_queue.cc:260] T cdb766efec08440aa6a296edb76f84b7 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:24.132155 14529 tablet_bootstrap.cc:492] T 4fb6646299c54e4da1938ea864806be4 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:24.132647 14532 tablet_bootstrap.cc:654] T cdb766efec08440aa6a296edb76f84b7 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.132755 9028 tablet_service.cc:1467] Processing CreateTablet for tablet f55cb3da9c874387a65eff5de0903331 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=aa1ce1266eff43659598655cc5166948]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:24.133037 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f55cb3da9c874387a65eff5de0903331. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.132966 14531 ts_tablet_manager.cc:1398] T cdb766efec08440aa6a296edb76f84b7 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.008s user 0.000s sys 0.001s
I20240503 02:00:24.133363 14531 tablet_bootstrap.cc:492] T f55cb3da9c874387a65eff5de0903331 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:24.134406 14534 tablet_bootstrap.cc:492] T 4fb6646299c54e4da1938ea864806be4 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:24.135797 14534 tablet_bootstrap.cc:654] T 4fb6646299c54e4da1938ea864806be4 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.135890 14531 tablet_bootstrap.cc:654] T f55cb3da9c874387a65eff5de0903331 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.137274 14529 tablet_bootstrap.cc:654] T 4fb6646299c54e4da1938ea864806be4 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.137817 14531 tablet_bootstrap.cc:492] T f55cb3da9c874387a65eff5de0903331 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:24.137956 14534 tablet_bootstrap.cc:492] T 4fb6646299c54e4da1938ea864806be4 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:24.138037 14531 ts_tablet_manager.cc:1367] T f55cb3da9c874387a65eff5de0903331 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20240503 02:00:24.138068 14534 ts_tablet_manager.cc:1367] T 4fb6646299c54e4da1938ea864806be4 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:24.138561 14534 raft_consensus.cc:348] T 4fb6646299c54e4da1938ea864806be4 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:24.138685 14534 raft_consensus.cc:374] T 4fb6646299c54e4da1938ea864806be4 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.138721 14534 raft_consensus.cc:729] T 4fb6646299c54e4da1938ea864806be4 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.138721 14531 raft_consensus.cc:348] T f55cb3da9c874387a65eff5de0903331 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:24.138841 14531 raft_consensus.cc:374] T f55cb3da9c874387a65eff5de0903331 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.138835 14534 consensus_queue.cc:260] T 4fb6646299c54e4da1938ea864806be4 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:24.138885 14531 raft_consensus.cc:729] T f55cb3da9c874387a65eff5de0903331 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.139071 14531 consensus_queue.cc:260] T f55cb3da9c874387a65eff5de0903331 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:24.139125 14534 ts_tablet_manager.cc:1398] T 4fb6646299c54e4da1938ea864806be4 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:24.142235 14531 ts_tablet_manager.cc:1398] T f55cb3da9c874387a65eff5de0903331 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240503 02:00:24.142473 14537 tablet_bootstrap.cc:492] T f55cb3da9c874387a65eff5de0903331 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:24.142402 14529 tablet_bootstrap.cc:492] T 4fb6646299c54e4da1938ea864806be4 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:24.143380 14529 ts_tablet_manager.cc:1367] T 4fb6646299c54e4da1938ea864806be4 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 02:00:24.143718 14537 tablet_bootstrap.cc:654] T f55cb3da9c874387a65eff5de0903331 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.145359 14529 raft_consensus.cc:348] T 4fb6646299c54e4da1938ea864806be4 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:24.145503 14529 raft_consensus.cc:374] T 4fb6646299c54e4da1938ea864806be4 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.145536 14529 raft_consensus.cc:729] T 4fb6646299c54e4da1938ea864806be4 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.145709 14529 consensus_queue.cc:260] T 4fb6646299c54e4da1938ea864806be4 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:24.145789 14532 tablet_bootstrap.cc:492] T cdb766efec08440aa6a296edb76f84b7 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:24.145787 14528 tablet_bootstrap.cc:492] T f55cb3da9c874387a65eff5de0903331 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:24.145879 14528 ts_tablet_manager.cc:1367] T f55cb3da9c874387a65eff5de0903331 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.002s
I20240503 02:00:24.145880 14532 ts_tablet_manager.cc:1367] T cdb766efec08440aa6a296edb76f84b7 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240503 02:00:24.146024 14529 ts_tablet_manager.cc:1398] T 4fb6646299c54e4da1938ea864806be4 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:24.146342 14528 raft_consensus.cc:348] T f55cb3da9c874387a65eff5de0903331 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:24.146448 14528 raft_consensus.cc:374] T f55cb3da9c874387a65eff5de0903331 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.146384 14532 raft_consensus.cc:348] T cdb766efec08440aa6a296edb76f84b7 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:24.146476 14528 raft_consensus.cc:729] T f55cb3da9c874387a65eff5de0903331 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.146503 14532 raft_consensus.cc:374] T cdb766efec08440aa6a296edb76f84b7 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.146528 14532 raft_consensus.cc:729] T cdb766efec08440aa6a296edb76f84b7 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.146597 14528 consensus_queue.cc:260] T f55cb3da9c874387a65eff5de0903331 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:24.146868 14528 ts_tablet_manager.cc:1398] T f55cb3da9c874387a65eff5de0903331 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:24.147204 14532 consensus_queue.cc:260] T cdb766efec08440aa6a296edb76f84b7 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:24.147773 14537 tablet_bootstrap.cc:492] T f55cb3da9c874387a65eff5de0903331 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:24.147899 14537 ts_tablet_manager.cc:1367] T f55cb3da9c874387a65eff5de0903331 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:00:24.148608 14537 raft_consensus.cc:348] T f55cb3da9c874387a65eff5de0903331 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:24.148784 14537 raft_consensus.cc:374] T f55cb3da9c874387a65eff5de0903331 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.148835 14537 raft_consensus.cc:729] T f55cb3da9c874387a65eff5de0903331 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.149012 14537 consensus_queue.cc:260] T f55cb3da9c874387a65eff5de0903331 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:24.149355 14537 ts_tablet_manager.cc:1398] T f55cb3da9c874387a65eff5de0903331 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:24.150733 14532 ts_tablet_manager.cc:1398] T cdb766efec08440aa6a296edb76f84b7 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
W20240503 02:00:24.157666 6523 consensus_peers.cc:485] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:00:24.166128 14459 raft_consensus.cc:482] T f55cb3da9c874387a65eff5de0903331 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:24.166278 14459 raft_consensus.cc:504] T f55cb3da9c874387a65eff5de0903331 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:24.166747 14459 leader_election.cc:288] T f55cb3da9c874387a65eff5de0903331 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:24.167201 6719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55cb3da9c874387a65eff5de0903331" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 02:00:24.167290 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55cb3da9c874387a65eff5de0903331" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:24.167338 6719 raft_consensus.cc:2454] T f55cb3da9c874387a65eff5de0903331 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 0.
I20240503 02:00:24.167428 9048 raft_consensus.cc:2454] T f55cb3da9c874387a65eff5de0903331 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 0.
I20240503 02:00:24.167706 8851 leader_election.cc:302] T f55cb3da9c874387a65eff5de0903331 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9942306855404cfd891adf227dd65eb2, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:24.167892 14459 raft_consensus.cc:2781] T f55cb3da9c874387a65eff5de0903331 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:24.167958 14459 raft_consensus.cc:482] T f55cb3da9c874387a65eff5de0903331 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:24.167990 14459 raft_consensus.cc:3037] T f55cb3da9c874387a65eff5de0903331 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:24.169148 14459 raft_consensus.cc:504] T f55cb3da9c874387a65eff5de0903331 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:24.169987 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55cb3da9c874387a65eff5de0903331" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:24.170068 6719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55cb3da9c874387a65eff5de0903331" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 02:00:24.170127 9048 raft_consensus.cc:3037] T f55cb3da9c874387a65eff5de0903331 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:24.170202 6719 raft_consensus.cc:3037] T f55cb3da9c874387a65eff5de0903331 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:24.171115 6719 raft_consensus.cc:2454] T f55cb3da9c874387a65eff5de0903331 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 1.
I20240503 02:00:24.171168 9048 raft_consensus.cc:2454] T f55cb3da9c874387a65eff5de0903331 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 1.
I20240503 02:00:24.169526 14459 leader_election.cc:288] T f55cb3da9c874387a65eff5de0903331 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 election: Requested vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:24.171603 8853 leader_election.cc:302] T f55cb3da9c874387a65eff5de0903331 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9942306855404cfd891adf227dd65eb2, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:24.171787 14459 raft_consensus.cc:2781] T f55cb3da9c874387a65eff5de0903331 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:24.171994 14459 raft_consensus.cc:686] T f55cb3da9c874387a65eff5de0903331 P 9942306855404cfd891adf227dd65eb2 [term 1 LEADER]: Becoming Leader. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Running, Role: LEADER
I20240503 02:00:24.172160 14459 consensus_queue.cc:237] T f55cb3da9c874387a65eff5de0903331 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:24.175002 6314 catalog_manager.cc:5484] T f55cb3da9c874387a65eff5de0903331 P 9942306855404cfd891adf227dd65eb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9942306855404cfd891adf227dd65eb2 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "9942306855404cfd891adf227dd65eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: HEALTHY } } }
W20240503 02:00:24.193240 8982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240503 02:00:24.193980 8982 consensus_peers.cc:485] T 13756a6e44814fe4b09a6b0f087546c3 P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:00:24.202240 6314 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
W20240503 02:00:24.208055 8851 consensus_peers.cc:485] T 5f51fe5f799f4e479b62f5917a2c2606 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:00:24.208961 14521 tablet.cc:1722] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:24.209756 14523 tablet.cc:1722] T 0401cac53a9c44ec83de9643aac3e5dc P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:24.213809 14522 tablet.cc:1722] T 0401cac53a9c44ec83de9643aac3e5dc P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:24.217094 6314 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=9a306ff2611d406e912ebbeffa5b62a4] alter complete (version 3)
I20240503 02:00:24.318459 6308 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:24.318615 6308 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:24.320868 6308 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet 0401cac53a9c44ec83de9643aac3e5dc
I20240503 02:00:24.321520 6436 tablet_service.cc:1514] Processing DeleteTablet for tablet 0401cac53a9c44ec83de9643aac3e5dc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:24 UTC) from {username='slave'} at 127.0.0.1:51338
I20240503 02:00:24.321599 6568 tablet_service.cc:1514] Processing DeleteTablet for tablet 0401cac53a9c44ec83de9643aac3e5dc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:24 UTC) from {username='slave'} at 127.0.0.1:59846
I20240503 02:00:24.321875 14544 tablet_replica.cc:330] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a: stopping tablet replica
I20240503 02:00:24.321897 8880 tablet_service.cc:1514] Processing DeleteTablet for tablet 0401cac53a9c44ec83de9643aac3e5dc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:24 UTC) from {username='slave'} at 127.0.0.1:40296
I20240503 02:00:24.321955 14545 tablet_replica.cc:330] T 0401cac53a9c44ec83de9643aac3e5dc P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 02:00:24.322124 14545 raft_consensus.cc:2229] T 0401cac53a9c44ec83de9643aac3e5dc P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:24.322181 14544 raft_consensus.cc:2229] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:24.322260 14546 tablet_replica.cc:330] T 0401cac53a9c44ec83de9643aac3e5dc P 9942306855404cfd891adf227dd65eb2: stopping tablet replica
I20240503 02:00:24.322299 14545 raft_consensus.cc:2258] T 0401cac53a9c44ec83de9643aac3e5dc P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:24.322415 14546 raft_consensus.cc:2229] T 0401cac53a9c44ec83de9643aac3e5dc P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:24.322486 14544 raft_consensus.cc:2258] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:24.322551 14546 raft_consensus.cc:2258] T 0401cac53a9c44ec83de9643aac3e5dc P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:24.322978 14545 ts_tablet_manager.cc:1875] T 0401cac53a9c44ec83de9643aac3e5dc P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:24.323065 14544 ts_tablet_manager.cc:1875] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:24.323340 14546 ts_tablet_manager.cc:1875] T 0401cac53a9c44ec83de9643aac3e5dc P 9942306855404cfd891adf227dd65eb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:24.325343 14544 ts_tablet_manager.cc:1888] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:24.325429 14544 log.cc:1196] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/0401cac53a9c44ec83de9643aac3e5dc
I20240503 02:00:24.325492 14545 ts_tablet_manager.cc:1888] T 0401cac53a9c44ec83de9643aac3e5dc P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:24.325587 14545 log.cc:1196] T 0401cac53a9c44ec83de9643aac3e5dc P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/0401cac53a9c44ec83de9643aac3e5dc
I20240503 02:00:24.325755 14544 ts_tablet_manager.cc:1909] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a: Deleting consensus metadata
I20240503 02:00:24.325757 14546 ts_tablet_manager.cc:1888] T 0401cac53a9c44ec83de9643aac3e5dc P 9942306855404cfd891adf227dd65eb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:24.325839 14546 log.cc:1196] T 0401cac53a9c44ec83de9643aac3e5dc P 9942306855404cfd891adf227dd65eb2: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/0401cac53a9c44ec83de9643aac3e5dc
I20240503 02:00:24.325961 14545 ts_tablet_manager.cc:1909] T 0401cac53a9c44ec83de9643aac3e5dc P 80fd18ef4d604c86b909d5c008a46b4f: Deleting consensus metadata
I20240503 02:00:24.326115 14546 ts_tablet_manager.cc:1909] T 0401cac53a9c44ec83de9643aac3e5dc P 9942306855404cfd891adf227dd65eb2: Deleting consensus metadata
I20240503 02:00:24.326673 6300 catalog_manager.cc:4830] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): tablet 0401cac53a9c44ec83de9643aac3e5dc (table rebalancer_dynamic_table_new_name [id=9a306ff2611d406e912ebbeffa5b62a4]) successfully deleted
I20240503 02:00:24.327013 6303 catalog_manager.cc:4830] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211): tablet 0401cac53a9c44ec83de9643aac3e5dc (table rebalancer_dynamic_table_new_name [id=9a306ff2611d406e912ebbeffa5b62a4]) successfully deleted
I20240503 02:00:24.327819 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet 0401cac53a9c44ec83de9643aac3e5dc (table rebalancer_dynamic_table_new_name [id=9a306ff2611d406e912ebbeffa5b62a4]) successfully deleted
W20240503 02:00:24.344172 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
W20240503 02:00:24.386687 8851 consensus_peers.cc:485] T 7da8f614a70e4177b70e7c2f31d22a10 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:24.400113 8982 consensus_peers.cc:485] T beeea71f02ae4c89a1af13d3584ef70a P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:24.401465 8982 consensus_peers.cc:485] T 1a7ee79312ea4bad816264375ca5bb2d P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:24.422132 9113 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240503 02:00:24.423074 6308 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:00:24.428803 9157 tablet_service.cc:1467] Processing CreateTablet for tablet ecaa46386dee476492bae905d05b3dad (DEFAULT_TABLE table=rebalancer_dynamic_table [id=eb2282ddd2e94eadb12f14d01b283020]), partition=
I20240503 02:00:24.428828 9028 tablet_service.cc:1467] Processing CreateTablet for tablet ecaa46386dee476492bae905d05b3dad (DEFAULT_TABLE table=rebalancer_dynamic_table [id=eb2282ddd2e94eadb12f14d01b283020]), partition=
I20240503 02:00:24.429109 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecaa46386dee476492bae905d05b3dad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.429131 9157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecaa46386dee476492bae905d05b3dad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.430341 14472 raft_consensus.cc:482] T 4fb6646299c54e4da1938ea864806be4 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:24.430464 14472 raft_consensus.cc:504] T 4fb6646299c54e4da1938ea864806be4 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:24.431411 14472 leader_election.cc:288] T 4fb6646299c54e4da1938ea864806be4 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:24.431550 6829 tablet_service.cc:1467] Processing CreateTablet for tablet ecaa46386dee476492bae905d05b3dad (DEFAULT_TABLE table=rebalancer_dynamic_table [id=eb2282ddd2e94eadb12f14d01b283020]), partition=
I20240503 02:00:24.431777 6829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecaa46386dee476492bae905d05b3dad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.431951 6719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fb6646299c54e4da1938ea864806be4" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 02:00:24.432123 6719 raft_consensus.cc:2454] T 4fb6646299c54e4da1938ea864806be4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 02:00:24.432137 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fb6646299c54e4da1938ea864806be4" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:24.432245 9179 raft_consensus.cc:2454] T 4fb6646299c54e4da1938ea864806be4 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 02:00:24.432533 6784 leader_election.cc:302] T 4fb6646299c54e4da1938ea864806be4 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 492448570cc94ed1879feb2ec3b9051d; no voters:
I20240503 02:00:24.432763 14472 raft_consensus.cc:2781] T 4fb6646299c54e4da1938ea864806be4 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:24.432809 14472 raft_consensus.cc:482] T 4fb6646299c54e4da1938ea864806be4 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:24.432844 14472 raft_consensus.cc:3037] T 4fb6646299c54e4da1938ea864806be4 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:24.433926 14472 raft_consensus.cc:504] T 4fb6646299c54e4da1938ea864806be4 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:24.434245 14529 tablet_bootstrap.cc:492] T ecaa46386dee476492bae905d05b3dad P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:24.434315 14534 tablet_bootstrap.cc:492] T ecaa46386dee476492bae905d05b3dad P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:24.434809 14472 leader_election.cc:288] T 4fb6646299c54e4da1938ea864806be4 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Requested vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:24.435019 6719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fb6646299c54e4da1938ea864806be4" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 02:00:24.435148 6719 raft_consensus.cc:3037] T 4fb6646299c54e4da1938ea864806be4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:24.435281 14534 tablet_bootstrap.cc:654] T ecaa46386dee476492bae905d05b3dad P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.435387 14529 tablet_bootstrap.cc:654] T ecaa46386dee476492bae905d05b3dad P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.435881 14537 tablet_bootstrap.cc:492] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:24.436345 6719 raft_consensus.cc:2454] T 4fb6646299c54e4da1938ea864806be4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 02:00:24.436734 6787 leader_election.cc:302] T 4fb6646299c54e4da1938ea864806be4 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:24.436726 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fb6646299c54e4da1938ea864806be4" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:24.436854 9179 raft_consensus.cc:3037] T 4fb6646299c54e4da1938ea864806be4 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:24.436904 14472 raft_consensus.cc:2781] T 4fb6646299c54e4da1938ea864806be4 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:24.436951 14537 tablet_bootstrap.cc:654] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.437302 14472 raft_consensus.cc:686] T 4fb6646299c54e4da1938ea864806be4 P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Becoming Leader. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Running, Role: LEADER
I20240503 02:00:24.437485 14472 consensus_queue.cc:237] T 4fb6646299c54e4da1938ea864806be4 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:24.437902 9179 raft_consensus.cc:2454] T 4fb6646299c54e4da1938ea864806be4 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 02:00:24.439076 14529 tablet_bootstrap.cc:492] T ecaa46386dee476492bae905d05b3dad P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:24.439194 14529 ts_tablet_manager.cc:1367] T ecaa46386dee476492bae905d05b3dad P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:00:24.439656 14534 tablet_bootstrap.cc:492] T ecaa46386dee476492bae905d05b3dad P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:24.439659 14529 raft_consensus.cc:348] T ecaa46386dee476492bae905d05b3dad P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:24.439751 14534 ts_tablet_manager.cc:1367] T ecaa46386dee476492bae905d05b3dad P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:00:24.439772 14529 raft_consensus.cc:374] T ecaa46386dee476492bae905d05b3dad P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.439801 14529 raft_consensus.cc:729] T ecaa46386dee476492bae905d05b3dad P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.439913 14529 consensus_queue.cc:260] T ecaa46386dee476492bae905d05b3dad P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:24.440197 14529 ts_tablet_manager.cc:1398] T ecaa46386dee476492bae905d05b3dad P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:24.440342 14534 raft_consensus.cc:348] T ecaa46386dee476492bae905d05b3dad P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:24.440467 14534 raft_consensus.cc:374] T ecaa46386dee476492bae905d05b3dad P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.440501 14534 raft_consensus.cc:729] T ecaa46386dee476492bae905d05b3dad P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.440626 14534 consensus_queue.cc:260] T ecaa46386dee476492bae905d05b3dad P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:24.440908 14534 ts_tablet_manager.cc:1398] T ecaa46386dee476492bae905d05b3dad P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:24.441504 6308 catalog_manager.cc:5484] T 4fb6646299c54e4da1938ea864806be4 P 492448570cc94ed1879feb2ec3b9051d reported cstate change: term changed from 0 to 1, leader changed from <none> to 492448570cc94ed1879feb2ec3b9051d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:24.443783 14537 tablet_bootstrap.cc:492] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:24.443893 14537 ts_tablet_manager.cc:1367] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20240503 02:00:24.444412 14537 raft_consensus.cc:348] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:24.444540 14537 raft_consensus.cc:374] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.444571 14537 raft_consensus.cc:729] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.444705 14537 consensus_queue.cc:260] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:24.444998 14537 ts_tablet_manager.cc:1398] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:24.453763 14462 raft_consensus.cc:482] T cdb766efec08440aa6a296edb76f84b7 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:24.453886 14462 raft_consensus.cc:504] T cdb766efec08440aa6a296edb76f84b7 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:24.454306 14462 leader_election.cc:288] T cdb766efec08440aa6a296edb76f84b7 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:24.454830 8917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdb766efec08440aa6a296edb76f84b7" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:24.454978 6719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdb766efec08440aa6a296edb76f84b7" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 02:00:24.454990 8917 raft_consensus.cc:2454] T cdb766efec08440aa6a296edb76f84b7 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:24.455111 6719 raft_consensus.cc:2454] T cdb766efec08440aa6a296edb76f84b7 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:24.455375 6525 leader_election.cc:302] T cdb766efec08440aa6a296edb76f84b7 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:24.455543 14462 raft_consensus.cc:2781] T cdb766efec08440aa6a296edb76f84b7 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:24.455613 14462 raft_consensus.cc:482] T cdb766efec08440aa6a296edb76f84b7 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:24.455662 14462 raft_consensus.cc:3037] T cdb766efec08440aa6a296edb76f84b7 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:24.456552 14462 raft_consensus.cc:504] T cdb766efec08440aa6a296edb76f84b7 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:24.456871 14462 leader_election.cc:288] T cdb766efec08440aa6a296edb76f84b7 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Requested vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:24.457286 6719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdb766efec08440aa6a296edb76f84b7" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 02:00:24.457269 8917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdb766efec08440aa6a296edb76f84b7" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:24.457415 8917 raft_consensus.cc:3037] T cdb766efec08440aa6a296edb76f84b7 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:24.457415 6719 raft_consensus.cc:3037] T cdb766efec08440aa6a296edb76f84b7 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:24.458410 8917 raft_consensus.cc:2454] T cdb766efec08440aa6a296edb76f84b7 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:24.458410 6719 raft_consensus.cc:2454] T cdb766efec08440aa6a296edb76f84b7 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:24.458721 6525 leader_election.cc:302] T cdb766efec08440aa6a296edb76f84b7 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:24.458902 14462 raft_consensus.cc:2781] T cdb766efec08440aa6a296edb76f84b7 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:24.458992 14462 raft_consensus.cc:686] T cdb766efec08440aa6a296edb76f84b7 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Becoming Leader. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Running, Role: LEADER
I20240503 02:00:24.459136 14462 consensus_queue.cc:237] T cdb766efec08440aa6a296edb76f84b7 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:24.461820 6308 catalog_manager.cc:5484] T cdb766efec08440aa6a296edb76f84b7 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } }
W20240503 02:00:24.494791 6523 consensus_peers.cc:485] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:00:24.502236 14552 raft_consensus.cc:482] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:24.502377 14552 raft_consensus.cc:504] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:24.502823 14552 leader_election.cc:288] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:24.503144 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecaa46386dee476492bae905d05b3dad" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:24.503173 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecaa46386dee476492bae905d05b3dad" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 02:00:24.503283 9179 raft_consensus.cc:2454] T ecaa46386dee476492bae905d05b3dad P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:24.503340 6849 raft_consensus.cc:2454] T ecaa46386dee476492bae905d05b3dad P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:24.503573 8981 leader_election.cc:302] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:24.503757 14552 raft_consensus.cc:2781] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:24.503818 14552 raft_consensus.cc:482] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:24.503845 14552 raft_consensus.cc:3037] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:24.504853 14552 raft_consensus.cc:504] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:24.505225 14552 leader_election.cc:288] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Requested vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:24.505571 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecaa46386dee476492bae905d05b3dad" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:24.505605 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecaa46386dee476492bae905d05b3dad" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 02:00:24.505690 9179 raft_consensus.cc:3037] T ecaa46386dee476492bae905d05b3dad P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:24.505720 6849 raft_consensus.cc:3037] T ecaa46386dee476492bae905d05b3dad P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:24.506547 6849 raft_consensus.cc:2454] T ecaa46386dee476492bae905d05b3dad P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:24.506549 9179 raft_consensus.cc:2454] T ecaa46386dee476492bae905d05b3dad P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:24.506875 8984 leader_election.cc:302] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:24.507053 14552 raft_consensus.cc:2781] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:24.507140 14552 raft_consensus.cc:686] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Becoming Leader. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Running, Role: LEADER
I20240503 02:00:24.507284 14552 consensus_queue.cc:237] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:24.510113 6308 catalog_manager.cc:5484] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b reported cstate change: term changed from 0 to 1, leader changed from <none> to 99ead02915954b4fab23b033ad6d546b (127.0.99.134). New cstate: current_term: 1 leader_uuid: "99ead02915954b4fab23b033ad6d546b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:24.614895 6308 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:24.619764 6849 raft_consensus.cc:1261] T ecaa46386dee476492bae905d05b3dad P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer 99ead02915954b4fab23b033ad6d546b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:24.620337 14431 consensus_queue.cc:1035] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:24.620476 9179 raft_consensus.cc:1261] T ecaa46386dee476492bae905d05b3dad P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer 99ead02915954b4fab23b033ad6d546b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:24.620970 14552 consensus_queue.cc:1035] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:24.623487 14559 tablet.cc:1722] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:24.623515 14554 mvcc.cc:205] Tried to move back new op lower bound from 7023417854437900288 to 7023417853983911936. Current Snapshot: MvccSnapshot[applied={T|T < 7023417854437900288}]
I20240503 02:00:24.624240 14559 tablet.cc:1620] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b: MemRowSet was empty: no flush needed.
I20240503 02:00:24.624585 6308 catalog_manager.cc:5576] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079) does not have the latest schema for tablet ecaa46386dee476492bae905d05b3dad (table rebalancer_dynamic_table [id=eb2282ddd2e94eadb12f14d01b283020]). Expected version 1 got 0
I20240503 02:00:24.625159 14560 tablet.cc:1722] T ecaa46386dee476492bae905d05b3dad P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:24.625535 14560 tablet.cc:1620] T ecaa46386dee476492bae905d05b3dad P 492448570cc94ed1879feb2ec3b9051d: MemRowSet was empty: no flush needed.
I20240503 02:00:24.626775 6309 catalog_manager.cc:5576] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719) does not have the latest schema for tablet ecaa46386dee476492bae905d05b3dad (table rebalancer_dynamic_table [id=eb2282ddd2e94eadb12f14d01b283020]). Expected version 1 got 0
I20240503 02:00:24.630683 6313 catalog_manager.cc:5576] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079) does not have the latest schema for tablet ecaa46386dee476492bae905d05b3dad (table rebalancer_dynamic_table [id=eb2282ddd2e94eadb12f14d01b283020]). Expected version 1 got 0
I20240503 02:00:24.631242 14561 tablet.cc:1722] T ecaa46386dee476492bae905d05b3dad P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:24.631600 14561 tablet.cc:1620] T ecaa46386dee476492bae905d05b3dad P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
I20240503 02:00:24.631685 6308 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=eb2282ddd2e94eadb12f14d01b283020] alter complete (version 1)
I20240503 02:00:24.724740 14380 consensus_queue.cc:1035] T f55cb3da9c874387a65eff5de0903331 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:24.729820 14459 consensus_queue.cc:1035] T f55cb3da9c874387a65eff5de0903331 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:24.735761 6313 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:24.741564 14559 tablet.cc:1722] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:24.741919 14559 tablet.cc:1620] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b: MemRowSet was empty: no flush needed.
I20240503 02:00:24.742115 14560 tablet.cc:1722] T ecaa46386dee476492bae905d05b3dad P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:24.742406 14560 tablet.cc:1620] T ecaa46386dee476492bae905d05b3dad P 492448570cc94ed1879feb2ec3b9051d: MemRowSet was empty: no flush needed.
I20240503 02:00:24.742700 14561 tablet.cc:1722] T ecaa46386dee476492bae905d05b3dad P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:24.742983 14561 tablet.cc:1620] T ecaa46386dee476492bae905d05b3dad P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
I20240503 02:00:24.750229 6309 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=eb2282ddd2e94eadb12f14d01b283020] alter complete (version 2)
I20240503 02:00:24.851697 6313 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:24.857203 14559 tablet.cc:1722] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:24.857982 14560 tablet.cc:1722] T ecaa46386dee476492bae905d05b3dad P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:24.859521 14561 tablet.cc:1722] T ecaa46386dee476492bae905d05b3dad P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:24.864426 6313 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=eb2282ddd2e94eadb12f14d01b283020] alter complete (version 3)
I20240503 02:00:24.897468 14562 consensus_queue.cc:1035] T cdb766efec08440aa6a296edb76f84b7 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:24.901619 14462 consensus_queue.cc:1035] T cdb766efec08440aa6a296edb76f84b7 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:24.914868 14324 consensus_queue.cc:1035] T 4fb6646299c54e4da1938ea864806be4 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:24.919044 14571 consensus_queue.cc:1035] T 4fb6646299c54e4da1938ea864806be4 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:00:24.965845 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:24.966524 6309 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:24.966676 6309 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:24.968804 6309 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet ecaa46386dee476492bae905d05b3dad
I20240503 02:00:24.969519 9028 tablet_service.cc:1514] Processing DeleteTablet for tablet ecaa46386dee476492bae905d05b3dad with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:24 UTC) from {username='slave'} at 127.0.0.1:52584
I20240503 02:00:24.969834 14573 tablet_replica.cc:330] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b: stopping tablet replica
I20240503 02:00:24.969830 9157 tablet_service.cc:1514] Processing DeleteTablet for tablet ecaa46386dee476492bae905d05b3dad with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:24 UTC) from {username='slave'} at 127.0.0.1:51940
I20240503 02:00:24.969987 14573 raft_consensus.cc:2229] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:24.970094 6829 tablet_service.cc:1514] Processing DeleteTablet for tablet ecaa46386dee476492bae905d05b3dad with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:24 UTC) from {username='slave'} at 127.0.0.1:52812
I20240503 02:00:24.970299 14573 raft_consensus.cc:2258] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:24.970415 14575 tablet_replica.cc:330] T ecaa46386dee476492bae905d05b3dad P 492448570cc94ed1879feb2ec3b9051d: stopping tablet replica
I20240503 02:00:24.970588 14575 raft_consensus.cc:2229] T ecaa46386dee476492bae905d05b3dad P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:24.970793 14575 raft_consensus.cc:2258] T ecaa46386dee476492bae905d05b3dad P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:24.971014 14574 tablet_replica.cc:330] T ecaa46386dee476492bae905d05b3dad P 2a01ace80baf4057b6bf1995cf88468b: stopping tablet replica
I20240503 02:00:24.971185 14574 raft_consensus.cc:2229] T ecaa46386dee476492bae905d05b3dad P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:24.971326 14574 raft_consensus.cc:2258] T ecaa46386dee476492bae905d05b3dad P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:24.971585 14575 ts_tablet_manager.cc:1875] T ecaa46386dee476492bae905d05b3dad P 492448570cc94ed1879feb2ec3b9051d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:24.971704 14573 ts_tablet_manager.cc:1875] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:24.971988 14574 ts_tablet_manager.cc:1875] T ecaa46386dee476492bae905d05b3dad P 2a01ace80baf4057b6bf1995cf88468b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:24.974551 14574 ts_tablet_manager.cc:1888] T ecaa46386dee476492bae905d05b3dad P 2a01ace80baf4057b6bf1995cf88468b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:24.974674 14574 log.cc:1196] T ecaa46386dee476492bae905d05b3dad P 2a01ace80baf4057b6bf1995cf88468b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/ecaa46386dee476492bae905d05b3dad
I20240503 02:00:24.974613 14575 ts_tablet_manager.cc:1888] T ecaa46386dee476492bae905d05b3dad P 492448570cc94ed1879feb2ec3b9051d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:24.974861 14575 log.cc:1196] T ecaa46386dee476492bae905d05b3dad P 492448570cc94ed1879feb2ec3b9051d: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/ecaa46386dee476492bae905d05b3dad
I20240503 02:00:24.974951 14574 ts_tablet_manager.cc:1909] T ecaa46386dee476492bae905d05b3dad P 2a01ace80baf4057b6bf1995cf88468b: Deleting consensus metadata
I20240503 02:00:24.975283 14575 ts_tablet_manager.cc:1909] T ecaa46386dee476492bae905d05b3dad P 492448570cc94ed1879feb2ec3b9051d: Deleting consensus metadata
I20240503 02:00:24.975690 6300 catalog_manager.cc:4830] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079): tablet ecaa46386dee476492bae905d05b3dad (table rebalancer_dynamic_table_new_name [id=eb2282ddd2e94eadb12f14d01b283020]) successfully deleted
I20240503 02:00:24.976487 6303 catalog_manager.cc:4830] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719): tablet ecaa46386dee476492bae905d05b3dad (table rebalancer_dynamic_table_new_name [id=eb2282ddd2e94eadb12f14d01b283020]) successfully deleted
I20240503 02:00:24.974382 14573 ts_tablet_manager.cc:1888] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:24.976727 14573 log.cc:1196] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/ecaa46386dee476492bae905d05b3dad
I20240503 02:00:24.977008 14573 ts_tablet_manager.cc:1909] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b: Deleting consensus metadata
I20240503 02:00:24.977707 6301 catalog_manager.cc:4830] TS 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157): tablet ecaa46386dee476492bae905d05b3dad (table rebalancer_dynamic_table_new_name [id=eb2282ddd2e94eadb12f14d01b283020]) successfully deleted
I20240503 02:00:24.990407 6309 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_extra_table_64"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:24.999305 6568 tablet_service.cc:1467] Processing CreateTablet for tablet 898eb10f12924bd196108b9bb4ba6b77 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=4389ec3b2b7e429ca5a81a91fc4f7c6d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:24.999812 6568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 898eb10f12924bd196108b9bb4ba6b77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.000572 6829 tablet_service.cc:1467] Processing CreateTablet for tablet 898eb10f12924bd196108b9bb4ba6b77 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=4389ec3b2b7e429ca5a81a91fc4f7c6d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:25.000850 6829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 898eb10f12924bd196108b9bb4ba6b77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.000875 6436 tablet_service.cc:1467] Processing CreateTablet for tablet 5946d453a0ea4f5491d98c41cc99efd9 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=4389ec3b2b7e429ca5a81a91fc4f7c6d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:25.001128 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5946d453a0ea4f5491d98c41cc99efd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.002745 8880 tablet_service.cc:1467] Processing CreateTablet for tablet 898eb10f12924bd196108b9bb4ba6b77 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=4389ec3b2b7e429ca5a81a91fc4f7c6d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:25.002851 14576 tablet_bootstrap.cc:492] T 898eb10f12924bd196108b9bb4ba6b77 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:25.003026 8880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 898eb10f12924bd196108b9bb4ba6b77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.003374 6438 tablet_service.cc:1467] Processing CreateTablet for tablet 197ec14bb8fa494da81637271913393f (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=4389ec3b2b7e429ca5a81a91fc4f7c6d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:25.003628 6438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 197ec14bb8fa494da81637271913393f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.004067 14576 tablet_bootstrap.cc:654] T 898eb10f12924bd196108b9bb4ba6b77 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.004101 14578 tablet_bootstrap.cc:492] T 898eb10f12924bd196108b9bb4ba6b77 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:25.005100 14578 tablet_bootstrap.cc:654] T 898eb10f12924bd196108b9bb4ba6b77 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.005535 14580 tablet_bootstrap.cc:492] T 898eb10f12924bd196108b9bb4ba6b77 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:25.006572 14580 tablet_bootstrap.cc:654] T 898eb10f12924bd196108b9bb4ba6b77 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.006743 6829 tablet_service.cc:1467] Processing CreateTablet for tablet 5946d453a0ea4f5491d98c41cc99efd9 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=4389ec3b2b7e429ca5a81a91fc4f7c6d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:25.006743 8887 tablet_service.cc:1467] Processing CreateTablet for tablet 197ec14bb8fa494da81637271913393f (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=4389ec3b2b7e429ca5a81a91fc4f7c6d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:25.006996 6829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5946d453a0ea4f5491d98c41cc99efd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.006996 8887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 197ec14bb8fa494da81637271913393f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.007529 6831 tablet_service.cc:1467] Processing CreateTablet for tablet 197ec14bb8fa494da81637271913393f (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=4389ec3b2b7e429ca5a81a91fc4f7c6d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:25.007781 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 197ec14bb8fa494da81637271913393f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.009820 14576 tablet_bootstrap.cc:492] T 898eb10f12924bd196108b9bb4ba6b77 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:25.009932 14576 ts_tablet_manager.cc:1367] T 898eb10f12924bd196108b9bb4ba6b77 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20240503 02:00:25.010398 14576 raft_consensus.cc:348] T 898eb10f12924bd196108b9bb4ba6b77 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:25.010557 14576 raft_consensus.cc:374] T 898eb10f12924bd196108b9bb4ba6b77 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.010596 14576 raft_consensus.cc:729] T 898eb10f12924bd196108b9bb4ba6b77 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.010758 14576 consensus_queue.cc:260] T 898eb10f12924bd196108b9bb4ba6b77 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:25.011054 14576 ts_tablet_manager.cc:1398] T 898eb10f12924bd196108b9bb4ba6b77 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:25.011608 14580 tablet_bootstrap.cc:492] T 898eb10f12924bd196108b9bb4ba6b77 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:25.011734 14580 ts_tablet_manager.cc:1367] T 898eb10f12924bd196108b9bb4ba6b77 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:00:25.012147 8880 tablet_service.cc:1467] Processing CreateTablet for tablet 5946d453a0ea4f5491d98c41cc99efd9 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=4389ec3b2b7e429ca5a81a91fc4f7c6d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:25.012212 14580 raft_consensus.cc:348] T 898eb10f12924bd196108b9bb4ba6b77 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:25.012348 14580 raft_consensus.cc:374] T 898eb10f12924bd196108b9bb4ba6b77 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.012403 14580 raft_consensus.cc:729] T 898eb10f12924bd196108b9bb4ba6b77 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.012447 8880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5946d453a0ea4f5491d98c41cc99efd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.012537 14580 consensus_queue.cc:260] T 898eb10f12924bd196108b9bb4ba6b77 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:25.012894 14580 ts_tablet_manager.cc:1398] T 898eb10f12924bd196108b9bb4ba6b77 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:25.013875 14580 tablet_bootstrap.cc:492] T 197ec14bb8fa494da81637271913393f P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:25.014907 14580 tablet_bootstrap.cc:654] T 197ec14bb8fa494da81637271913393f P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.015424 14577 tablet_bootstrap.cc:492] T 5946d453a0ea4f5491d98c41cc99efd9 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:25.016778 14577 tablet_bootstrap.cc:654] T 5946d453a0ea4f5491d98c41cc99efd9 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.018183 14577 tablet_bootstrap.cc:492] T 5946d453a0ea4f5491d98c41cc99efd9 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:25.018364 14577 ts_tablet_manager.cc:1367] T 5946d453a0ea4f5491d98c41cc99efd9 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:25.018746 14578 tablet_bootstrap.cc:492] T 898eb10f12924bd196108b9bb4ba6b77 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:25.018986 14577 raft_consensus.cc:348] T 5946d453a0ea4f5491d98c41cc99efd9 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:25.019181 14577 raft_consensus.cc:374] T 5946d453a0ea4f5491d98c41cc99efd9 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.019256 14577 raft_consensus.cc:729] T 5946d453a0ea4f5491d98c41cc99efd9 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.019454 14577 consensus_queue.cc:260] T 5946d453a0ea4f5491d98c41cc99efd9 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:25.019832 14577 ts_tablet_manager.cc:1398] T 5946d453a0ea4f5491d98c41cc99efd9 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:25.020681 14578 ts_tablet_manager.cc:1367] T 898eb10f12924bd196108b9bb4ba6b77 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240503 02:00:25.021314 14578 raft_consensus.cc:348] T 898eb10f12924bd196108b9bb4ba6b77 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:25.021517 14577 tablet_bootstrap.cc:492] T 197ec14bb8fa494da81637271913393f P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:25.021512 14578 raft_consensus.cc:374] T 898eb10f12924bd196108b9bb4ba6b77 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.021555 14578 raft_consensus.cc:729] T 898eb10f12924bd196108b9bb4ba6b77 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.021695 14578 consensus_queue.cc:260] T 898eb10f12924bd196108b9bb4ba6b77 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:25.022059 14578 ts_tablet_manager.cc:1398] T 898eb10f12924bd196108b9bb4ba6b77 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:25.023360 14578 tablet_bootstrap.cc:492] T 5946d453a0ea4f5491d98c41cc99efd9 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:25.023846 14577 tablet_bootstrap.cc:654] T 197ec14bb8fa494da81637271913393f P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.024363 14578 tablet_bootstrap.cc:654] T 5946d453a0ea4f5491d98c41cc99efd9 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.025115 14577 tablet_bootstrap.cc:492] T 197ec14bb8fa494da81637271913393f P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:25.025207 14577 ts_tablet_manager.cc:1367] T 197ec14bb8fa494da81637271913393f P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:25.025692 14577 raft_consensus.cc:348] T 197ec14bb8fa494da81637271913393f P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:25.025806 14577 raft_consensus.cc:374] T 197ec14bb8fa494da81637271913393f P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.025832 14577 raft_consensus.cc:729] T 197ec14bb8fa494da81637271913393f P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.025955 14577 consensus_queue.cc:260] T 197ec14bb8fa494da81637271913393f P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:25.026240 14577 ts_tablet_manager.cc:1398] T 197ec14bb8fa494da81637271913393f P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:25.029258 14578 tablet_bootstrap.cc:492] T 5946d453a0ea4f5491d98c41cc99efd9 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:25.029450 14578 ts_tablet_manager.cc:1367] T 5946d453a0ea4f5491d98c41cc99efd9 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240503 02:00:25.030120 14578 raft_consensus.cc:348] T 5946d453a0ea4f5491d98c41cc99efd9 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:25.030292 14578 raft_consensus.cc:374] T 5946d453a0ea4f5491d98c41cc99efd9 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.030344 14578 raft_consensus.cc:729] T 5946d453a0ea4f5491d98c41cc99efd9 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.030498 14578 consensus_queue.cc:260] T 5946d453a0ea4f5491d98c41cc99efd9 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:25.030954 14578 ts_tablet_manager.cc:1398] T 5946d453a0ea4f5491d98c41cc99efd9 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:25.031193 14578 tablet_bootstrap.cc:492] T 197ec14bb8fa494da81637271913393f P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:25.032496 14578 tablet_bootstrap.cc:654] T 197ec14bb8fa494da81637271913393f P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.033192 14580 tablet_bootstrap.cc:492] T 197ec14bb8fa494da81637271913393f P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:25.033389 14580 ts_tablet_manager.cc:1367] T 197ec14bb8fa494da81637271913393f P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.002s
I20240503 02:00:25.034148 14580 raft_consensus.cc:348] T 197ec14bb8fa494da81637271913393f P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:25.034312 14580 raft_consensus.cc:374] T 197ec14bb8fa494da81637271913393f P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.034343 14580 raft_consensus.cc:729] T 197ec14bb8fa494da81637271913393f P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.035782 14580 consensus_queue.cc:260] T 197ec14bb8fa494da81637271913393f P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:25.036641 14580 ts_tablet_manager.cc:1398] T 197ec14bb8fa494da81637271913393f P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 02:00:25.036999 14578 tablet_bootstrap.cc:492] T 197ec14bb8fa494da81637271913393f P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:25.037376 14578 ts_tablet_manager.cc:1367] T 197ec14bb8fa494da81637271913393f P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20240503 02:00:25.037398 14580 tablet_bootstrap.cc:492] T 5946d453a0ea4f5491d98c41cc99efd9 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:25.038064 14578 raft_consensus.cc:348] T 197ec14bb8fa494da81637271913393f P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:25.038233 14578 raft_consensus.cc:374] T 197ec14bb8fa494da81637271913393f P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.038301 14578 raft_consensus.cc:729] T 197ec14bb8fa494da81637271913393f P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.038522 14578 consensus_queue.cc:260] T 197ec14bb8fa494da81637271913393f P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:25.038632 14580 tablet_bootstrap.cc:654] T 5946d453a0ea4f5491d98c41cc99efd9 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.039921 14580 tablet_bootstrap.cc:492] T 5946d453a0ea4f5491d98c41cc99efd9 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:25.039927 14578 ts_tablet_manager.cc:1398] T 197ec14bb8fa494da81637271913393f P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:00:25.040026 14580 ts_tablet_manager.cc:1367] T 5946d453a0ea4f5491d98c41cc99efd9 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:00:25.040534 14580 raft_consensus.cc:348] T 5946d453a0ea4f5491d98c41cc99efd9 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:25.040643 14580 raft_consensus.cc:374] T 5946d453a0ea4f5491d98c41cc99efd9 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.040668 14580 raft_consensus.cc:729] T 5946d453a0ea4f5491d98c41cc99efd9 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.040772 14580 consensus_queue.cc:260] T 5946d453a0ea4f5491d98c41cc99efd9 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:25.041041 14580 ts_tablet_manager.cc:1398] T 5946d453a0ea4f5491d98c41cc99efd9 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:25.050038 14472 raft_consensus.cc:482] T 898eb10f12924bd196108b9bb4ba6b77 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:25.050159 14472 raft_consensus.cc:504] T 898eb10f12924bd196108b9bb4ba6b77 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:25.050511 14472 leader_election.cc:288] T 898eb10f12924bd196108b9bb4ba6b77 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:25.050904 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "898eb10f12924bd196108b9bb4ba6b77" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:25.051131 8916 raft_consensus.cc:2454] T 898eb10f12924bd196108b9bb4ba6b77 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 02:00:25.051112 6587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "898eb10f12924bd196108b9bb4ba6b77" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 02:00:25.051252 6587 raft_consensus.cc:2454] T 898eb10f12924bd196108b9bb4ba6b77 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 02:00:25.051482 6787 leader_election.cc:302] T 898eb10f12924bd196108b9bb4ba6b77 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:25.052443 14472 raft_consensus.cc:2781] T 898eb10f12924bd196108b9bb4ba6b77 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:25.052524 14472 raft_consensus.cc:482] T 898eb10f12924bd196108b9bb4ba6b77 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:25.052567 14472 raft_consensus.cc:3037] T 898eb10f12924bd196108b9bb4ba6b77 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:25.053711 14472 raft_consensus.cc:504] T 898eb10f12924bd196108b9bb4ba6b77 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:25.054121 14472 leader_election.cc:288] T 898eb10f12924bd196108b9bb4ba6b77 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Requested vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:25.054937 6587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "898eb10f12924bd196108b9bb4ba6b77" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 02:00:25.055074 6587 raft_consensus.cc:3037] T 898eb10f12924bd196108b9bb4ba6b77 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:25.056110 6587 raft_consensus.cc:2454] T 898eb10f12924bd196108b9bb4ba6b77 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 02:00:25.056800 6786 leader_election.cc:302] T 898eb10f12924bd196108b9bb4ba6b77 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 80fd18ef4d604c86b909d5c008a46b4f; no voters:
I20240503 02:00:25.057302 14571 raft_consensus.cc:2781] T 898eb10f12924bd196108b9bb4ba6b77 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:25.057397 14571 raft_consensus.cc:686] T 898eb10f12924bd196108b9bb4ba6b77 P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Becoming Leader. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Running, Role: LEADER
I20240503 02:00:25.057543 14571 consensus_queue.cc:237] T 898eb10f12924bd196108b9bb4ba6b77 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:25.058722 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "898eb10f12924bd196108b9bb4ba6b77" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:25.058880 8916 raft_consensus.cc:3037] T 898eb10f12924bd196108b9bb4ba6b77 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:25.059823 8916 raft_consensus.cc:2454] T 898eb10f12924bd196108b9bb4ba6b77 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 02:00:25.061959 6309 catalog_manager.cc:5484] T 898eb10f12924bd196108b9bb4ba6b77 P 492448570cc94ed1879feb2ec3b9051d reported cstate change: term changed from 0 to 1, leader changed from <none> to 492448570cc94ed1879feb2ec3b9051d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:25.071422 6309 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:00:25.078240 6700 tablet_service.cc:1467] Processing CreateTablet for tablet 65ca02bf14bf4cec8b5cd90c305e349e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e9e0570a970943cb845b71a46c761153]), partition=
I20240503 02:00:25.078368 9157 tablet_service.cc:1467] Processing CreateTablet for tablet 65ca02bf14bf4cec8b5cd90c305e349e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e9e0570a970943cb845b71a46c761153]), partition=
I20240503 02:00:25.078395 6438 tablet_service.cc:1467] Processing CreateTablet for tablet 65ca02bf14bf4cec8b5cd90c305e349e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e9e0570a970943cb845b71a46c761153]), partition=
I20240503 02:00:25.078553 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65ca02bf14bf4cec8b5cd90c305e349e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.078583 9157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65ca02bf14bf4cec8b5cd90c305e349e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.078645 6438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65ca02bf14bf4cec8b5cd90c305e349e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.080827 14577 tablet_bootstrap.cc:492] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:25.081269 14591 tablet_bootstrap.cc:492] T 65ca02bf14bf4cec8b5cd90c305e349e P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:25.082190 14577 tablet_bootstrap.cc:654] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.082271 14591 tablet_bootstrap.cc:654] T 65ca02bf14bf4cec8b5cd90c305e349e P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.083448 14577 tablet_bootstrap.cc:492] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:25.083586 14577 ts_tablet_manager.cc:1367] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240503 02:00:25.083634 14591 tablet_bootstrap.cc:492] T 65ca02bf14bf4cec8b5cd90c305e349e P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:25.083721 14591 ts_tablet_manager.cc:1367] T 65ca02bf14bf4cec8b5cd90c305e349e P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:25.084091 14577 raft_consensus.cc:348] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:25.084209 14577 raft_consensus.cc:374] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.084239 14577 raft_consensus.cc:729] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.084172 14591 raft_consensus.cc:348] T 65ca02bf14bf4cec8b5cd90c305e349e P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:25.084281 14591 raft_consensus.cc:374] T 65ca02bf14bf4cec8b5cd90c305e349e P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.084312 14591 raft_consensus.cc:729] T 65ca02bf14bf4cec8b5cd90c305e349e P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.084468 14591 consensus_queue.cc:260] T 65ca02bf14bf4cec8b5cd90c305e349e P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:25.084498 14577 consensus_queue.cc:260] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:25.084820 14591 ts_tablet_manager.cc:1398] T 65ca02bf14bf4cec8b5cd90c305e349e P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:25.084936 14577 ts_tablet_manager.cc:1398] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:25.086676 14594 tablet_bootstrap.cc:492] T 65ca02bf14bf4cec8b5cd90c305e349e P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:25.087801 14594 tablet_bootstrap.cc:654] T 65ca02bf14bf4cec8b5cd90c305e349e P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.090245 14594 tablet_bootstrap.cc:492] T 65ca02bf14bf4cec8b5cd90c305e349e P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:25.090435 14594 ts_tablet_manager.cc:1367] T 65ca02bf14bf4cec8b5cd90c305e349e P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:25.091329 14594 raft_consensus.cc:348] T 65ca02bf14bf4cec8b5cd90c305e349e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:25.091467 14594 raft_consensus.cc:374] T 65ca02bf14bf4cec8b5cd90c305e349e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.091500 14594 raft_consensus.cc:729] T 65ca02bf14bf4cec8b5cd90c305e349e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.091673 14594 consensus_queue.cc:260] T 65ca02bf14bf4cec8b5cd90c305e349e P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:25.092325 14594 ts_tablet_manager.cc:1398] T 65ca02bf14bf4cec8b5cd90c305e349e P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:25.103395 14459 raft_consensus.cc:482] T 5946d453a0ea4f5491d98c41cc99efd9 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:25.103549 14459 raft_consensus.cc:504] T 5946d453a0ea4f5491d98c41cc99efd9 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:25.104146 14459 leader_election.cc:288] T 5946d453a0ea4f5491d98c41cc99efd9 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:25.104660 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5946d453a0ea4f5491d98c41cc99efd9" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 02:00:25.104715 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5946d453a0ea4f5491d98c41cc99efd9" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 02:00:25.104866 6458 raft_consensus.cc:2454] T 5946d453a0ea4f5491d98c41cc99efd9 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 0.
I20240503 02:00:25.104882 6849 raft_consensus.cc:2454] T 5946d453a0ea4f5491d98c41cc99efd9 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 0.
I20240503 02:00:25.105317 8853 leader_election.cc:302] T 5946d453a0ea4f5491d98c41cc99efd9 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:25.105576 14459 raft_consensus.cc:2781] T 5946d453a0ea4f5491d98c41cc99efd9 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:25.105630 14459 raft_consensus.cc:482] T 5946d453a0ea4f5491d98c41cc99efd9 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:25.105665 14459 raft_consensus.cc:3037] T 5946d453a0ea4f5491d98c41cc99efd9 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:25.106946 14459 raft_consensus.cc:504] T 5946d453a0ea4f5491d98c41cc99efd9 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:25.107312 14459 leader_election.cc:288] T 5946d453a0ea4f5491d98c41cc99efd9 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 election: Requested vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:25.107796 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5946d453a0ea4f5491d98c41cc99efd9" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 02:00:25.107808 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5946d453a0ea4f5491d98c41cc99efd9" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 02:00:25.107942 6849 raft_consensus.cc:3037] T 5946d453a0ea4f5491d98c41cc99efd9 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:25.107942 6458 raft_consensus.cc:3037] T 5946d453a0ea4f5491d98c41cc99efd9 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:25.108963 6458 raft_consensus.cc:2454] T 5946d453a0ea4f5491d98c41cc99efd9 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 1.
I20240503 02:00:25.108961 6849 raft_consensus.cc:2454] T 5946d453a0ea4f5491d98c41cc99efd9 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 1.
I20240503 02:00:25.109354 8853 leader_election.cc:302] T 5946d453a0ea4f5491d98c41cc99efd9 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:25.109558 14459 raft_consensus.cc:2781] T 5946d453a0ea4f5491d98c41cc99efd9 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:25.109648 14459 raft_consensus.cc:686] T 5946d453a0ea4f5491d98c41cc99efd9 P 9942306855404cfd891adf227dd65eb2 [term 1 LEADER]: Becoming Leader. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Running, Role: LEADER
I20240503 02:00:25.109787 14459 consensus_queue.cc:237] T 5946d453a0ea4f5491d98c41cc99efd9 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:25.112542 6314 catalog_manager.cc:5484] T 5946d453a0ea4f5491d98c41cc99efd9 P 9942306855404cfd891adf227dd65eb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9942306855404cfd891adf227dd65eb2 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "9942306855404cfd891adf227dd65eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:25.168205 14472 raft_consensus.cc:482] T 197ec14bb8fa494da81637271913393f P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:25.168342 14472 raft_consensus.cc:504] T 197ec14bb8fa494da81637271913393f P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:25.168771 14472 leader_election.cc:288] T 197ec14bb8fa494da81637271913393f P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:25.169386 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "197ec14bb8fa494da81637271913393f" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:25.169539 8916 raft_consensus.cc:2454] T 197ec14bb8fa494da81637271913393f P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 02:00:25.169744 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "197ec14bb8fa494da81637271913393f" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 02:00:25.169826 6787 leader_election.cc:302] T 197ec14bb8fa494da81637271913393f P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:25.169885 6458 raft_consensus.cc:2454] T 197ec14bb8fa494da81637271913393f P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 02:00:25.170006 14472 raft_consensus.cc:2781] T 197ec14bb8fa494da81637271913393f P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:25.170061 14472 raft_consensus.cc:482] T 197ec14bb8fa494da81637271913393f P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:25.170097 14472 raft_consensus.cc:3037] T 197ec14bb8fa494da81637271913393f P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:25.170981 14472 raft_consensus.cc:504] T 197ec14bb8fa494da81637271913393f P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:25.171329 14472 leader_election.cc:288] T 197ec14bb8fa494da81637271913393f P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Requested vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:25.171684 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "197ec14bb8fa494da81637271913393f" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:25.171703 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "197ec14bb8fa494da81637271913393f" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 02:00:25.171818 8916 raft_consensus.cc:3037] T 197ec14bb8fa494da81637271913393f P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:25.171929 6458 raft_consensus.cc:3037] T 197ec14bb8fa494da81637271913393f P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:25.172941 8916 raft_consensus.cc:2454] T 197ec14bb8fa494da81637271913393f P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 02:00:25.172941 6458 raft_consensus.cc:2454] T 197ec14bb8fa494da81637271913393f P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 02:00:25.173344 6784 leader_election.cc:302] T 197ec14bb8fa494da81637271913393f P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:25.173557 14472 raft_consensus.cc:2781] T 197ec14bb8fa494da81637271913393f P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:25.173713 14472 raft_consensus.cc:686] T 197ec14bb8fa494da81637271913393f P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Becoming Leader. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Running, Role: LEADER
I20240503 02:00:25.173864 14472 consensus_queue.cc:237] T 197ec14bb8fa494da81637271913393f P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:25.176651 6314 catalog_manager.cc:5484] T 197ec14bb8fa494da81637271913393f P 492448570cc94ed1879feb2ec3b9051d reported cstate change: term changed from 0 to 1, leader changed from <none> to 492448570cc94ed1879feb2ec3b9051d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:25.334816 14585 raft_consensus.cc:482] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:25.334954 14585 raft_consensus.cc:504] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:25.335462 14585 leader_election.cc:288] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:25.335880 6720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65ca02bf14bf4cec8b5cd90c305e349e" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 02:00:25.335906 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65ca02bf14bf4cec8b5cd90c305e349e" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:25.336017 6720 raft_consensus.cc:2454] T 65ca02bf14bf4cec8b5cd90c305e349e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:25.336052 9179 raft_consensus.cc:2454] T 65ca02bf14bf4cec8b5cd90c305e349e P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:25.336329 6394 leader_election.cc:302] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2ac2eb7cc264047b9ab32fe3a4fc025, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:25.336537 14585 raft_consensus.cc:2781] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:25.336607 14585 raft_consensus.cc:482] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:25.336652 14585 raft_consensus.cc:3037] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:25.337620 14585 raft_consensus.cc:504] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:25.338107 14585 leader_election.cc:288] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Requested vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:25.338491 6720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65ca02bf14bf4cec8b5cd90c305e349e" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 02:00:25.338491 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65ca02bf14bf4cec8b5cd90c305e349e" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:25.338641 6720 raft_consensus.cc:3037] T 65ca02bf14bf4cec8b5cd90c305e349e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:25.338796 9179 raft_consensus.cc:3037] T 65ca02bf14bf4cec8b5cd90c305e349e P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:25.339716 6720 raft_consensus.cc:2454] T 65ca02bf14bf4cec8b5cd90c305e349e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:25.339716 9179 raft_consensus.cc:2454] T 65ca02bf14bf4cec8b5cd90c305e349e P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:25.340026 6391 leader_election.cc:302] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:25.340399 14585 raft_consensus.cc:2781] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:25.340492 14585 raft_consensus.cc:686] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Becoming Leader. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Running, Role: LEADER
I20240503 02:00:25.340629 14585 consensus_queue.cc:237] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:25.342850 6314 catalog_manager.cc:5484] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: term changed from 0 to 1, leader changed from <none> to dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129). New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } }
W20240503 02:00:25.352340 6785 consensus_peers.cc:485] T c3485701bf3b48f5ae2e1568afeed133 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:25.403998 8982 consensus_peers.cc:485] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:00:25.481150 14472 consensus_queue.cc:1035] T 898eb10f12924bd196108b9bb4ba6b77 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:25.482618 6314 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:25.488212 14571 consensus_queue.cc:1035] T 898eb10f12924bd196108b9bb4ba6b77 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:25.489696 9178 raft_consensus.cc:1261] T 65ca02bf14bf4cec8b5cd90c305e349e P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:25.490495 6720 raft_consensus.cc:1261] T 65ca02bf14bf4cec8b5cd90c305e349e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:25.491276 14153 consensus_queue.cc:1035] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:25.491592 14153 consensus_queue.cc:1035] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:25.497066 6314 catalog_manager.cc:5576] TS c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655) does not have the latest schema for tablet 65ca02bf14bf4cec8b5cd90c305e349e (table rebalancer_dynamic_table [id=e9e0570a970943cb845b71a46c761153]). Expected version 1 got 0
I20240503 02:00:25.497748 14610 tablet.cc:1722] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:25.498122 14610 tablet.cc:1620] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a: MemRowSet was empty: no flush needed.
I20240503 02:00:25.499749 6314 catalog_manager.cc:5576] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093) does not have the latest schema for tablet 65ca02bf14bf4cec8b5cd90c305e349e (table rebalancer_dynamic_table [id=e9e0570a970943cb845b71a46c761153]). Expected version 1 got 0
I20240503 02:00:25.501207 6313 catalog_manager.cc:5576] TS c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655) does not have the latest schema for tablet 65ca02bf14bf4cec8b5cd90c305e349e (table rebalancer_dynamic_table [id=e9e0570a970943cb845b71a46c761153]). Expected version 1 got 0
I20240503 02:00:25.504089 6313 catalog_manager.cc:5576] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079) does not have the latest schema for tablet 65ca02bf14bf4cec8b5cd90c305e349e (table rebalancer_dynamic_table [id=e9e0570a970943cb845b71a46c761153]). Expected version 1 got 0
I20240503 02:00:25.503880 14611 tablet.cc:1722] T 65ca02bf14bf4cec8b5cd90c305e349e P c2ac2eb7cc264047b9ab32fe3a4fc025: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:25.504458 14611 tablet.cc:1620] T 65ca02bf14bf4cec8b5cd90c305e349e P c2ac2eb7cc264047b9ab32fe3a4fc025: MemRowSet was empty: no flush needed.
I20240503 02:00:25.507692 6313 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=e9e0570a970943cb845b71a46c761153] alter complete (version 1)
I20240503 02:00:25.511083 14613 tablet.cc:1722] T 65ca02bf14bf4cec8b5cd90c305e349e P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:25.511744 14613 tablet.cc:1620] T 65ca02bf14bf4cec8b5cd90c305e349e P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
W20240503 02:00:25.557039 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:25.586073 14598 consensus_queue.cc:1035] T 5946d453a0ea4f5491d98c41cc99efd9 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:25.591269 14380 consensus_queue.cc:1035] T 5946d453a0ea4f5491d98c41cc99efd9 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:25.604135 14571 consensus_queue.cc:1035] T 197ec14bb8fa494da81637271913393f P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:25.608285 14571 consensus_queue.cc:1035] T 197ec14bb8fa494da81637271913393f P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:25.612740 6314 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
W20240503 02:00:25.612957 6301 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
W20240503 02:00:25.615257 6301 catalog_manager.cc:4573] TS e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3d3463f1ff2b41a396bb1b039ddf1946: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:25.619374 14611 tablet.cc:1722] T 65ca02bf14bf4cec8b5cd90c305e349e P c2ac2eb7cc264047b9ab32fe3a4fc025: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:25.619891 14611 tablet.cc:1620] T 65ca02bf14bf4cec8b5cd90c305e349e P c2ac2eb7cc264047b9ab32fe3a4fc025: MemRowSet was empty: no flush needed.
I20240503 02:00:25.619886 14610 tablet.cc:1722] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:25.620255 14613 tablet.cc:1722] T 65ca02bf14bf4cec8b5cd90c305e349e P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:25.620548 14610 tablet.cc:1620] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a: MemRowSet was empty: no flush needed.
I20240503 02:00:25.620584 14613 tablet.cc:1620] T 65ca02bf14bf4cec8b5cd90c305e349e P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
I20240503 02:00:25.626791 6313 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=e9e0570a970943cb845b71a46c761153] alter complete (version 2)
W20240503 02:00:25.665956 6785 consensus_peers.cc:485] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:25.666424 6654 consensus_peers.cc:485] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 02:00:25.707300 6309 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_extra_table_66"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:25.714625 6438 tablet_service.cc:1467] Processing CreateTablet for tablet 1dfbfcb60a3c4537a32f496e6772c916 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=45810e9ec99140aa91640e149c149851]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:25.714733 9028 tablet_service.cc:1467] Processing CreateTablet for tablet 1dfbfcb60a3c4537a32f496e6772c916 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=45810e9ec99140aa91640e149c149851]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:25.714733 6831 tablet_service.cc:1467] Processing CreateTablet for tablet 1dfbfcb60a3c4537a32f496e6772c916 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=45810e9ec99140aa91640e149c149851]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:25.714983 6438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dfbfcb60a3c4537a32f496e6772c916. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.715044 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dfbfcb60a3c4537a32f496e6772c916. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.715039 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dfbfcb60a3c4537a32f496e6772c916. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.717712 14623 tablet_bootstrap.cc:492] T 1dfbfcb60a3c4537a32f496e6772c916 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:25.717491 14621 tablet_bootstrap.cc:492] T 1dfbfcb60a3c4537a32f496e6772c916 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:25.718637 6568 tablet_service.cc:1467] Processing CreateTablet for tablet 9c00b9bef94d4890ae17b04b8ff3f517 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=45810e9ec99140aa91640e149c149851]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:25.718637 9157 tablet_service.cc:1467] Processing CreateTablet for tablet 9c00b9bef94d4890ae17b04b8ff3f517 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=45810e9ec99140aa91640e149c149851]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:25.719076 6568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c00b9bef94d4890ae17b04b8ff3f517. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.719076 9157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c00b9bef94d4890ae17b04b8ff3f517. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.719501 9159 tablet_service.cc:1467] Processing CreateTablet for tablet 41b2718b68ea4714b5d9a0deac130030 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=45810e9ec99140aa91640e149c149851]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:25.720063 14621 tablet_bootstrap.cc:654] T 1dfbfcb60a3c4537a32f496e6772c916 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.720860 14622 tablet_bootstrap.cc:492] T 1dfbfcb60a3c4537a32f496e6772c916 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:25.721030 9028 tablet_service.cc:1467] Processing CreateTablet for tablet 41b2718b68ea4714b5d9a0deac130030 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=45810e9ec99140aa91640e149c149851]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:25.721258 9159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41b2718b68ea4714b5d9a0deac130030. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.721288 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41b2718b68ea4714b5d9a0deac130030. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.722259 14623 tablet_bootstrap.cc:654] T 1dfbfcb60a3c4537a32f496e6772c916 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.723006 14624 tablet_bootstrap.cc:492] T 9c00b9bef94d4890ae17b04b8ff3f517 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:25.724332 14622 tablet_bootstrap.cc:654] T 1dfbfcb60a3c4537a32f496e6772c916 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.724318 6700 tablet_service.cc:1467] Processing CreateTablet for tablet 41b2718b68ea4714b5d9a0deac130030 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=45810e9ec99140aa91640e149c149851]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:25.724660 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41b2718b68ea4714b5d9a0deac130030. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.726016 14626 tablet_bootstrap.cc:492] T 41b2718b68ea4714b5d9a0deac130030 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:25.726555 6438 tablet_service.cc:1467] Processing CreateTablet for tablet 9c00b9bef94d4890ae17b04b8ff3f517 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=45810e9ec99140aa91640e149c149851]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:25.726822 6438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c00b9bef94d4890ae17b04b8ff3f517. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.726974 14621 tablet_bootstrap.cc:492] T 1dfbfcb60a3c4537a32f496e6772c916 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:25.727053 14621 ts_tablet_manager.cc:1367] T 1dfbfcb60a3c4537a32f496e6772c916 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.002s
I20240503 02:00:25.727607 14622 tablet_bootstrap.cc:492] T 1dfbfcb60a3c4537a32f496e6772c916 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:25.727710 14622 ts_tablet_manager.cc:1367] T 1dfbfcb60a3c4537a32f496e6772c916 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:00:25.728219 14622 raft_consensus.cc:348] T 1dfbfcb60a3c4537a32f496e6772c916 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:25.728334 14622 raft_consensus.cc:374] T 1dfbfcb60a3c4537a32f496e6772c916 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.728364 14622 raft_consensus.cc:729] T 1dfbfcb60a3c4537a32f496e6772c916 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.728890 14621 raft_consensus.cc:348] T 1dfbfcb60a3c4537a32f496e6772c916 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:25.729022 14621 raft_consensus.cc:374] T 1dfbfcb60a3c4537a32f496e6772c916 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.729060 14621 raft_consensus.cc:729] T 1dfbfcb60a3c4537a32f496e6772c916 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.729226 14621 consensus_queue.cc:260] T 1dfbfcb60a3c4537a32f496e6772c916 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:25.729651 14622 consensus_queue.cc:260] T 1dfbfcb60a3c4537a32f496e6772c916 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:25.730234 14624 tablet_bootstrap.cc:654] T 9c00b9bef94d4890ae17b04b8ff3f517 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.730945 6313 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:25.731729 14622 ts_tablet_manager.cc:1398] T 1dfbfcb60a3c4537a32f496e6772c916 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 02:00:25.731938 14622 tablet_bootstrap.cc:492] T 41b2718b68ea4714b5d9a0deac130030 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:25.733117 14622 tablet_bootstrap.cc:654] T 41b2718b68ea4714b5d9a0deac130030 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.733350 14621 ts_tablet_manager.cc:1398] T 1dfbfcb60a3c4537a32f496e6772c916 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240503 02:00:25.733619 14621 tablet_bootstrap.cc:492] T 9c00b9bef94d4890ae17b04b8ff3f517 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:25.735510 14624 tablet_bootstrap.cc:492] T 9c00b9bef94d4890ae17b04b8ff3f517 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:25.735639 14624 ts_tablet_manager.cc:1367] T 9c00b9bef94d4890ae17b04b8ff3f517 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 02:00:25.736166 14624 raft_consensus.cc:348] T 9c00b9bef94d4890ae17b04b8ff3f517 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:25.736321 14624 raft_consensus.cc:374] T 9c00b9bef94d4890ae17b04b8ff3f517 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.736357 14624 raft_consensus.cc:729] T 9c00b9bef94d4890ae17b04b8ff3f517 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.736508 14624 consensus_queue.cc:260] T 9c00b9bef94d4890ae17b04b8ff3f517 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:25.736828 14624 ts_tablet_manager.cc:1398] T 9c00b9bef94d4890ae17b04b8ff3f517 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:25.739120 14629 tablet_bootstrap.cc:492] T 41b2718b68ea4714b5d9a0deac130030 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:25.739740 14626 tablet_bootstrap.cc:654] T 41b2718b68ea4714b5d9a0deac130030 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.740859 14629 tablet_bootstrap.cc:654] T 41b2718b68ea4714b5d9a0deac130030 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.741042 14621 tablet_bootstrap.cc:654] T 9c00b9bef94d4890ae17b04b8ff3f517 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.742266 14623 tablet_bootstrap.cc:492] T 1dfbfcb60a3c4537a32f496e6772c916 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:25.742374 14623 ts_tablet_manager.cc:1367] T 1dfbfcb60a3c4537a32f496e6772c916 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.000s
I20240503 02:00:25.742884 14623 raft_consensus.cc:348] T 1dfbfcb60a3c4537a32f496e6772c916 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:25.743019 14623 raft_consensus.cc:374] T 1dfbfcb60a3c4537a32f496e6772c916 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.743052 14623 raft_consensus.cc:729] T 1dfbfcb60a3c4537a32f496e6772c916 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.743216 14629 tablet_bootstrap.cc:492] T 41b2718b68ea4714b5d9a0deac130030 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:25.743304 14629 ts_tablet_manager.cc:1367] T 41b2718b68ea4714b5d9a0deac130030 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:25.743515 14623 consensus_queue.cc:260] T 1dfbfcb60a3c4537a32f496e6772c916 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:25.743839 14623 ts_tablet_manager.cc:1398] T 1dfbfcb60a3c4537a32f496e6772c916 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:25.743814 14610 tablet.cc:1722] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:25.744155 14629 raft_consensus.cc:348] T 41b2718b68ea4714b5d9a0deac130030 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:25.744266 14629 raft_consensus.cc:374] T 41b2718b68ea4714b5d9a0deac130030 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.744295 14629 raft_consensus.cc:729] T 41b2718b68ea4714b5d9a0deac130030 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.744601 14629 consensus_queue.cc:260] T 41b2718b68ea4714b5d9a0deac130030 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:25.744858 14611 tablet.cc:1722] T 65ca02bf14bf4cec8b5cd90c305e349e P c2ac2eb7cc264047b9ab32fe3a4fc025: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:25.745522 14629 ts_tablet_manager.cc:1398] T 41b2718b68ea4714b5d9a0deac130030 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:25.745893 14621 tablet_bootstrap.cc:492] T 9c00b9bef94d4890ae17b04b8ff3f517 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:25.745980 14621 ts_tablet_manager.cc:1367] T 9c00b9bef94d4890ae17b04b8ff3f517 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.002s
I20240503 02:00:25.746492 14621 raft_consensus.cc:348] T 9c00b9bef94d4890ae17b04b8ff3f517 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:25.746603 14621 raft_consensus.cc:374] T 9c00b9bef94d4890ae17b04b8ff3f517 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.746631 14621 raft_consensus.cc:729] T 9c00b9bef94d4890ae17b04b8ff3f517 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.746742 14621 consensus_queue.cc:260] T 9c00b9bef94d4890ae17b04b8ff3f517 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:25.747057 14621 ts_tablet_manager.cc:1398] T 9c00b9bef94d4890ae17b04b8ff3f517 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:25.752158 14626 tablet_bootstrap.cc:492] T 41b2718b68ea4714b5d9a0deac130030 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:25.752288 14626 ts_tablet_manager.cc:1367] T 41b2718b68ea4714b5d9a0deac130030 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.000s
I20240503 02:00:25.752858 14626 raft_consensus.cc:348] T 41b2718b68ea4714b5d9a0deac130030 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:25.753001 14626 raft_consensus.cc:374] T 41b2718b68ea4714b5d9a0deac130030 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.753037 14626 raft_consensus.cc:729] T 41b2718b68ea4714b5d9a0deac130030 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.753170 14626 consensus_queue.cc:260] T 41b2718b68ea4714b5d9a0deac130030 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:25.753492 14626 ts_tablet_manager.cc:1398] T 41b2718b68ea4714b5d9a0deac130030 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:25.753659 14626 tablet_bootstrap.cc:492] T 9c00b9bef94d4890ae17b04b8ff3f517 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:25.755906 6313 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=e9e0570a970943cb845b71a46c761153] alter complete (version 3)
I20240503 02:00:25.758108 14622 tablet_bootstrap.cc:492] T 41b2718b68ea4714b5d9a0deac130030 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:25.758168 14462 raft_consensus.cc:482] T 9c00b9bef94d4890ae17b04b8ff3f517 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:25.758231 14622 ts_tablet_manager.cc:1367] T 41b2718b68ea4714b5d9a0deac130030 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.000s
I20240503 02:00:25.758256 14462 raft_consensus.cc:504] T 9c00b9bef94d4890ae17b04b8ff3f517 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:25.758678 14462 leader_election.cc:288] T 9c00b9bef94d4890ae17b04b8ff3f517 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:25.758869 14622 raft_consensus.cc:348] T 41b2718b68ea4714b5d9a0deac130030 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:25.758985 14626 tablet_bootstrap.cc:654] T 9c00b9bef94d4890ae17b04b8ff3f517 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.758991 14622 raft_consensus.cc:374] T 41b2718b68ea4714b5d9a0deac130030 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.759044 14622 raft_consensus.cc:729] T 41b2718b68ea4714b5d9a0deac130030 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.759171 14622 consensus_queue.cc:260] T 41b2718b68ea4714b5d9a0deac130030 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:25.759238 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c00b9bef94d4890ae17b04b8ff3f517" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 02:00:25.759382 6456 raft_consensus.cc:2454] T 9c00b9bef94d4890ae17b04b8ff3f517 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:25.759469 14622 ts_tablet_manager.cc:1398] T 41b2718b68ea4714b5d9a0deac130030 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:25.760177 14626 tablet_bootstrap.cc:492] T 9c00b9bef94d4890ae17b04b8ff3f517 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:25.760283 14626 ts_tablet_manager.cc:1367] T 9c00b9bef94d4890ae17b04b8ff3f517 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240503 02:00:25.760586 6522 leader_election.cc:302] T 9c00b9bef94d4890ae17b04b8ff3f517 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:25.760598 9177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c00b9bef94d4890ae17b04b8ff3f517" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:25.760675 14613 tablet.cc:1722] T 65ca02bf14bf4cec8b5cd90c305e349e P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:25.760787 14626 raft_consensus.cc:348] T 9c00b9bef94d4890ae17b04b8ff3f517 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:25.760876 14626 raft_consensus.cc:374] T 9c00b9bef94d4890ae17b04b8ff3f517 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.760905 14626 raft_consensus.cc:729] T 9c00b9bef94d4890ae17b04b8ff3f517 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.760902 14462 raft_consensus.cc:2781] T 9c00b9bef94d4890ae17b04b8ff3f517 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:25.760954 14462 raft_consensus.cc:482] T 9c00b9bef94d4890ae17b04b8ff3f517 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:25.760988 14462 raft_consensus.cc:3037] T 9c00b9bef94d4890ae17b04b8ff3f517 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:25.761037 14626 consensus_queue.cc:260] T 9c00b9bef94d4890ae17b04b8ff3f517 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:25.761318 14626 ts_tablet_manager.cc:1398] T 9c00b9bef94d4890ae17b04b8ff3f517 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:25.762126 14462 raft_consensus.cc:504] T 9c00b9bef94d4890ae17b04b8ff3f517 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:25.762487 14462 leader_election.cc:288] T 9c00b9bef94d4890ae17b04b8ff3f517 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Requested vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:25.763386 14552 raft_consensus.cc:482] T 1dfbfcb60a3c4537a32f496e6772c916 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:25.763486 14552 raft_consensus.cc:504] T 1dfbfcb60a3c4537a32f496e6772c916 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
W20240503 02:00:25.763823 6522 leader_election.cc:341] T 9c00b9bef94d4890ae17b04b8ff3f517 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:00:25.763870 14552 leader_election.cc:288] T 1dfbfcb60a3c4537a32f496e6772c916 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 02:00:25.764250 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c00b9bef94d4890ae17b04b8ff3f517" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 02:00:25.764407 6456 raft_consensus.cc:3037] T 9c00b9bef94d4890ae17b04b8ff3f517 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:25.764412 9177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c00b9bef94d4890ae17b04b8ff3f517" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:25.764537 9177 raft_consensus.cc:3037] T 9c00b9bef94d4890ae17b04b8ff3f517 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:25.764678 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dfbfcb60a3c4537a32f496e6772c916" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 02:00:25.764777 6458 raft_consensus.cc:2454] T 1dfbfcb60a3c4537a32f496e6772c916 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:25.765249 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dfbfcb60a3c4537a32f496e6772c916" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 02:00:25.765401 6849 raft_consensus.cc:2454] T 1dfbfcb60a3c4537a32f496e6772c916 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:25.765532 6456 raft_consensus.cc:2454] T 9c00b9bef94d4890ae17b04b8ff3f517 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:25.765605 9177 raft_consensus.cc:2454] T 9c00b9bef94d4890ae17b04b8ff3f517 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:25.766320 8981 leader_election.cc:302] T 1dfbfcb60a3c4537a32f496e6772c916 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99ead02915954b4fab23b033ad6d546b, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:25.767789 14552 raft_consensus.cc:2781] T 1dfbfcb60a3c4537a32f496e6772c916 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:25.767863 14552 raft_consensus.cc:482] T 1dfbfcb60a3c4537a32f496e6772c916 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:25.767899 14552 raft_consensus.cc:3037] T 1dfbfcb60a3c4537a32f496e6772c916 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:25.768566 6522 leader_election.cc:302] T 9c00b9bef94d4890ae17b04b8ff3f517 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 80fd18ef4d604c86b909d5c008a46b4f; no voters:
I20240503 02:00:25.768900 14462 raft_consensus.cc:2781] T 9c00b9bef94d4890ae17b04b8ff3f517 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:25.768980 14462 raft_consensus.cc:686] T 9c00b9bef94d4890ae17b04b8ff3f517 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Becoming Leader. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Running, Role: LEADER
I20240503 02:00:25.768960 14552 raft_consensus.cc:504] T 1dfbfcb60a3c4537a32f496e6772c916 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:25.769116 14462 consensus_queue.cc:237] T 9c00b9bef94d4890ae17b04b8ff3f517 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:25.769345 14552 leader_election.cc:288] T 1dfbfcb60a3c4537a32f496e6772c916 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Requested vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 02:00:25.772054 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dfbfcb60a3c4537a32f496e6772c916" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 02:00:25.772194 6849 raft_consensus.cc:3037] T 1dfbfcb60a3c4537a32f496e6772c916 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:25.773339 6849 raft_consensus.cc:2454] T 1dfbfcb60a3c4537a32f496e6772c916 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:25.774602 8984 leader_election.cc:302] T 1dfbfcb60a3c4537a32f496e6772c916 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:25.776607 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dfbfcb60a3c4537a32f496e6772c916" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 02:00:25.776746 6456 raft_consensus.cc:3037] T 1dfbfcb60a3c4537a32f496e6772c916 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:25.777216 6309 catalog_manager.cc:5484] T 9c00b9bef94d4890ae17b04b8ff3f517 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:25.777876 6456 raft_consensus.cc:2454] T 1dfbfcb60a3c4537a32f496e6772c916 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:25.782222 14195 raft_consensus.cc:2781] T 1dfbfcb60a3c4537a32f496e6772c916 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:25.782426 14195 raft_consensus.cc:686] T 1dfbfcb60a3c4537a32f496e6772c916 P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Becoming Leader. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Running, Role: LEADER
I20240503 02:00:25.782568 14195 consensus_queue.cc:237] T 1dfbfcb60a3c4537a32f496e6772c916 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:25.789299 6313 catalog_manager.cc:5484] T 1dfbfcb60a3c4537a32f496e6772c916 P 99ead02915954b4fab23b033ad6d546b reported cstate change: term changed from 0 to 1, leader changed from <none> to 99ead02915954b4fab23b033ad6d546b (127.0.99.134). New cstate: current_term: 1 leader_uuid: "99ead02915954b4fab23b033ad6d546b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:25.863520 6313 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:25.863705 6313 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:25.865805 6313 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet 65ca02bf14bf4cec8b5cd90c305e349e
I20240503 02:00:25.866464 6700 tablet_service.cc:1514] Processing DeleteTablet for tablet 65ca02bf14bf4cec8b5cd90c305e349e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:25 UTC) from {username='slave'} at 127.0.0.1:40336
I20240503 02:00:25.866519 9157 tablet_service.cc:1514] Processing DeleteTablet for tablet 65ca02bf14bf4cec8b5cd90c305e349e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:25 UTC) from {username='slave'} at 127.0.0.1:51940
I20240503 02:00:25.866524 6438 tablet_service.cc:1514] Processing DeleteTablet for tablet 65ca02bf14bf4cec8b5cd90c305e349e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:25 UTC) from {username='slave'} at 127.0.0.1:51338
I20240503 02:00:25.866865 14639 tablet_replica.cc:330] T 65ca02bf14bf4cec8b5cd90c305e349e P c2ac2eb7cc264047b9ab32fe3a4fc025: stopping tablet replica
I20240503 02:00:25.867038 14641 tablet_replica.cc:330] T 65ca02bf14bf4cec8b5cd90c305e349e P 2a01ace80baf4057b6bf1995cf88468b: stopping tablet replica
I20240503 02:00:25.867076 14639 raft_consensus.cc:2229] T 65ca02bf14bf4cec8b5cd90c305e349e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:25.867071 14640 tablet_replica.cc:330] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a: stopping tablet replica
I20240503 02:00:25.867182 14641 raft_consensus.cc:2229] T 65ca02bf14bf4cec8b5cd90c305e349e P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:25.867219 14639 raft_consensus.cc:2258] T 65ca02bf14bf4cec8b5cd90c305e349e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:25.867331 14641 raft_consensus.cc:2258] T 65ca02bf14bf4cec8b5cd90c305e349e P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:25.867550 14640 raft_consensus.cc:2229] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:25.867821 14639 ts_tablet_manager.cc:1875] T 65ca02bf14bf4cec8b5cd90c305e349e P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:25.867832 14641 ts_tablet_manager.cc:1875] T 65ca02bf14bf4cec8b5cd90c305e349e P 2a01ace80baf4057b6bf1995cf88468b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:25.867861 14640 raft_consensus.cc:2258] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:25.868423 14640 ts_tablet_manager.cc:1875] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:25.870321 14641 ts_tablet_manager.cc:1888] T 65ca02bf14bf4cec8b5cd90c305e349e P 2a01ace80baf4057b6bf1995cf88468b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:25.870321 14639 ts_tablet_manager.cc:1888] T 65ca02bf14bf4cec8b5cd90c305e349e P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:25.870405 14639 log.cc:1196] T 65ca02bf14bf4cec8b5cd90c305e349e P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/65ca02bf14bf4cec8b5cd90c305e349e
I20240503 02:00:25.870455 14641 log.cc:1196] T 65ca02bf14bf4cec8b5cd90c305e349e P 2a01ace80baf4057b6bf1995cf88468b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/65ca02bf14bf4cec8b5cd90c305e349e
I20240503 02:00:25.870760 14641 ts_tablet_manager.cc:1909] T 65ca02bf14bf4cec8b5cd90c305e349e P 2a01ace80baf4057b6bf1995cf88468b: Deleting consensus metadata
I20240503 02:00:25.870770 14639 ts_tablet_manager.cc:1909] T 65ca02bf14bf4cec8b5cd90c305e349e P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting consensus metadata
I20240503 02:00:25.871698 14640 ts_tablet_manager.cc:1888] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:25.871829 14640 log.cc:1196] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/65ca02bf14bf4cec8b5cd90c305e349e
I20240503 02:00:25.871804 6300 catalog_manager.cc:4830] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079): tablet 65ca02bf14bf4cec8b5cd90c305e349e (table rebalancer_dynamic_table_new_name [id=e9e0570a970943cb845b71a46c761153]) successfully deleted
I20240503 02:00:25.871917 6303 catalog_manager.cc:4830] TS c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655): tablet 65ca02bf14bf4cec8b5cd90c305e349e (table rebalancer_dynamic_table_new_name [id=e9e0570a970943cb845b71a46c761153]) successfully deleted
I20240503 02:00:25.872298 14640 ts_tablet_manager.cc:1909] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a: Deleting consensus metadata
I20240503 02:00:25.873080 6300 catalog_manager.cc:4830] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): tablet 65ca02bf14bf4cec8b5cd90c305e349e (table rebalancer_dynamic_table_new_name [id=e9e0570a970943cb845b71a46c761153]) successfully deleted
I20240503 02:00:25.968089 6313 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:00:25.974514 6700 tablet_service.cc:1467] Processing CreateTablet for tablet b73df7c0cc5643f3b77f8f276acb475f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1b4f9b18e1ba4a5c97eb865e166f296e]), partition=
I20240503 02:00:25.974854 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b73df7c0cc5643f3b77f8f276acb475f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.976835 6831 tablet_service.cc:1467] Processing CreateTablet for tablet b73df7c0cc5643f3b77f8f276acb475f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1b4f9b18e1ba4a5c97eb865e166f296e]), partition=
I20240503 02:00:25.977128 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b73df7c0cc5643f3b77f8f276acb475f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.977470 14629 tablet_bootstrap.cc:492] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:25.979085 14629 tablet_bootstrap.cc:654] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.979147 14623 tablet_bootstrap.cc:492] T b73df7c0cc5643f3b77f8f276acb475f P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:25.979887 6568 tablet_service.cc:1467] Processing CreateTablet for tablet b73df7c0cc5643f3b77f8f276acb475f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1b4f9b18e1ba4a5c97eb865e166f296e]), partition=
I20240503 02:00:25.980203 6568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b73df7c0cc5643f3b77f8f276acb475f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.980826 14623 tablet_bootstrap.cc:654] T b73df7c0cc5643f3b77f8f276acb475f P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.982398 14623 tablet_bootstrap.cc:492] T b73df7c0cc5643f3b77f8f276acb475f P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:25.982523 14623 ts_tablet_manager.cc:1367] T b73df7c0cc5643f3b77f8f276acb475f P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:25.982677 14624 tablet_bootstrap.cc:492] T b73df7c0cc5643f3b77f8f276acb475f P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:25.983074 14623 raft_consensus.cc:348] T b73df7c0cc5643f3b77f8f276acb475f P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:25.983246 14623 raft_consensus.cc:374] T b73df7c0cc5643f3b77f8f276acb475f P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.983283 14623 raft_consensus.cc:729] T b73df7c0cc5643f3b77f8f276acb475f P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.983436 14623 consensus_queue.cc:260] T b73df7c0cc5643f3b77f8f276acb475f P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:25.983695 14624 tablet_bootstrap.cc:654] T b73df7c0cc5643f3b77f8f276acb475f P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.983736 14623 ts_tablet_manager.cc:1398] T b73df7c0cc5643f3b77f8f276acb475f P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:25.985011 14624 tablet_bootstrap.cc:492] T b73df7c0cc5643f3b77f8f276acb475f P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:25.985113 14624 ts_tablet_manager.cc:1367] T b73df7c0cc5643f3b77f8f276acb475f P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:25.985143 14629 tablet_bootstrap.cc:492] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:25.985268 14629 ts_tablet_manager.cc:1367] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240503 02:00:25.985550 14624 raft_consensus.cc:348] T b73df7c0cc5643f3b77f8f276acb475f P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:25.985644 14624 raft_consensus.cc:374] T b73df7c0cc5643f3b77f8f276acb475f P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.985664 14624 raft_consensus.cc:729] T b73df7c0cc5643f3b77f8f276acb475f P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.985774 14624 consensus_queue.cc:260] T b73df7c0cc5643f3b77f8f276acb475f P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:25.985777 14629 raft_consensus.cc:348] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:25.985888 14629 raft_consensus.cc:374] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.986047 14624 ts_tablet_manager.cc:1398] T b73df7c0cc5643f3b77f8f276acb475f P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:25.985917 14629 raft_consensus.cc:729] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.987157 14629 consensus_queue.cc:260] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:25.988981 14629 ts_tablet_manager.cc:1398] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
W20240503 02:00:25.993088 6654 consensus_peers.cc:485] T ba17d4d3e0754b2798eb95484d37229a P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:25.995294 8851 consensus_peers.cc:485] T 067f54011a37432d99157884e70fc601 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:26.041015 9113 consensus_peers.cc:485] T f44b59277c3b47adbd4dd6aff9350af1 P 2a01ace80baf4057b6bf1995cf88468b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:26.102427 6523 consensus_peers.cc:485] T 4385d51d11574192883d01376e2d5620 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:26.121910 6523 consensus_peers.cc:485] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:26.122246 8982 consensus_peers.cc:485] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:26.136173 6785 consensus_peers.cc:485] T 8246488dec204bf6b09e868d7951ac5a P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 02:00:26.144038 14386 raft_consensus.cc:482] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:26.144276 14386 raft_consensus.cc:504] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
W20240503 02:00:26.144830 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:26.145053 14386 leader_election.cc:288] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:26.145597 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b73df7c0cc5643f3b77f8f276acb475f" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 02:00:26.145624 6587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b73df7c0cc5643f3b77f8f276acb475f" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 02:00:26.145828 6587 raft_consensus.cc:2454] T b73df7c0cc5643f3b77f8f276acb475f P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:26.145835 6849 raft_consensus.cc:2454] T b73df7c0cc5643f3b77f8f276acb475f P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:26.146206 6655 leader_election.cc:302] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 80fd18ef4d604c86b909d5c008a46b4f, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:26.146863 14386 raft_consensus.cc:2781] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:26.146939 14386 raft_consensus.cc:482] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:26.147001 14386 raft_consensus.cc:3037] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.148154 14386 raft_consensus.cc:504] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:26.148572 14386 leader_election.cc:288] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Requested vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:26.149008 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b73df7c0cc5643f3b77f8f276acb475f" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 02:00:26.149160 6589 raft_consensus.cc:3037] T b73df7c0cc5643f3b77f8f276acb475f P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.149451 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b73df7c0cc5643f3b77f8f276acb475f" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 02:00:26.149585 6849 raft_consensus.cc:3037] T b73df7c0cc5643f3b77f8f276acb475f P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.150118 6589 raft_consensus.cc:2454] T b73df7c0cc5643f3b77f8f276acb475f P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:26.150494 6849 raft_consensus.cc:2454] T b73df7c0cc5643f3b77f8f276acb475f P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:26.151021 6656 leader_election.cc:302] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:26.151417 14190 raft_consensus.cc:2781] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:26.151564 14190 raft_consensus.cc:686] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Becoming Leader. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Running, Role: LEADER
I20240503 02:00:26.151719 14190 consensus_queue.cc:237] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:26.155071 6308 catalog_manager.cc:5484] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:26.163467 14386 raft_consensus.cc:482] T 41b2718b68ea4714b5d9a0deac130030 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:26.163859 14386 raft_consensus.cc:504] T 41b2718b68ea4714b5d9a0deac130030 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:26.164744 14386 leader_election.cc:288] T 41b2718b68ea4714b5d9a0deac130030 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:26.164886 14562 consensus_queue.cc:1035] T 9c00b9bef94d4890ae17b04b8ff3f517 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:26.166245 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41b2718b68ea4714b5d9a0deac130030" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:26.166393 9178 raft_consensus.cc:2454] T 41b2718b68ea4714b5d9a0deac130030 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:26.166708 6653 leader_election.cc:302] T 41b2718b68ea4714b5d9a0deac130030 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:26.169003 14386 raft_consensus.cc:2781] T 41b2718b68ea4714b5d9a0deac130030 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:26.169099 14386 raft_consensus.cc:482] T 41b2718b68ea4714b5d9a0deac130030 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:26.169023 9047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41b2718b68ea4714b5d9a0deac130030" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:26.169148 14386 raft_consensus.cc:3037] T 41b2718b68ea4714b5d9a0deac130030 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.169206 9047 raft_consensus.cc:2454] T 41b2718b68ea4714b5d9a0deac130030 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:26.171553 14462 consensus_queue.cc:1035] T 9c00b9bef94d4890ae17b04b8ff3f517 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:26.175287 14386 raft_consensus.cc:504] T 41b2718b68ea4714b5d9a0deac130030 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:26.176406 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41b2718b68ea4714b5d9a0deac130030" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:26.176541 9178 raft_consensus.cc:3037] T 41b2718b68ea4714b5d9a0deac130030 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.177424 14386 leader_election.cc:288] T 41b2718b68ea4714b5d9a0deac130030 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Requested vote from peers 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:26.177488 9047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41b2718b68ea4714b5d9a0deac130030" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:26.177548 9178 raft_consensus.cc:2454] T 41b2718b68ea4714b5d9a0deac130030 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:26.177601 9047 raft_consensus.cc:3037] T 41b2718b68ea4714b5d9a0deac130030 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.177995 6653 leader_election.cc:302] T 41b2718b68ea4714b5d9a0deac130030 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:26.178236 14386 raft_consensus.cc:2781] T 41b2718b68ea4714b5d9a0deac130030 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:26.178400 14386 raft_consensus.cc:686] T 41b2718b68ea4714b5d9a0deac130030 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Becoming Leader. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Running, Role: LEADER
I20240503 02:00:26.178571 14386 consensus_queue.cc:237] T 41b2718b68ea4714b5d9a0deac130030 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:26.178684 9047 raft_consensus.cc:2454] T 41b2718b68ea4714b5d9a0deac130030 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:26.181385 6309 catalog_manager.cc:5484] T 41b2718b68ea4714b5d9a0deac130030 P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } }
W20240503 02:00:26.247449 6785 consensus_peers.cc:485] T 46ad540deb624a25b97d3b967fe1f6f4 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:26.265272 6785 consensus_peers.cc:485] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:00:26.285496 6309 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:26.289940 6589 raft_consensus.cc:1261] T b73df7c0cc5643f3b77f8f276acb475f P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:26.290189 6849 raft_consensus.cc:1261] T b73df7c0cc5643f3b77f8f276acb475f P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:26.290637 14190 consensus_queue.cc:1035] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:26.290999 14190 consensus_queue.cc:1035] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:26.294981 14659 tablet.cc:1722] T b73df7c0cc5643f3b77f8f276acb475f P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:26.295634 14659 tablet.cc:1620] T b73df7c0cc5643f3b77f8f276acb475f P 80fd18ef4d604c86b909d5c008a46b4f: MemRowSet was empty: no flush needed.
I20240503 02:00:26.296214 6308 catalog_manager.cc:5576] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719) does not have the latest schema for tablet b73df7c0cc5643f3b77f8f276acb475f (table rebalancer_dynamic_table [id=1b4f9b18e1ba4a5c97eb865e166f296e]). Expected version 1 got 0
I20240503 02:00:26.294782 14658 tablet.cc:1722] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:26.297613 14658 tablet.cc:1620] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025: MemRowSet was empty: no flush needed.
I20240503 02:00:26.298339 6313 catalog_manager.cc:5576] TS c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655) does not have the latest schema for tablet b73df7c0cc5643f3b77f8f276acb475f (table rebalancer_dynamic_table [id=1b4f9b18e1ba4a5c97eb865e166f296e]). Expected version 1 got 0
I20240503 02:00:26.298650 6309 catalog_manager.cc:5576] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863) does not have the latest schema for tablet b73df7c0cc5643f3b77f8f276acb475f (table rebalancer_dynamic_table [id=1b4f9b18e1ba4a5c97eb865e166f296e]). Expected version 1 got 0
I20240503 02:00:26.299626 14660 tablet.cc:1722] T b73df7c0cc5643f3b77f8f276acb475f P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:26.299896 14660 tablet.cc:1620] T b73df7c0cc5643f3b77f8f276acb475f P 492448570cc94ed1879feb2ec3b9051d: MemRowSet was empty: no flush needed.
I20240503 02:00:26.301301 6313 catalog_manager.cc:5576] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719) does not have the latest schema for tablet b73df7c0cc5643f3b77f8f276acb475f (table rebalancer_dynamic_table [id=1b4f9b18e1ba4a5c97eb865e166f296e]). Expected version 1 got 0
I20240503 02:00:26.304632 6308 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=1b4f9b18e1ba4a5c97eb865e166f296e] alter complete (version 1)
W20240503 02:00:26.332499 6523 consensus_peers.cc:485] T 925612b660ec471da5b3c548802ede6e P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 02:00:26.358970 14552 consensus_queue.cc:1035] T 1dfbfcb60a3c4537a32f496e6772c916 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:26.364965 14552 consensus_queue.cc:1035] T 1dfbfcb60a3c4537a32f496e6772c916 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:00:26.380420 9113 consensus_peers.cc:485] T b1740b877c1e4372a0337d99f5bcab0e P 2a01ace80baf4057b6bf1995cf88468b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:26.388870 6785 consensus_peers.cc:485] T d897c21e630d4770832bb10f43778e82 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:26.405189 8851 consensus_peers.cc:485] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 02:00:26.410620 6308 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:26.416625 14658 tablet.cc:1722] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:26.416998 14658 tablet.cc:1620] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025: MemRowSet was empty: no flush needed.
I20240503 02:00:26.417070 14659 tablet.cc:1722] T b73df7c0cc5643f3b77f8f276acb475f P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:26.417364 14659 tablet.cc:1620] T b73df7c0cc5643f3b77f8f276acb475f P 80fd18ef4d604c86b909d5c008a46b4f: MemRowSet was empty: no flush needed.
I20240503 02:00:26.417362 14660 tablet.cc:1722] T b73df7c0cc5643f3b77f8f276acb475f P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:26.417670 14660 tablet.cc:1620] T b73df7c0cc5643f3b77f8f276acb475f P 492448570cc94ed1879feb2ec3b9051d: MemRowSet was empty: no flush needed.
I20240503 02:00:26.423130 6308 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=1b4f9b18e1ba4a5c97eb865e166f296e] alter complete (version 2)
W20240503 02:00:26.436861 6654 consensus_peers.cc:485] T cea1ea50fa434d31b42451f84f26bc76 P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 02:00:26.526170 6309 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:26.532155 14658 tablet.cc:1722] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:26.532887 14659 tablet.cc:1722] T b73df7c0cc5643f3b77f8f276acb475f P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:26.534089 14660 tablet.cc:1722] T b73df7c0cc5643f3b77f8f276acb475f P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
W20240503 02:00:26.535934 6523 consensus_peers.cc:485] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:00:26.538813 6314 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=1b4f9b18e1ba4a5c97eb865e166f296e] alter complete (version 3)
W20240503 02:00:26.571951 6523 consensus_peers.cc:485] T d0d5dd42db3e4883af7fd16d8813b550 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 02:00:26.642629 6308 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:26.642818 6308 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:26.645216 6308 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet b73df7c0cc5643f3b77f8f276acb475f
I20240503 02:00:26.645849 6700 tablet_service.cc:1514] Processing DeleteTablet for tablet b73df7c0cc5643f3b77f8f276acb475f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:26 UTC) from {username='slave'} at 127.0.0.1:40336
I20240503 02:00:26.645850 6568 tablet_service.cc:1514] Processing DeleteTablet for tablet b73df7c0cc5643f3b77f8f276acb475f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:26 UTC) from {username='slave'} at 127.0.0.1:59846
I20240503 02:00:26.645936 6831 tablet_service.cc:1514] Processing DeleteTablet for tablet b73df7c0cc5643f3b77f8f276acb475f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:26 UTC) from {username='slave'} at 127.0.0.1:52812
I20240503 02:00:26.646176 14666 tablet_replica.cc:330] T b73df7c0cc5643f3b77f8f276acb475f P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 02:00:26.646236 14667 tablet_replica.cc:330] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025: stopping tablet replica
I20240503 02:00:26.646329 14666 raft_consensus.cc:2229] T b73df7c0cc5643f3b77f8f276acb475f P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:26.646384 14667 raft_consensus.cc:2229] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:26.646467 14666 raft_consensus.cc:2258] T b73df7c0cc5643f3b77f8f276acb475f P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:26.646587 14667 raft_consensus.cc:2258] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:26.646827 14668 tablet_replica.cc:330] T b73df7c0cc5643f3b77f8f276acb475f P 492448570cc94ed1879feb2ec3b9051d: stopping tablet replica
I20240503 02:00:26.647049 14666 ts_tablet_manager.cc:1875] T b73df7c0cc5643f3b77f8f276acb475f P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:26.647085 14668 raft_consensus.cc:2229] T b73df7c0cc5643f3b77f8f276acb475f P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:26.647267 14668 raft_consensus.cc:2258] T b73df7c0cc5643f3b77f8f276acb475f P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:26.647277 14667 ts_tablet_manager.cc:1875] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:26.647832 14668 ts_tablet_manager.cc:1875] T b73df7c0cc5643f3b77f8f276acb475f P 492448570cc94ed1879feb2ec3b9051d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:26.649488 14666 ts_tablet_manager.cc:1888] T b73df7c0cc5643f3b77f8f276acb475f P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:26.649637 14666 log.cc:1196] T b73df7c0cc5643f3b77f8f276acb475f P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/b73df7c0cc5643f3b77f8f276acb475f
I20240503 02:00:26.650070 14666 ts_tablet_manager.cc:1909] T b73df7c0cc5643f3b77f8f276acb475f P 80fd18ef4d604c86b909d5c008a46b4f: Deleting consensus metadata
I20240503 02:00:26.650667 14668 ts_tablet_manager.cc:1888] T b73df7c0cc5643f3b77f8f276acb475f P 492448570cc94ed1879feb2ec3b9051d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:26.650774 14667 ts_tablet_manager.cc:1888] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:26.650823 14668 log.cc:1196] T b73df7c0cc5643f3b77f8f276acb475f P 492448570cc94ed1879feb2ec3b9051d: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/b73df7c0cc5643f3b77f8f276acb475f
I20240503 02:00:26.650882 14667 log.cc:1196] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/b73df7c0cc5643f3b77f8f276acb475f
I20240503 02:00:26.651006 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet b73df7c0cc5643f3b77f8f276acb475f (table rebalancer_dynamic_table_new_name [id=1b4f9b18e1ba4a5c97eb865e166f296e]) successfully deleted
I20240503 02:00:26.651198 14667 ts_tablet_manager.cc:1909] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting consensus metadata
I20240503 02:00:26.651198 14668 ts_tablet_manager.cc:1909] T b73df7c0cc5643f3b77f8f276acb475f P 492448570cc94ed1879feb2ec3b9051d: Deleting consensus metadata
I20240503 02:00:26.652294 6303 catalog_manager.cc:4830] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719): tablet b73df7c0cc5643f3b77f8f276acb475f (table rebalancer_dynamic_table_new_name [id=1b4f9b18e1ba4a5c97eb865e166f296e]) successfully deleted
I20240503 02:00:26.652503 6303 catalog_manager.cc:4830] TS c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655): tablet b73df7c0cc5643f3b77f8f276acb475f (table rebalancer_dynamic_table_new_name [id=1b4f9b18e1ba4a5c97eb865e166f296e]) successfully deleted
I20240503 02:00:26.656827 14664 consensus_queue.cc:1035] T 41b2718b68ea4714b5d9a0deac130030 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:26.659860 14664 consensus_queue.cc:1035] T 41b2718b68ea4714b5d9a0deac130030 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:00:26.664561 6301 catalog_manager.cc:4573] TS e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): CreateTablet RPC failed for tablet 56824b553788471c9a2fcd6ec82d507e: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
W20240503 02:00:26.681623 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:26.711624 6314 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_extra_table_68"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:26.718322 6700 tablet_service.cc:1467] Processing CreateTablet for tablet 3c1ce8f08de449c7b4e54aca59a187bc (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=af68bb4f304a47f9b6790678a88690e9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:26.718577 6438 tablet_service.cc:1467] Processing CreateTablet for tablet 3c1ce8f08de449c7b4e54aca59a187bc (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=af68bb4f304a47f9b6790678a88690e9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:26.718672 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c1ce8f08de449c7b4e54aca59a187bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.718842 6438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c1ce8f08de449c7b4e54aca59a187bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.720079 6436 tablet_service.cc:1467] Processing CreateTablet for tablet 81911ae22dfa45c886c1126cc77769dd (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=af68bb4f304a47f9b6790678a88690e9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:26.720351 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81911ae22dfa45c886c1126cc77769dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.720829 6437 tablet_service.cc:1467] Processing CreateTablet for tablet f6ce658b7326427583fd3d2e5d0b53a7 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=af68bb4f304a47f9b6790678a88690e9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:26.721045 6437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6ce658b7326427583fd3d2e5d0b53a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.721199 14672 tablet_bootstrap.cc:492] T 3c1ce8f08de449c7b4e54aca59a187bc P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:26.721421 8880 tablet_service.cc:1467] Processing CreateTablet for tablet 81911ae22dfa45c886c1126cc77769dd (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=af68bb4f304a47f9b6790678a88690e9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:26.721676 8880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81911ae22dfa45c886c1126cc77769dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.722435 14672 tablet_bootstrap.cc:654] T 3c1ce8f08de449c7b4e54aca59a187bc P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.722860 9157 tablet_service.cc:1467] Processing CreateTablet for tablet f6ce658b7326427583fd3d2e5d0b53a7 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=af68bb4f304a47f9b6790678a88690e9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:26.723145 9157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6ce658b7326427583fd3d2e5d0b53a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.724014 14675 tablet_bootstrap.cc:492] T 81911ae22dfa45c886c1126cc77769dd P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:26.724217 8887 tablet_service.cc:1467] Processing CreateTablet for tablet f6ce658b7326427583fd3d2e5d0b53a7 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=af68bb4f304a47f9b6790678a88690e9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:26.724299 14673 tablet_bootstrap.cc:492] T 81911ae22dfa45c886c1126cc77769dd P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:26.724498 8887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6ce658b7326427583fd3d2e5d0b53a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.725098 14675 tablet_bootstrap.cc:654] T 81911ae22dfa45c886c1126cc77769dd P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.725226 14673 tablet_bootstrap.cc:654] T 81911ae22dfa45c886c1126cc77769dd P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.725935 14678 tablet_bootstrap.cc:492] T f6ce658b7326427583fd3d2e5d0b53a7 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:26.726245 6568 tablet_service.cc:1467] Processing CreateTablet for tablet 81911ae22dfa45c886c1126cc77769dd (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=af68bb4f304a47f9b6790678a88690e9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:26.726516 6568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81911ae22dfa45c886c1126cc77769dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.726877 14678 tablet_bootstrap.cc:654] T f6ce658b7326427583fd3d2e5d0b53a7 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.728281 14678 tablet_bootstrap.cc:492] T f6ce658b7326427583fd3d2e5d0b53a7 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:26.728425 14678 ts_tablet_manager.cc:1367] T f6ce658b7326427583fd3d2e5d0b53a7 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:26.728956 14678 raft_consensus.cc:348] T f6ce658b7326427583fd3d2e5d0b53a7 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:26.729063 14678 raft_consensus.cc:374] T f6ce658b7326427583fd3d2e5d0b53a7 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.729092 14678 raft_consensus.cc:729] T f6ce658b7326427583fd3d2e5d0b53a7 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.729163 14675 tablet_bootstrap.cc:492] T 81911ae22dfa45c886c1126cc77769dd P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:26.729204 14678 consensus_queue.cc:260] T f6ce658b7326427583fd3d2e5d0b53a7 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:26.729255 14675 ts_tablet_manager.cc:1367] T 81911ae22dfa45c886c1126cc77769dd P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:00:26.729506 14678 ts_tablet_manager.cc:1398] T f6ce658b7326427583fd3d2e5d0b53a7 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:26.729718 14675 raft_consensus.cc:348] T 81911ae22dfa45c886c1126cc77769dd P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:26.729816 14675 raft_consensus.cc:374] T 81911ae22dfa45c886c1126cc77769dd P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.729846 14675 raft_consensus.cc:729] T 81911ae22dfa45c886c1126cc77769dd P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.729969 14675 consensus_queue.cc:260] T 81911ae22dfa45c886c1126cc77769dd P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:26.730242 14675 ts_tablet_manager.cc:1398] T 81911ae22dfa45c886c1126cc77769dd P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:26.730410 14675 tablet_bootstrap.cc:492] T f6ce658b7326427583fd3d2e5d0b53a7 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
W20240503 02:00:26.730767 8982 consensus_peers.cc:485] T 1a7ee79312ea4bad816264375ca5bb2d P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 02:00:26.731472 14675 tablet_bootstrap.cc:654] T f6ce658b7326427583fd3d2e5d0b53a7 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.731684 9028 tablet_service.cc:1467] Processing CreateTablet for tablet 3c1ce8f08de449c7b4e54aca59a187bc (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=af68bb4f304a47f9b6790678a88690e9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:26.731959 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c1ce8f08de449c7b4e54aca59a187bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.733464 14680 tablet_bootstrap.cc:492] T 81911ae22dfa45c886c1126cc77769dd P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:26.734112 14675 tablet_bootstrap.cc:492] T f6ce658b7326427583fd3d2e5d0b53a7 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:26.734208 14675 ts_tablet_manager.cc:1367] T f6ce658b7326427583fd3d2e5d0b53a7 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:26.735436 14675 raft_consensus.cc:348] T f6ce658b7326427583fd3d2e5d0b53a7 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:26.735894 14675 raft_consensus.cc:374] T f6ce658b7326427583fd3d2e5d0b53a7 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.735940 14675 raft_consensus.cc:729] T f6ce658b7326427583fd3d2e5d0b53a7 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.736083 14675 consensus_queue.cc:260] T f6ce658b7326427583fd3d2e5d0b53a7 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:26.736426 14675 ts_tablet_manager.cc:1398] T f6ce658b7326427583fd3d2e5d0b53a7 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:26.736845 14680 tablet_bootstrap.cc:654] T 81911ae22dfa45c886c1126cc77769dd P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.738224 14672 tablet_bootstrap.cc:492] T 3c1ce8f08de449c7b4e54aca59a187bc P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:26.738330 14672 ts_tablet_manager.cc:1367] T 3c1ce8f08de449c7b4e54aca59a187bc P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240503 02:00:26.738802 14672 raft_consensus.cc:348] T 3c1ce8f08de449c7b4e54aca59a187bc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:26.738930 14672 raft_consensus.cc:374] T 3c1ce8f08de449c7b4e54aca59a187bc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.738958 14672 raft_consensus.cc:729] T 3c1ce8f08de449c7b4e54aca59a187bc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.739053 14673 tablet_bootstrap.cc:492] T 81911ae22dfa45c886c1126cc77769dd P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:26.739063 14672 consensus_queue.cc:260] T 3c1ce8f08de449c7b4e54aca59a187bc P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:26.739271 14673 ts_tablet_manager.cc:1367] T 81911ae22dfa45c886c1126cc77769dd P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240503 02:00:26.739346 14672 ts_tablet_manager.cc:1398] T 3c1ce8f08de449c7b4e54aca59a187bc P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:26.739768 14673 raft_consensus.cc:348] T 81911ae22dfa45c886c1126cc77769dd P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:26.739856 14673 raft_consensus.cc:374] T 81911ae22dfa45c886c1126cc77769dd P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.739883 14673 raft_consensus.cc:729] T 81911ae22dfa45c886c1126cc77769dd P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.740264 14673 consensus_queue.cc:260] T 81911ae22dfa45c886c1126cc77769dd P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:26.740552 14673 ts_tablet_manager.cc:1398] T 81911ae22dfa45c886c1126cc77769dd P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:26.741261 14673 tablet_bootstrap.cc:492] T f6ce658b7326427583fd3d2e5d0b53a7 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
W20240503 02:00:26.742780 8982 consensus_peers.cc:485] T 13756a6e44814fe4b09a6b0f087546c3 P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 02:00:26.743142 14673 tablet_bootstrap.cc:654] T f6ce658b7326427583fd3d2e5d0b53a7 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.743388 14682 tablet_bootstrap.cc:492] T 3c1ce8f08de449c7b4e54aca59a187bc P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:26.745317 14680 tablet_bootstrap.cc:492] T 81911ae22dfa45c886c1126cc77769dd P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:26.745446 14680 ts_tablet_manager.cc:1367] T 81911ae22dfa45c886c1126cc77769dd P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 02:00:26.746013 14673 tablet_bootstrap.cc:492] T f6ce658b7326427583fd3d2e5d0b53a7 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:26.746114 14673 ts_tablet_manager.cc:1367] T f6ce658b7326427583fd3d2e5d0b53a7 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240503 02:00:26.746497 14680 raft_consensus.cc:348] T 81911ae22dfa45c886c1126cc77769dd P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:26.746634 14680 raft_consensus.cc:374] T 81911ae22dfa45c886c1126cc77769dd P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.746668 14680 raft_consensus.cc:729] T 81911ae22dfa45c886c1126cc77769dd P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.746606 14673 raft_consensus.cc:348] T f6ce658b7326427583fd3d2e5d0b53a7 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:26.746717 14673 raft_consensus.cc:374] T f6ce658b7326427583fd3d2e5d0b53a7 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.746753 14673 raft_consensus.cc:729] T f6ce658b7326427583fd3d2e5d0b53a7 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.746932 14682 tablet_bootstrap.cc:654] T 3c1ce8f08de449c7b4e54aca59a187bc P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.746788 14680 consensus_queue.cc:260] T 81911ae22dfa45c886c1126cc77769dd P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:26.747498 14680 ts_tablet_manager.cc:1398] T 81911ae22dfa45c886c1126cc77769dd P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:00:26.747740 14673 consensus_queue.cc:260] T f6ce658b7326427583fd3d2e5d0b53a7 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:26.748677 14673 ts_tablet_manager.cc:1398] T f6ce658b7326427583fd3d2e5d0b53a7 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:26.748924 6314 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:00:26.749248 14682 tablet_bootstrap.cc:492] T 3c1ce8f08de449c7b4e54aca59a187bc P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:26.749394 14682 ts_tablet_manager.cc:1367] T 3c1ce8f08de449c7b4e54aca59a187bc P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240503 02:00:26.752283 14682 raft_consensus.cc:348] T 3c1ce8f08de449c7b4e54aca59a187bc P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:26.752450 14682 raft_consensus.cc:374] T 3c1ce8f08de449c7b4e54aca59a187bc P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.752559 14673 tablet_bootstrap.cc:492] T 3c1ce8f08de449c7b4e54aca59a187bc P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:26.754281 14673 tablet_bootstrap.cc:654] T 3c1ce8f08de449c7b4e54aca59a187bc P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.754411 14682 raft_consensus.cc:729] T 3c1ce8f08de449c7b4e54aca59a187bc P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.754632 14682 consensus_queue.cc:260] T 3c1ce8f08de449c7b4e54aca59a187bc P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:26.754951 14682 ts_tablet_manager.cc:1398] T 3c1ce8f08de449c7b4e54aca59a187bc P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:00:26.756423 14673 tablet_bootstrap.cc:492] T 3c1ce8f08de449c7b4e54aca59a187bc P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:26.756546 14673 ts_tablet_manager.cc:1367] T 3c1ce8f08de449c7b4e54aca59a187bc P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20240503 02:00:26.757145 14673 raft_consensus.cc:348] T 3c1ce8f08de449c7b4e54aca59a187bc P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:26.757289 14673 raft_consensus.cc:374] T 3c1ce8f08de449c7b4e54aca59a187bc P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.757325 14673 raft_consensus.cc:729] T 3c1ce8f08de449c7b4e54aca59a187bc P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.757488 14673 consensus_queue.cc:260] T 3c1ce8f08de449c7b4e54aca59a187bc P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:26.757793 14673 ts_tablet_manager.cc:1398] T 3c1ce8f08de449c7b4e54aca59a187bc P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:26.761672 9028 tablet_service.cc:1467] Processing CreateTablet for tablet 61a3076a42dc47859a4b9ffbd873110c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=249868ae11634abf8527ed751495bcc6]), partition=
I20240503 02:00:26.761999 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61a3076a42dc47859a4b9ffbd873110c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.762239 9157 tablet_service.cc:1467] Processing CreateTablet for tablet 61a3076a42dc47859a4b9ffbd873110c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=249868ae11634abf8527ed751495bcc6]), partition=
I20240503 02:00:26.762482 9157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61a3076a42dc47859a4b9ffbd873110c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.764127 6831 tablet_service.cc:1467] Processing CreateTablet for tablet 61a3076a42dc47859a4b9ffbd873110c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=249868ae11634abf8527ed751495bcc6]), partition=
I20240503 02:00:26.764277 14678 tablet_bootstrap.cc:492] T 61a3076a42dc47859a4b9ffbd873110c P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:26.764348 14682 tablet_bootstrap.cc:492] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:26.764405 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61a3076a42dc47859a4b9ffbd873110c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.765246 14682 tablet_bootstrap.cc:654] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.765246 14678 tablet_bootstrap.cc:654] T 61a3076a42dc47859a4b9ffbd873110c P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.766904 14678 tablet_bootstrap.cc:492] T 61a3076a42dc47859a4b9ffbd873110c P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:26.766904 14682 tablet_bootstrap.cc:492] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:26.767002 14682 ts_tablet_manager.cc:1367] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:26.767002 14678 ts_tablet_manager.cc:1367] T 61a3076a42dc47859a4b9ffbd873110c P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:26.767457 14682 raft_consensus.cc:348] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:26.767491 14678 raft_consensus.cc:348] T 61a3076a42dc47859a4b9ffbd873110c P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:26.767567 14682 raft_consensus.cc:374] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.767573 14678 raft_consensus.cc:374] T 61a3076a42dc47859a4b9ffbd873110c P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.767597 14682 raft_consensus.cc:729] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.767602 14678 raft_consensus.cc:729] T 61a3076a42dc47859a4b9ffbd873110c P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.767722 14678 consensus_queue.cc:260] T 61a3076a42dc47859a4b9ffbd873110c P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:26.767741 14682 consensus_queue.cc:260] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:26.768002 14678 ts_tablet_manager.cc:1398] T 61a3076a42dc47859a4b9ffbd873110c P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:26.768013 14682 ts_tablet_manager.cc:1398] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:26.769730 14689 tablet_bootstrap.cc:492] T 61a3076a42dc47859a4b9ffbd873110c P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:26.771317 14689 tablet_bootstrap.cc:654] T 61a3076a42dc47859a4b9ffbd873110c P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.773133 14689 tablet_bootstrap.cc:492] T 61a3076a42dc47859a4b9ffbd873110c P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:26.773279 14689 ts_tablet_manager.cc:1367] T 61a3076a42dc47859a4b9ffbd873110c P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:00:26.773983 14689 raft_consensus.cc:348] T 61a3076a42dc47859a4b9ffbd873110c P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:26.774130 14689 raft_consensus.cc:374] T 61a3076a42dc47859a4b9ffbd873110c P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.774176 14689 raft_consensus.cc:729] T 61a3076a42dc47859a4b9ffbd873110c P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.774308 14689 consensus_queue.cc:260] T 61a3076a42dc47859a4b9ffbd873110c P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:26.774597 14689 ts_tablet_manager.cc:1398] T 61a3076a42dc47859a4b9ffbd873110c P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240503 02:00:26.776242 8851 consensus_peers.cc:485] T 5f51fe5f799f4e479b62f5917a2c2606 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 02:00:26.776906 14612 raft_consensus.cc:482] T 81911ae22dfa45c886c1126cc77769dd P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:26.777019 14612 raft_consensus.cc:504] T 81911ae22dfa45c886c1126cc77769dd P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:26.777480 14612 leader_election.cc:288] T 81911ae22dfa45c886c1126cc77769dd P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:26.777810 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81911ae22dfa45c886c1126cc77769dd" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 02:00:26.777967 6589 raft_consensus.cc:2454] T 81911ae22dfa45c886c1126cc77769dd P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:26.778061 8912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81911ae22dfa45c886c1126cc77769dd" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:26.778198 8912 raft_consensus.cc:2454] T 81911ae22dfa45c886c1126cc77769dd P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:26.778410 6393 leader_election.cc:302] T 81911ae22dfa45c886c1126cc77769dd P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:26.778600 14612 raft_consensus.cc:2781] T 81911ae22dfa45c886c1126cc77769dd P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:26.778662 14612 raft_consensus.cc:482] T 81911ae22dfa45c886c1126cc77769dd P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:26.778698 14612 raft_consensus.cc:3037] T 81911ae22dfa45c886c1126cc77769dd P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.779933 14612 raft_consensus.cc:504] T 81911ae22dfa45c886c1126cc77769dd P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:26.780370 14612 leader_election.cc:288] T 81911ae22dfa45c886c1126cc77769dd P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Requested vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:26.781610 8912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81911ae22dfa45c886c1126cc77769dd" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:26.781746 8912 raft_consensus.cc:3037] T 81911ae22dfa45c886c1126cc77769dd P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.781769 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81911ae22dfa45c886c1126cc77769dd" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 02:00:26.781903 6589 raft_consensus.cc:3037] T 81911ae22dfa45c886c1126cc77769dd P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.782672 8912 raft_consensus.cc:2454] T 81911ae22dfa45c886c1126cc77769dd P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:26.782713 6589 raft_consensus.cc:2454] T 81911ae22dfa45c886c1126cc77769dd P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:26.782991 6394 leader_election.cc:302] T 81911ae22dfa45c886c1126cc77769dd P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9942306855404cfd891adf227dd65eb2, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:26.783200 14612 raft_consensus.cc:2781] T 81911ae22dfa45c886c1126cc77769dd P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:26.783277 14612 raft_consensus.cc:686] T 81911ae22dfa45c886c1126cc77769dd P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Becoming Leader. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Running, Role: LEADER
I20240503 02:00:26.783406 14612 consensus_queue.cc:237] T 81911ae22dfa45c886c1126cc77769dd P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:26.791822 6313 catalog_manager.cc:5484] T 81911ae22dfa45c886c1126cc77769dd P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: term changed from 0 to 1, leader changed from <none> to dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129). New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } }
W20240503 02:00:26.803767 8851 consensus_peers.cc:485] T f28586e70bd9473fbdc6b9126b666fa9 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 02:00:26.840749 14612 raft_consensus.cc:482] T f6ce658b7326427583fd3d2e5d0b53a7 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:26.840893 14612 raft_consensus.cc:504] T f6ce658b7326427583fd3d2e5d0b53a7 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:26.841300 14612 leader_election.cc:288] T f6ce658b7326427583fd3d2e5d0b53a7 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:26.841652 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ce658b7326427583fd3d2e5d0b53a7" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:26.841722 8912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ce658b7326427583fd3d2e5d0b53a7" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:26.841784 9178 raft_consensus.cc:2454] T f6ce658b7326427583fd3d2e5d0b53a7 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:26.841871 8912 raft_consensus.cc:2454] T f6ce658b7326427583fd3d2e5d0b53a7 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:26.842105 6391 leader_election.cc:302] T f6ce658b7326427583fd3d2e5d0b53a7 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:26.842258 14612 raft_consensus.cc:2781] T f6ce658b7326427583fd3d2e5d0b53a7 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:26.842300 14612 raft_consensus.cc:482] T f6ce658b7326427583fd3d2e5d0b53a7 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:26.842332 14612 raft_consensus.cc:3037] T f6ce658b7326427583fd3d2e5d0b53a7 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.843436 14612 raft_consensus.cc:504] T f6ce658b7326427583fd3d2e5d0b53a7 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:26.843824 14612 leader_election.cc:288] T f6ce658b7326427583fd3d2e5d0b53a7 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Requested vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:26.844262 8912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ce658b7326427583fd3d2e5d0b53a7" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:26.844261 9177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ce658b7326427583fd3d2e5d0b53a7" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:26.844405 9177 raft_consensus.cc:3037] T f6ce658b7326427583fd3d2e5d0b53a7 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.844417 8912 raft_consensus.cc:3037] T f6ce658b7326427583fd3d2e5d0b53a7 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.845263 9177 raft_consensus.cc:2454] T f6ce658b7326427583fd3d2e5d0b53a7 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:26.845243 8912 raft_consensus.cc:2454] T f6ce658b7326427583fd3d2e5d0b53a7 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:26.845556 6391 leader_election.cc:302] T f6ce658b7326427583fd3d2e5d0b53a7 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:26.845734 14612 raft_consensus.cc:2781] T f6ce658b7326427583fd3d2e5d0b53a7 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:26.845820 14612 raft_consensus.cc:686] T f6ce658b7326427583fd3d2e5d0b53a7 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Becoming Leader. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Running, Role: LEADER
I20240503 02:00:26.845971 14612 consensus_queue.cc:237] T f6ce658b7326427583fd3d2e5d0b53a7 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:26.848733 6313 catalog_manager.cc:5484] T f6ce658b7326427583fd3d2e5d0b53a7 P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: term changed from 0 to 1, leader changed from <none> to dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129). New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:26.875725 14664 raft_consensus.cc:482] T 3c1ce8f08de449c7b4e54aca59a187bc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:26.875864 14664 raft_consensus.cc:504] T 3c1ce8f08de449c7b4e54aca59a187bc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:26.876291 14664 leader_election.cc:288] T 3c1ce8f08de449c7b4e54aca59a187bc P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:26.876716 9047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1ce8f08de449c7b4e54aca59a187bc" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:26.876871 9047 raft_consensus.cc:2454] T 3c1ce8f08de449c7b4e54aca59a187bc P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:26.876978 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1ce8f08de449c7b4e54aca59a187bc" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 02:00:26.877102 6456 raft_consensus.cc:2454] T 3c1ce8f08de449c7b4e54aca59a187bc P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:26.877177 6654 leader_election.cc:302] T 3c1ce8f08de449c7b4e54aca59a187bc P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99ead02915954b4fab23b033ad6d546b, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:26.877393 14664 raft_consensus.cc:2781] T 3c1ce8f08de449c7b4e54aca59a187bc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:26.877465 14664 raft_consensus.cc:482] T 3c1ce8f08de449c7b4e54aca59a187bc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:26.877506 14664 raft_consensus.cc:3037] T 3c1ce8f08de449c7b4e54aca59a187bc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.878322 14664 raft_consensus.cc:504] T 3c1ce8f08de449c7b4e54aca59a187bc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:26.878696 14664 leader_election.cc:288] T 3c1ce8f08de449c7b4e54aca59a187bc P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Requested vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:26.879153 9047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1ce8f08de449c7b4e54aca59a187bc" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:26.879123 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1ce8f08de449c7b4e54aca59a187bc" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 02:00:26.879289 9047 raft_consensus.cc:3037] T 3c1ce8f08de449c7b4e54aca59a187bc P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.879590 6456 raft_consensus.cc:3037] T 3c1ce8f08de449c7b4e54aca59a187bc P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.880395 9047 raft_consensus.cc:2454] T 3c1ce8f08de449c7b4e54aca59a187bc P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:26.880785 6456 raft_consensus.cc:2454] T 3c1ce8f08de449c7b4e54aca59a187bc P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:26.880970 6654 leader_election.cc:302] T 3c1ce8f08de449c7b4e54aca59a187bc P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99ead02915954b4fab23b033ad6d546b, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:26.881193 14664 raft_consensus.cc:2781] T 3c1ce8f08de449c7b4e54aca59a187bc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:26.881280 14664 raft_consensus.cc:686] T 3c1ce8f08de449c7b4e54aca59a187bc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Becoming Leader. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Running, Role: LEADER
I20240503 02:00:26.881423 14664 consensus_queue.cc:237] T 3c1ce8f08de449c7b4e54aca59a187bc P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:26.884482 6313 catalog_manager.cc:5484] T 3c1ce8f08de449c7b4e54aca59a187bc P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:26.896873 14552 raft_consensus.cc:482] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:26.897001 14552 raft_consensus.cc:504] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:26.897382 14552 leader_election.cc:288] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:26.897735 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61a3076a42dc47859a4b9ffbd873110c" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 02:00:26.897768 9177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61a3076a42dc47859a4b9ffbd873110c" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:26.897882 9177 raft_consensus.cc:2454] T 61a3076a42dc47859a4b9ffbd873110c P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:26.897882 6849 raft_consensus.cc:2454] T 61a3076a42dc47859a4b9ffbd873110c P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:26.898214 8984 leader_election.cc:302] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:26.898389 14552 raft_consensus.cc:2781] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:26.898458 14552 raft_consensus.cc:482] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:26.898496 14552 raft_consensus.cc:3037] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.899328 14552 raft_consensus.cc:504] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:26.899709 14552 leader_election.cc:288] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Requested vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:26.900096 9177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61a3076a42dc47859a4b9ffbd873110c" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:26.900067 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61a3076a42dc47859a4b9ffbd873110c" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 02:00:26.900216 6849 raft_consensus.cc:3037] T 61a3076a42dc47859a4b9ffbd873110c P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.900216 9177 raft_consensus.cc:3037] T 61a3076a42dc47859a4b9ffbd873110c P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.901278 6849 raft_consensus.cc:2454] T 61a3076a42dc47859a4b9ffbd873110c P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:26.901278 9177 raft_consensus.cc:2454] T 61a3076a42dc47859a4b9ffbd873110c P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:26.901607 8981 leader_election.cc:302] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:26.901798 14552 raft_consensus.cc:2781] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:26.901909 14552 raft_consensus.cc:686] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Becoming Leader. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Running, Role: LEADER
I20240503 02:00:26.902123 14552 consensus_queue.cc:237] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:26.904633 6313 catalog_manager.cc:5484] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b reported cstate change: term changed from 0 to 1, leader changed from <none> to 99ead02915954b4fab23b033ad6d546b (127.0.99.134). New cstate: current_term: 1 leader_uuid: "99ead02915954b4fab23b033ad6d546b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } }
W20240503 02:00:26.940264 8982 consensus_peers.cc:485] T beeea71f02ae4c89a1af13d3584ef70a P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:26.950264 6523 consensus_peers.cc:485] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 02:00:27.010188 6313 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:27.015036 6849 raft_consensus.cc:1261] T 61a3076a42dc47859a4b9ffbd873110c P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer 99ead02915954b4fab23b033ad6d546b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:27.015125 9177 raft_consensus.cc:1261] T 61a3076a42dc47859a4b9ffbd873110c P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer 99ead02915954b4fab23b033ad6d546b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:27.015540 14665 consensus_queue.cc:1035] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:27.015727 14552 consensus_queue.cc:1035] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:27.019229 6313 catalog_manager.cc:5576] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719) does not have the latest schema for tablet 61a3076a42dc47859a4b9ffbd873110c (table rebalancer_dynamic_table [id=249868ae11634abf8527ed751495bcc6]). Expected version 1 got 0
I20240503 02:00:27.019362 14700 tablet.cc:1722] T 61a3076a42dc47859a4b9ffbd873110c P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:27.019721 14700 tablet.cc:1620] T 61a3076a42dc47859a4b9ffbd873110c P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
I20240503 02:00:27.019999 14699 tablet.cc:1722] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:27.020063 6314 catalog_manager.cc:5576] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079) does not have the latest schema for tablet 61a3076a42dc47859a4b9ffbd873110c (table rebalancer_dynamic_table [id=249868ae11634abf8527ed751495bcc6]). Expected version 1 got 0
I20240503 02:00:27.020401 14699 tablet.cc:1620] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b: MemRowSet was empty: no flush needed.
I20240503 02:00:27.025668 6308 catalog_manager.cc:5576] TS 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157) does not have the latest schema for tablet 61a3076a42dc47859a4b9ffbd873110c (table rebalancer_dynamic_table [id=249868ae11634abf8527ed751495bcc6]). Expected version 1 got 0
I20240503 02:00:27.025787 14660 tablet.cc:1722] T 61a3076a42dc47859a4b9ffbd873110c P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:27.026114 14660 tablet.cc:1620] T 61a3076a42dc47859a4b9ffbd873110c P 492448570cc94ed1879feb2ec3b9051d: MemRowSet was empty: no flush needed.
I20240503 02:00:27.027678 6313 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=249868ae11634abf8527ed751495bcc6] alter complete (version 1)
W20240503 02:00:27.099258 8851 consensus_peers.cc:485] T 7da8f614a70e4177b70e7c2f31d22a10 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 02:00:27.132761 6313 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:27.139647 14699 tablet.cc:1722] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:27.141150 14660 tablet.cc:1722] T 61a3076a42dc47859a4b9ffbd873110c P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:27.142071 14660 tablet.cc:1620] T 61a3076a42dc47859a4b9ffbd873110c P 492448570cc94ed1879feb2ec3b9051d: MemRowSet was empty: no flush needed.
I20240503 02:00:27.142074 14700 tablet.cc:1722] T 61a3076a42dc47859a4b9ffbd873110c P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:27.142127 14699 tablet.cc:1620] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b: MemRowSet was empty: no flush needed.
I20240503 02:00:27.142354 14700 tablet.cc:1620] T 61a3076a42dc47859a4b9ffbd873110c P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
I20240503 02:00:27.150228 6314 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=249868ae11634abf8527ed751495bcc6] alter complete (version 2)
W20240503 02:00:27.155472 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:27.226264 14693 consensus_queue.cc:1035] T 81911ae22dfa45c886c1126cc77769dd P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:27.231954 14612 consensus_queue.cc:1035] T 81911ae22dfa45c886c1126cc77769dd P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:27.254369 6313 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:27.260732 14699 tablet.cc:1722] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:27.261152 14700 tablet.cc:1722] T 61a3076a42dc47859a4b9ffbd873110c P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:27.263800 14660 tablet.cc:1722] T 61a3076a42dc47859a4b9ffbd873110c P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:27.267881 6313 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=249868ae11634abf8527ed751495bcc6] alter complete (version 3)
I20240503 02:00:27.295207 14612 consensus_queue.cc:1035] T f6ce658b7326427583fd3d2e5d0b53a7 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:27.299674 14693 consensus_queue.cc:1035] T f6ce658b7326427583fd3d2e5d0b53a7 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:27.371126 6313 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:27.371305 6313 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:27.373970 6313 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet 61a3076a42dc47859a4b9ffbd873110c
I20240503 02:00:27.374572 9028 tablet_service.cc:1514] Processing DeleteTablet for tablet 61a3076a42dc47859a4b9ffbd873110c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:27 UTC) from {username='slave'} at 127.0.0.1:52584
I20240503 02:00:27.374660 9157 tablet_service.cc:1514] Processing DeleteTablet for tablet 61a3076a42dc47859a4b9ffbd873110c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:27 UTC) from {username='slave'} at 127.0.0.1:51940
I20240503 02:00:27.374665 6831 tablet_service.cc:1514] Processing DeleteTablet for tablet 61a3076a42dc47859a4b9ffbd873110c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:27 UTC) from {username='slave'} at 127.0.0.1:52812
I20240503 02:00:27.374971 14713 tablet_replica.cc:330] T 61a3076a42dc47859a4b9ffbd873110c P 492448570cc94ed1879feb2ec3b9051d: stopping tablet replica
I20240503 02:00:27.374994 14711 tablet_replica.cc:330] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b: stopping tablet replica
I20240503 02:00:27.375114 14711 raft_consensus.cc:2229] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:27.375118 14713 raft_consensus.cc:2229] T 61a3076a42dc47859a4b9ffbd873110c P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:27.375236 14713 raft_consensus.cc:2258] T 61a3076a42dc47859a4b9ffbd873110c P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:27.375270 14711 raft_consensus.cc:2258] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:27.375864 14711 ts_tablet_manager.cc:1875] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:27.375967 14713 ts_tablet_manager.cc:1875] T 61a3076a42dc47859a4b9ffbd873110c P 492448570cc94ed1879feb2ec3b9051d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:27.377074 14712 tablet_replica.cc:330] T 61a3076a42dc47859a4b9ffbd873110c P 2a01ace80baf4057b6bf1995cf88468b: stopping tablet replica
I20240503 02:00:27.377379 14712 raft_consensus.cc:2229] T 61a3076a42dc47859a4b9ffbd873110c P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:27.377653 14712 raft_consensus.cc:2258] T 61a3076a42dc47859a4b9ffbd873110c P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:27.378379 14713 ts_tablet_manager.cc:1888] T 61a3076a42dc47859a4b9ffbd873110c P 492448570cc94ed1879feb2ec3b9051d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:27.378453 14711 ts_tablet_manager.cc:1888] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:27.378515 14713 log.cc:1196] T 61a3076a42dc47859a4b9ffbd873110c P 492448570cc94ed1879feb2ec3b9051d: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/61a3076a42dc47859a4b9ffbd873110c
I20240503 02:00:27.378639 14711 log.cc:1196] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/61a3076a42dc47859a4b9ffbd873110c
I20240503 02:00:27.378870 14713 ts_tablet_manager.cc:1909] T 61a3076a42dc47859a4b9ffbd873110c P 492448570cc94ed1879feb2ec3b9051d: Deleting consensus metadata
I20240503 02:00:27.379061 14712 ts_tablet_manager.cc:1875] T 61a3076a42dc47859a4b9ffbd873110c P 2a01ace80baf4057b6bf1995cf88468b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:27.379258 14711 ts_tablet_manager.cc:1909] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b: Deleting consensus metadata
I20240503 02:00:27.379714 6303 catalog_manager.cc:4830] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719): tablet 61a3076a42dc47859a4b9ffbd873110c (table rebalancer_dynamic_table_new_name [id=249868ae11634abf8527ed751495bcc6]) successfully deleted
I20240503 02:00:27.380584 6301 catalog_manager.cc:4830] TS 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157): tablet 61a3076a42dc47859a4b9ffbd873110c (table rebalancer_dynamic_table_new_name [id=249868ae11634abf8527ed751495bcc6]) successfully deleted
I20240503 02:00:27.382052 14712 ts_tablet_manager.cc:1888] T 61a3076a42dc47859a4b9ffbd873110c P 2a01ace80baf4057b6bf1995cf88468b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:27.382123 14712 log.cc:1196] T 61a3076a42dc47859a4b9ffbd873110c P 2a01ace80baf4057b6bf1995cf88468b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/61a3076a42dc47859a4b9ffbd873110c
I20240503 02:00:27.382339 14712 ts_tablet_manager.cc:1909] T 61a3076a42dc47859a4b9ffbd873110c P 2a01ace80baf4057b6bf1995cf88468b: Deleting consensus metadata
I20240503 02:00:27.383241 6300 catalog_manager.cc:4830] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079): tablet 61a3076a42dc47859a4b9ffbd873110c (table rebalancer_dynamic_table_new_name [id=249868ae11634abf8527ed751495bcc6]) successfully deleted
I20240503 02:00:27.410357 6313 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_extra_table_70"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:27.416814 6568 tablet_service.cc:1467] Processing CreateTablet for tablet 676e4bd2e5ed4cbea577a2f4210eb781 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=c3d4f630569149c79fc93cb8000a3cd1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:27.416813 6831 tablet_service.cc:1467] Processing CreateTablet for tablet 676e4bd2e5ed4cbea577a2f4210eb781 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=c3d4f630569149c79fc93cb8000a3cd1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:27.416863 8887 tablet_service.cc:1467] Processing CreateTablet for tablet 676e4bd2e5ed4cbea577a2f4210eb781 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=c3d4f630569149c79fc93cb8000a3cd1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:27.417152 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 676e4bd2e5ed4cbea577a2f4210eb781. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:27.417156 6568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 676e4bd2e5ed4cbea577a2f4210eb781. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:27.417163 8887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 676e4bd2e5ed4cbea577a2f4210eb781. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:27.420142 14715 tablet_bootstrap.cc:492] T 676e4bd2e5ed4cbea577a2f4210eb781 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:27.420552 9157 tablet_service.cc:1467] Processing CreateTablet for tablet ab3a5e2e96c44759bca819b1a79a5dc9 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=c3d4f630569149c79fc93cb8000a3cd1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:27.420853 9157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab3a5e2e96c44759bca819b1a79a5dc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:27.421463 14715 tablet_bootstrap.cc:654] T 676e4bd2e5ed4cbea577a2f4210eb781 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:27.422132 14716 tablet_bootstrap.cc:492] T 676e4bd2e5ed4cbea577a2f4210eb781 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:27.422143 6438 tablet_service.cc:1467] Processing CreateTablet for tablet ab3a5e2e96c44759bca819b1a79a5dc9 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=c3d4f630569149c79fc93cb8000a3cd1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:27.422461 6438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab3a5e2e96c44759bca819b1a79a5dc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:27.423206 14716 tablet_bootstrap.cc:654] T 676e4bd2e5ed4cbea577a2f4210eb781 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:27.423758 14717 tablet_bootstrap.cc:492] T ab3a5e2e96c44759bca819b1a79a5dc9 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:27.423868 9028 tablet_service.cc:1467] Processing CreateTablet for tablet ec7210e0a04549fab1200b7ab6a133bf (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=c3d4f630569149c79fc93cb8000a3cd1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:27.424208 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec7210e0a04549fab1200b7ab6a133bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:27.424506 14716 tablet_bootstrap.cc:492] T 676e4bd2e5ed4cbea577a2f4210eb781 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:27.424610 14716 ts_tablet_manager.cc:1367] T 676e4bd2e5ed4cbea577a2f4210eb781 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:27.425062 14716 raft_consensus.cc:348] T 676e4bd2e5ed4cbea577a2f4210eb781 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:27.425165 14716 raft_consensus.cc:374] T 676e4bd2e5ed4cbea577a2f4210eb781 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:27.425191 14716 raft_consensus.cc:729] T 676e4bd2e5ed4cbea577a2f4210eb781 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:27.425303 14716 consensus_queue.cc:260] T 676e4bd2e5ed4cbea577a2f4210eb781 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:27.425581 14716 ts_tablet_manager.cc:1398] T 676e4bd2e5ed4cbea577a2f4210eb781 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:27.427067 6700 tablet_service.cc:1467] Processing CreateTablet for tablet ec7210e0a04549fab1200b7ab6a133bf (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=c3d4f630569149c79fc93cb8000a3cd1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:27.427470 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec7210e0a04549fab1200b7ab6a133bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:27.427595 14717 tablet_bootstrap.cc:654] T ab3a5e2e96c44759bca819b1a79a5dc9 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:27.428673 14714 tablet_bootstrap.cc:492] T 676e4bd2e5ed4cbea577a2f4210eb781 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:27.430017 14714 tablet_bootstrap.cc:654] T 676e4bd2e5ed4cbea577a2f4210eb781 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:27.430599 14717 tablet_bootstrap.cc:492] T ab3a5e2e96c44759bca819b1a79a5dc9 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:27.430704 14717 ts_tablet_manager.cc:1367] T ab3a5e2e96c44759bca819b1a79a5dc9 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20240503 02:00:27.430728 6568 tablet_service.cc:1467] Processing CreateTablet for tablet ab3a5e2e96c44759bca819b1a79a5dc9 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=c3d4f630569149c79fc93cb8000a3cd1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:27.431035 6568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab3a5e2e96c44759bca819b1a79a5dc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:27.431308 14722 tablet_bootstrap.cc:492] T ec7210e0a04549fab1200b7ab6a133bf P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:27.432618 14719 tablet_bootstrap.cc:492] T ab3a5e2e96c44759bca819b1a79a5dc9 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:27.432618 14722 tablet_bootstrap.cc:654] T ec7210e0a04549fab1200b7ab6a133bf P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:27.433498 14716 tablet_bootstrap.cc:492] T ab3a5e2e96c44759bca819b1a79a5dc9 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:27.433826 14719 tablet_bootstrap.cc:654] T ab3a5e2e96c44759bca819b1a79a5dc9 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:27.434458 14716 tablet_bootstrap.cc:654] T ab3a5e2e96c44759bca819b1a79a5dc9 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:27.434422 14717 raft_consensus.cc:348] T ab3a5e2e96c44759bca819b1a79a5dc9 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:27.434562 14717 raft_consensus.cc:374] T ab3a5e2e96c44759bca819b1a79a5dc9 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:27.434595 14717 raft_consensus.cc:729] T ab3a5e2e96c44759bca819b1a79a5dc9 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:27.434724 14717 consensus_queue.cc:260] T ab3a5e2e96c44759bca819b1a79a5dc9 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:27.430728 6569 tablet_service.cc:1467] Processing CreateTablet for tablet ec7210e0a04549fab1200b7ab6a133bf (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=c3d4f630569149c79fc93cb8000a3cd1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:27.435331 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec7210e0a04549fab1200b7ab6a133bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:27.435482 14715 tablet_bootstrap.cc:492] T 676e4bd2e5ed4cbea577a2f4210eb781 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:27.435508 14714 tablet_bootstrap.cc:492] T 676e4bd2e5ed4cbea577a2f4210eb781 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:27.435575 14715 ts_tablet_manager.cc:1367] T 676e4bd2e5ed4cbea577a2f4210eb781 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240503 02:00:27.435582 14714 ts_tablet_manager.cc:1367] T 676e4bd2e5ed4cbea577a2f4210eb781 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:00:27.436062 14715 raft_consensus.cc:348] T 676e4bd2e5ed4cbea577a2f4210eb781 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:27.436169 14715 raft_consensus.cc:374] T 676e4bd2e5ed4cbea577a2f4210eb781 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:27.436199 14715 raft_consensus.cc:729] T 676e4bd2e5ed4cbea577a2f4210eb781 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:27.436224 14714 raft_consensus.cc:348] T 676e4bd2e5ed4cbea577a2f4210eb781 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:27.436328 14715 consensus_queue.cc:260] T 676e4bd2e5ed4cbea577a2f4210eb781 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:27.436657 14715 ts_tablet_manager.cc:1398] T 676e4bd2e5ed4cbea577a2f4210eb781 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:27.436369 14714 raft_consensus.cc:374] T 676e4bd2e5ed4cbea577a2f4210eb781 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:27.436761 14714 raft_consensus.cc:729] T 676e4bd2e5ed4cbea577a2f4210eb781 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:27.436923 14714 consensus_queue.cc:260] T 676e4bd2e5ed4cbea577a2f4210eb781 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:27.437345 14714 ts_tablet_manager.cc:1398] T 676e4bd2e5ed4cbea577a2f4210eb781 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:27.438195 14716 tablet_bootstrap.cc:492] T ab3a5e2e96c44759bca819b1a79a5dc9 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:27.438287 14716 ts_tablet_manager.cc:1367] T ab3a5e2e96c44759bca819b1a79a5dc9 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:00:27.438768 14717 ts_tablet_manager.cc:1398] T ab3a5e2e96c44759bca819b1a79a5dc9 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.008s user 0.000s sys 0.001s
I20240503 02:00:27.438831 14716 raft_consensus.cc:348] T ab3a5e2e96c44759bca819b1a79a5dc9 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:27.438918 14716 raft_consensus.cc:374] T ab3a5e2e96c44759bca819b1a79a5dc9 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:27.438941 14716 raft_consensus.cc:729] T ab3a5e2e96c44759bca819b1a79a5dc9 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:27.439067 14716 consensus_queue.cc:260] T ab3a5e2e96c44759bca819b1a79a5dc9 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:27.439107 14724 tablet_bootstrap.cc:492] T ec7210e0a04549fab1200b7ab6a133bf P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:27.439334 14716 ts_tablet_manager.cc:1398] T ab3a5e2e96c44759bca819b1a79a5dc9 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:27.439522 14716 tablet_bootstrap.cc:492] T ec7210e0a04549fab1200b7ab6a133bf P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:27.440441 14724 tablet_bootstrap.cc:654] T ec7210e0a04549fab1200b7ab6a133bf P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:27.441329 14716 tablet_bootstrap.cc:654] T ec7210e0a04549fab1200b7ab6a133bf P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:27.442155 14724 tablet_bootstrap.cc:492] T ec7210e0a04549fab1200b7ab6a133bf P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:27.442255 14724 ts_tablet_manager.cc:1367] T ec7210e0a04549fab1200b7ab6a133bf P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:27.442742 14724 raft_consensus.cc:348] T ec7210e0a04549fab1200b7ab6a133bf P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:27.442875 14724 raft_consensus.cc:374] T ec7210e0a04549fab1200b7ab6a133bf P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:27.442909 14724 raft_consensus.cc:729] T ec7210e0a04549fab1200b7ab6a133bf P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:27.443030 14724 consensus_queue.cc:260] T ec7210e0a04549fab1200b7ab6a133bf P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:27.443454 14724 ts_tablet_manager.cc:1398] T ec7210e0a04549fab1200b7ab6a133bf P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:27.444506 14716 tablet_bootstrap.cc:492] T ec7210e0a04549fab1200b7ab6a133bf P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:27.444749 14716 ts_tablet_manager.cc:1367] T ec7210e0a04549fab1200b7ab6a133bf P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240503 02:00:27.445312 14716 raft_consensus.cc:348] T ec7210e0a04549fab1200b7ab6a133bf P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:27.445726 14716 raft_consensus.cc:374] T ec7210e0a04549fab1200b7ab6a133bf P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:27.445808 14716 raft_consensus.cc:729] T ec7210e0a04549fab1200b7ab6a133bf P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:27.446167 14716 consensus_queue.cc:260] T ec7210e0a04549fab1200b7ab6a133bf P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:27.446548 14716 ts_tablet_manager.cc:1398] T ec7210e0a04549fab1200b7ab6a133bf P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:00:27.448444 14722 tablet_bootstrap.cc:492] T ec7210e0a04549fab1200b7ab6a133bf P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:27.448619 14722 ts_tablet_manager.cc:1367] T ec7210e0a04549fab1200b7ab6a133bf P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20240503 02:00:27.449885 14719 tablet_bootstrap.cc:492] T ab3a5e2e96c44759bca819b1a79a5dc9 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:27.450006 14719 ts_tablet_manager.cc:1367] T ab3a5e2e96c44759bca819b1a79a5dc9 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240503 02:00:27.450026 14722 raft_consensus.cc:348] T ec7210e0a04549fab1200b7ab6a133bf P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:27.450153 14722 raft_consensus.cc:374] T ec7210e0a04549fab1200b7ab6a133bf P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:27.450191 14722 raft_consensus.cc:729] T ec7210e0a04549fab1200b7ab6a133bf P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:27.450313 14722 consensus_queue.cc:260] T ec7210e0a04549fab1200b7ab6a133bf P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:27.450647 14722 ts_tablet_manager.cc:1398] T ec7210e0a04549fab1200b7ab6a133bf P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:27.450654 14719 raft_consensus.cc:348] T ab3a5e2e96c44759bca819b1a79a5dc9 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:27.450795 14719 raft_consensus.cc:374] T ab3a5e2e96c44759bca819b1a79a5dc9 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:27.450863 14719 raft_consensus.cc:729] T ab3a5e2e96c44759bca819b1a79a5dc9 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:27.450990 14719 consensus_queue.cc:260] T ab3a5e2e96c44759bca819b1a79a5dc9 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:27.451261 14719 ts_tablet_manager.cc:1398] T ab3a5e2e96c44759bca819b1a79a5dc9 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:27.477633 6314 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:00:27.483633 8887 tablet_service.cc:1467] Processing CreateTablet for tablet 3d7f87f912be4c9daf0848002c50bfeb (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ec3c6861e0924592b1687dc1dc8245cf]), partition=
I20240503 02:00:27.483711 6831 tablet_service.cc:1467] Processing CreateTablet for tablet 3d7f87f912be4c9daf0848002c50bfeb (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ec3c6861e0924592b1687dc1dc8245cf]), partition=
I20240503 02:00:27.483711 6700 tablet_service.cc:1467] Processing CreateTablet for tablet 3d7f87f912be4c9daf0848002c50bfeb (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ec3c6861e0924592b1687dc1dc8245cf]), partition=
I20240503 02:00:27.483898 8887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d7f87f912be4c9daf0848002c50bfeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:27.483968 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d7f87f912be4c9daf0848002c50bfeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:27.483984 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d7f87f912be4c9daf0848002c50bfeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:27.486099 14714 tablet_bootstrap.cc:492] T 3d7f87f912be4c9daf0848002c50bfeb P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:27.486336 14671 raft_consensus.cc:482] T ec7210e0a04549fab1200b7ab6a133bf P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:27.486469 14671 raft_consensus.cc:504] T ec7210e0a04549fab1200b7ab6a133bf P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:27.486696 14715 tablet_bootstrap.cc:492] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:27.486958 14671 leader_election.cc:288] T ec7210e0a04549fab1200b7ab6a133bf P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:27.487257 14714 tablet_bootstrap.cc:654] T 3d7f87f912be4c9daf0848002c50bfeb P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:27.487634 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7210e0a04549fab1200b7ab6a133bf" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:27.487776 9048 raft_consensus.cc:2454] T ec7210e0a04549fab1200b7ab6a133bf P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:27.487977 14715 tablet_bootstrap.cc:654] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:27.488003 6720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7210e0a04549fab1200b7ab6a133bf" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 02:00:27.488147 6720 raft_consensus.cc:2454] T ec7210e0a04549fab1200b7ab6a133bf P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:27.488518 6523 leader_election.cc:302] T ec7210e0a04549fab1200b7ab6a133bf P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:27.488687 14671 raft_consensus.cc:2781] T ec7210e0a04549fab1200b7ab6a133bf P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:27.488734 14671 raft_consensus.cc:482] T ec7210e0a04549fab1200b7ab6a133bf P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:27.488770 14671 raft_consensus.cc:3037] T ec7210e0a04549fab1200b7ab6a133bf P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:27.489890 14671 raft_consensus.cc:504] T ec7210e0a04549fab1200b7ab6a133bf P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:27.489972 14715 tablet_bootstrap.cc:492] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:27.490046 14715 ts_tablet_manager.cc:1367] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:00:27.490259 14671 leader_election.cc:288] T ec7210e0a04549fab1200b7ab6a133bf P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Requested vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:27.490578 14715 raft_consensus.cc:348] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:27.490749 14715 raft_consensus.cc:374] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:27.490805 14715 raft_consensus.cc:729] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:27.490783 6720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7210e0a04549fab1200b7ab6a133bf" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 02:00:27.490909 6720 raft_consensus.cc:3037] T ec7210e0a04549fab1200b7ab6a133bf P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:27.490950 14715 consensus_queue.cc:260] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:27.491242 14715 ts_tablet_manager.cc:1398] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:27.491771 6720 raft_consensus.cc:2454] T ec7210e0a04549fab1200b7ab6a133bf P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:27.492056 6525 leader_election.cc:302] T ec7210e0a04549fab1200b7ab6a133bf P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:27.492122 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7210e0a04549fab1200b7ab6a133bf" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:27.492215 14671 raft_consensus.cc:2781] T ec7210e0a04549fab1200b7ab6a133bf P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:27.492264 9048 raft_consensus.cc:3037] T ec7210e0a04549fab1200b7ab6a133bf P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:27.492302 14671 raft_consensus.cc:686] T ec7210e0a04549fab1200b7ab6a133bf P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Becoming Leader. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Running, Role: LEADER
I20240503 02:00:27.492478 14671 consensus_queue.cc:237] T ec7210e0a04549fab1200b7ab6a133bf P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:27.493317 9048 raft_consensus.cc:2454] T ec7210e0a04549fab1200b7ab6a133bf P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:27.494212 14722 tablet_bootstrap.cc:492] T 3d7f87f912be4c9daf0848002c50bfeb P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:27.495584 14722 tablet_bootstrap.cc:654] T 3d7f87f912be4c9daf0848002c50bfeb P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:27.495976 6308 catalog_manager.cc:5484] T ec7210e0a04549fab1200b7ab6a133bf P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:27.496734 14722 tablet_bootstrap.cc:492] T 3d7f87f912be4c9daf0848002c50bfeb P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:27.496822 14722 ts_tablet_manager.cc:1367] T 3d7f87f912be4c9daf0848002c50bfeb P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:27.497267 14722 raft_consensus.cc:348] T 3d7f87f912be4c9daf0848002c50bfeb P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:27.497373 14722 raft_consensus.cc:374] T 3d7f87f912be4c9daf0848002c50bfeb P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:27.497402 14722 raft_consensus.cc:729] T 3d7f87f912be4c9daf0848002c50bfeb P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:27.497519 14722 consensus_queue.cc:260] T 3d7f87f912be4c9daf0848002c50bfeb P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:27.497783 14722 ts_tablet_manager.cc:1398] T 3d7f87f912be4c9daf0848002c50bfeb P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:27.499271 14714 tablet_bootstrap.cc:492] T 3d7f87f912be4c9daf0848002c50bfeb P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:27.499377 14714 ts_tablet_manager.cc:1367] T 3d7f87f912be4c9daf0848002c50bfeb P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 02:00:27.499871 14714 raft_consensus.cc:348] T 3d7f87f912be4c9daf0848002c50bfeb P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:27.499990 14714 raft_consensus.cc:374] T 3d7f87f912be4c9daf0848002c50bfeb P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:27.500011 14714 raft_consensus.cc:729] T 3d7f87f912be4c9daf0848002c50bfeb P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:27.500113 14714 consensus_queue.cc:260] T 3d7f87f912be4c9daf0848002c50bfeb P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:27.500336 14714 ts_tablet_manager.cc:1398] T 3d7f87f912be4c9daf0848002c50bfeb P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:27.505743 14386 consensus_queue.cc:1035] T 3c1ce8f08de449c7b4e54aca59a187bc P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:27.506256 14386 consensus_queue.cc:1035] T 3c1ce8f08de449c7b4e54aca59a187bc P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:27.537775 14618 raft_consensus.cc:482] T ab3a5e2e96c44759bca819b1a79a5dc9 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:27.537936 14618 raft_consensus.cc:504] T ab3a5e2e96c44759bca819b1a79a5dc9 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:27.538399 14618 leader_election.cc:288] T ab3a5e2e96c44759bca819b1a79a5dc9 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:27.538817 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab3a5e2e96c44759bca819b1a79a5dc9" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 02:00:27.539008 6456 raft_consensus.cc:2454] T ab3a5e2e96c44759bca819b1a79a5dc9 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:27.539008 6587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab3a5e2e96c44759bca819b1a79a5dc9" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 02:00:27.539137 6587 raft_consensus.cc:2454] T ab3a5e2e96c44759bca819b1a79a5dc9 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:27.539347 9112 leader_election.cc:302] T ab3a5e2e96c44759bca819b1a79a5dc9 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:27.539577 14618 raft_consensus.cc:2781] T ab3a5e2e96c44759bca819b1a79a5dc9 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:27.539628 14618 raft_consensus.cc:482] T ab3a5e2e96c44759bca819b1a79a5dc9 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:27.539664 14618 raft_consensus.cc:3037] T ab3a5e2e96c44759bca819b1a79a5dc9 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:27.540565 14618 raft_consensus.cc:504] T ab3a5e2e96c44759bca819b1a79a5dc9 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:27.540953 14618 leader_election.cc:288] T ab3a5e2e96c44759bca819b1a79a5dc9 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Requested vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:27.541541 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab3a5e2e96c44759bca819b1a79a5dc9" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 02:00:27.541669 6456 raft_consensus.cc:3037] T ab3a5e2e96c44759bca819b1a79a5dc9 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:27.541680 6587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab3a5e2e96c44759bca819b1a79a5dc9" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 02:00:27.541779 6587 raft_consensus.cc:3037] T ab3a5e2e96c44759bca819b1a79a5dc9 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:27.542711 6587 raft_consensus.cc:2454] T ab3a5e2e96c44759bca819b1a79a5dc9 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:27.542711 6456 raft_consensus.cc:2454] T ab3a5e2e96c44759bca819b1a79a5dc9 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:27.543009 9114 leader_election.cc:302] T ab3a5e2e96c44759bca819b1a79a5dc9 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 80fd18ef4d604c86b909d5c008a46b4f; no voters:
I20240503 02:00:27.543197 14618 raft_consensus.cc:2781] T ab3a5e2e96c44759bca819b1a79a5dc9 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:27.543303 14618 raft_consensus.cc:686] T ab3a5e2e96c44759bca819b1a79a5dc9 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Becoming Leader. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Running, Role: LEADER
I20240503 02:00:27.543476 14618 consensus_queue.cc:237] T ab3a5e2e96c44759bca819b1a79a5dc9 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:27.546191 6314 catalog_manager.cc:5484] T ab3a5e2e96c44759bca819b1a79a5dc9 P 2a01ace80baf4057b6bf1995cf88468b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135). New cstate: current_term: 1 leader_uuid: "2a01ace80baf4057b6bf1995cf88468b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:27.561421 14671 raft_consensus.cc:482] T 676e4bd2e5ed4cbea577a2f4210eb781 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:27.561545 14671 raft_consensus.cc:504] T 676e4bd2e5ed4cbea577a2f4210eb781 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:27.561918 14671 leader_election.cc:288] T 676e4bd2e5ed4cbea577a2f4210eb781 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:27.562294 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "676e4bd2e5ed4cbea577a2f4210eb781" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 02:00:27.562426 6851 raft_consensus.cc:2454] T 676e4bd2e5ed4cbea577a2f4210eb781 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:27.562379 8912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "676e4bd2e5ed4cbea577a2f4210eb781" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:27.562520 8912 raft_consensus.cc:2454] T 676e4bd2e5ed4cbea577a2f4210eb781 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:27.562667 6525 leader_election.cc:302] T 676e4bd2e5ed4cbea577a2f4210eb781 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 80fd18ef4d604c86b909d5c008a46b4f; no voters:
I20240503 02:00:27.562848 14671 raft_consensus.cc:2781] T 676e4bd2e5ed4cbea577a2f4210eb781 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:27.562916 14671 raft_consensus.cc:482] T 676e4bd2e5ed4cbea577a2f4210eb781 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:27.562956 14671 raft_consensus.cc:3037] T 676e4bd2e5ed4cbea577a2f4210eb781 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:27.564059 14671 raft_consensus.cc:504] T 676e4bd2e5ed4cbea577a2f4210eb781 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:27.564464 14671 leader_election.cc:288] T 676e4bd2e5ed4cbea577a2f4210eb781 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Requested vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:27.564951 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "676e4bd2e5ed4cbea577a2f4210eb781" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 02:00:27.565074 6849 raft_consensus.cc:3037] T 676e4bd2e5ed4cbea577a2f4210eb781 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:27.565135 8912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "676e4bd2e5ed4cbea577a2f4210eb781" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:27.565261 8912 raft_consensus.cc:3037] T 676e4bd2e5ed4cbea577a2f4210eb781 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:27.566110 6849 raft_consensus.cc:2454] T 676e4bd2e5ed4cbea577a2f4210eb781 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:27.566082 8912 raft_consensus.cc:2454] T 676e4bd2e5ed4cbea577a2f4210eb781 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:27.566515 6525 leader_election.cc:302] T 676e4bd2e5ed4cbea577a2f4210eb781 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:27.566864 14671 raft_consensus.cc:2781] T 676e4bd2e5ed4cbea577a2f4210eb781 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:27.566967 14671 raft_consensus.cc:686] T 676e4bd2e5ed4cbea577a2f4210eb781 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Becoming Leader. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Running, Role: LEADER
I20240503 02:00:27.567095 14671 consensus_queue.cc:237] T 676e4bd2e5ed4cbea577a2f4210eb781 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:27.569877 6314 catalog_manager.cc:5484] T 676e4bd2e5ed4cbea577a2f4210eb781 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } }
W20240503 02:00:27.708192 8982 consensus_peers.cc:485] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:27.713827 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:27.742408 14706 raft_consensus.cc:482] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:27.742544 14706 raft_consensus.cc:504] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:27.742945 14706 leader_election.cc:288] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:27.743229 8912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d7f87f912be4c9daf0848002c50bfeb" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:27.743400 8912 raft_consensus.cc:2454] T 3d7f87f912be4c9daf0848002c50bfeb P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 02:00:27.743527 6720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d7f87f912be4c9daf0848002c50bfeb" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 02:00:27.743705 6720 raft_consensus.cc:2454] T 3d7f87f912be4c9daf0848002c50bfeb P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 02:00:27.744009 6787 leader_election.cc:302] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:27.744174 14706 raft_consensus.cc:2781] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:27.744227 14706 raft_consensus.cc:482] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:27.744267 14706 raft_consensus.cc:3037] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:27.746053 14706 raft_consensus.cc:504] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:27.746441 14706 leader_election.cc:288] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Requested vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:27.746745 8917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d7f87f912be4c9daf0848002c50bfeb" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:27.746928 6720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d7f87f912be4c9daf0848002c50bfeb" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 02:00:27.746974 8917 raft_consensus.cc:3037] T 3d7f87f912be4c9daf0848002c50bfeb P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:27.747046 6720 raft_consensus.cc:3037] T 3d7f87f912be4c9daf0848002c50bfeb P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:27.747977 8917 raft_consensus.cc:2454] T 3d7f87f912be4c9daf0848002c50bfeb P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 02:00:27.747977 6720 raft_consensus.cc:2454] T 3d7f87f912be4c9daf0848002c50bfeb P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 02:00:27.748309 6787 leader_election.cc:302] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:27.748524 14706 raft_consensus.cc:2781] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:27.748621 14706 raft_consensus.cc:686] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Becoming Leader. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Running, Role: LEADER
I20240503 02:00:27.748826 14706 consensus_queue.cc:237] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:27.751428 6314 catalog_manager.cc:5484] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d reported cstate change: term changed from 0 to 1, leader changed from <none> to 492448570cc94ed1879feb2ec3b9051d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:27.886631 6314 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:27.890774 8917 raft_consensus.cc:1261] T 3d7f87f912be4c9daf0848002c50bfeb P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:27.890774 6720 raft_consensus.cc:1261] T 3d7f87f912be4c9daf0848002c50bfeb P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:27.891194 14706 consensus_queue.cc:1035] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:27.891353 14472 consensus_queue.cc:1035] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:27.894731 6314 catalog_manager.cc:5576] TS c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655) does not have the latest schema for tablet 3d7f87f912be4c9daf0848002c50bfeb (table rebalancer_dynamic_table [id=ec3c6861e0924592b1687dc1dc8245cf]). Expected version 1 got 0
I20240503 02:00:27.895094 6309 catalog_manager.cc:5576] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211) does not have the latest schema for tablet 3d7f87f912be4c9daf0848002c50bfeb (table rebalancer_dynamic_table [id=ec3c6861e0924592b1687dc1dc8245cf]). Expected version 1 got 0
I20240503 02:00:27.895969 14744 tablet.cc:1722] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:27.896399 14744 tablet.cc:1620] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d: MemRowSet was empty: no flush needed.
I20240503 02:00:27.896762 14745 tablet.cc:1722] T 3d7f87f912be4c9daf0848002c50bfeb P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:27.897197 14745 tablet.cc:1620] T 3d7f87f912be4c9daf0848002c50bfeb P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
I20240503 02:00:27.899051 14746 tablet.cc:1722] T 3d7f87f912be4c9daf0848002c50bfeb P c2ac2eb7cc264047b9ab32fe3a4fc025: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:27.899359 14746 tablet.cc:1620] T 3d7f87f912be4c9daf0848002c50bfeb P c2ac2eb7cc264047b9ab32fe3a4fc025: MemRowSet was empty: no flush needed.
I20240503 02:00:27.899910 6314 catalog_manager.cc:5576] TS c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655) does not have the latest schema for tablet 3d7f87f912be4c9daf0848002c50bfeb (table rebalancer_dynamic_table [id=ec3c6861e0924592b1687dc1dc8245cf]). Expected version 1 got 0
I20240503 02:00:27.903146 6309 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=ec3c6861e0924592b1687dc1dc8245cf] alter complete (version 1)
I20240503 02:00:27.936008 14649 consensus_queue.cc:1035] T ec7210e0a04549fab1200b7ab6a133bf P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:27.941413 14671 consensus_queue.cc:1035] T ec7210e0a04549fab1200b7ab6a133bf P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:27.983474 14618 consensus_queue.cc:1035] T ab3a5e2e96c44759bca819b1a79a5dc9 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:27.988204 14751 consensus_queue.cc:1035] T ab3a5e2e96c44759bca819b1a79a5dc9 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:28.008492 6314 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:28.014006 14744 tablet.cc:1722] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:28.014381 14744 tablet.cc:1620] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d: MemRowSet was empty: no flush needed.
I20240503 02:00:28.014743 14746 tablet.cc:1722] T 3d7f87f912be4c9daf0848002c50bfeb P c2ac2eb7cc264047b9ab32fe3a4fc025: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:28.015033 14745 tablet.cc:1722] T 3d7f87f912be4c9daf0848002c50bfeb P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:28.015110 14746 tablet.cc:1620] T 3d7f87f912be4c9daf0848002c50bfeb P c2ac2eb7cc264047b9ab32fe3a4fc025: MemRowSet was empty: no flush needed.
I20240503 02:00:28.015319 14745 tablet.cc:1620] T 3d7f87f912be4c9daf0848002c50bfeb P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
I20240503 02:00:28.022046 6308 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=ec3c6861e0924592b1687dc1dc8245cf] alter complete (version 2)
W20240503 02:00:28.061223 6785 consensus_peers.cc:485] T c3485701bf3b48f5ae2e1568afeed133 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 02:00:28.098096 6314 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_extra_table_72"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:28.105091 9028 tablet_service.cc:1467] Processing CreateTablet for tablet 2a3c9f1f41dd43d9878f97388523a2cf (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=e3918a1ea1744c97a837f7f2acb7a8d8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:28.105091 6438 tablet_service.cc:1467] Processing CreateTablet for tablet bdc99a9b4d144059bfda89af3c033803 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=e3918a1ea1744c97a837f7f2acb7a8d8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:28.105195 6831 tablet_service.cc:1467] Processing CreateTablet for tablet bdc99a9b4d144059bfda89af3c033803 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=e3918a1ea1744c97a837f7f2acb7a8d8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:28.105412 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a3c9f1f41dd43d9878f97388523a2cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:28.105412 6438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdc99a9b4d144059bfda89af3c033803. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:28.105482 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdc99a9b4d144059bfda89af3c033803. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:28.108029 9157 tablet_service.cc:1467] Processing CreateTablet for tablet bdc99a9b4d144059bfda89af3c033803 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=e3918a1ea1744c97a837f7f2acb7a8d8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:28.108343 9157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdc99a9b4d144059bfda89af3c033803. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:28.108748 14756 tablet_bootstrap.cc:492] T 2a3c9f1f41dd43d9878f97388523a2cf P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:28.109019 14755 tablet_bootstrap.cc:492] T bdc99a9b4d144059bfda89af3c033803 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:28.109191 14754 tablet_bootstrap.cc:492] T bdc99a9b4d144059bfda89af3c033803 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:28.109282 6569 tablet_service.cc:1467] Processing CreateTablet for tablet 2a3c9f1f41dd43d9878f97388523a2cf (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=e3918a1ea1744c97a837f7f2acb7a8d8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:28.109539 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a3c9f1f41dd43d9878f97388523a2cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:28.110321 14754 tablet_bootstrap.cc:654] T bdc99a9b4d144059bfda89af3c033803 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:28.110867 14755 tablet_bootstrap.cc:654] T bdc99a9b4d144059bfda89af3c033803 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:28.112135 14756 tablet_bootstrap.cc:654] T 2a3c9f1f41dd43d9878f97388523a2cf P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:28.112560 14759 tablet_bootstrap.cc:492] T 2a3c9f1f41dd43d9878f97388523a2cf P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:28.112653 9157 tablet_service.cc:1467] Processing CreateTablet for tablet 2a3c9f1f41dd43d9878f97388523a2cf (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=e3918a1ea1744c97a837f7f2acb7a8d8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:28.112916 9157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a3c9f1f41dd43d9878f97388523a2cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:28.113624 14759 tablet_bootstrap.cc:654] T 2a3c9f1f41dd43d9878f97388523a2cf P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:28.113817 6438 tablet_service.cc:1467] Processing CreateTablet for tablet 2e87caa8a93d43a7b61e1596b12b0906 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=e3918a1ea1744c97a837f7f2acb7a8d8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:28.114105 6438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e87caa8a93d43a7b61e1596b12b0906. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:28.114259 9159 tablet_service.cc:1467] Processing CreateTablet for tablet 2e87caa8a93d43a7b61e1596b12b0906 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=e3918a1ea1744c97a837f7f2acb7a8d8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:28.114508 9159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e87caa8a93d43a7b61e1596b12b0906. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:28.116690 6700 tablet_service.cc:1467] Processing CreateTablet for tablet 2e87caa8a93d43a7b61e1596b12b0906 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=e3918a1ea1744c97a837f7f2acb7a8d8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:28.116984 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e87caa8a93d43a7b61e1596b12b0906. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:00:28.117487 6654 consensus_peers.cc:485] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 02:00:28.117823 14757 tablet_bootstrap.cc:492] T bdc99a9b4d144059bfda89af3c033803 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:28.119045 14757 tablet_bootstrap.cc:654] T bdc99a9b4d144059bfda89af3c033803 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:28.120404 14762 tablet_bootstrap.cc:492] T 2e87caa8a93d43a7b61e1596b12b0906 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:28.120692 14757 tablet_bootstrap.cc:492] T bdc99a9b4d144059bfda89af3c033803 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:28.120815 14757 ts_tablet_manager.cc:1367] T bdc99a9b4d144059bfda89af3c033803 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:28.121338 14757 raft_consensus.cc:348] T bdc99a9b4d144059bfda89af3c033803 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:28.121459 14757 raft_consensus.cc:374] T bdc99a9b4d144059bfda89af3c033803 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:28.121474 14762 tablet_bootstrap.cc:654] T 2e87caa8a93d43a7b61e1596b12b0906 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:28.121490 14757 raft_consensus.cc:729] T bdc99a9b4d144059bfda89af3c033803 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:28.121613 14757 consensus_queue.cc:260] T bdc99a9b4d144059bfda89af3c033803 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:28.121717 14755 tablet_bootstrap.cc:492] T bdc99a9b4d144059bfda89af3c033803 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:28.121824 14755 ts_tablet_manager.cc:1367] T bdc99a9b4d144059bfda89af3c033803 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 02:00:28.121896 14756 tablet_bootstrap.cc:492] T 2a3c9f1f41dd43d9878f97388523a2cf P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:28.121891 14757 ts_tablet_manager.cc:1398] T bdc99a9b4d144059bfda89af3c033803 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:28.121978 14756 ts_tablet_manager.cc:1367] T 2a3c9f1f41dd43d9878f97388523a2cf P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.001s
I20240503 02:00:28.122064 14757 tablet_bootstrap.cc:492] T 2e87caa8a93d43a7b61e1596b12b0906 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:28.122335 14755 raft_consensus.cc:348] T bdc99a9b4d144059bfda89af3c033803 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:28.122443 14755 raft_consensus.cc:374] T bdc99a9b4d144059bfda89af3c033803 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:28.122473 14755 raft_consensus.cc:729] T bdc99a9b4d144059bfda89af3c033803 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:28.122582 14756 raft_consensus.cc:348] T 2a3c9f1f41dd43d9878f97388523a2cf P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:28.122714 14756 raft_consensus.cc:374] T 2a3c9f1f41dd43d9878f97388523a2cf P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:28.122746 14756 raft_consensus.cc:729] T 2a3c9f1f41dd43d9878f97388523a2cf P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:28.122736 14755 consensus_queue.cc:260] T bdc99a9b4d144059bfda89af3c033803 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:28.122990 14757 tablet_bootstrap.cc:654] T 2e87caa8a93d43a7b61e1596b12b0906 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:28.123059 14756 consensus_queue.cc:260] T 2a3c9f1f41dd43d9878f97388523a2cf P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:28.123337 14755 ts_tablet_manager.cc:1398] T bdc99a9b4d144059bfda89af3c033803 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:28.123525 14755 tablet_bootstrap.cc:492] T 2e87caa8a93d43a7b61e1596b12b0906 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:28.124644 14755 tablet_bootstrap.cc:654] T 2e87caa8a93d43a7b61e1596b12b0906 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:28.125591 14756 ts_tablet_manager.cc:1398] T 2a3c9f1f41dd43d9878f97388523a2cf P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240503 02:00:28.127133 14754 tablet_bootstrap.cc:492] T bdc99a9b4d144059bfda89af3c033803 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:28.127254 14754 ts_tablet_manager.cc:1367] T bdc99a9b4d144059bfda89af3c033803 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240503 02:00:28.127740 14754 raft_consensus.cc:348] T bdc99a9b4d144059bfda89af3c033803 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:28.127859 14754 raft_consensus.cc:374] T bdc99a9b4d144059bfda89af3c033803 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:28.127892 14754 raft_consensus.cc:729] T bdc99a9b4d144059bfda89af3c033803 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:28.127995 14754 consensus_queue.cc:260] T bdc99a9b4d144059bfda89af3c033803 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:28.128273 14754 ts_tablet_manager.cc:1398] T bdc99a9b4d144059bfda89af3c033803 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:28.130029 14757 tablet_bootstrap.cc:492] T 2e87caa8a93d43a7b61e1596b12b0906 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:28.130131 14757 ts_tablet_manager.cc:1367] T 2e87caa8a93d43a7b61e1596b12b0906 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 02:00:28.130753 14757 raft_consensus.cc:348] T 2e87caa8a93d43a7b61e1596b12b0906 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:28.130898 14757 raft_consensus.cc:374] T 2e87caa8a93d43a7b61e1596b12b0906 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:28.130934 14757 raft_consensus.cc:729] T 2e87caa8a93d43a7b61e1596b12b0906 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:28.131052 14757 consensus_queue.cc:260] T 2e87caa8a93d43a7b61e1596b12b0906 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:28.131345 14757 ts_tablet_manager.cc:1398] T 2e87caa8a93d43a7b61e1596b12b0906 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:28.131562 14757 tablet_bootstrap.cc:492] T 2a3c9f1f41dd43d9878f97388523a2cf P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:28.132869 14757 tablet_bootstrap.cc:654] T 2a3c9f1f41dd43d9878f97388523a2cf P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:28.134801 14757 tablet_bootstrap.cc:492] T 2a3c9f1f41dd43d9878f97388523a2cf P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:28.135047 14757 ts_tablet_manager.cc:1367] T 2a3c9f1f41dd43d9878f97388523a2cf P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
W20240503 02:00:28.135251 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:28.135919 14757 raft_consensus.cc:348] T 2a3c9f1f41dd43d9878f97388523a2cf P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:28.136585 14757 raft_consensus.cc:374] T 2a3c9f1f41dd43d9878f97388523a2cf P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:28.136699 14757 raft_consensus.cc:729] T 2a3c9f1f41dd43d9878f97388523a2cf P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:28.137076 14757 consensus_queue.cc:260] T 2a3c9f1f41dd43d9878f97388523a2cf P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:28.137405 14755 tablet_bootstrap.cc:492] T 2e87caa8a93d43a7b61e1596b12b0906 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:28.137660 14755 ts_tablet_manager.cc:1367] T 2e87caa8a93d43a7b61e1596b12b0906 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 02:00:28.137733 14757 ts_tablet_manager.cc:1398] T 2a3c9f1f41dd43d9878f97388523a2cf P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:28.137928 14671 consensus_queue.cc:1035] T 676e4bd2e5ed4cbea577a2f4210eb781 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:28.138316 14755 raft_consensus.cc:348] T 2e87caa8a93d43a7b61e1596b12b0906 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:28.138437 14755 raft_consensus.cc:374] T 2e87caa8a93d43a7b61e1596b12b0906 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:28.138471 14755 raft_consensus.cc:729] T 2e87caa8a93d43a7b61e1596b12b0906 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:28.138585 14755 consensus_queue.cc:260] T 2e87caa8a93d43a7b61e1596b12b0906 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:28.138995 14755 ts_tablet_manager.cc:1398] T 2e87caa8a93d43a7b61e1596b12b0906 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240503 02:00:28.139750 6785 consensus_peers.cc:485] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 02:00:28.140404 6308 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:28.141166 14691 raft_consensus.cc:482] T 2e87caa8a93d43a7b61e1596b12b0906 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:28.141276 14691 raft_consensus.cc:504] T 2e87caa8a93d43a7b61e1596b12b0906 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:28.141695 14691 leader_election.cc:288] T 2e87caa8a93d43a7b61e1596b12b0906 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:28.142423 6720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e87caa8a93d43a7b61e1596b12b0906" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
W20240503 02:00:28.142866 9115 leader_election.cc:341] T 2e87caa8a93d43a7b61e1596b12b0906 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:00:28.143303 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e87caa8a93d43a7b61e1596b12b0906" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 02:00:28.143435 6456 raft_consensus.cc:2454] T 2e87caa8a93d43a7b61e1596b12b0906 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:28.143759 9112 leader_election.cc:302] T 2e87caa8a93d43a7b61e1596b12b0906 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, dec3e4a04d604f7eaaa3f077d081218a; no voters: c2ac2eb7cc264047b9ab32fe3a4fc025
I20240503 02:00:28.143980 14691 raft_consensus.cc:2781] T 2e87caa8a93d43a7b61e1596b12b0906 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:28.144042 14691 raft_consensus.cc:482] T 2e87caa8a93d43a7b61e1596b12b0906 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:28.144080 14691 raft_consensus.cc:3037] T 2e87caa8a93d43a7b61e1596b12b0906 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:28.145161 14691 raft_consensus.cc:504] T 2e87caa8a93d43a7b61e1596b12b0906 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:28.145560 14691 leader_election.cc:288] T 2e87caa8a93d43a7b61e1596b12b0906 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Requested vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:28.146287 6720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e87caa8a93d43a7b61e1596b12b0906" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
W20240503 02:00:28.146807 9115 leader_election.cc:341] T 2e87caa8a93d43a7b61e1596b12b0906 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:00:28.146994 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e87caa8a93d43a7b61e1596b12b0906" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 02:00:28.147114 6456 raft_consensus.cc:3037] T 2e87caa8a93d43a7b61e1596b12b0906 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:28.148177 6456 raft_consensus.cc:2454] T 2e87caa8a93d43a7b61e1596b12b0906 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:28.149161 14744 tablet.cc:1722] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:28.150575 9112 leader_election.cc:302] T 2e87caa8a93d43a7b61e1596b12b0906 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, dec3e4a04d604f7eaaa3f077d081218a; no voters: c2ac2eb7cc264047b9ab32fe3a4fc025
I20240503 02:00:28.150861 14759 tablet_bootstrap.cc:492] T 2a3c9f1f41dd43d9878f97388523a2cf P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:28.150943 14759 ts_tablet_manager.cc:1367] T 2a3c9f1f41dd43d9878f97388523a2cf P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.038s user 0.002s sys 0.000s
I20240503 02:00:28.151384 14759 raft_consensus.cc:348] T 2a3c9f1f41dd43d9878f97388523a2cf P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:28.151484 14759 raft_consensus.cc:374] T 2a3c9f1f41dd43d9878f97388523a2cf P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:28.151504 14759 raft_consensus.cc:729] T 2a3c9f1f41dd43d9878f97388523a2cf P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:28.151592 14759 consensus_queue.cc:260] T 2a3c9f1f41dd43d9878f97388523a2cf P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:28.151865 14759 ts_tablet_manager.cc:1398] T 2a3c9f1f41dd43d9878f97388523a2cf P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:28.153400 14649 consensus_queue.cc:1035] T 676e4bd2e5ed4cbea577a2f4210eb781 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:28.155495 14691 raft_consensus.cc:2781] T 2e87caa8a93d43a7b61e1596b12b0906 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:28.155611 14691 raft_consensus.cc:686] T 2e87caa8a93d43a7b61e1596b12b0906 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Becoming Leader. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Running, Role: LEADER
I20240503 02:00:28.155748 14691 consensus_queue.cc:237] T 2e87caa8a93d43a7b61e1596b12b0906 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:28.155807 14745 tablet.cc:1722] T 3d7f87f912be4c9daf0848002c50bfeb P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:28.157253 14746 tablet.cc:1722] T 3d7f87f912be4c9daf0848002c50bfeb P c2ac2eb7cc264047b9ab32fe3a4fc025: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:28.158708 14762 tablet_bootstrap.cc:492] T 2e87caa8a93d43a7b61e1596b12b0906 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:28.158814 14762 ts_tablet_manager.cc:1367] T 2e87caa8a93d43a7b61e1596b12b0906 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.038s user 0.002s sys 0.000s
I20240503 02:00:28.162905 14762 raft_consensus.cc:348] T 2e87caa8a93d43a7b61e1596b12b0906 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:28.163057 14762 raft_consensus.cc:374] T 2e87caa8a93d43a7b61e1596b12b0906 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:28.163090 14762 raft_consensus.cc:729] T 2e87caa8a93d43a7b61e1596b12b0906 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:28.163223 14762 consensus_queue.cc:260] T 2e87caa8a93d43a7b61e1596b12b0906 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:28.163547 14762 ts_tablet_manager.cc:1398] T 2e87caa8a93d43a7b61e1596b12b0906 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240503 02:00:28.163930 6314 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=ec3c6861e0924592b1687dc1dc8245cf] alter complete (version 3)
I20240503 02:00:28.170946 6313 catalog_manager.cc:5484] T 2e87caa8a93d43a7b61e1596b12b0906 P 2a01ace80baf4057b6bf1995cf88468b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135). New cstate: current_term: 1 leader_uuid: "2a01ace80baf4057b6bf1995cf88468b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:28.206408 14751 raft_consensus.cc:482] T bdc99a9b4d144059bfda89af3c033803 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:28.206553 14751 raft_consensus.cc:504] T bdc99a9b4d144059bfda89af3c033803 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:28.206936 14751 leader_election.cc:288] T bdc99a9b4d144059bfda89af3c033803 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:28.207228 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdc99a9b4d144059bfda89af3c033803" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 02:00:28.207362 6456 raft_consensus.cc:2454] T bdc99a9b4d144059bfda89af3c033803 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:28.207520 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdc99a9b4d144059bfda89af3c033803" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 02:00:28.207634 9112 leader_election.cc:302] T bdc99a9b4d144059bfda89af3c033803 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:28.207653 6849 raft_consensus.cc:2454] T bdc99a9b4d144059bfda89af3c033803 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:28.207798 14751 raft_consensus.cc:2781] T bdc99a9b4d144059bfda89af3c033803 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:28.207847 14751 raft_consensus.cc:482] T bdc99a9b4d144059bfda89af3c033803 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:28.207882 14751 raft_consensus.cc:3037] T bdc99a9b4d144059bfda89af3c033803 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:28.208952 14751 raft_consensus.cc:504] T bdc99a9b4d144059bfda89af3c033803 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:28.209293 14751 leader_election.cc:288] T bdc99a9b4d144059bfda89af3c033803 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Requested vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:28.210923 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdc99a9b4d144059bfda89af3c033803" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 02:00:28.211053 6849 raft_consensus.cc:3037] T bdc99a9b4d144059bfda89af3c033803 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:28.211258 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdc99a9b4d144059bfda89af3c033803" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 02:00:28.211380 6456 raft_consensus.cc:3037] T bdc99a9b4d144059bfda89af3c033803 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:28.212173 6456 raft_consensus.cc:2454] T bdc99a9b4d144059bfda89af3c033803 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:28.212180 6849 raft_consensus.cc:2454] T bdc99a9b4d144059bfda89af3c033803 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:28.212572 9115 leader_election.cc:302] T bdc99a9b4d144059bfda89af3c033803 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 492448570cc94ed1879feb2ec3b9051d; no voters:
I20240503 02:00:28.212901 14751 raft_consensus.cc:2781] T bdc99a9b4d144059bfda89af3c033803 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:28.212990 14751 raft_consensus.cc:686] T bdc99a9b4d144059bfda89af3c033803 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Becoming Leader. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Running, Role: LEADER
I20240503 02:00:28.213131 14751 consensus_queue.cc:237] T bdc99a9b4d144059bfda89af3c033803 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:28.215942 6309 catalog_manager.cc:5484] T bdc99a9b4d144059bfda89af3c033803 P 2a01ace80baf4057b6bf1995cf88468b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135). New cstate: current_term: 1 leader_uuid: "2a01ace80baf4057b6bf1995cf88468b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:28.221616 14649 raft_consensus.cc:482] T 2a3c9f1f41dd43d9878f97388523a2cf P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:28.221733 14649 raft_consensus.cc:504] T 2a3c9f1f41dd43d9878f97388523a2cf P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:28.222119 14649 leader_election.cc:288] T 2a3c9f1f41dd43d9878f97388523a2cf P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:28.222559 9177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a3c9f1f41dd43d9878f97388523a2cf" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:28.222576 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a3c9f1f41dd43d9878f97388523a2cf" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:28.222710 9048 raft_consensus.cc:2454] T 2a3c9f1f41dd43d9878f97388523a2cf P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:28.222705 9177 raft_consensus.cc:2454] T 2a3c9f1f41dd43d9878f97388523a2cf P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:28.223043 6522 leader_election.cc:302] T 2a3c9f1f41dd43d9878f97388523a2cf P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 80fd18ef4d604c86b909d5c008a46b4f; no voters:
I20240503 02:00:28.223343 14649 raft_consensus.cc:2781] T 2a3c9f1f41dd43d9878f97388523a2cf P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:28.223397 14649 raft_consensus.cc:482] T 2a3c9f1f41dd43d9878f97388523a2cf P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:28.223430 14649 raft_consensus.cc:3037] T 2a3c9f1f41dd43d9878f97388523a2cf P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:28.224694 14649 raft_consensus.cc:504] T 2a3c9f1f41dd43d9878f97388523a2cf P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:28.225078 14649 leader_election.cc:288] T 2a3c9f1f41dd43d9878f97388523a2cf P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Requested vote from peers 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:28.225437 9177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a3c9f1f41dd43d9878f97388523a2cf" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:28.225557 9177 raft_consensus.cc:3037] T 2a3c9f1f41dd43d9878f97388523a2cf P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:28.225605 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a3c9f1f41dd43d9878f97388523a2cf" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:28.225708 9048 raft_consensus.cc:3037] T 2a3c9f1f41dd43d9878f97388523a2cf P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:28.226581 9177 raft_consensus.cc:2454] T 2a3c9f1f41dd43d9878f97388523a2cf P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:28.226581 9048 raft_consensus.cc:2454] T 2a3c9f1f41dd43d9878f97388523a2cf P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:28.226934 6522 leader_election.cc:302] T 2a3c9f1f41dd43d9878f97388523a2cf P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 80fd18ef4d604c86b909d5c008a46b4f; no voters:
I20240503 02:00:28.227316 14671 raft_consensus.cc:2781] T 2a3c9f1f41dd43d9878f97388523a2cf P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:28.227403 14671 raft_consensus.cc:686] T 2a3c9f1f41dd43d9878f97388523a2cf P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Becoming Leader. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Running, Role: LEADER
I20240503 02:00:28.227528 14671 consensus_queue.cc:237] T 2a3c9f1f41dd43d9878f97388523a2cf P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:28.231774 6309 catalog_manager.cc:5484] T 2a3c9f1f41dd43d9878f97388523a2cf P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:28.267802 6309 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:28.267961 6309 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:28.270243 6309 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet 3d7f87f912be4c9daf0848002c50bfeb
I20240503 02:00:28.271003 6831 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d7f87f912be4c9daf0848002c50bfeb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:28 UTC) from {username='slave'} at 127.0.0.1:52812
I20240503 02:00:28.271073 8887 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d7f87f912be4c9daf0848002c50bfeb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:28 UTC) from {username='slave'} at 127.0.0.1:40296
I20240503 02:00:28.271131 6700 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d7f87f912be4c9daf0848002c50bfeb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:28 UTC) from {username='slave'} at 127.0.0.1:40336
I20240503 02:00:28.271430 14775 tablet_replica.cc:330] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d: stopping tablet replica
I20240503 02:00:28.271479 14777 tablet_replica.cc:330] T 3d7f87f912be4c9daf0848002c50bfeb P c2ac2eb7cc264047b9ab32fe3a4fc025: stopping tablet replica
I20240503 02:00:28.271764 14777 raft_consensus.cc:2229] T 3d7f87f912be4c9daf0848002c50bfeb P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:28.271765 14775 raft_consensus.cc:2229] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:28.271983 14777 raft_consensus.cc:2258] T 3d7f87f912be4c9daf0848002c50bfeb P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:28.272027 14775 raft_consensus.cc:2258] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:28.272293 14776 tablet_replica.cc:330] T 3d7f87f912be4c9daf0848002c50bfeb P 9942306855404cfd891adf227dd65eb2: stopping tablet replica
I20240503 02:00:28.272467 14776 raft_consensus.cc:2229] T 3d7f87f912be4c9daf0848002c50bfeb P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:28.272612 14776 raft_consensus.cc:2258] T 3d7f87f912be4c9daf0848002c50bfeb P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:28.272954 14777 ts_tablet_manager.cc:1875] T 3d7f87f912be4c9daf0848002c50bfeb P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:28.273129 14775 ts_tablet_manager.cc:1875] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:28.274989 14776 ts_tablet_manager.cc:1875] T 3d7f87f912be4c9daf0848002c50bfeb P 9942306855404cfd891adf227dd65eb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:28.275708 14777 ts_tablet_manager.cc:1888] T 3d7f87f912be4c9daf0848002c50bfeb P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:28.275961 14777 log.cc:1196] T 3d7f87f912be4c9daf0848002c50bfeb P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/3d7f87f912be4c9daf0848002c50bfeb
I20240503 02:00:28.276329 14777 ts_tablet_manager.cc:1909] T 3d7f87f912be4c9daf0848002c50bfeb P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting consensus metadata
I20240503 02:00:28.277468 6303 catalog_manager.cc:4830] TS c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655): tablet 3d7f87f912be4c9daf0848002c50bfeb (table rebalancer_dynamic_table_new_name [id=ec3c6861e0924592b1687dc1dc8245cf]) successfully deleted
I20240503 02:00:28.277947 14776 ts_tablet_manager.cc:1888] T 3d7f87f912be4c9daf0848002c50bfeb P 9942306855404cfd891adf227dd65eb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:28.278038 14776 log.cc:1196] T 3d7f87f912be4c9daf0848002c50bfeb P 9942306855404cfd891adf227dd65eb2: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/3d7f87f912be4c9daf0848002c50bfeb
I20240503 02:00:28.278302 14776 ts_tablet_manager.cc:1909] T 3d7f87f912be4c9daf0848002c50bfeb P 9942306855404cfd891adf227dd65eb2: Deleting consensus metadata
I20240503 02:00:28.279558 14775 ts_tablet_manager.cc:1888] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:28.279652 14775 log.cc:1196] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/3d7f87f912be4c9daf0848002c50bfeb
I20240503 02:00:28.279992 14775 ts_tablet_manager.cc:1909] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d: Deleting consensus metadata
I20240503 02:00:28.280754 6303 catalog_manager.cc:4830] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211): tablet 3d7f87f912be4c9daf0848002c50bfeb (table rebalancer_dynamic_table_new_name [id=ec3c6861e0924592b1687dc1dc8245cf]) successfully deleted
I20240503 02:00:28.280946 6303 catalog_manager.cc:4830] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719): tablet 3d7f87f912be4c9daf0848002c50bfeb (table rebalancer_dynamic_table_new_name [id=ec3c6861e0924592b1687dc1dc8245cf]) successfully deleted
W20240503 02:00:28.330543 6523 consensus_peers.cc:485] T 4385d51d11574192883d01376e2d5620 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 02:00:28.365609 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu with pid 6376
I20240503 02:00:28.376087 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu with pid 6507
I20240503 02:00:28.386612 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu with pid 6638
W20240503 02:00:28.390647 8850 consensus_peers.cc:485] T 7da8f614a70e4177b70e7c2f31d22a10 P 9942306855404cfd891adf227dd65eb2 -> Peer dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): Couldn't send request to peer dec3e4a04d604f7eaaa3f077d081218a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33093: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:28.392160 8981 consensus_peers.cc:485] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): Couldn't send request to peer dec3e4a04d604f7eaaa3f077d081218a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33093: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:28.397145 8853 connection.cc:535] client connection to 127.0.99.131:42655 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:00:28.397188 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu with pid 6769
W20240503 02:00:28.399657 8853 consensus_peers.cc:485] T f55cb3da9c874387a65eff5de0903331 P 9942306855404cfd891adf227dd65eb2 -> Peer c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655): Couldn't send request to peer c2ac2eb7cc264047b9ab32fe3a4fc025. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:42655: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:28.408092 8853 consensus_peers.cc:485] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2 -> Peer c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655): Couldn't send request to peer c2ac2eb7cc264047b9ab32fe3a4fc025. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:42655: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:28.409587 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu with pid 8835
W20240503 02:00:28.416628 9115 connection.cc:535] client connection to 127.0.99.133:33211 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:00:28.417423 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu with pid 8966
W20240503 02:00:28.418488 9115 consensus_peers.cc:485] T b1740b877c1e4372a0337d99f5bcab0e P 2a01ace80baf4057b6bf1995cf88468b -> Peer 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211): Couldn't send request to peer 9942306855404cfd891adf227dd65eb2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33211: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:28.424235 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu with pid 9097
I20240503 02:00:28.431548 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu with pid 6285
2024-05-03T02:00:28Z chronyd exiting
I20240503 02:00:28.448989 398 test_util.cc:175] -----------------------------------------------
I20240503 02:00:28.449059 398 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0
LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer/idx8_rf4_t8_l_3_1_1_1: /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tools/rebalancer_tool-test.cc:2186: Failure
Value of: s.ok()
Actual: false
Expected: true
Runtime error: /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Locations load summary:
Location | Load
----------+-----------
/A | 24.000000
/B | 0.000000
/C | 24.000000
/D | 0.000000
--------------------------------------------------
Location: /A
--------------------------------------------------
Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 24
Maximum Replica Count | 24
Average Replica Count | 24.000000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 0
Maximum | 0
Average | 0.000000
--------------------------------------------------
Location: /B
--------------------------------------------------
Per-server replica distribution summary:
Statistic | Value
-----------------------+----------
Minimum Replica Count | 0
Maximum Replica Count | 0
Average Replica Count | 0.000000
Per-table replica distribution summary:
Replica Skew | Value
--------------+-------
N/A | N/A
--------------------------------------------------
Location: /C
--------------------------------------------------
Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 24
Maximum Replica Count | 24
Average Replica Count | 24.000000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 0
Maximum | 0
Average | 0.000000
--------------------------------------------------
Location: /D
--------------------------------------------------
Per-server replica distribution summary:
Statistic | Value
-----------------------+----------
Minimum Replica Count | 0
Maximum Replica Count | 0
Average Replica Count | 0.000000
Per-table replica distribution summary:
Replica Skew | Value
--------------+-------
N/A | N/A
Placement policy violations:
Location | Number of non-complying tables | Number of non-complying tablets
----------+--------------------------------+---------------------------------
/A | 8 | 24
stderr: W20240503 02:01:55.678258 25214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:01:55.678570 25214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:01:55.682066 25214 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:01:55.978865 25214 placement_policy_util.cc:399] tablet 8a90cf7d371747eba4191be5d3b1ef07: detected 3 of 4 replicas at location /A
I20240503 02:01:55.979004 25214 placement_policy_util.cc:399] tablet 57346221ef5e4b4fa69c127345c69821: detected 3 of 4 replicas at location /A
I20240503 02:01:55.979038 25214 placement_policy_util.cc:399] tablet 43643b2168db4925875aa995ce71ee94: detected 3 of 4 replicas at location /A
I20240503 02:01:55.979060 25214 placement_policy_util.cc:399] tablet 43cdbbc431c243619f5322ceff8cf403: detected 3 of 4 replicas at location /A
I20240503 02:01:55.979086 25214 placement_policy_util.cc:399] tablet d44684f0c09f4ebc98204e592921699a: detected 3 of 4 replicas at location /A
I20240503 02:01:55.979120 25214 placement_policy_util.cc:399] tablet cf54dd831d41447cba6d8d4f8789900b: detected 3 of 4 replicas at location /A
I20240503 02:01:55.979152 25214 placement_policy_util.cc:399] tablet 7ac40a74e5064395b92050be4f9493ed: detected 3 of 4 replicas at location /A
I20240503 02:01:55.979173 25214 placement_policy_util.cc:399] tablet d397654530844ca9aef380669fd0ad7e: detected 3 of 4 replicas at location /A
I20240503 02:01:55.979193 25214 placement_policy_util.cc:399] tablet 917c41c20407484886c6ecca69f2a99e: detected 3 of 4 replicas at location /A
I20240503 02:01:55.979218 25214 placement_policy_util.cc:399] tablet b59d722c2cdf40a5a023f5a1eecfc69a: detected 3 of 4 replicas at location /A
I20240503 02:01:55.979238 25214 placement_policy_util.cc:399] tablet cf858dafc71d45dcb649c5a09f55ab23: detected 3 of 4 replicas at location /A
I20240503 02:01:55.979259 25214 placement_policy_util.cc:399] tablet f6e2af0050854514b0f6844dc79eed90: detected 3 of 4 replicas at location /A
I20240503 02:01:55.979277 25214 placement_policy_util.cc:399] tablet 9e63310e1e2b48d5bf34d4957b53438c: detected 3 of 4 replicas at location /A
I20240503 02:01:55.979298 25214 placement_policy_util.cc:399] tablet ba26bfbf4f424b83b0093cc55ed70355: detected 3 of 4 replicas at location /A
I20240503 02:01:55.979318 25214 placement_policy_util.cc:399] tablet cbdf4ac35a10417f8887dc3c4eed6634: detected 3 of 4 replicas at location /A
I20240503 02:01:55.979339 25214 placement_policy_util.cc:399] tablet f82fbb08e52244d8b3b3905f12112759: detected 3 of 4 replicas at location /A
I20240503 02:01:55.979359 25214 placement_policy_util.cc:399] tablet 2e10afd0dfdc4e498f01affb07af2184: detected 3 of 4 replicas at location /A
I20240503 02:01:55.979387 25214 placement_policy_util.cc:399] tablet f1c576a257ea4b879cc0fcf755e3cfc0: detected 3 of 4 replicas at location /A
I20240503 02:01:55.979418 25214 placement_policy_util.cc:399] tablet 8acad5c6a5d0485fa426a19b753e3198: detected 3 of 4 replicas at location /A
I20240503 02:01:55.979442 25214 placement_policy_util.cc:399] tablet dbeb93f606ff4807b3fdb5cd84cc4ad2: detected 3 of 4 replicas at location /A
I20240503 02:01:55.979470 25214 placement_policy_util.cc:399] tablet 98dddb26ec8647a4bc1c18fcc4f9a1a8: detected 3 of 4 replicas at location /A
I20240503 02:01:55.979501 25214 placement_policy_util.cc:399] tablet 6ad14d2ab8b74ad69ac419cc5794dd6f: detected 3 of 4 replicas at location /A
I20240503 02:01:55.979528 25214 placement_policy_util.cc:399] tablet fcb339e7a813465093a6d86ab5ff6119: detected 3 of 4 replicas at location /A
I20240503 02:01:55.979550 25214 placement_policy_util.cc:399] tablet 85150ba83b484b4b90aa4385105851bd: detected 3 of 4 replicas at location /A
I20240503 02:01:55.981209 25214 rebalancer_tool.cc:237] fixing placement policy violations
I20240503 02:01:56.130853 25214 placement_policy_util.cc:399] tablet 98dddb26ec8647a4bc1c18fcc4f9a1a8: detected 3 of 4 replicas at location /A
I20240503 02:01:56.130991 25214 placement_policy_util.cc:399] tablet 917c41c20407484886c6ecca69f2a99e: detected 3 of 4 replicas at location /A
I20240503 02:01:56.131014 25214 placement_policy_util.cc:399] tablet d397654530844ca9aef380669fd0ad7e: detected 3 of 4 replicas at location /A
I20240503 02:01:56.131037 25214 placement_policy_util.cc:399] tablet 57346221ef5e4b4fa69c127345c69821: detected 3 of 4 replicas at location /A
I20240503 02:01:56.131062 25214 placement_policy_util.cc:399] tablet 43643b2168db4925875aa995ce71ee94: detected 3 of 4 replicas at location /A
I20240503 02:01:56.131084 25214 placement_policy_util.cc:399] tablet 8a90cf7d371747eba4191be5d3b1ef07: detected 3 of 4 replicas at location /A
I20240503 02:01:56.131105 25214 placement_policy_util.cc:399] tablet f1c576a257ea4b879cc0fcf755e3cfc0: detected 3 of 4 replicas at location /A
I20240503 02:01:56.131126 25214 placement_policy_util.cc:399] tablet 8acad5c6a5d0485fa426a19b753e3198: detected 3 of 4 replicas at location /A
I20240503 02:01:56.131146 25214 placement_policy_util.cc:399] tablet b59d722c2cdf40a5a023f5a1eecfc69a: detected 3 of 4 replicas at location /A
I20240503 02:01:56.131170 25214 placement_policy_util.cc:399] tablet cf858dafc71d45dcb649c5a09f55ab23: detected 3 of 4 replicas at location /A
I20240503 02:01:56.131189 25214 placement_policy_util.cc:399] tablet dbeb93f606ff4807b3fdb5cd84cc4ad2: detected 3 of 4 replicas at location /A
I20240503 02:01:56.131209 25214 placement_policy_util.cc:399] tablet f82fbb08e52244d8b3b3905f12112759: detected 3 of 4 replicas at location /A
I20240503 02:01:56.131230 25214 placement_policy_util.cc:399] tablet ba26bfbf4f424b83b0093cc55ed70355: detected 3 of 4 replicas at location /A
I20240503 02:01:56.131250 25214 placement_policy_util.cc:399] tablet cbdf4ac35a10417f8887dc3c4eed6634: detected 3 of 4 replicas at location /A
I20240503 02:01:56.131270 25214 placement_policy_util.cc:399] tablet 2e10afd0dfdc4e498f01affb07af2184: detected 3 of 4 replicas at location /A
I20240503 02:01:56.131290 25214 placement_policy_util.cc:399] tablet 9e63310e1e2b48d5bf34d4957b53438c: detected 3 of 4 replicas at location /A
I20240503 02:01:56.131311 25214 placement_policy_util.cc:399] tablet f6e2af0050854514b0f6844dc79eed90: detected 3 of 4 replicas at location /A
I20240503 02:01:56.131337 25214 placement_policy_util.cc:399] tablet 43cdbbc431c243619f5322ceff8cf403: detected 3 of 4 replicas at location /A
I20240503 02:01:56.131357 25214 placement_policy_util.cc:399] tablet d44684f0c09f4ebc98204e592921699a: detected 3 of 4 replicas at location /A
I20240503 02:01:56.131377 25214 placement_policy_util.cc:399] tablet 6ad14d2ab8b74ad69ac419cc5794dd6f: detected 3 of 4 replicas at location /A
I20240503 02:01:56.131398 25214 placement_policy_util.cc:399] tablet fcb339e7a813465093a6d86ab5ff6119: detected 3 of 4 replicas at location /A
I20240503 02:01:56.131412 25214 placement_policy_util.cc:399] tablet 85150ba83b484b4b90aa4385105851bd: detected 3 of 4 replicas at location /A
I20240503 02:01:56.131433 25214 placement_policy_util.cc:399] tablet 7ac40a74e5064395b92050be4f9493ed: detected 3 of 4 replicas at location /A
I20240503 02:01:56.131453 25214 placement_policy_util.cc:399] tablet cf54dd831d41447cba6d8d4f8789900b: detected 3 of 4 replicas at location /A
I20240503 02:01:56.150710 25214 rebalancer_tool.cc:1397] tablet cf54dd831d41447cba6d8d4f8789900b: 'e9b9ef47abee4728ad1322c7e5006685' -> '?' move scheduled
I20240503 02:01:56.174716 25214 rebalancer_tool.cc:1397] tablet 7ac40a74e5064395b92050be4f9493ed: 'ca68eff1c707415e8981067ac1d7aec6' -> '?' move scheduled
I20240503 02:01:56.200831 25214 rebalancer_tool.cc:1397] tablet fcb339e7a813465093a6d86ab5ff6119: 'e9b9ef47abee4728ad1322c7e5006685' -> '?' move scheduled
I20240503 02:01:56.219807 25214 rebalancer_tool.cc:1397] tablet 85150ba83b484b4b90aa4385105851bd: 'ca68eff1c707415e8981067ac1d7aec6' -> '?' move scheduled
I20240503 02:01:56.243521 25214 rebalancer_tool.cc:1397] tablet d44684f0c09f4ebc98204e592921699a: 'e9b9ef47abee4728ad1322c7e5006685' -> '?' move scheduled
I20240503 02:01:56.261097 25214 rebalancer_tool.cc:1397] tablet 6ad14d2ab8b74ad69ac419cc5794dd6f: 'ca68eff1c707415e8981067ac1d7aec6' -> '?' move scheduled
I20240503 02:01:57.518081 25214 rebalancer_tool.cc:1438] tablet 7ac40a74e5064395b92050be4f9493ed: 'ca68eff1c707415e8981067ac1d7aec6' -> '?' move completed: OK
I20240503 02:01:57.735827 25214 rebalancer_tool.cc:1397] tablet dbeb93f606ff4807b3fdb5cd84cc4ad2: 'ca68eff1c707415e8981067ac1d7aec6' -> '?' move scheduled
I20240503 02:01:57.773384 25214 rebalancer_tool.cc:1438] tablet d44684f0c09f4ebc98204e592921699a: 'e9b9ef47abee4728ad1322c7e5006685' -> '?' move completed: OK
I20240503 02:01:58.041929 25214 rebalancer_tool.cc:1397] tablet 43cdbbc431c243619f5322ceff8cf403: 'e9b9ef47abee4728ad1322c7e5006685' -> '?' move scheduled
I20240503 02:01:58.081423 25214 rebalancer_tool.cc:1438] tablet 85150ba83b484b4b90aa4385105851bd: 'ca68eff1c707415e8981067ac1d7aec6' -> '?' move completed: OK
I20240503 02:01:58.095360 25214 rebalancer_tool.cc:1438] tablet fcb339e7a813465093a6d86ab5ff6119: 'e9b9ef47abee4728ad1322c7e5006685' -> '?' move completed: OK
I20240503 02:01:58.102999 25214 rebalancer_tool.cc:1438] tablet cf54dd831d41447cba6d8d4f8789900b: 'e9b9ef47abee4728ad1322c7e5006685' -> '?' move completed: OK
I20240503 02:01:58.109401 25214 rebalancer_tool.cc:1438] tablet 6ad14d2ab8b74ad69ac419cc5794dd6f: 'ca68eff1c707415e8981067ac1d7aec6' -> '?' move completed: OK
I20240503 02:01:58.331768 25214 rebalancer_tool.cc:1397] tablet f6e2af0050854514b0f6844dc79eed90: 'e9b9ef47abee4728ad1322c7e5006685' -> '?' move scheduled
I20240503 02:01:58.356549 25214 rebalancer_tool.cc:1397] tablet cf858dafc71d45dcb649c5a09f55ab23: 'ca68eff1c707415e8981067ac1d7aec6' -> '?' move scheduled
I20240503 02:01:58.394207 25214 rebalancer_tool.cc:1397] tablet 9e63310e1e2b48d5bf34d4957b53438c: 'e9b9ef47abee4728ad1322c7e5006685' -> '?' move scheduled
I20240503 02:01:58.419072 25214 rebalancer_tool.cc:1397] tablet f1c576a257ea4b879cc0fcf755e3cfc0: 'ca68eff1c707415e8981067ac1d7aec6' -> '?' move scheduled
I20240503 02:01:59.427316 25214 rebalancer_tool.cc:1438] tablet 43cdbbc431c243619f5322ceff8cf403: 'e9b9ef47abee4728ad1322c7e5006685' -> '?' move completed: OK
I20240503 02:01:59.440160 25214 rebalancer_tool.cc:1438] tablet dbeb93f606ff4807b3fdb5cd84cc4ad2: 'ca68eff1c707415e8981067ac1d7aec6' -> '?' move completed: OK
I20240503 02:01:59.653925 25214 rebalancer_tool.cc:1397] tablet 2e10afd0dfdc4e498f01affb07af2184: 'e9b9ef47abee4728ad1322c7e5006685' -> '?' move scheduled
I20240503 02:01:59.683038 25214 rebalancer_tool.cc:1397] tablet 8a90cf7d371747eba4191be5d3b1ef07: 'ca68eff1c707415e8981067ac1d7aec6' -> '?' move scheduled
I20240503 02:01:59.947351 25214 rebalancer_tool.cc:1438] tablet f1c576a257ea4b879cc0fcf755e3cfc0: 'ca68eff1c707415e8981067ac1d7aec6' -> '?' move completed: OK
I20240503 02:01:59.953632 25214 rebalancer_tool.cc:1438] tablet 9e63310e1e2b48d5bf34d4957b53438c: 'e9b9ef47abee4728ad1322c7e5006685' -> '?' move completed: OK
I20240503 02:01:59.959698 25214 rebalancer_tool.cc:1438] tablet f6e2af0050854514b0f6844dc79eed90: 'e9b9ef47abee4728ad1322c7e5006685' -> '?' move completed: OK
I20240503 02:01:59.971935 25214 rebalancer_tool.cc:1438] tablet cf858dafc71d45dcb649c5a09f55ab23: 'ca68eff1c707415e8981067ac1d7aec6' -> '?' move completed: OK
I20240503 02:02:00.189129 25214 rebalancer_tool.cc:1397] tablet cbdf4ac35a10417f8887dc3c4eed6634: 'e9b9ef47abee4728ad1322c7e5006685' -> '?' move scheduled
I20240503 02:02:00.206701 25214 rebalancer_tool.cc:1397] tablet 43643b2168db4925875aa995ce71ee94: 'ca68eff1c707415e8981067ac1d7aec6' -> '?' move scheduled
I20240503 02:02:00.230562 25214 rebalancer_tool.cc:1397] tablet ba26bfbf4f424b83b0093cc55ed70355: 'e9b9ef47abee4728ad1322c7e5006685' -> '?' move scheduled
I20240503 02:02:00.254426 25214 rebalancer_tool.cc:1397] tablet 917c41c20407484886c6ecca69f2a99e: 'ca68eff1c707415e8981067ac1d7aec6' -> '?' move scheduled
I20240503 02:02:01.292174 25214 rebalancer_tool.cc:1438] tablet 8a90cf7d371747eba4191be5d3b1ef07: 'ca68eff1c707415e8981067ac1d7aec6' -> '?' move completed: OK
I20240503 02:02:01.310796 25214 rebalancer_tool.cc:1438] tablet 2e10afd0dfdc4e498f01affb07af2184: 'e9b9ef47abee4728ad1322c7e5006685' -> '?' move completed: OK
I20240503 02:02:01.525144 25214 rebalancer_tool.cc:1397] tablet f82fbb08e52244d8b3b3905f12112759: 'e9b9ef47abee4728ad1322c7e5006685' -> '?' move scheduled
I20240503 02:02:01.795850 25214 rebalancer_tool.cc:1438] tablet 917c41c20407484886c6ecca69f2a99e: 'ca68eff1c707415e8981067ac1d7aec6' -> '?' move completed: OK
I20240503 02:02:01.805378 25214 rebalancer_tool.cc:1438] tablet ba26bfbf4f424b83b0093cc55ed70355: 'e9b9ef47abee4728ad1322c7e5006685' -> '?' move completed: OK
I20240503 02:02:01.811745 25214 rebalancer_tool.cc:1438] tablet 43643b2168db4925875aa995ce71ee94: 'ca68eff1c707415e8981067ac1d7aec6' -> '?' move completed: OK
I20240503 02:02:02.034747 25214 rebalancer_tool.cc:1397] tablet b59d722c2cdf40a5a023f5a1eecfc69a: 'e9b9ef47abee4728ad1322c7e5006685' -> '?' move scheduled
I20240503 02:02:02.067873 25214 rebalancer_tool.cc:1438] tablet cbdf4ac35a10417f8887dc3c4eed6634: 'e9b9ef47abee4728ad1322c7e5006685' -> '?' move completed: OK
I20240503 02:02:02.282862 25214 rebalancer_tool.cc:1397] tablet 8acad5c6a5d0485fa426a19b753e3198: 'e9b9ef47abee4728ad1322c7e5006685' -> '?' move scheduled
I20240503 02:02:02.969242 25214 rebalancer_tool.cc:1438] tablet f82fbb08e52244d8b3b3905f12112759: 'e9b9ef47abee4728ad1322c7e5006685' -> '?' move completed: OK
I20240503 02:02:03.185321 25214 rebalancer_tool.cc:1397] tablet 57346221ef5e4b4fa69c127345c69821: 'e9b9ef47abee4728ad1322c7e5006685' -> '?' move scheduled
I20240503 02:02:03.654953 25214 rebalancer_tool.cc:1438] tablet b59d722c2cdf40a5a023f5a1eecfc69a: 'e9b9ef47abee4728ad1322c7e5006685' -> '?' move completed: OK
I20240503 02:02:03.873020 25214 rebalancer_tool.cc:1397] tablet d397654530844ca9aef380669fd0ad7e: 'e9b9ef47abee4728ad1322c7e5006685' -> '?' move scheduled
I20240503 02:02:04.122185 25214 rebalancer_tool.cc:1438] tablet 8acad5c6a5d0485fa426a19b753e3198: 'e9b9ef47abee4728ad1322c7e5006685' -> '?' move completed: OK
I20240503 02:02:04.340360 25214 rebalancer_tool.cc:1397] tablet 98dddb26ec8647a4bc1c18fcc4f9a1a8: 'e9b9ef47abee4728ad1322c7e5006685' -> '?' move scheduled
I20240503 02:02:04.604519 25214 rebalancer_tool.cc:1438] tablet 57346221ef5e4b4fa69c127345c69821: 'e9b9ef47abee4728ad1322c7e5006685' -> '?' move completed: OK
I20240503 02:02:05.684196 25214 rebalancer_tool.cc:1438] tablet d397654530844ca9aef380669fd0ad7e: 'e9b9ef47abee4728ad1322c7e5006685' -> '?' move completed: OK
I20240503 02:02:06.103374 25214 rebalancer_tool.cc:1438] tablet 98dddb26ec8647a4bc1c18fcc4f9a1a8: 'e9b9ef47abee4728ad1322c7e5006685' -> '?' move completed: OK
I20240503 02:02:06.447417 25214 placement_policy_util.cc:399] tablet 57346221ef5e4b4fa69c127345c69821: detected 2 of 4 replicas at location /A
I20240503 02:02:06.447506 25214 placement_policy_util.cc:399] tablet 43643b2168db4925875aa995ce71ee94: detected 2 of 4 replicas at location /A
I20240503 02:02:06.447532 25214 placement_policy_util.cc:399] tablet 8a90cf7d371747eba4191be5d3b1ef07: detected 2 of 4 replicas at location /A
I20240503 02:02:06.447556 25214 placement_policy_util.cc:399] tablet 6ad14d2ab8b74ad69ac419cc5794dd6f: detected 2 of 4 replicas at location /A
I20240503 02:02:06.447585 25214 placement_policy_util.cc:399] tablet 8acad5c6a5d0485fa426a19b753e3198: detected 2 of 4 replicas at location /A
I20240503 02:02:06.447619 25214 placement_policy_util.cc:399] tablet f1c576a257ea4b879cc0fcf755e3cfc0: detected 2 of 4 replicas at location /A
I20240503 02:02:06.447647 25214 placement_policy_util.cc:399] tablet b59d722c2cdf40a5a023f5a1eecfc69a: detected 2 of 4 replicas at location /A
I20240503 02:02:06.447677 25214 placement_policy_util.cc:399] tablet cf858dafc71d45dcb649c5a09f55ab23: detected 2 of 4 replicas at location /A
I20240503 02:02:06.447706 25214 placement_policy_util.cc:399] tablet dbeb93f606ff4807b3fdb5cd84cc4ad2: detected 2 of 4 replicas at location /A
I20240503 02:02:06.447741 25214 placement_policy_util.cc:399] tablet 2e10afd0dfdc4e498f01affb07af2184: detected 2 of 4 replicas at location /A
I20240503 02:02:06.447772 25214 placement_policy_util.cc:399] tablet 9e63310e1e2b48d5bf34d4957b53438c: detected 2 of 4 replicas at location /A
I20240503 02:02:06.447803 25214 placement_policy_util.cc:399] tablet ba26bfbf4f424b83b0093cc55ed70355: detected 2 of 4 replicas at location /A
I20240503 02:02:06.447831 25214 placement_policy_util.cc:399] tablet cbdf4ac35a10417f8887dc3c4eed6634: detected 2 of 4 replicas at location /A
I20240503 02:02:06.447860 25214 placement_policy_util.cc:399] tablet f82fbb08e52244d8b3b3905f12112759: detected 2 of 4 replicas at location /A
I20240503 02:02:06.447888 25214 placement_policy_util.cc:399] tablet f6e2af0050854514b0f6844dc79eed90: detected 2 of 4 replicas at location /A
I20240503 02:02:06.447917 25214 placement_policy_util.cc:399] tablet d397654530844ca9aef380669fd0ad7e: detected 2 of 4 replicas at location /A
I20240503 02:02:06.447945 25214 placement_policy_util.cc:399] tablet 917c41c20407484886c6ecca69f2a99e: detected 2 of 4 replicas at location /A
I20240503 02:02:06.447983 25214 placement_policy_util.cc:399] tablet cf54dd831d41447cba6d8d4f8789900b: detected 2 of 4 replicas at location /A
I20240503 02:02:06.448014 25214 placement_policy_util.cc:399] tablet 7ac40a74e5064395b92050be4f9493ed: detected 2 of 4 replicas at location /A
I20240503 02:02:06.448042 25214 placement_policy_util.cc:399] tablet 43cdbbc431c243619f5322ceff8cf403: detected 2 of 4 replicas at location /A
I20240503 02:02:06.448069 25214 placement_policy_util.cc:399] tablet 98dddb26ec8647a4bc1c18fcc4f9a1a8: detected 2 of 4 replicas at location /A
I20240503 02:02:06.448098 25214 placement_policy_util.cc:399] tablet d44684f0c09f4ebc98204e592921699a: detected 2 of 4 replicas at location /A
I20240503 02:02:06.448127 25214 placement_policy_util.cc:399] tablet fcb339e7a813465093a6d86ab5ff6119: detected 2 of 4 replicas at location /A
I20240503 02:02:06.448163 25214 placement_policy_util.cc:399] tablet 85150ba83b484b4b90aa4385105851bd: detected 2 of 4 replicas at location /A
I20240503 02:02:06.465162 25214 rebalancer_tool.cc:1397] tablet fcb339e7a813465093a6d86ab5ff6119: 'ca68eff1c707415e8981067ac1d7aec6' -> '?' move scheduled
I20240503 02:02:06.492798 25214 rebalancer_tool.cc:1397] tablet 85150ba83b484b4b90aa4385105851bd: 'e9b9ef47abee4728ad1322c7e5006685' -> '?' move scheduled
I20240503 02:02:06.516866 25214 rebalancer_tool.cc:1397] tablet d44684f0c09f4ebc98204e592921699a: 'ca68eff1c707415e8981067ac1d7aec6' -> '?' move scheduled
I20240503 02:02:06.538199 25214 rebalancer_tool.cc:1397] tablet 7ac40a74e5064395b92050be4f9493ed: 'e9b9ef47abee4728ad1322c7e5006685' -> '?' move scheduled
I20240503 02:02:06.562809 25214 rebalancer_tool.cc:1397] tablet 98dddb26ec8647a4bc1c18fcc4f9a1a8: 'ca68eff1c707415e8981067ac1d7aec6' -> '?' move scheduled
I20240503 02:02:06.590008 25214 rebalancer_tool.cc:1397] tablet 917c41c20407484886c6ecca69f2a99e: 'e9b9ef47abee4728ad1322c7e5006685' -> '?' move scheduled
W20240503 02:07:08.104871 1522 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 02:07:07.954528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.129:43197 (local address 127.0.0.1:60626)
0503 02:07:07.954798 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0503 02:07:07.954813 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0503 02:07:07.954912 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 02:07:08.104077 (+149165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.129:43197: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":97,"threads_started":1}
I20240503 02:07:08.105201 25214 rebalancer_tool.cc:1429] tablet 917c41c20407484886c6ecca69f2a99e: 'e9b9ef47abee4728ad1322c7e5006685' -> '?' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.0.99.129:43197: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:07:08.107813 25214 rebalancer_tool.cc:1429] tablet 7ac40a74e5064395b92050be4f9493ed: 'e9b9ef47abee4728ad1322c7e5006685' -> '?' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.0.99.129:43197: connect: Connection refused (error 111)
I20240503 02:07:08.116222 25214 rebalancer_tool.cc:1429] tablet 85150ba83b484b4b90aa4385105851bd: 'e9b9ef47abee4728ad1322c7e5006685' -> '?' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.0.99.129:43197: connect: Connection refused (error 111)
I20240503 02:07:08.122720 25214 rebalancer_tool.cc:748] detected a staleness period of 301.533s
Incomplete: stalled with no progress for more than 300.000s seconds, aborting
I20240503 02:07:08.132805 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu with pid 23605
I20240503 02:07:08.140323 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu with pid 23737
I20240503 02:07:08.148049 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu with pid 24001
I20240503 02:07:08.155148 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu with pid 23383
2024-05-03T02:07:08Z chronyd exiting
I20240503 02:07:08.172875 398 test_util.cc:175] -----------------------------------------------
I20240503 02:07:08.172942 398 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx8_rf4_t8_l_3_1_1_1.1714701555658200-398-0
Note: This is test shard 2 of 8.
[==========] Running 8 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from RebalanceIgnoredTserversTest
[ RUN ] RebalanceIgnoredTserversTest.Basic
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240503 01:59:15.671057 398 test_util.cc:268] Using random seed: 287252472
I20240503 01:59:15.678675 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 01:59:15.678822 398 ts_itest-base.cc:113] --------------
I20240503 01:59:15.678877 398 ts_itest-base.cc:114] 5 tablet servers
I20240503 01:59:15.678943 398 ts_itest-base.cc:115] 3 replicas per TS
I20240503 01:59:15.678997 398 ts_itest-base.cc:116] --------------
2024-05-03T01:59:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T01:59:15Z Disabled control of system clock
I20240503 01:59:15.689960 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
/tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:34021
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:41121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:34021
--enable_range_replica_placement=false with env {}
W20240503 01:59:15.833736 410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:15.834018 410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:15.834062 410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:15.837512 410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 01:59:15.837602 410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:15.837622 410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 01:59:15.837639 410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 01:59:15.841651 410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:34021
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:34021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:15.842675 410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:15.843995 410 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:15.852985 416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:15.853096 410 server_base.cc:1034] running on GCE node
W20240503 01:59:15.852985 415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:15.852985 418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:15.853842 410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:15.854983 410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:15.856207 410 hybrid_clock.cc:648] HybridClock initialized: now 1714701555856136 us; error 96 us; skew 500 ppm
I20240503 01:59:15.858873 410 webserver.cc:460] Webserver started at http://127.0.99.190:34907/ using document root <none> and password file <none>
I20240503 01:59:15.859526 410 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:15.859601 410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:15.859793 410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:15.861724 410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "3ce705400ef0467c87b1537bdacc0925"
format_stamp: "Formatted at 2024-05-03 01:59:15 on dist-test-slave-pz5w"
I20240503 01:59:15.862259 410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "3ce705400ef0467c87b1537bdacc0925"
format_stamp: "Formatted at 2024-05-03 01:59:15 on dist-test-slave-pz5w"
I20240503 01:59:15.866075 410 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 01:59:15.869063 424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:15.870620 410 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 01:59:15.870810 410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "3ce705400ef0467c87b1537bdacc0925"
format_stamp: "Formatted at 2024-05-03 01:59:15 on dist-test-slave-pz5w"
I20240503 01:59:15.870932 410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:15.887575 410 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:15.888322 410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:15.888553 410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:15.897135 410 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:34021
I20240503 01:59:15.897156 476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:34021 every 8 connection(s)
I20240503 01:59:15.898694 410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 01:59:15.901675 477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:15.907305 477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ce705400ef0467c87b1537bdacc0925: Bootstrap starting.
I20240503 01:59:15.908030 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu as pid 410
I20240503 01:59:15.908159 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 01:59:15.909897 477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ce705400ef0467c87b1537bdacc0925: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:15.910669 477 log.cc:824] T 00000000000000000000000000000000 P 3ce705400ef0467c87b1537bdacc0925: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:15.912959 477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ce705400ef0467c87b1537bdacc0925: No bootstrap required, opened a new log
I20240503 01:59:15.916419 477 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ce705400ef0467c87b1537bdacc0925 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ce705400ef0467c87b1537bdacc0925" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 34021 } }
I20240503 01:59:15.916703 477 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ce705400ef0467c87b1537bdacc0925 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:15.916814 477 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ce705400ef0467c87b1537bdacc0925 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ce705400ef0467c87b1537bdacc0925, State: Initialized, Role: FOLLOWER
I20240503 01:59:15.917344 477 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ce705400ef0467c87b1537bdacc0925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ce705400ef0467c87b1537bdacc0925" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 34021 } }
I20240503 01:59:15.917501 477 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ce705400ef0467c87b1537bdacc0925 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 01:59:15.917572 477 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ce705400ef0467c87b1537bdacc0925 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 01:59:15.917680 477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ce705400ef0467c87b1537bdacc0925 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:15.928689 477 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ce705400ef0467c87b1537bdacc0925 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ce705400ef0467c87b1537bdacc0925" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 34021 } }
I20240503 01:59:15.929255 477 leader_election.cc:302] T 00000000000000000000000000000000 P 3ce705400ef0467c87b1537bdacc0925 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ce705400ef0467c87b1537bdacc0925; no voters:
I20240503 01:59:15.929682 477 leader_election.cc:288] T 00000000000000000000000000000000 P 3ce705400ef0467c87b1537bdacc0925 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 01:59:15.929996 482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ce705400ef0467c87b1537bdacc0925 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:15.930233 482 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ce705400ef0467c87b1537bdacc0925 [term 1 LEADER]: Becoming Leader. State: Replica: 3ce705400ef0467c87b1537bdacc0925, State: Running, Role: LEADER
I20240503 01:59:15.930606 482 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ce705400ef0467c87b1537bdacc0925 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ce705400ef0467c87b1537bdacc0925" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 34021 } }
I20240503 01:59:15.931072 477 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ce705400ef0467c87b1537bdacc0925 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 01:59:15.932097 483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ce705400ef0467c87b1537bdacc0925 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ce705400ef0467c87b1537bdacc0925" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ce705400ef0467c87b1537bdacc0925" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 34021 } } }
I20240503 01:59:15.932251 483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ce705400ef0467c87b1537bdacc0925 [sys.catalog]: This master's current role is: LEADER
I20240503 01:59:15.932591 484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ce705400ef0467c87b1537bdacc0925 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ce705400ef0467c87b1537bdacc0925. Latest consensus state: current_term: 1 leader_uuid: "3ce705400ef0467c87b1537bdacc0925" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ce705400ef0467c87b1537bdacc0925" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 34021 } } }
I20240503 01:59:15.932669 484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ce705400ef0467c87b1537bdacc0925 [sys.catalog]: This master's current role is: LEADER
I20240503 01:59:15.933071 491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 01:59:15.936463 491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 01:59:15.942608 491 catalog_manager.cc:1347] Generated new cluster ID: 167003c47c8748aba23afe480fefec51
I20240503 01:59:15.942714 491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 01:59:15.973565 491 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 01:59:15.974565 491 catalog_manager.cc:1504] Loading token signing keys...
I20240503 01:59:15.983011 491 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 3ce705400ef0467c87b1537bdacc0925: Generated new TSK 0
I20240503 01:59:15.983794 491 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 01:59:15.993690 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
/tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34021
--builtin_ntp_servers=127.0.99.148:41121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20240503 01:59:16.112273 501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:16.112566 501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:16.112628 501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:16.115949 501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:16.116106 501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 01:59:16.120200 501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34021
--never_fsync=true
--heap_profile_path=/tmp/kudu.501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:16.121229 501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:16.122457 501 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:16.131439 509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:16.131444 507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:16.132261 506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:16.132247 501 server_base.cc:1034] running on GCE node
I20240503 01:59:16.132763 501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:16.133443 501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:16.134634 501 hybrid_clock.cc:648] HybridClock initialized: now 1714701556134591 us; error 64 us; skew 500 ppm
I20240503 01:59:16.136849 501 webserver.cc:460] Webserver started at http://127.0.99.129:46631/ using document root <none> and password file <none>
I20240503 01:59:16.137445 501 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:16.137509 501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:16.137691 501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:16.139509 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "231ffd8a64274c199d4a4cbe280942e7"
format_stamp: "Formatted at 2024-05-03 01:59:16 on dist-test-slave-pz5w"
I20240503 01:59:16.140064 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "231ffd8a64274c199d4a4cbe280942e7"
format_stamp: "Formatted at 2024-05-03 01:59:16 on dist-test-slave-pz5w"
I20240503 01:59:16.143839 501 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20240503 01:59:16.146301 515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:16.147451 501 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240503 01:59:16.147593 501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "231ffd8a64274c199d4a4cbe280942e7"
format_stamp: "Formatted at 2024-05-03 01:59:16 on dist-test-slave-pz5w"
I20240503 01:59:16.147696 501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:16.162799 501 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:16.163503 501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:16.163676 501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:16.164247 501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:16.165295 501 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:16.165349 501 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:16.165408 501 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:16.165434 501 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:16.174408 501 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:39287
I20240503 01:59:16.174443 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:39287 every 8 connection(s)
I20240503 01:59:16.175513 501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 01:59:16.180001 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu as pid 501
I20240503 01:59:16.180111 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 01:59:16.182433 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
/tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34021
--builtin_ntp_servers=127.0.99.148:41121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240503 01:59:16.185370 629 heartbeater.cc:347] Connected to a master server at 127.0.99.190:34021
I20240503 01:59:16.185616 629 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:16.186203 629 heartbeater.cc:511] Master 127.0.99.190:34021 requested a full tablet report, sending...
I20240503 01:59:16.188007 439 ts_manager.cc:194] Registered new tserver with Master: 231ffd8a64274c199d4a4cbe280942e7 (127.0.99.129:39287)
I20240503 01:59:16.189848 439 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:46441
W20240503 01:59:16.301525 632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:16.301782 632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:16.301827 632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:16.305424 632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:16.305562 632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 01:59:16.309619 632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34021
--never_fsync=true
--heap_profile_path=/tmp/kudu.632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:16.310626 632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:16.311808 632 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:16.320816 640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:16.320817 637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:16.320817 638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:16.321177 632 server_base.cc:1034] running on GCE node
I20240503 01:59:16.321632 632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:16.322290 632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:16.323485 632 hybrid_clock.cc:648] HybridClock initialized: now 1714701556323485 us; error 67 us; skew 500 ppm
I20240503 01:59:16.326018 632 webserver.cc:460] Webserver started at http://127.0.99.130:34661/ using document root <none> and password file <none>
I20240503 01:59:16.326666 632 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:16.327188 632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:16.327433 632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:16.329562 632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "796ea4d7162a46e19166f41ab0d8cc19"
format_stamp: "Formatted at 2024-05-03 01:59:16 on dist-test-slave-pz5w"
I20240503 01:59:16.330157 632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "796ea4d7162a46e19166f41ab0d8cc19"
format_stamp: "Formatted at 2024-05-03 01:59:16 on dist-test-slave-pz5w"
I20240503 01:59:16.334371 632 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 01:59:16.337294 646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:16.338573 632 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 01:59:16.338727 632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "796ea4d7162a46e19166f41ab0d8cc19"
format_stamp: "Formatted at 2024-05-03 01:59:16 on dist-test-slave-pz5w"
I20240503 01:59:16.338833 632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:16.353893 632 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:16.354655 632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:16.354847 632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:16.355463 632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:16.356504 632 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:16.356556 632 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:16.356616 632 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:16.356643 632 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:16.368244 632 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:39195
I20240503 01:59:16.368278 759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:39195 every 8 connection(s)
I20240503 01:59:16.369283 632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 01:59:16.379228 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu as pid 632
I20240503 01:59:16.379367 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 01:59:16.379495 760 heartbeater.cc:347] Connected to a master server at 127.0.99.190:34021
I20240503 01:59:16.379828 760 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:16.380566 760 heartbeater.cc:511] Master 127.0.99.190:34021 requested a full tablet report, sending...
I20240503 01:59:16.381786 439 ts_manager.cc:194] Registered new tserver with Master: 796ea4d7162a46e19166f41ab0d8cc19 (127.0.99.130:39195)
I20240503 01:59:16.381847 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
/tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34021
--builtin_ntp_servers=127.0.99.148:41121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240503 01:59:16.382552 439 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:39779
W20240503 01:59:16.504002 763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:16.504271 763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:16.504316 763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:16.507613 763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:16.507756 763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 01:59:16.511878 763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34021
--never_fsync=true
--heap_profile_path=/tmp/kudu.763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:16.513186 763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:16.514747 763 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:16.525595 768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:16.525596 769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:16.525648 771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:16.529717 763 server_base.cc:1034] running on GCE node
I20240503 01:59:16.530366 763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:16.531203 763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:16.532418 763 hybrid_clock.cc:648] HybridClock initialized: now 1714701556532398 us; error 45 us; skew 500 ppm
I20240503 01:59:16.534806 763 webserver.cc:460] Webserver started at http://127.0.99.131:44987/ using document root <none> and password file <none>
I20240503 01:59:16.535367 763 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:16.535450 763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:16.535616 763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:16.537493 763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "660ba05d01284c07b672e386370434ca"
format_stamp: "Formatted at 2024-05-03 01:59:16 on dist-test-slave-pz5w"
I20240503 01:59:16.538170 763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "660ba05d01284c07b672e386370434ca"
format_stamp: "Formatted at 2024-05-03 01:59:16 on dist-test-slave-pz5w"
I20240503 01:59:16.541957 763 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240503 01:59:16.545373 777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:16.546823 763 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 01:59:16.546967 763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "660ba05d01284c07b672e386370434ca"
format_stamp: "Formatted at 2024-05-03 01:59:16 on dist-test-slave-pz5w"
I20240503 01:59:16.547082 763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:16.572614 763 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:16.573316 763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:16.573487 763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:16.574064 763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:16.575030 763 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:16.575079 763 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:16.575136 763 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:16.575162 763 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:16.585455 763 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:45787
I20240503 01:59:16.585484 890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:45787 every 8 connection(s)
I20240503 01:59:16.586516 763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 01:59:16.591931 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu as pid 763
I20240503 01:59:16.592054 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 01:59:16.594295 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
/tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34021
--builtin_ntp_servers=127.0.99.148:41121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240503 01:59:16.597048 891 heartbeater.cc:347] Connected to a master server at 127.0.99.190:34021
I20240503 01:59:16.597347 891 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:16.597970 891 heartbeater.cc:511] Master 127.0.99.190:34021 requested a full tablet report, sending...
I20240503 01:59:16.599220 439 ts_manager.cc:194] Registered new tserver with Master: 660ba05d01284c07b672e386370434ca (127.0.99.131:45787)
I20240503 01:59:16.599823 439 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:35561
W20240503 01:59:16.713388 894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:16.713734 894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:16.713796 894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:16.717546 894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:16.717684 894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 01:59:16.722841 894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34021
--never_fsync=true
--heap_profile_path=/tmp/kudu.894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:16.724474 894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:16.726269 894 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:16.735610 899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:16.735749 902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:16.736028 900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:16.736124 894 server_base.cc:1034] running on GCE node
I20240503 01:59:16.736560 894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:16.737180 894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:16.738348 894 hybrid_clock.cc:648] HybridClock initialized: now 1714701556738336 us; error 37 us; skew 500 ppm
I20240503 01:59:16.740345 894 webserver.cc:460] Webserver started at http://127.0.99.132:42669/ using document root <none> and password file <none>
I20240503 01:59:16.740996 894 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:16.741062 894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:16.741256 894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:16.743120 894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "3bdc12f8eb324dbdb62051dfe235f299"
format_stamp: "Formatted at 2024-05-03 01:59:16 on dist-test-slave-pz5w"
I20240503 01:59:16.743660 894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "3bdc12f8eb324dbdb62051dfe235f299"
format_stamp: "Formatted at 2024-05-03 01:59:16 on dist-test-slave-pz5w"
I20240503 01:59:16.747435 894 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240503 01:59:16.750046 908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:16.751396 894 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240503 01:59:16.751590 894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "3bdc12f8eb324dbdb62051dfe235f299"
format_stamp: "Formatted at 2024-05-03 01:59:16 on dist-test-slave-pz5w"
I20240503 01:59:16.751715 894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:16.781034 894 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:16.781795 894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:16.781971 894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:16.782528 894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:16.783492 894 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:16.783550 894 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:16.783607 894 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:16.783633 894 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:16.793303 894 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:42765
I20240503 01:59:16.793331 1021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:42765 every 8 connection(s)
I20240503 01:59:16.794333 894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 01:59:16.801869 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu as pid 894
I20240503 01:59:16.801998 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 01:59:16.804157 1022 heartbeater.cc:347] Connected to a master server at 127.0.99.190:34021
I20240503 01:59:16.804453 1022 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:16.804463 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
/tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34021
--builtin_ntp_servers=127.0.99.148:41121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240503 01:59:16.805080 1022 heartbeater.cc:511] Master 127.0.99.190:34021 requested a full tablet report, sending...
I20240503 01:59:16.806274 439 ts_manager.cc:194] Registered new tserver with Master: 3bdc12f8eb324dbdb62051dfe235f299 (127.0.99.132:42765)
I20240503 01:59:16.806851 439 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:45251
W20240503 01:59:16.925225 1025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:16.925485 1025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:16.925547 1025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:16.929782 1025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:16.929947 1025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 01:59:16.936492 1025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34021
--never_fsync=true
--heap_profile_path=/tmp/kudu.1025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:16.937829 1025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:16.939443 1025 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:16.950723 1030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:16.950971 1033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:16.951364 1031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:16.951683 1025 server_base.cc:1034] running on GCE node
I20240503 01:59:16.952247 1025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:16.953091 1025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:16.954254 1025 hybrid_clock.cc:648] HybridClock initialized: now 1714701556954245 us; error 37 us; skew 500 ppm
I20240503 01:59:16.957008 1025 webserver.cc:460] Webserver started at http://127.0.99.133:33179/ using document root <none> and password file <none>
I20240503 01:59:16.957840 1025 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:16.957940 1025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:16.958178 1025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:16.960520 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "7f69193525a4450db0e10a79c6f1fe62"
format_stamp: "Formatted at 2024-05-03 01:59:16 on dist-test-slave-pz5w"
I20240503 01:59:16.961076 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "7f69193525a4450db0e10a79c6f1fe62"
format_stamp: "Formatted at 2024-05-03 01:59:16 on dist-test-slave-pz5w"
I20240503 01:59:16.964962 1025 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240503 01:59:16.967752 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:16.969364 1025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 01:59:16.969569 1025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "7f69193525a4450db0e10a79c6f1fe62"
format_stamp: "Formatted at 2024-05-03 01:59:16 on dist-test-slave-pz5w"
I20240503 01:59:16.969683 1025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:16.987649 1025 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:16.988426 1025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:16.988602 1025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:16.989176 1025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:16.990162 1025 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:16.990213 1025 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:16.990271 1025 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:16.990300 1025 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:17.000638 1025 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:36137
I20240503 01:59:17.000654 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:36137 every 8 connection(s)
I20240503 01:59:17.001814 1025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 01:59:17.011279 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu as pid 1025
I20240503 01:59:17.011415 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 01:59:17.012161 1153 heartbeater.cc:347] Connected to a master server at 127.0.99.190:34021
I20240503 01:59:17.012591 1153 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:17.013283 1153 heartbeater.cc:511] Master 127.0.99.190:34021 requested a full tablet report, sending...
I20240503 01:59:17.014575 439 ts_manager.cc:194] Registered new tserver with Master: 7f69193525a4450db0e10a79c6f1fe62 (127.0.99.133:36137)
I20240503 01:59:17.015189 439 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:35997
I20240503 01:59:17.026480 398 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 01:59:17.041086 439 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51506:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 01:59:17.061834 563 tablet_service.cc:1467] Processing CreateTablet for tablet 3600000d084149d6b4063721e6cf3646 (DEFAULT_TABLE table=TestTable [id=94b8bbcc49d6475b96f26e8b26a5d962]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:17.063038 563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3600000d084149d6b4063721e6cf3646. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:17.063349 825 tablet_service.cc:1467] Processing CreateTablet for tablet 3600000d084149d6b4063721e6cf3646 (DEFAULT_TABLE table=TestTable [id=94b8bbcc49d6475b96f26e8b26a5d962]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:17.064359 825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3600000d084149d6b4063721e6cf3646. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:17.066650 1086 tablet_service.cc:1467] Processing CreateTablet for tablet 3600000d084149d6b4063721e6cf3646 (DEFAULT_TABLE table=TestTable [id=94b8bbcc49d6475b96f26e8b26a5d962]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:17.067523 1086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3600000d084149d6b4063721e6cf3646. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:17.070740 1172 tablet_bootstrap.cc:492] T 3600000d084149d6b4063721e6cf3646 P 660ba05d01284c07b672e386370434ca: Bootstrap starting.
I20240503 01:59:17.073302 1172 tablet_bootstrap.cc:654] T 3600000d084149d6b4063721e6cf3646 P 660ba05d01284c07b672e386370434ca: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:17.073807 1173 tablet_bootstrap.cc:492] T 3600000d084149d6b4063721e6cf3646 P 7f69193525a4450db0e10a79c6f1fe62: Bootstrap starting.
I20240503 01:59:17.074244 1172 log.cc:824] T 3600000d084149d6b4063721e6cf3646 P 660ba05d01284c07b672e386370434ca: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:17.075081 1171 tablet_bootstrap.cc:492] T 3600000d084149d6b4063721e6cf3646 P 231ffd8a64274c199d4a4cbe280942e7: Bootstrap starting.
I20240503 01:59:17.076129 1173 tablet_bootstrap.cc:654] T 3600000d084149d6b4063721e6cf3646 P 7f69193525a4450db0e10a79c6f1fe62: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:17.077047 1173 log.cc:824] T 3600000d084149d6b4063721e6cf3646 P 7f69193525a4450db0e10a79c6f1fe62: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:17.077277 1171 tablet_bootstrap.cc:654] T 3600000d084149d6b4063721e6cf3646 P 231ffd8a64274c199d4a4cbe280942e7: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:17.078085 1171 log.cc:824] T 3600000d084149d6b4063721e6cf3646 P 231ffd8a64274c199d4a4cbe280942e7: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:17.078224 1172 tablet_bootstrap.cc:492] T 3600000d084149d6b4063721e6cf3646 P 660ba05d01284c07b672e386370434ca: No bootstrap required, opened a new log
I20240503 01:59:17.078438 1172 ts_tablet_manager.cc:1367] T 3600000d084149d6b4063721e6cf3646 P 660ba05d01284c07b672e386370434ca: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240503 01:59:17.079147 1173 tablet_bootstrap.cc:492] T 3600000d084149d6b4063721e6cf3646 P 7f69193525a4450db0e10a79c6f1fe62: No bootstrap required, opened a new log
I20240503 01:59:17.079502 1173 ts_tablet_manager.cc:1367] T 3600000d084149d6b4063721e6cf3646 P 7f69193525a4450db0e10a79c6f1fe62: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240503 01:59:17.080210 1171 tablet_bootstrap.cc:492] T 3600000d084149d6b4063721e6cf3646 P 231ffd8a64274c199d4a4cbe280942e7: No bootstrap required, opened a new log
I20240503 01:59:17.080439 1171 ts_tablet_manager.cc:1367] T 3600000d084149d6b4063721e6cf3646 P 231ffd8a64274c199d4a4cbe280942e7: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240503 01:59:17.082571 1173 raft_consensus.cc:348] T 3600000d084149d6b4063721e6cf3646 P 7f69193525a4450db0e10a79c6f1fe62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231ffd8a64274c199d4a4cbe280942e7" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39287 } } peers { permanent_uuid: "660ba05d01284c07b672e386370434ca" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45787 } } peers { permanent_uuid: "7f69193525a4450db0e10a79c6f1fe62" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36137 } }
I20240503 01:59:17.082553 1172 raft_consensus.cc:348] T 3600000d084149d6b4063721e6cf3646 P 660ba05d01284c07b672e386370434ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231ffd8a64274c199d4a4cbe280942e7" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39287 } } peers { permanent_uuid: "660ba05d01284c07b672e386370434ca" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45787 } } peers { permanent_uuid: "7f69193525a4450db0e10a79c6f1fe62" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36137 } }
I20240503 01:59:17.082788 1172 raft_consensus.cc:374] T 3600000d084149d6b4063721e6cf3646 P 660ba05d01284c07b672e386370434ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:17.082789 1173 raft_consensus.cc:374] T 3600000d084149d6b4063721e6cf3646 P 7f69193525a4450db0e10a79c6f1fe62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:17.082866 1173 raft_consensus.cc:729] T 3600000d084149d6b4063721e6cf3646 P 7f69193525a4450db0e10a79c6f1fe62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f69193525a4450db0e10a79c6f1fe62, State: Initialized, Role: FOLLOWER
I20240503 01:59:17.082898 1172 raft_consensus.cc:729] T 3600000d084149d6b4063721e6cf3646 P 660ba05d01284c07b672e386370434ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 660ba05d01284c07b672e386370434ca, State: Initialized, Role: FOLLOWER
I20240503 01:59:17.083352 1173 consensus_queue.cc:260] T 3600000d084149d6b4063721e6cf3646 P 7f69193525a4450db0e10a79c6f1fe62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231ffd8a64274c199d4a4cbe280942e7" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39287 } } peers { permanent_uuid: "660ba05d01284c07b672e386370434ca" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45787 } } peers { permanent_uuid: "7f69193525a4450db0e10a79c6f1fe62" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36137 } }
I20240503 01:59:17.083392 1172 consensus_queue.cc:260] T 3600000d084149d6b4063721e6cf3646 P 660ba05d01284c07b672e386370434ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231ffd8a64274c199d4a4cbe280942e7" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39287 } } peers { permanent_uuid: "660ba05d01284c07b672e386370434ca" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45787 } } peers { permanent_uuid: "7f69193525a4450db0e10a79c6f1fe62" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36137 } }
I20240503 01:59:17.084116 1153 heartbeater.cc:503] Master 127.0.99.190:34021 was elected leader, sending a full tablet report...
I20240503 01:59:17.084298 1173 ts_tablet_manager.cc:1398] T 3600000d084149d6b4063721e6cf3646 P 7f69193525a4450db0e10a79c6f1fe62: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20240503 01:59:17.084316 1172 ts_tablet_manager.cc:1398] T 3600000d084149d6b4063721e6cf3646 P 660ba05d01284c07b672e386370434ca: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240503 01:59:17.084659 891 heartbeater.cc:503] Master 127.0.99.190:34021 was elected leader, sending a full tablet report...
I20240503 01:59:17.083746 1171 raft_consensus.cc:348] T 3600000d084149d6b4063721e6cf3646 P 231ffd8a64274c199d4a4cbe280942e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231ffd8a64274c199d4a4cbe280942e7" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39287 } } peers { permanent_uuid: "660ba05d01284c07b672e386370434ca" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45787 } } peers { permanent_uuid: "7f69193525a4450db0e10a79c6f1fe62" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36137 } }
I20240503 01:59:17.085229 1171 raft_consensus.cc:374] T 3600000d084149d6b4063721e6cf3646 P 231ffd8a64274c199d4a4cbe280942e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:17.085350 1171 raft_consensus.cc:729] T 3600000d084149d6b4063721e6cf3646 P 231ffd8a64274c199d4a4cbe280942e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231ffd8a64274c199d4a4cbe280942e7, State: Initialized, Role: FOLLOWER
I20240503 01:59:17.086093 1171 consensus_queue.cc:260] T 3600000d084149d6b4063721e6cf3646 P 231ffd8a64274c199d4a4cbe280942e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231ffd8a64274c199d4a4cbe280942e7" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39287 } } peers { permanent_uuid: "660ba05d01284c07b672e386370434ca" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45787 } } peers { permanent_uuid: "7f69193525a4450db0e10a79c6f1fe62" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36137 } }
I20240503 01:59:17.086999 629 heartbeater.cc:503] Master 127.0.99.190:34021 was elected leader, sending a full tablet report...
W20240503 01:59:17.087644 892 tablet.cc:2369] T 3600000d084149d6b4063721e6cf3646 P 660ba05d01284c07b672e386370434ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 01:59:17.088899 1171 ts_tablet_manager.cc:1398] T 3600000d084149d6b4063721e6cf3646 P 231ffd8a64274c199d4a4cbe280942e7: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
W20240503 01:59:17.177057 630 tablet.cc:2369] T 3600000d084149d6b4063721e6cf3646 P 231ffd8a64274c199d4a4cbe280942e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 01:59:17.197947 1177 raft_consensus.cc:482] T 3600000d084149d6b4063721e6cf3646 P 7f69193525a4450db0e10a79c6f1fe62 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:17.198155 1177 raft_consensus.cc:504] T 3600000d084149d6b4063721e6cf3646 P 7f69193525a4450db0e10a79c6f1fe62 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231ffd8a64274c199d4a4cbe280942e7" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39287 } } peers { permanent_uuid: "660ba05d01284c07b672e386370434ca" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45787 } } peers { permanent_uuid: "7f69193525a4450db0e10a79c6f1fe62" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36137 } }
I20240503 01:59:17.199280 1177 leader_election.cc:288] T 3600000d084149d6b4063721e6cf3646 P 7f69193525a4450db0e10a79c6f1fe62 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 231ffd8a64274c199d4a4cbe280942e7 (127.0.99.129:39287), 660ba05d01284c07b672e386370434ca (127.0.99.131:45787)
I20240503 01:59:17.203404 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3600000d084149d6b4063721e6cf3646" candidate_uuid: "7f69193525a4450db0e10a79c6f1fe62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231ffd8a64274c199d4a4cbe280942e7" is_pre_election: true
I20240503 01:59:17.203399 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3600000d084149d6b4063721e6cf3646" candidate_uuid: "7f69193525a4450db0e10a79c6f1fe62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "660ba05d01284c07b672e386370434ca" is_pre_election: true
I20240503 01:59:17.203719 845 raft_consensus.cc:2454] T 3600000d084149d6b4063721e6cf3646 P 660ba05d01284c07b672e386370434ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f69193525a4450db0e10a79c6f1fe62 in term 0.
I20240503 01:59:17.203719 583 raft_consensus.cc:2454] T 3600000d084149d6b4063721e6cf3646 P 231ffd8a64274c199d4a4cbe280942e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f69193525a4450db0e10a79c6f1fe62 in term 0.
I20240503 01:59:17.204254 1040 leader_election.cc:302] T 3600000d084149d6b4063721e6cf3646 P 7f69193525a4450db0e10a79c6f1fe62 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231ffd8a64274c199d4a4cbe280942e7, 7f69193525a4450db0e10a79c6f1fe62; no voters:
I20240503 01:59:17.204643 1177 raft_consensus.cc:2781] T 3600000d084149d6b4063721e6cf3646 P 7f69193525a4450db0e10a79c6f1fe62 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:17.204736 1177 raft_consensus.cc:482] T 3600000d084149d6b4063721e6cf3646 P 7f69193525a4450db0e10a79c6f1fe62 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:17.204764 1177 raft_consensus.cc:3037] T 3600000d084149d6b4063721e6cf3646 P 7f69193525a4450db0e10a79c6f1fe62 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:17.205632 1177 raft_consensus.cc:504] T 3600000d084149d6b4063721e6cf3646 P 7f69193525a4450db0e10a79c6f1fe62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231ffd8a64274c199d4a4cbe280942e7" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39287 } } peers { permanent_uuid: "660ba05d01284c07b672e386370434ca" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45787 } } peers { permanent_uuid: "7f69193525a4450db0e10a79c6f1fe62" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36137 } }
I20240503 01:59:17.205989 1177 leader_election.cc:288] T 3600000d084149d6b4063721e6cf3646 P 7f69193525a4450db0e10a79c6f1fe62 [CANDIDATE]: Term 1 election: Requested vote from peers 231ffd8a64274c199d4a4cbe280942e7 (127.0.99.129:39287), 660ba05d01284c07b672e386370434ca (127.0.99.131:45787)
I20240503 01:59:17.206449 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3600000d084149d6b4063721e6cf3646" candidate_uuid: "7f69193525a4450db0e10a79c6f1fe62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "660ba05d01284c07b672e386370434ca"
I20240503 01:59:17.206444 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3600000d084149d6b4063721e6cf3646" candidate_uuid: "7f69193525a4450db0e10a79c6f1fe62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231ffd8a64274c199d4a4cbe280942e7"
I20240503 01:59:17.206583 583 raft_consensus.cc:3037] T 3600000d084149d6b4063721e6cf3646 P 231ffd8a64274c199d4a4cbe280942e7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:17.206583 845 raft_consensus.cc:3037] T 3600000d084149d6b4063721e6cf3646 P 660ba05d01284c07b672e386370434ca [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:17.207680 845 raft_consensus.cc:2454] T 3600000d084149d6b4063721e6cf3646 P 660ba05d01284c07b672e386370434ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f69193525a4450db0e10a79c6f1fe62 in term 1.
I20240503 01:59:17.207680 583 raft_consensus.cc:2454] T 3600000d084149d6b4063721e6cf3646 P 231ffd8a64274c199d4a4cbe280942e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f69193525a4450db0e10a79c6f1fe62 in term 1.
I20240503 01:59:17.208070 1041 leader_election.cc:302] T 3600000d084149d6b4063721e6cf3646 P 7f69193525a4450db0e10a79c6f1fe62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 231ffd8a64274c199d4a4cbe280942e7, 660ba05d01284c07b672e386370434ca, 7f69193525a4450db0e10a79c6f1fe62; no voters:
I20240503 01:59:17.208287 1177 raft_consensus.cc:2781] T 3600000d084149d6b4063721e6cf3646 P 7f69193525a4450db0e10a79c6f1fe62 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:17.208567 1177 raft_consensus.cc:686] T 3600000d084149d6b4063721e6cf3646 P 7f69193525a4450db0e10a79c6f1fe62 [term 1 LEADER]: Becoming Leader. State: Replica: 7f69193525a4450db0e10a79c6f1fe62, State: Running, Role: LEADER
I20240503 01:59:17.209014 1177 consensus_queue.cc:237] T 3600000d084149d6b4063721e6cf3646 P 7f69193525a4450db0e10a79c6f1fe62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231ffd8a64274c199d4a4cbe280942e7" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39287 } } peers { permanent_uuid: "660ba05d01284c07b672e386370434ca" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45787 } } peers { permanent_uuid: "7f69193525a4450db0e10a79c6f1fe62" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36137 } }
I20240503 01:59:17.212237 439 catalog_manager.cc:5484] T 3600000d084149d6b4063721e6cf3646 P 7f69193525a4450db0e10a79c6f1fe62 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f69193525a4450db0e10a79c6f1fe62 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "7f69193525a4450db0e10a79c6f1fe62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231ffd8a64274c199d4a4cbe280942e7" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "660ba05d01284c07b672e386370434ca" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f69193525a4450db0e10a79c6f1fe62" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36137 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:17.251641 398 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
W20240503 01:59:17.252756 1154 tablet.cc:2369] T 3600000d084149d6b4063721e6cf3646 P 7f69193525a4450db0e10a79c6f1fe62: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 01:59:17.253754 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 231ffd8a64274c199d4a4cbe280942e7 to finish bootstrapping
I20240503 01:59:17.259938 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 796ea4d7162a46e19166f41ab0d8cc19 to finish bootstrapping
I20240503 01:59:17.266291 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 660ba05d01284c07b672e386370434ca to finish bootstrapping
I20240503 01:59:17.271528 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 3bdc12f8eb324dbdb62051dfe235f299 to finish bootstrapping
I20240503 01:59:17.277936 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 7f69193525a4450db0e10a79c6f1fe62 to finish bootstrapping
I20240503 01:59:17.385185 760 heartbeater.cc:503] Master 127.0.99.190:34021 was elected leader, sending a full tablet report...
I20240503 01:59:17.483237 956 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:17.483716 563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:17.484720 1086 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:17.485306 825 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:17.487563 694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:17.681803 1177 consensus_queue.cc:1035] T 3600000d084149d6b4063721e6cf3646 P 7f69193525a4450db0e10a79c6f1fe62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "231ffd8a64274c199d4a4cbe280942e7" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:17.690917 1177 consensus_queue.cc:1035] T 3600000d084149d6b4063721e6cf3646 P 7f69193525a4450db0e10a79c6f1fe62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "660ba05d01284c07b672e386370434ca" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:17.737278 825 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:17.737726 563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:17.738569 1086 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:17.739430 956 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:17.739492 694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:17.809079 1022 heartbeater.cc:503] Master 127.0.99.190:34021 was elected leader, sending a full tablet report...
I20240503 01:59:17.972733 825 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:17.972908 694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:17.973771 563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:17.974124 956 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:17.989449 1086 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:18.074087 825 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:18.074314 956 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:18.075649 694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:18.077629 563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:18.095175 1086 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:18.212051 956 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:18.215469 825 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:18.216054 694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:18.216663 563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:18.224953 1086 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:18.409322 439 ts_manager.cc:292] Set tserver state for 231ffd8a64274c199d4a4cbe280942e7 to MAINTENANCE_MODE
I20240503 01:59:18.610721 1086 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:18.610939 956 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:18.611359 825 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:18.612110 563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:18.612718 694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:18.709321 563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:18.709992 825 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:18.710532 1086 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:18.712038 956 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:18.723373 694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:18.831521 825 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:18.832432 956 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:18.834903 1086 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:18.840643 563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:18.841485 694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:18.874915 1107 consensus_queue.cc:237] T 3600000d084149d6b4063721e6cf3646 P 7f69193525a4450db0e10a79c6f1fe62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "231ffd8a64274c199d4a4cbe280942e7" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39287 } attrs { replace: true } } peers { permanent_uuid: "660ba05d01284c07b672e386370434ca" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45787 } } peers { permanent_uuid: "7f69193525a4450db0e10a79c6f1fe62" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36137 } }
I20240503 01:59:18.876816 583 raft_consensus.cc:1261] T 3600000d084149d6b4063721e6cf3646 P 231ffd8a64274c199d4a4cbe280942e7 [term 1 FOLLOWER]: Refusing update from remote peer 7f69193525a4450db0e10a79c6f1fe62: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:18.877279 845 raft_consensus.cc:1261] T 3600000d084149d6b4063721e6cf3646 P 660ba05d01284c07b672e386370434ca [term 1 FOLLOWER]: Refusing update from remote peer 7f69193525a4450db0e10a79c6f1fe62: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:18.877386 1473 consensus_queue.cc:1035] T 3600000d084149d6b4063721e6cf3646 P 7f69193525a4450db0e10a79c6f1fe62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "231ffd8a64274c199d4a4cbe280942e7" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39287 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 01:59:18.877871 1473 consensus_queue.cc:1035] T 3600000d084149d6b4063721e6cf3646 P 7f69193525a4450db0e10a79c6f1fe62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "660ba05d01284c07b672e386370434ca" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 01:59:18.880012 1354 raft_consensus.cc:2932] T 3600000d084149d6b4063721e6cf3646 P 7f69193525a4450db0e10a79c6f1fe62 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "231ffd8a64274c199d4a4cbe280942e7" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39287 } attrs { replace: true } } peers { permanent_uuid: "660ba05d01284c07b672e386370434ca" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45787 } } peers { permanent_uuid: "7f69193525a4450db0e10a79c6f1fe62" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36137 } } }
I20240503 01:59:18.881258 845 raft_consensus.cc:2932] T 3600000d084149d6b4063721e6cf3646 P 660ba05d01284c07b672e386370434ca [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "231ffd8a64274c199d4a4cbe280942e7" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39287 } attrs { replace: true } } peers { permanent_uuid: "660ba05d01284c07b672e386370434ca" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45787 } } peers { permanent_uuid: "7f69193525a4450db0e10a79c6f1fe62" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36137 } } }
I20240503 01:59:18.882017 583 raft_consensus.cc:2932] T 3600000d084149d6b4063721e6cf3646 P 231ffd8a64274c199d4a4cbe280942e7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "231ffd8a64274c199d4a4cbe280942e7" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39287 } attrs { replace: true } } peers { permanent_uuid: "660ba05d01284c07b672e386370434ca" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45787 } } peers { permanent_uuid: "7f69193525a4450db0e10a79c6f1fe62" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36137 } } }
I20240503 01:59:18.882910 439 catalog_manager.cc:5484] T 3600000d084149d6b4063721e6cf3646 P 7f69193525a4450db0e10a79c6f1fe62 reported cstate change: config changed from index -1 to 2. New cstate: current_term: 1 leader_uuid: "7f69193525a4450db0e10a79c6f1fe62" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "231ffd8a64274c199d4a4cbe280942e7" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39287 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "660ba05d01284c07b672e386370434ca" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f69193525a4450db0e10a79c6f1fe62" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36137 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:18.886859 1107 consensus_queue.cc:237] T 3600000d084149d6b4063721e6cf3646 P 7f69193525a4450db0e10a79c6f1fe62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "231ffd8a64274c199d4a4cbe280942e7" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39287 } attrs { replace: true } } peers { permanent_uuid: "660ba05d01284c07b672e386370434ca" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45787 } } peers { permanent_uuid: "7f69193525a4450db0e10a79c6f1fe62" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36137 } } peers { permanent_uuid: "3bdc12f8eb324dbdb62051dfe235f299" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 42765 } attrs { promote: true } }
I20240503 01:59:18.888517 583 raft_consensus.cc:1261] T 3600000d084149d6b4063721e6cf3646 P 231ffd8a64274c199d4a4cbe280942e7 [term 1 FOLLOWER]: Refusing update from remote peer 7f69193525a4450db0e10a79c6f1fe62: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 01:59:18.889039 1354 consensus_queue.cc:1035] T 3600000d084149d6b4063721e6cf3646 P 7f69193525a4450db0e10a79c6f1fe62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "231ffd8a64274c199d4a4cbe280942e7" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39287 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 01:59:18.890301 845 raft_consensus.cc:1261] T 3600000d084149d6b4063721e6cf3646 P 660ba05d01284c07b672e386370434ca [term 1 FOLLOWER]: Refusing update from remote peer 7f69193525a4450db0e10a79c6f1fe62: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 01:59:18.890730 1354 consensus_queue.cc:1035] T 3600000d084149d6b4063721e6cf3646 P 7f69193525a4450db0e10a79c6f1fe62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "660ba05d01284c07b672e386370434ca" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 01:59:18.891153 1482 raft_consensus.cc:2932] T 3600000d084149d6b4063721e6cf3646 P 7f69193525a4450db0e10a79c6f1fe62 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 3bdc12f8eb324dbdb62051dfe235f299 (127.0.99.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "231ffd8a64274c199d4a4cbe280942e7" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39287 } attrs { replace: true } } peers { permanent_uuid: "660ba05d01284c07b672e386370434ca" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45787 } } peers { permanent_uuid: "7f69193525a4450db0e10a79c6f1fe62" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36137 } } peers { permanent_uuid: "3bdc12f8eb324dbdb62051dfe235f299" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 42765 } attrs { promote: true } } }
I20240503 01:59:18.891585 583 raft_consensus.cc:2932] T 3600000d084149d6b4063721e6cf3646 P 231ffd8a64274c199d4a4cbe280942e7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 3bdc12f8eb324dbdb62051dfe235f299 (127.0.99.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "231ffd8a64274c199d4a4cbe280942e7" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39287 } attrs { replace: true } } peers { permanent_uuid: "660ba05d01284c07b672e386370434ca" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45787 } } peers { permanent_uuid: "7f69193525a4450db0e10a79c6f1fe62" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36137 } } peers { permanent_uuid: "3bdc12f8eb324dbdb62051dfe235f299" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 42765 } attrs { promote: true } } }
W20240503 01:59:18.892977 1042 consensus_peers.cc:485] T 3600000d084149d6b4063721e6cf3646 P 7f69193525a4450db0e10a79c6f1fe62 -> Peer 3bdc12f8eb324dbdb62051dfe235f299 (127.0.99.132:42765): Couldn't send request to peer 3bdc12f8eb324dbdb62051dfe235f299. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3600000d084149d6b4063721e6cf3646. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:18.894248 441 catalog_manager.cc:5484] T 3600000d084149d6b4063721e6cf3646 P 231ffd8a64274c199d4a4cbe280942e7 reported cstate change: config changed from index 2 to 3, NON_VOTER 3bdc12f8eb324dbdb62051dfe235f299 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "7f69193525a4450db0e10a79c6f1fe62" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "231ffd8a64274c199d4a4cbe280942e7" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39287 } attrs { replace: true } } peers { permanent_uuid: "660ba05d01284c07b672e386370434ca" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45787 } } peers { permanent_uuid: "7f69193525a4450db0e10a79c6f1fe62" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36137 } } peers { permanent_uuid: "3bdc12f8eb324dbdb62051dfe235f299" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 42765 } attrs { promote: true } } }
I20240503 01:59:18.894655 845 raft_consensus.cc:2932] T 3600000d084149d6b4063721e6cf3646 P 660ba05d01284c07b672e386370434ca [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 3bdc12f8eb324dbdb62051dfe235f299 (127.0.99.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "231ffd8a64274c199d4a4cbe280942e7" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39287 } attrs { replace: true } } peers { permanent_uuid: "660ba05d01284c07b672e386370434ca" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45787 } } peers { permanent_uuid: "7f69193525a4450db0e10a79c6f1fe62" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36137 } } peers { permanent_uuid: "3bdc12f8eb324dbdb62051dfe235f299" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 42765 } attrs { promote: true } } }
I20240503 01:59:18.895097 425 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3600000d084149d6b4063721e6cf3646 with cas_config_opid_index 2: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 01:59:19.429540 1495 ts_tablet_manager.cc:902] T 3600000d084149d6b4063721e6cf3646 P 3bdc12f8eb324dbdb62051dfe235f299: Initiating tablet copy from peer 7f69193525a4450db0e10a79c6f1fe62 (127.0.99.133:36137)
I20240503 01:59:19.430275 1495 tablet_copy_client.cc:308] T 3600000d084149d6b4063721e6cf3646 P 3bdc12f8eb324dbdb62051dfe235f299: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:36137
I20240503 01:59:19.434136 1126 tablet_copy_service.cc:140] P 7f69193525a4450db0e10a79c6f1fe62: Received BeginTabletCopySession request for tablet 3600000d084149d6b4063721e6cf3646 from peer 3bdc12f8eb324dbdb62051dfe235f299 ({username='slave'} at 127.0.99.132:43411)
I20240503 01:59:19.434303 1126 tablet_copy_service.cc:161] P 7f69193525a4450db0e10a79c6f1fe62: Beginning new tablet copy session on tablet 3600000d084149d6b4063721e6cf3646 from peer 3bdc12f8eb324dbdb62051dfe235f299 at {username='slave'} at 127.0.99.132:43411: session id = 3bdc12f8eb324dbdb62051dfe235f299-3600000d084149d6b4063721e6cf3646
I20240503 01:59:19.435516 1126 tablet_copy_source_session.cc:205] T 3600000d084149d6b4063721e6cf3646 P 7f69193525a4450db0e10a79c6f1fe62: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:19.438620 1495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3600000d084149d6b4063721e6cf3646. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:19.443742 1495 tablet_copy_client.cc:783] T 3600000d084149d6b4063721e6cf3646 P 3bdc12f8eb324dbdb62051dfe235f299: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:19.443970 1495 tablet_copy_client.cc:647] T 3600000d084149d6b4063721e6cf3646 P 3bdc12f8eb324dbdb62051dfe235f299: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:19.445982 1495 tablet_copy_client.cc:523] T 3600000d084149d6b4063721e6cf3646 P 3bdc12f8eb324dbdb62051dfe235f299: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:19.447818 1495 tablet_bootstrap.cc:492] T 3600000d084149d6b4063721e6cf3646 P 3bdc12f8eb324dbdb62051dfe235f299: Bootstrap starting.
I20240503 01:59:19.452838 1495 log.cc:824] T 3600000d084149d6b4063721e6cf3646 P 3bdc12f8eb324dbdb62051dfe235f299: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:19.455695 1495 tablet_bootstrap.cc:492] T 3600000d084149d6b4063721e6cf3646 P 3bdc12f8eb324dbdb62051dfe235f299: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:19.455999 1495 tablet_bootstrap.cc:492] T 3600000d084149d6b4063721e6cf3646 P 3bdc12f8eb324dbdb62051dfe235f299: Bootstrap complete.
I20240503 01:59:19.456269 1495 ts_tablet_manager.cc:1367] T 3600000d084149d6b4063721e6cf3646 P 3bdc12f8eb324dbdb62051dfe235f299: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20240503 01:59:19.458817 1495 raft_consensus.cc:348] T 3600000d084149d6b4063721e6cf3646 P 3bdc12f8eb324dbdb62051dfe235f299 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "231ffd8a64274c199d4a4cbe280942e7" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39287 } attrs { replace: true } } peers { permanent_uuid: "660ba05d01284c07b672e386370434ca" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45787 } } peers { permanent_uuid: "7f69193525a4450db0e10a79c6f1fe62" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36137 } } peers { permanent_uuid: "3bdc12f8eb324dbdb62051dfe235f299" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 42765 } attrs { promote: true } }
I20240503 01:59:19.459071 1495 raft_consensus.cc:729] T 3600000d084149d6b4063721e6cf3646 P 3bdc12f8eb324dbdb62051dfe235f299 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3bdc12f8eb324dbdb62051dfe235f299, State: Initialized, Role: LEARNER
I20240503 01:59:19.459367 1495 consensus_queue.cc:260] T 3600000d084149d6b4063721e6cf3646 P 3bdc12f8eb324dbdb62051dfe235f299 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "231ffd8a64274c199d4a4cbe280942e7" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39287 } attrs { replace: true } } peers { permanent_uuid: "660ba05d01284c07b672e386370434ca" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45787 } } peers { permanent_uuid: "7f69193525a4450db0e10a79c6f1fe62" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36137 } } peers { permanent_uuid: "3bdc12f8eb324dbdb62051dfe235f299" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 42765 } attrs { promote: true } }
I20240503 01:59:19.460239 1495 ts_tablet_manager.cc:1398] T 3600000d084149d6b4063721e6cf3646 P 3bdc12f8eb324dbdb62051dfe235f299: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20240503 01:59:19.461110 1126 tablet_copy_service.cc:342] P 7f69193525a4450db0e10a79c6f1fe62: Request end of tablet copy session 3bdc12f8eb324dbdb62051dfe235f299-3600000d084149d6b4063721e6cf3646 received from {username='slave'} at 127.0.99.132:43411
I20240503 01:59:19.461225 1126 tablet_copy_service.cc:433] P 7f69193525a4450db0e10a79c6f1fe62: ending tablet copy session 3bdc12f8eb324dbdb62051dfe235f299-3600000d084149d6b4063721e6cf3646 on tablet 3600000d084149d6b4063721e6cf3646 with peer 3bdc12f8eb324dbdb62051dfe235f299
I20240503 01:59:20.007798 976 raft_consensus.cc:1203] T 3600000d084149d6b4063721e6cf3646 P 3bdc12f8eb324dbdb62051dfe235f299 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3] Dedup: 1.3->[]
I20240503 01:59:20.629878 1536 raft_consensus.cc:1050] T 3600000d084149d6b4063721e6cf3646 P 7f69193525a4450db0e10a79c6f1fe62: attempting to promote NON_VOTER 3bdc12f8eb324dbdb62051dfe235f299 to VOTER
I20240503 01:59:20.630338 1536 consensus_queue.cc:237] T 3600000d084149d6b4063721e6cf3646 P 7f69193525a4450db0e10a79c6f1fe62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231ffd8a64274c199d4a4cbe280942e7" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39287 } attrs { replace: true } } peers { permanent_uuid: "660ba05d01284c07b672e386370434ca" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45787 } } peers { permanent_uuid: "7f69193525a4450db0e10a79c6f1fe62" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36137 } } peers { permanent_uuid: "3bdc12f8eb324dbdb62051dfe235f299" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42765 } attrs { promote: false } }
I20240503 01:59:20.631808 583 raft_consensus.cc:1261] T 3600000d084149d6b4063721e6cf3646 P 231ffd8a64274c199d4a4cbe280942e7 [term 1 FOLLOWER]: Refusing update from remote peer 7f69193525a4450db0e10a79c6f1fe62: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:20.631855 845 raft_consensus.cc:1261] T 3600000d084149d6b4063721e6cf3646 P 660ba05d01284c07b672e386370434ca [term 1 FOLLOWER]: Refusing update from remote peer 7f69193525a4450db0e10a79c6f1fe62: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:20.632323 976 raft_consensus.cc:1261] T 3600000d084149d6b4063721e6cf3646 P 3bdc12f8eb324dbdb62051dfe235f299 [term 1 LEARNER]: Refusing update from remote peer 7f69193525a4450db0e10a79c6f1fe62: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:20.632308 1539 consensus_queue.cc:1035] T 3600000d084149d6b4063721e6cf3646 P 7f69193525a4450db0e10a79c6f1fe62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "231ffd8a64274c199d4a4cbe280942e7" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39287 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:20.632725 1538 consensus_queue.cc:1035] T 3600000d084149d6b4063721e6cf3646 P 7f69193525a4450db0e10a79c6f1fe62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "660ba05d01284c07b672e386370434ca" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:20.632879 1539 consensus_queue.cc:1035] T 3600000d084149d6b4063721e6cf3646 P 7f69193525a4450db0e10a79c6f1fe62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bdc12f8eb324dbdb62051dfe235f299" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42765 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:20.634996 1539 raft_consensus.cc:2932] T 3600000d084149d6b4063721e6cf3646 P 7f69193525a4450db0e10a79c6f1fe62 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 3bdc12f8eb324dbdb62051dfe235f299 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231ffd8a64274c199d4a4cbe280942e7" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39287 } attrs { replace: true } } peers { permanent_uuid: "660ba05d01284c07b672e386370434ca" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45787 } } peers { permanent_uuid: "7f69193525a4450db0e10a79c6f1fe62" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36137 } } peers { permanent_uuid: "3bdc12f8eb324dbdb62051dfe235f299" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42765 } attrs { promote: false } } }
I20240503 01:59:20.635530 583 raft_consensus.cc:2932] T 3600000d084149d6b4063721e6cf3646 P 231ffd8a64274c199d4a4cbe280942e7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 3bdc12f8eb324dbdb62051dfe235f299 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231ffd8a64274c199d4a4cbe280942e7" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39287 } attrs { replace: true } } peers { permanent_uuid: "660ba05d01284c07b672e386370434ca" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45787 } } peers { permanent_uuid: "7f69193525a4450db0e10a79c6f1fe62" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36137 } } peers { permanent_uuid: "3bdc12f8eb324dbdb62051dfe235f299" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42765 } attrs { promote: false } } }
I20240503 01:59:20.636884 845 raft_consensus.cc:2932] T 3600000d084149d6b4063721e6cf3646 P 660ba05d01284c07b672e386370434ca [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 3bdc12f8eb324dbdb62051dfe235f299 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231ffd8a64274c199d4a4cbe280942e7" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39287 } attrs { replace: true } } peers { permanent_uuid: "660ba05d01284c07b672e386370434ca" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45787 } } peers { permanent_uuid: "7f69193525a4450db0e10a79c6f1fe62" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36137 } } peers { permanent_uuid: "3bdc12f8eb324dbdb62051dfe235f299" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42765 } attrs { promote: false } } }
I20240503 01:59:20.638070 438 catalog_manager.cc:5484] T 3600000d084149d6b4063721e6cf3646 P 231ffd8a64274c199d4a4cbe280942e7 reported cstate change: config changed from index 3 to 4, 3bdc12f8eb324dbdb62051dfe235f299 (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7f69193525a4450db0e10a79c6f1fe62" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231ffd8a64274c199d4a4cbe280942e7" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39287 } attrs { replace: true } } peers { permanent_uuid: "660ba05d01284c07b672e386370434ca" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45787 } } peers { permanent_uuid: "7f69193525a4450db0e10a79c6f1fe62" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36137 } } peers { permanent_uuid: "3bdc12f8eb324dbdb62051dfe235f299" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42765 } attrs { promote: false } } }
I20240503 01:59:20.638231 976 raft_consensus.cc:2932] T 3600000d084149d6b4063721e6cf3646 P 3bdc12f8eb324dbdb62051dfe235f299 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 3bdc12f8eb324dbdb62051dfe235f299 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "231ffd8a64274c199d4a4cbe280942e7" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39287 } attrs { replace: true } } peers { permanent_uuid: "660ba05d01284c07b672e386370434ca" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45787 } } peers { permanent_uuid: "7f69193525a4450db0e10a79c6f1fe62" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36137 } } peers { permanent_uuid: "3bdc12f8eb324dbdb62051dfe235f299" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42765 } attrs { promote: false } } }
I20240503 01:59:20.644479 1107 consensus_queue.cc:237] T 3600000d084149d6b4063721e6cf3646 P 7f69193525a4450db0e10a79c6f1fe62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "660ba05d01284c07b672e386370434ca" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45787 } } peers { permanent_uuid: "7f69193525a4450db0e10a79c6f1fe62" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36137 } } peers { permanent_uuid: "3bdc12f8eb324dbdb62051dfe235f299" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42765 } attrs { promote: false } }
I20240503 01:59:20.645871 845 raft_consensus.cc:1261] T 3600000d084149d6b4063721e6cf3646 P 660ba05d01284c07b672e386370434ca [term 1 FOLLOWER]: Refusing update from remote peer 7f69193525a4450db0e10a79c6f1fe62: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:20.645941 976 raft_consensus.cc:1261] T 3600000d084149d6b4063721e6cf3646 P 3bdc12f8eb324dbdb62051dfe235f299 [term 1 FOLLOWER]: Refusing update from remote peer 7f69193525a4450db0e10a79c6f1fe62: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:20.646328 1536 consensus_queue.cc:1035] T 3600000d084149d6b4063721e6cf3646 P 7f69193525a4450db0e10a79c6f1fe62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "660ba05d01284c07b672e386370434ca" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:20.646559 1538 consensus_queue.cc:1035] T 3600000d084149d6b4063721e6cf3646 P 7f69193525a4450db0e10a79c6f1fe62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bdc12f8eb324dbdb62051dfe235f299" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42765 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:20.648313 1519 raft_consensus.cc:2932] T 3600000d084149d6b4063721e6cf3646 P 7f69193525a4450db0e10a79c6f1fe62 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 231ffd8a64274c199d4a4cbe280942e7 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "660ba05d01284c07b672e386370434ca" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45787 } } peers { permanent_uuid: "7f69193525a4450db0e10a79c6f1fe62" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36137 } } peers { permanent_uuid: "3bdc12f8eb324dbdb62051dfe235f299" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42765 } attrs { promote: false } } }
I20240503 01:59:20.648811 845 raft_consensus.cc:2932] T 3600000d084149d6b4063721e6cf3646 P 660ba05d01284c07b672e386370434ca [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 231ffd8a64274c199d4a4cbe280942e7 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "660ba05d01284c07b672e386370434ca" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45787 } } peers { permanent_uuid: "7f69193525a4450db0e10a79c6f1fe62" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36137 } } peers { permanent_uuid: "3bdc12f8eb324dbdb62051dfe235f299" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42765 } attrs { promote: false } } }
I20240503 01:59:20.649760 425 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3600000d084149d6b4063721e6cf3646 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:20.650207 976 raft_consensus.cc:2932] T 3600000d084149d6b4063721e6cf3646 P 3bdc12f8eb324dbdb62051dfe235f299 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 231ffd8a64274c199d4a4cbe280942e7 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "660ba05d01284c07b672e386370434ca" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45787 } } peers { permanent_uuid: "7f69193525a4450db0e10a79c6f1fe62" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36137 } } peers { permanent_uuid: "3bdc12f8eb324dbdb62051dfe235f299" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42765 } attrs { promote: false } } }
I20240503 01:59:20.650916 440 catalog_manager.cc:5484] T 3600000d084149d6b4063721e6cf3646 P 7f69193525a4450db0e10a79c6f1fe62 reported cstate change: config changed from index 4 to 5, VOTER 231ffd8a64274c199d4a4cbe280942e7 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "7f69193525a4450db0e10a79c6f1fe62" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "660ba05d01284c07b672e386370434ca" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f69193525a4450db0e10a79c6f1fe62" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36137 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3bdc12f8eb324dbdb62051dfe235f299" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42765 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:20.654402 563 tablet_service.cc:1514] Processing DeleteTablet for tablet 3600000d084149d6b4063721e6cf3646 with delete_type TABLET_DATA_TOMBSTONED (TS 231ffd8a64274c199d4a4cbe280942e7 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:59120
I20240503 01:59:20.654985 1550 tablet_replica.cc:330] T 3600000d084149d6b4063721e6cf3646 P 231ffd8a64274c199d4a4cbe280942e7: stopping tablet replica
I20240503 01:59:20.655439 1550 raft_consensus.cc:2229] T 3600000d084149d6b4063721e6cf3646 P 231ffd8a64274c199d4a4cbe280942e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:20.655730 1550 raft_consensus.cc:2258] T 3600000d084149d6b4063721e6cf3646 P 231ffd8a64274c199d4a4cbe280942e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:20.656797 1550 ts_tablet_manager.cc:1875] T 3600000d084149d6b4063721e6cf3646 P 231ffd8a64274c199d4a4cbe280942e7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:20.659022 1550 ts_tablet_manager.cc:1888] T 3600000d084149d6b4063721e6cf3646 P 231ffd8a64274c199d4a4cbe280942e7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 01:59:20.659111 1550 log.cc:1196] T 3600000d084149d6b4063721e6cf3646 P 231ffd8a64274c199d4a4cbe280942e7: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/3600000d084149d6b4063721e6cf3646
I20240503 01:59:20.659685 425 catalog_manager.cc:4830] TS 231ffd8a64274c199d4a4cbe280942e7 (127.0.99.129:39287): tablet 3600000d084149d6b4063721e6cf3646 (table TestTable [id=94b8bbcc49d6475b96f26e8b26a5d962]) successfully deleted
I20240503 01:59:21.015214 956 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:21.015573 694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:21.016551 825 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:21.024928 563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:21.025609 1086 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:21.110599 563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:21.110880 956 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:21.112653 825 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:21.113580 694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:21.120936 1086 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:21.198838 563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:21.199434 825 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:21.201223 956 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:21.205248 694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:21.206188 1086 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:21.238610 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu with pid 632
I20240503 01:59:21.434787 956 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:21.435714 563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:21.437171 825 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:21.440642 1086 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:21.513803 825 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:21.514503 956 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:21.514891 563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:21.515400 1086 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:21.630875 563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:21.631682 956 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:21.632520 825 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:21.632939 1086 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:21.721639 563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:21.723559 956 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:21.725243 825 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:21.728777 1086 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:21.766547 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu with pid 501
I20240503 01:59:21.772265 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu with pid 763
I20240503 01:59:21.777464 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu with pid 894
I20240503 01:59:21.782754 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu with pid 1025
I20240503 01:59:21.787683 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu with pid 410
2024-05-03T01:59:21Z chronyd exiting
[ OK ] RebalanceIgnoredTserversTest.Basic (6139 ms)
[----------] 1 test from RebalanceIgnoredTserversTest (6139 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.TabletServerIsDown/1
I20240503 01:59:21.809337 398 test_util.cc:268] Using random seed: 293390771
I20240503 01:59:21.810114 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 01:59:21.810158 398 ts_itest-base.cc:113] --------------
I20240503 01:59:21.810168 398 ts_itest-base.cc:114] 5 tablet servers
I20240503 01:59:21.810179 398 ts_itest-base.cc:115] 3 replicas per TS
I20240503 01:59:21.810187 398 ts_itest-base.cc:116] --------------
2024-05-03T01:59:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T01:59:21Z Disabled control of system clock
I20240503 01:59:21.817957 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
/tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:43525
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:39513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:43525
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240503 01:59:21.932299 1769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:21.932605 1769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:21.932664 1769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:21.936067 1769 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:21.936203 1769 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 01:59:21.936231 1769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:21.936257 1769 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 01:59:21.936297 1769 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 01:59:21.940941 1769 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:43525
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:43525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.1769
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:21.942049 1769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:21.943262 1769 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:21.951035 1777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:21.950990 1774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:21.951339 1775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:21.951534 1769 server_base.cc:1034] running on GCE node
I20240503 01:59:21.952136 1769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:21.953270 1769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:21.954475 1769 hybrid_clock.cc:648] HybridClock initialized: now 1714701561954450 us; error 32 us; skew 500 ppm
I20240503 01:59:21.956739 1769 webserver.cc:460] Webserver started at http://127.0.99.190:37185/ using document root <none> and password file <none>
I20240503 01:59:21.957336 1769 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:21.957399 1769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:21.957582 1769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:21.959370 1769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "75d4a116cf214506bd206c7d16a30d2a"
format_stamp: "Formatted at 2024-05-03 01:59:21 on dist-test-slave-pz5w"
I20240503 01:59:21.959869 1769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "75d4a116cf214506bd206c7d16a30d2a"
format_stamp: "Formatted at 2024-05-03 01:59:21 on dist-test-slave-pz5w"
I20240503 01:59:21.963606 1769 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20240503 01:59:21.966281 1783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:21.967631 1769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:21.967798 1769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "75d4a116cf214506bd206c7d16a30d2a"
format_stamp: "Formatted at 2024-05-03 01:59:21 on dist-test-slave-pz5w"
I20240503 01:59:21.967909 1769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:21.988551 1769 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:21.989265 1769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:21.989439 1769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:21.996024 1835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:43525 every 8 connection(s)
I20240503 01:59:21.996021 1769 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:43525
I20240503 01:59:21.997150 1769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 01:59:21.999815 1836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:22.003948 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu as pid 1769
I20240503 01:59:22.004076 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 01:59:22.005407 1836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75d4a116cf214506bd206c7d16a30d2a: Bootstrap starting.
I20240503 01:59:22.007699 1836 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75d4a116cf214506bd206c7d16a30d2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:22.008453 1836 log.cc:824] T 00000000000000000000000000000000 P 75d4a116cf214506bd206c7d16a30d2a: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:22.010358 1836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75d4a116cf214506bd206c7d16a30d2a: No bootstrap required, opened a new log
I20240503 01:59:22.013432 1836 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75d4a116cf214506bd206c7d16a30d2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75d4a116cf214506bd206c7d16a30d2a" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 43525 } }
I20240503 01:59:22.013650 1836 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75d4a116cf214506bd206c7d16a30d2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:22.013701 1836 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75d4a116cf214506bd206c7d16a30d2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75d4a116cf214506bd206c7d16a30d2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:22.014158 1836 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75d4a116cf214506bd206c7d16a30d2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75d4a116cf214506bd206c7d16a30d2a" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 43525 } }
I20240503 01:59:22.014292 1836 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75d4a116cf214506bd206c7d16a30d2a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 01:59:22.014360 1836 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75d4a116cf214506bd206c7d16a30d2a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 01:59:22.014465 1836 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75d4a116cf214506bd206c7d16a30d2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:22.038655 1836 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75d4a116cf214506bd206c7d16a30d2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75d4a116cf214506bd206c7d16a30d2a" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 43525 } }
I20240503 01:59:22.039181 1836 leader_election.cc:302] T 00000000000000000000000000000000 P 75d4a116cf214506bd206c7d16a30d2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75d4a116cf214506bd206c7d16a30d2a; no voters:
I20240503 01:59:22.039572 1836 leader_election.cc:288] T 00000000000000000000000000000000 P 75d4a116cf214506bd206c7d16a30d2a [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 01:59:22.039700 1841 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75d4a116cf214506bd206c7d16a30d2a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:22.039929 1841 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75d4a116cf214506bd206c7d16a30d2a [term 1 LEADER]: Becoming Leader. State: Replica: 75d4a116cf214506bd206c7d16a30d2a, State: Running, Role: LEADER
I20240503 01:59:22.040369 1841 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75d4a116cf214506bd206c7d16a30d2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75d4a116cf214506bd206c7d16a30d2a" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 43525 } }
I20240503 01:59:22.040824 1836 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75d4a116cf214506bd206c7d16a30d2a [sys.catalog]: configured and running, proceeding with master startup.
I20240503 01:59:22.042162 1842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75d4a116cf214506bd206c7d16a30d2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75d4a116cf214506bd206c7d16a30d2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75d4a116cf214506bd206c7d16a30d2a" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 43525 } } }
I20240503 01:59:22.042179 1843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75d4a116cf214506bd206c7d16a30d2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75d4a116cf214506bd206c7d16a30d2a. Latest consensus state: current_term: 1 leader_uuid: "75d4a116cf214506bd206c7d16a30d2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75d4a116cf214506bd206c7d16a30d2a" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 43525 } } }
I20240503 01:59:22.042323 1843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75d4a116cf214506bd206c7d16a30d2a [sys.catalog]: This master's current role is: LEADER
I20240503 01:59:22.042323 1842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75d4a116cf214506bd206c7d16a30d2a [sys.catalog]: This master's current role is: LEADER
I20240503 01:59:22.042722 1850 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 01:59:22.046090 1850 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 01:59:22.051762 1850 catalog_manager.cc:1347] Generated new cluster ID: e3112c5a530f4a7caef95f6df851763f
I20240503 01:59:22.051869 1850 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 01:59:22.068284 1850 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 01:59:22.069482 1850 catalog_manager.cc:1504] Loading token signing keys...
I20240503 01:59:22.076474 1850 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 75d4a116cf214506bd206c7d16a30d2a: Generated new TSK 0
I20240503 01:59:22.077288 1850 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 01:59:22.090714 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
/tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43525
--builtin_ntp_servers=127.0.99.148:39513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 01:59:22.209959 1860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:22.210299 1860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:22.210377 1860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:22.213779 1860 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:22.213871 1860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:22.213951 1860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 01:59:22.218205 1860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43525
--never_fsync=true
--heap_profile_path=/tmp/kudu.1860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:22.219220 1860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:22.220485 1860 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:22.229216 1866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:22.229216 1868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:22.229463 1860 server_base.cc:1034] running on GCE node
W20240503 01:59:22.229216 1865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:22.229986 1860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:22.230640 1860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:22.231843 1860 hybrid_clock.cc:648] HybridClock initialized: now 1714701562231813 us; error 50 us; skew 500 ppm
I20240503 01:59:22.234222 1860 webserver.cc:460] Webserver started at http://127.0.99.129:40395/ using document root <none> and password file <none>
I20240503 01:59:22.234799 1860 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:22.234864 1860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:22.235069 1860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:22.236999 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "961b13c11f8d4ce8a7c5434f38951f49"
format_stamp: "Formatted at 2024-05-03 01:59:22 on dist-test-slave-pz5w"
I20240503 01:59:22.237519 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "961b13c11f8d4ce8a7c5434f38951f49"
format_stamp: "Formatted at 2024-05-03 01:59:22 on dist-test-slave-pz5w"
I20240503 01:59:22.241616 1860 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240503 01:59:22.244258 1874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:22.245587 1860 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:22.245735 1860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "961b13c11f8d4ce8a7c5434f38951f49"
format_stamp: "Formatted at 2024-05-03 01:59:22 on dist-test-slave-pz5w"
I20240503 01:59:22.245843 1860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:22.264694 1860 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:22.265376 1860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:22.265554 1860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:22.266103 1860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:22.267046 1860 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:22.267097 1860 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:22.267151 1860 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:22.267180 1860 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:22.275790 1860 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:40959
I20240503 01:59:22.275828 1987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:40959 every 8 connection(s)
I20240503 01:59:22.276806 1860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 01:59:22.277086 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu as pid 1860
I20240503 01:59:22.277184 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 01:59:22.279443 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
/tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43525
--builtin_ntp_servers=127.0.99.148:39513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 01:59:22.288440 1988 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43525
I20240503 01:59:22.288699 1988 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:22.289310 1988 heartbeater.cc:511] Master 127.0.99.190:43525 requested a full tablet report, sending...
I20240503 01:59:22.291038 1800 ts_manager.cc:194] Registered new tserver with Master: 961b13c11f8d4ce8a7c5434f38951f49 (127.0.99.129:40959)
I20240503 01:59:22.292846 1800 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:50741
W20240503 01:59:22.398219 1991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:22.398568 1991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:22.398644 1991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:22.402714 1991 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:22.402819 1991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:22.402925 1991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 01:59:22.407406 1991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43525
--never_fsync=true
--heap_profile_path=/tmp/kudu.1991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:22.408407 1991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:22.409585 1991 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:22.417919 1996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:22.417949 1999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:22.418082 1991 server_base.cc:1034] running on GCE node
W20240503 01:59:22.417930 1997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:22.418569 1991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:22.419194 1991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:22.420357 1991 hybrid_clock.cc:648] HybridClock initialized: now 1714701562420335 us; error 42 us; skew 500 ppm
I20240503 01:59:22.422397 1991 webserver.cc:460] Webserver started at http://127.0.99.130:38927/ using document root <none> and password file <none>
I20240503 01:59:22.422973 1991 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:22.423036 1991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:22.423211 1991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:22.424996 1991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "46f764d98fc64dde8e370450c4c3ff8e"
format_stamp: "Formatted at 2024-05-03 01:59:22 on dist-test-slave-pz5w"
I20240503 01:59:22.425516 1991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "46f764d98fc64dde8e370450c4c3ff8e"
format_stamp: "Formatted at 2024-05-03 01:59:22 on dist-test-slave-pz5w"
I20240503 01:59:22.429108 1991 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20240503 01:59:22.431442 2005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:22.432533 1991 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20240503 01:59:22.432655 1991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "46f764d98fc64dde8e370450c4c3ff8e"
format_stamp: "Formatted at 2024-05-03 01:59:22 on dist-test-slave-pz5w"
I20240503 01:59:22.432750 1991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:22.450130 1991 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:22.450817 1991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:22.450981 1991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:22.451524 1991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:22.452457 1991 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:22.452504 1991 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:22.452560 1991 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:22.452585 1991 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:22.461128 1991 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:40667
I20240503 01:59:22.461155 2118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:40667 every 8 connection(s)
I20240503 01:59:22.462131 1991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 01:59:22.466212 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu as pid 1991
I20240503 01:59:22.466328 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 01:59:22.468310 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
/tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43525
--builtin_ntp_servers=127.0.99.148:39513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 01:59:22.471443 2119 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43525
I20240503 01:59:22.471695 2119 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:22.472429 2119 heartbeater.cc:511] Master 127.0.99.190:43525 requested a full tablet report, sending...
I20240503 01:59:22.473567 1800 ts_manager.cc:194] Registered new tserver with Master: 46f764d98fc64dde8e370450c4c3ff8e (127.0.99.130:40667)
I20240503 01:59:22.474138 1800 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:45259
W20240503 01:59:22.580476 2122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:22.580753 2122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:22.580838 2122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:22.584077 2122 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:22.584169 2122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:22.584268 2122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 01:59:22.588157 2122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43525
--never_fsync=true
--heap_profile_path=/tmp/kudu.2122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:22.589181 2122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:22.590332 2122 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:22.598731 2130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:22.598804 2122 server_base.cc:1034] running on GCE node
W20240503 01:59:22.598731 2127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:22.598964 2128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:22.599387 2122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:22.599982 2122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:22.601167 2122 hybrid_clock.cc:648] HybridClock initialized: now 1714701562601131 us; error 58 us; skew 500 ppm
I20240503 01:59:22.603377 2122 webserver.cc:460] Webserver started at http://127.0.99.131:35889/ using document root <none> and password file <none>
I20240503 01:59:22.603951 2122 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:22.604010 2122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:22.604194 2122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:22.605976 2122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "6e31962e88794140906e86020cf2ff16"
format_stamp: "Formatted at 2024-05-03 01:59:22 on dist-test-slave-pz5w"
I20240503 01:59:22.606470 2122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "6e31962e88794140906e86020cf2ff16"
format_stamp: "Formatted at 2024-05-03 01:59:22 on dist-test-slave-pz5w"
I20240503 01:59:22.610086 2122 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 01:59:22.612529 2136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:22.613620 2122 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 01:59:22.613751 2122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "6e31962e88794140906e86020cf2ff16"
format_stamp: "Formatted at 2024-05-03 01:59:22 on dist-test-slave-pz5w"
I20240503 01:59:22.613866 2122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:22.637401 2122 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:22.638041 2122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:22.638212 2122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:22.638760 2122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:22.639681 2122 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:22.639730 2122 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:22.639787 2122 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:22.639811 2122 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:22.648555 2122 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:34445
I20240503 01:59:22.648578 2249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:34445 every 8 connection(s)
I20240503 01:59:22.649796 2122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 01:59:22.654513 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu as pid 2122
I20240503 01:59:22.654634 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 01:59:22.657254 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
/tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43525
--builtin_ntp_servers=127.0.99.148:39513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 01:59:22.660763 2250 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43525
I20240503 01:59:22.661026 2250 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:22.661628 2250 heartbeater.cc:511] Master 127.0.99.190:43525 requested a full tablet report, sending...
I20240503 01:59:22.662899 1800 ts_manager.cc:194] Registered new tserver with Master: 6e31962e88794140906e86020cf2ff16 (127.0.99.131:34445)
I20240503 01:59:22.663692 1800 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:49447
W20240503 01:59:22.768025 2253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:22.768303 2253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:22.768348 2253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:22.771543 2253 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:22.771628 2253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:22.771698 2253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 01:59:22.775508 2253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43525
--never_fsync=true
--heap_profile_path=/tmp/kudu.2253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:22.776515 2253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:22.777652 2253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:22.786190 2259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:22.786175 2258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:22.786263 2253 server_base.cc:1034] running on GCE node
W20240503 01:59:22.786175 2261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:22.786831 2253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:22.787415 2253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:22.788564 2253 hybrid_clock.cc:648] HybridClock initialized: now 1714701562788538 us; error 48 us; skew 500 ppm
I20240503 01:59:22.790640 2253 webserver.cc:460] Webserver started at http://127.0.99.132:34209/ using document root <none> and password file <none>
I20240503 01:59:22.791210 2253 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:22.791270 2253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:22.791468 2253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:22.793357 2253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "c6d019d21f184f50a523835d7f4ca7b3"
format_stamp: "Formatted at 2024-05-03 01:59:22 on dist-test-slave-pz5w"
I20240503 01:59:22.793851 2253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "c6d019d21f184f50a523835d7f4ca7b3"
format_stamp: "Formatted at 2024-05-03 01:59:22 on dist-test-slave-pz5w"
I20240503 01:59:22.797478 2253 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20240503 01:59:22.799897 2267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:22.801025 2253 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 01:59:22.801151 2253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "c6d019d21f184f50a523835d7f4ca7b3"
format_stamp: "Formatted at 2024-05-03 01:59:22 on dist-test-slave-pz5w"
I20240503 01:59:22.801266 2253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:22.816344 2253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:22.817031 2253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:22.817195 2253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:22.817761 2253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:22.818677 2253 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:22.818727 2253 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:22.818783 2253 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:22.818814 2253 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:22.827342 2253 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:45449
I20240503 01:59:22.827350 2380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:45449 every 8 connection(s)
I20240503 01:59:22.828341 2253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 01:59:22.833760 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu as pid 2253
I20240503 01:59:22.833899 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 01:59:22.836133 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
/tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43525
--builtin_ntp_servers=127.0.99.148:39513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 01:59:22.838176 2381 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43525
I20240503 01:59:22.838434 2381 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:22.839040 2381 heartbeater.cc:511] Master 127.0.99.190:43525 requested a full tablet report, sending...
I20240503 01:59:22.840153 1800 ts_manager.cc:194] Registered new tserver with Master: c6d019d21f184f50a523835d7f4ca7b3 (127.0.99.132:45449)
I20240503 01:59:22.840797 1800 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:43183
W20240503 01:59:22.946236 2384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:22.946506 2384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:22.946570 2384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:22.949820 2384 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:22.949909 2384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:22.950001 2384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 01:59:22.953929 2384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43525
--never_fsync=true
--heap_profile_path=/tmp/kudu.2384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:22.954929 2384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:22.956132 2384 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:22.964479 2390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:22.964510 2392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:22.964480 2389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:22.964900 2384 server_base.cc:1034] running on GCE node
I20240503 01:59:22.965334 2384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:22.965945 2384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:22.967145 2384 hybrid_clock.cc:648] HybridClock initialized: now 1714701562967111 us; error 47 us; skew 500 ppm
I20240503 01:59:22.969341 2384 webserver.cc:460] Webserver started at http://127.0.99.133:34561/ using document root <none> and password file <none>
I20240503 01:59:22.969933 2384 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:22.969998 2384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:22.970196 2384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:22.972081 2384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "d3ea3c5c177f44938e0e81020aa69da2"
format_stamp: "Formatted at 2024-05-03 01:59:22 on dist-test-slave-pz5w"
I20240503 01:59:22.972661 2384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "d3ea3c5c177f44938e0e81020aa69da2"
format_stamp: "Formatted at 2024-05-03 01:59:22 on dist-test-slave-pz5w"
I20240503 01:59:22.976229 2384 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 01:59:22.978682 2398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:22.979799 2384 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:22.979918 2384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "d3ea3c5c177f44938e0e81020aa69da2"
format_stamp: "Formatted at 2024-05-03 01:59:22 on dist-test-slave-pz5w"
I20240503 01:59:22.980020 2384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:23.005163 2384 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:23.005884 2384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:23.006067 2384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:23.006691 2384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:23.007768 2384 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:23.007830 2384 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:23.007881 2384 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:23.007907 2384 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:23.016862 2384 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:32927
I20240503 01:59:23.016892 2511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:32927 every 8 connection(s)
I20240503 01:59:23.017918 2384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 01:59:23.022408 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu as pid 2384
I20240503 01:59:23.022508 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 01:59:23.028754 2512 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43525
I20240503 01:59:23.029038 2512 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:23.029637 2512 heartbeater.cc:511] Master 127.0.99.190:43525 requested a full tablet report, sending...
I20240503 01:59:23.030725 1800 ts_manager.cc:194] Registered new tserver with Master: d3ea3c5c177f44938e0e81020aa69da2 (127.0.99.133:32927)
I20240503 01:59:23.031287 1800 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:41063
I20240503 01:59:23.037734 398 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 01:59:23.047689 1800 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47988:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 01:59:23.066474 1922 tablet_service.cc:1467] Processing CreateTablet for tablet b4f90d0644504e89ab392f71ba39e8b8 (DEFAULT_TABLE table=TestTable [id=d118e98bb0e1441e9bd0dd17e77d5fc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:23.067473 1922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4f90d0644504e89ab392f71ba39e8b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:23.067140 2315 tablet_service.cc:1467] Processing CreateTablet for tablet b4f90d0644504e89ab392f71ba39e8b8 (DEFAULT_TABLE table=TestTable [id=d118e98bb0e1441e9bd0dd17e77d5fc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:23.067140 2184 tablet_service.cc:1467] Processing CreateTablet for tablet b4f90d0644504e89ab392f71ba39e8b8 (DEFAULT_TABLE table=TestTable [id=d118e98bb0e1441e9bd0dd17e77d5fc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:23.068096 2184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4f90d0644504e89ab392f71ba39e8b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:23.068096 2315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4f90d0644504e89ab392f71ba39e8b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:23.073274 2529 tablet_bootstrap.cc:492] T b4f90d0644504e89ab392f71ba39e8b8 P 961b13c11f8d4ce8a7c5434f38951f49: Bootstrap starting.
I20240503 01:59:23.074268 2531 tablet_bootstrap.cc:492] T b4f90d0644504e89ab392f71ba39e8b8 P c6d019d21f184f50a523835d7f4ca7b3: Bootstrap starting.
I20240503 01:59:23.075824 2529 tablet_bootstrap.cc:654] T b4f90d0644504e89ab392f71ba39e8b8 P 961b13c11f8d4ce8a7c5434f38951f49: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:23.076612 2531 tablet_bootstrap.cc:654] T b4f90d0644504e89ab392f71ba39e8b8 P c6d019d21f184f50a523835d7f4ca7b3: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:23.076663 2530 tablet_bootstrap.cc:492] T b4f90d0644504e89ab392f71ba39e8b8 P 6e31962e88794140906e86020cf2ff16: Bootstrap starting.
I20240503 01:59:23.076750 2529 log.cc:824] T b4f90d0644504e89ab392f71ba39e8b8 P 961b13c11f8d4ce8a7c5434f38951f49: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:23.077610 2531 log.cc:824] T b4f90d0644504e89ab392f71ba39e8b8 P c6d019d21f184f50a523835d7f4ca7b3: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:23.078646 2529 tablet_bootstrap.cc:492] T b4f90d0644504e89ab392f71ba39e8b8 P 961b13c11f8d4ce8a7c5434f38951f49: No bootstrap required, opened a new log
I20240503 01:59:23.078662 2530 tablet_bootstrap.cc:654] T b4f90d0644504e89ab392f71ba39e8b8 P 6e31962e88794140906e86020cf2ff16: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:23.078842 2529 ts_tablet_manager.cc:1367] T b4f90d0644504e89ab392f71ba39e8b8 P 961b13c11f8d4ce8a7c5434f38951f49: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 01:59:23.079337 2531 tablet_bootstrap.cc:492] T b4f90d0644504e89ab392f71ba39e8b8 P c6d019d21f184f50a523835d7f4ca7b3: No bootstrap required, opened a new log
I20240503 01:59:23.079335 2530 log.cc:824] T b4f90d0644504e89ab392f71ba39e8b8 P 6e31962e88794140906e86020cf2ff16: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:23.079514 2531 ts_tablet_manager.cc:1367] T b4f90d0644504e89ab392f71ba39e8b8 P c6d019d21f184f50a523835d7f4ca7b3: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 01:59:23.080796 2530 tablet_bootstrap.cc:492] T b4f90d0644504e89ab392f71ba39e8b8 P 6e31962e88794140906e86020cf2ff16: No bootstrap required, opened a new log
I20240503 01:59:23.080988 2530 ts_tablet_manager.cc:1367] T b4f90d0644504e89ab392f71ba39e8b8 P 6e31962e88794140906e86020cf2ff16: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 01:59:23.082345 2529 raft_consensus.cc:348] T b4f90d0644504e89ab392f71ba39e8b8 P 961b13c11f8d4ce8a7c5434f38951f49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "961b13c11f8d4ce8a7c5434f38951f49" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40959 } } peers { permanent_uuid: "c6d019d21f184f50a523835d7f4ca7b3" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45449 } } peers { permanent_uuid: "6e31962e88794140906e86020cf2ff16" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34445 } }
I20240503 01:59:23.082617 2529 raft_consensus.cc:374] T b4f90d0644504e89ab392f71ba39e8b8 P 961b13c11f8d4ce8a7c5434f38951f49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:23.082679 2529 raft_consensus.cc:729] T b4f90d0644504e89ab392f71ba39e8b8 P 961b13c11f8d4ce8a7c5434f38951f49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 961b13c11f8d4ce8a7c5434f38951f49, State: Initialized, Role: FOLLOWER
I20240503 01:59:23.083173 2529 consensus_queue.cc:260] T b4f90d0644504e89ab392f71ba39e8b8 P 961b13c11f8d4ce8a7c5434f38951f49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "961b13c11f8d4ce8a7c5434f38951f49" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40959 } } peers { permanent_uuid: "c6d019d21f184f50a523835d7f4ca7b3" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45449 } } peers { permanent_uuid: "6e31962e88794140906e86020cf2ff16" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34445 } }
I20240503 01:59:23.083986 1988 heartbeater.cc:503] Master 127.0.99.190:43525 was elected leader, sending a full tablet report...
I20240503 01:59:23.084193 2529 ts_tablet_manager.cc:1398] T b4f90d0644504e89ab392f71ba39e8b8 P 961b13c11f8d4ce8a7c5434f38951f49: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240503 01:59:23.083648 2531 raft_consensus.cc:348] T b4f90d0644504e89ab392f71ba39e8b8 P c6d019d21f184f50a523835d7f4ca7b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "961b13c11f8d4ce8a7c5434f38951f49" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40959 } } peers { permanent_uuid: "c6d019d21f184f50a523835d7f4ca7b3" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45449 } } peers { permanent_uuid: "6e31962e88794140906e86020cf2ff16" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34445 } }
I20240503 01:59:23.084537 2531 raft_consensus.cc:374] T b4f90d0644504e89ab392f71ba39e8b8 P c6d019d21f184f50a523835d7f4ca7b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:23.084439 2530 raft_consensus.cc:348] T b4f90d0644504e89ab392f71ba39e8b8 P 6e31962e88794140906e86020cf2ff16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "961b13c11f8d4ce8a7c5434f38951f49" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40959 } } peers { permanent_uuid: "c6d019d21f184f50a523835d7f4ca7b3" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45449 } } peers { permanent_uuid: "6e31962e88794140906e86020cf2ff16" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34445 } }
I20240503 01:59:23.084615 2531 raft_consensus.cc:729] T b4f90d0644504e89ab392f71ba39e8b8 P c6d019d21f184f50a523835d7f4ca7b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6d019d21f184f50a523835d7f4ca7b3, State: Initialized, Role: FOLLOWER
I20240503 01:59:23.084652 2530 raft_consensus.cc:374] T b4f90d0644504e89ab392f71ba39e8b8 P 6e31962e88794140906e86020cf2ff16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:23.084717 2530 raft_consensus.cc:729] T b4f90d0644504e89ab392f71ba39e8b8 P 6e31962e88794140906e86020cf2ff16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e31962e88794140906e86020cf2ff16, State: Initialized, Role: FOLLOWER
I20240503 01:59:23.085067 2531 consensus_queue.cc:260] T b4f90d0644504e89ab392f71ba39e8b8 P c6d019d21f184f50a523835d7f4ca7b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "961b13c11f8d4ce8a7c5434f38951f49" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40959 } } peers { permanent_uuid: "c6d019d21f184f50a523835d7f4ca7b3" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45449 } } peers { permanent_uuid: "6e31962e88794140906e86020cf2ff16" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34445 } }
I20240503 01:59:23.085110 2530 consensus_queue.cc:260] T b4f90d0644504e89ab392f71ba39e8b8 P 6e31962e88794140906e86020cf2ff16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "961b13c11f8d4ce8a7c5434f38951f49" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40959 } } peers { permanent_uuid: "c6d019d21f184f50a523835d7f4ca7b3" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45449 } } peers { permanent_uuid: "6e31962e88794140906e86020cf2ff16" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34445 } }
I20240503 01:59:23.085769 2381 heartbeater.cc:503] Master 127.0.99.190:43525 was elected leader, sending a full tablet report...
I20240503 01:59:23.085971 2531 ts_tablet_manager.cc:1398] T b4f90d0644504e89ab392f71ba39e8b8 P c6d019d21f184f50a523835d7f4ca7b3: Time spent starting tablet: real 0.006s user 0.000s sys 0.006s
I20240503 01:59:23.085979 2530 ts_tablet_manager.cc:1398] T b4f90d0644504e89ab392f71ba39e8b8 P 6e31962e88794140906e86020cf2ff16: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240503 01:59:23.086309 2250 heartbeater.cc:503] Master 127.0.99.190:43525 was elected leader, sending a full tablet report...
I20240503 01:59:23.137463 2535 raft_consensus.cc:482] T b4f90d0644504e89ab392f71ba39e8b8 P 961b13c11f8d4ce8a7c5434f38951f49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:23.137713 2535 raft_consensus.cc:504] T b4f90d0644504e89ab392f71ba39e8b8 P 961b13c11f8d4ce8a7c5434f38951f49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "961b13c11f8d4ce8a7c5434f38951f49" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40959 } } peers { permanent_uuid: "c6d019d21f184f50a523835d7f4ca7b3" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45449 } } peers { permanent_uuid: "6e31962e88794140906e86020cf2ff16" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34445 } }
I20240503 01:59:23.138880 2535 leader_election.cc:288] T b4f90d0644504e89ab392f71ba39e8b8 P 961b13c11f8d4ce8a7c5434f38951f49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6d019d21f184f50a523835d7f4ca7b3 (127.0.99.132:45449), 6e31962e88794140906e86020cf2ff16 (127.0.99.131:34445)
I20240503 01:59:23.143036 2335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f90d0644504e89ab392f71ba39e8b8" candidate_uuid: "961b13c11f8d4ce8a7c5434f38951f49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6d019d21f184f50a523835d7f4ca7b3" is_pre_election: true
I20240503 01:59:23.143036 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f90d0644504e89ab392f71ba39e8b8" candidate_uuid: "961b13c11f8d4ce8a7c5434f38951f49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e31962e88794140906e86020cf2ff16" is_pre_election: true
I20240503 01:59:23.143368 2335 raft_consensus.cc:2454] T b4f90d0644504e89ab392f71ba39e8b8 P c6d019d21f184f50a523835d7f4ca7b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 961b13c11f8d4ce8a7c5434f38951f49 in term 0.
I20240503 01:59:23.143388 2204 raft_consensus.cc:2454] T b4f90d0644504e89ab392f71ba39e8b8 P 6e31962e88794140906e86020cf2ff16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 961b13c11f8d4ce8a7c5434f38951f49 in term 0.
I20240503 01:59:23.143849 1875 leader_election.cc:302] T b4f90d0644504e89ab392f71ba39e8b8 P 961b13c11f8d4ce8a7c5434f38951f49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 961b13c11f8d4ce8a7c5434f38951f49, c6d019d21f184f50a523835d7f4ca7b3; no voters:
I20240503 01:59:23.144110 2535 raft_consensus.cc:2781] T b4f90d0644504e89ab392f71ba39e8b8 P 961b13c11f8d4ce8a7c5434f38951f49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:23.144187 2535 raft_consensus.cc:482] T b4f90d0644504e89ab392f71ba39e8b8 P 961b13c11f8d4ce8a7c5434f38951f49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:23.144224 2535 raft_consensus.cc:3037] T b4f90d0644504e89ab392f71ba39e8b8 P 961b13c11f8d4ce8a7c5434f38951f49 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:23.145135 2535 raft_consensus.cc:504] T b4f90d0644504e89ab392f71ba39e8b8 P 961b13c11f8d4ce8a7c5434f38951f49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "961b13c11f8d4ce8a7c5434f38951f49" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40959 } } peers { permanent_uuid: "c6d019d21f184f50a523835d7f4ca7b3" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45449 } } peers { permanent_uuid: "6e31962e88794140906e86020cf2ff16" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34445 } }
I20240503 01:59:23.145536 2535 leader_election.cc:288] T b4f90d0644504e89ab392f71ba39e8b8 P 961b13c11f8d4ce8a7c5434f38951f49 [CANDIDATE]: Term 1 election: Requested vote from peers c6d019d21f184f50a523835d7f4ca7b3 (127.0.99.132:45449), 6e31962e88794140906e86020cf2ff16 (127.0.99.131:34445)
I20240503 01:59:23.145941 2335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f90d0644504e89ab392f71ba39e8b8" candidate_uuid: "961b13c11f8d4ce8a7c5434f38951f49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6d019d21f184f50a523835d7f4ca7b3"
I20240503 01:59:23.145984 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f90d0644504e89ab392f71ba39e8b8" candidate_uuid: "961b13c11f8d4ce8a7c5434f38951f49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e31962e88794140906e86020cf2ff16"
I20240503 01:59:23.146070 2335 raft_consensus.cc:3037] T b4f90d0644504e89ab392f71ba39e8b8 P c6d019d21f184f50a523835d7f4ca7b3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:23.146092 2204 raft_consensus.cc:3037] T b4f90d0644504e89ab392f71ba39e8b8 P 6e31962e88794140906e86020cf2ff16 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:23.146971 2204 raft_consensus.cc:2454] T b4f90d0644504e89ab392f71ba39e8b8 P 6e31962e88794140906e86020cf2ff16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 961b13c11f8d4ce8a7c5434f38951f49 in term 1.
I20240503 01:59:23.146972 2335 raft_consensus.cc:2454] T b4f90d0644504e89ab392f71ba39e8b8 P c6d019d21f184f50a523835d7f4ca7b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 961b13c11f8d4ce8a7c5434f38951f49 in term 1.
I20240503 01:59:23.147289 1877 leader_election.cc:302] T b4f90d0644504e89ab392f71ba39e8b8 P 961b13c11f8d4ce8a7c5434f38951f49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e31962e88794140906e86020cf2ff16, 961b13c11f8d4ce8a7c5434f38951f49; no voters:
I20240503 01:59:23.147486 2535 raft_consensus.cc:2781] T b4f90d0644504e89ab392f71ba39e8b8 P 961b13c11f8d4ce8a7c5434f38951f49 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:23.147739 2535 raft_consensus.cc:686] T b4f90d0644504e89ab392f71ba39e8b8 P 961b13c11f8d4ce8a7c5434f38951f49 [term 1 LEADER]: Becoming Leader. State: Replica: 961b13c11f8d4ce8a7c5434f38951f49, State: Running, Role: LEADER
I20240503 01:59:23.148087 2535 consensus_queue.cc:237] T b4f90d0644504e89ab392f71ba39e8b8 P 961b13c11f8d4ce8a7c5434f38951f49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "961b13c11f8d4ce8a7c5434f38951f49" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40959 } } peers { permanent_uuid: "c6d019d21f184f50a523835d7f4ca7b3" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45449 } } peers { permanent_uuid: "6e31962e88794140906e86020cf2ff16" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34445 } }
I20240503 01:59:23.151383 1798 catalog_manager.cc:5484] T b4f90d0644504e89ab392f71ba39e8b8 P 961b13c11f8d4ce8a7c5434f38951f49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 961b13c11f8d4ce8a7c5434f38951f49 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "961b13c11f8d4ce8a7c5434f38951f49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "961b13c11f8d4ce8a7c5434f38951f49" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6d019d21f184f50a523835d7f4ca7b3" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e31962e88794140906e86020cf2ff16" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34445 } health_report { overall_health: UNKNOWN } } }
W20240503 01:59:23.152487 2251 tablet.cc:2369] T b4f90d0644504e89ab392f71ba39e8b8 P 6e31962e88794140906e86020cf2ff16: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 01:59:23.167464 398 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 01:59:23.169102 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 961b13c11f8d4ce8a7c5434f38951f49 to finish bootstrapping
I20240503 01:59:23.174787 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 46f764d98fc64dde8e370450c4c3ff8e to finish bootstrapping
I20240503 01:59:23.181511 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 6e31962e88794140906e86020cf2ff16 to finish bootstrapping
I20240503 01:59:23.186982 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver c6d019d21f184f50a523835d7f4ca7b3 to finish bootstrapping
I20240503 01:59:23.192559 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver d3ea3c5c177f44938e0e81020aa69da2 to finish bootstrapping
I20240503 01:59:23.198490 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu with pid 1860
W20240503 01:59:23.329468 2382 tablet.cc:2369] T b4f90d0644504e89ab392f71ba39e8b8 P c6d019d21f184f50a523835d7f4ca7b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 01:59:23.400172 2053 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:23.400569 2315 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:23.401288 2446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:23.401525 2184 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:23.476264 2119 heartbeater.cc:503] Master 127.0.99.190:43525 was elected leader, sending a full tablet report...
I20240503 01:59:23.483426 2184 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:23.483585 2315 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:23.484215 2053 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:23.486717 2446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:23.583165 2053 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:23.583202 2184 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:23.583794 2446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:23.584707 2315 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:23.813805 2053 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:23.814353 2184 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:23.815330 2446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:23.820881 2315 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:23.895200 2184 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:23.896018 2053 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:23.896736 2315 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:23.897426 2446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:24.007907 2184 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:24.008931 2053 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:24.009297 2315 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:24.018539 2446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:24.033413 2512 heartbeater.cc:503] Master 127.0.99.190:43525 was elected leader, sending a full tablet report...
I20240503 01:59:24.051774 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu with pid 1991
I20240503 01:59:24.056912 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu with pid 2122
I20240503 01:59:24.061957 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu with pid 2253
I20240503 01:59:24.066849 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu with pid 2384
I20240503 01:59:24.072046 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu with pid 1769
2024-05-03T01:59:24Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.TabletServerIsDown/1 (2284 ms)
[----------] 1 test from RebalanceStartCriteriaTest (2284 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/1
I20240503 01:59:24.094321 398 test_util.cc:268] Using random seed: 295675754
I20240503 01:59:24.095176 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 01:59:24.095222 398 ts_itest-base.cc:113] --------------
I20240503 01:59:24.095242 398 ts_itest-base.cc:114] 7 tablet servers
I20240503 01:59:24.095257 398 ts_itest-base.cc:115] 1 replicas per TS
I20240503 01:59:24.095283 398 ts_itest-base.cc:116] --------------
2024-05-03T01:59:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T01:59:24Z Disabled control of system clock
I20240503 01:59:24.103354 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
/tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:43851
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:35751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:43851
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240503 01:59:24.212175 2724 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240503 01:59:24.212505 2724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:24.212545 2724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:24.212610 2724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:24.215869 2724 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:24.215977 2724 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 01:59:24.216010 2724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:24.216032 2724 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 01:59:24.216068 2724 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 01:59:24.220043 2724 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:43851
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:43851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.2724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:24.221071 2724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:24.222236 2724 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:24.229563 2729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:24.229671 2724 server_base.cc:1034] running on GCE node
W20240503 01:59:24.229563 2732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:24.229563 2730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:24.230533 2724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:24.231560 2724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:24.232721 2724 hybrid_clock.cc:648] HybridClock initialized: now 1714701564232704 us; error 42 us; skew 500 ppm
I20240503 01:59:24.234773 2724 webserver.cc:460] Webserver started at http://127.0.99.190:35687/ using document root <none> and password file <none>
I20240503 01:59:24.235388 2724 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:24.235457 2724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:24.235642 2724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:24.237519 2724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "667273152a48411eb72e6817f2e34b09"
format_stamp: "Formatted at 2024-05-03 01:59:24 on dist-test-slave-pz5w"
I20240503 01:59:24.238041 2724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "667273152a48411eb72e6817f2e34b09"
format_stamp: "Formatted at 2024-05-03 01:59:24 on dist-test-slave-pz5w"
I20240503 01:59:24.242012 2724 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 01:59:24.244546 2738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:24.245723 2724 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 01:59:24.245877 2724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "667273152a48411eb72e6817f2e34b09"
format_stamp: "Formatted at 2024-05-03 01:59:24 on dist-test-slave-pz5w"
I20240503 01:59:24.245990 2724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:24.271889 2724 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:24.272701 2724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:24.272886 2724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:24.279793 2724 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:43851
I20240503 01:59:24.279806 2790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:43851 every 8 connection(s)
I20240503 01:59:24.280867 2724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 01:59:24.283533 2791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:24.288903 2791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 667273152a48411eb72e6817f2e34b09: Bootstrap starting.
I20240503 01:59:24.289666 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu as pid 2724
I20240503 01:59:24.289752 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 01:59:24.291519 2791 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 667273152a48411eb72e6817f2e34b09: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:24.292412 2791 log.cc:824] T 00000000000000000000000000000000 P 667273152a48411eb72e6817f2e34b09: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:24.294437 2791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 667273152a48411eb72e6817f2e34b09: No bootstrap required, opened a new log
I20240503 01:59:24.297767 2791 raft_consensus.cc:348] T 00000000000000000000000000000000 P 667273152a48411eb72e6817f2e34b09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "667273152a48411eb72e6817f2e34b09" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 43851 } }
I20240503 01:59:24.298027 2791 raft_consensus.cc:374] T 00000000000000000000000000000000 P 667273152a48411eb72e6817f2e34b09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:24.298099 2791 raft_consensus.cc:729] T 00000000000000000000000000000000 P 667273152a48411eb72e6817f2e34b09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 667273152a48411eb72e6817f2e34b09, State: Initialized, Role: FOLLOWER
I20240503 01:59:24.298544 2791 consensus_queue.cc:260] T 00000000000000000000000000000000 P 667273152a48411eb72e6817f2e34b09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "667273152a48411eb72e6817f2e34b09" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 43851 } }
I20240503 01:59:24.298678 2791 raft_consensus.cc:388] T 00000000000000000000000000000000 P 667273152a48411eb72e6817f2e34b09 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 01:59:24.298744 2791 raft_consensus.cc:482] T 00000000000000000000000000000000 P 667273152a48411eb72e6817f2e34b09 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 01:59:24.298861 2791 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 667273152a48411eb72e6817f2e34b09 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:24.299633 2791 raft_consensus.cc:504] T 00000000000000000000000000000000 P 667273152a48411eb72e6817f2e34b09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "667273152a48411eb72e6817f2e34b09" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 43851 } }
I20240503 01:59:24.299976 2791 leader_election.cc:302] T 00000000000000000000000000000000 P 667273152a48411eb72e6817f2e34b09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 667273152a48411eb72e6817f2e34b09; no voters:
I20240503 01:59:24.300266 2791 leader_election.cc:288] T 00000000000000000000000000000000 P 667273152a48411eb72e6817f2e34b09 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 01:59:24.300638 2796 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 667273152a48411eb72e6817f2e34b09 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:24.300817 2796 raft_consensus.cc:686] T 00000000000000000000000000000000 P 667273152a48411eb72e6817f2e34b09 [term 1 LEADER]: Becoming Leader. State: Replica: 667273152a48411eb72e6817f2e34b09, State: Running, Role: LEADER
I20240503 01:59:24.301119 2796 consensus_queue.cc:237] T 00000000000000000000000000000000 P 667273152a48411eb72e6817f2e34b09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "667273152a48411eb72e6817f2e34b09" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 43851 } }
I20240503 01:59:24.301380 2791 sys_catalog.cc:564] T 00000000000000000000000000000000 P 667273152a48411eb72e6817f2e34b09 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 01:59:24.302515 2798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 667273152a48411eb72e6817f2e34b09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 667273152a48411eb72e6817f2e34b09. Latest consensus state: current_term: 1 leader_uuid: "667273152a48411eb72e6817f2e34b09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "667273152a48411eb72e6817f2e34b09" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 43851 } } }
I20240503 01:59:24.302639 2798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 667273152a48411eb72e6817f2e34b09 [sys.catalog]: This master's current role is: LEADER
I20240503 01:59:24.303004 2805 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 01:59:24.302964 2797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 667273152a48411eb72e6817f2e34b09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "667273152a48411eb72e6817f2e34b09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "667273152a48411eb72e6817f2e34b09" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 43851 } } }
I20240503 01:59:24.303112 2797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 667273152a48411eb72e6817f2e34b09 [sys.catalog]: This master's current role is: LEADER
I20240503 01:59:24.306252 2805 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 01:59:24.312139 2805 catalog_manager.cc:1347] Generated new cluster ID: 7d8e9f385f42486690c7d13364d3f05d
I20240503 01:59:24.312244 2805 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 01:59:24.346998 2805 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 01:59:24.347992 2805 catalog_manager.cc:1504] Loading token signing keys...
I20240503 01:59:24.359016 2805 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 667273152a48411eb72e6817f2e34b09: Generated new TSK 0
I20240503 01:59:24.359835 2805 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 01:59:24.367259 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
/tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43851
--builtin_ntp_servers=127.0.99.148:35751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 01:59:24.482721 2815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:24.482971 2815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:24.483016 2815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:24.486281 2815 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:24.486373 2815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:24.486452 2815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 01:59:24.490407 2815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43851
--never_fsync=true
--heap_profile_path=/tmp/kudu.2815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:24.491397 2815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:24.492566 2815 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:24.500882 2823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:24.500882 2821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:24.500882 2820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:24.501241 2815 server_base.cc:1034] running on GCE node
I20240503 01:59:24.501677 2815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:24.502287 2815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:24.503451 2815 hybrid_clock.cc:648] HybridClock initialized: now 1714701564503429 us; error 47 us; skew 500 ppm
I20240503 01:59:24.505487 2815 webserver.cc:460] Webserver started at http://127.0.99.129:33369/ using document root <none> and password file <none>
I20240503 01:59:24.506060 2815 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:24.506124 2815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:24.506309 2815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:24.508015 2815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "4d61ec80bf4c475ea86ae2bcb866fd05"
format_stamp: "Formatted at 2024-05-03 01:59:24 on dist-test-slave-pz5w"
I20240503 01:59:24.508554 2815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "4d61ec80bf4c475ea86ae2bcb866fd05"
format_stamp: "Formatted at 2024-05-03 01:59:24 on dist-test-slave-pz5w"
I20240503 01:59:24.511978 2815 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:24.514489 2829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:24.515710 2815 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:24.515858 2815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "4d61ec80bf4c475ea86ae2bcb866fd05"
format_stamp: "Formatted at 2024-05-03 01:59:24 on dist-test-slave-pz5w"
I20240503 01:59:24.515965 2815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:24.534582 2815 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:24.535355 2815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:24.535529 2815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:24.536092 2815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:24.537191 2815 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:24.537252 2815 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:24.537318 2815 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:24.537348 2815 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:24.546314 2815 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:38011
I20240503 01:59:24.546336 2942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:38011 every 8 connection(s)
I20240503 01:59:24.547260 2815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 01:59:24.553460 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu as pid 2815
I20240503 01:59:24.553586 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 01:59:24.556100 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
/tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43851
--builtin_ntp_servers=127.0.99.148:35751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 01:59:24.557473 2943 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43851
I20240503 01:59:24.557798 2943 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:24.558449 2943 heartbeater.cc:511] Master 127.0.99.190:43851 requested a full tablet report, sending...
I20240503 01:59:24.560220 2752 ts_manager.cc:194] Registered new tserver with Master: 4d61ec80bf4c475ea86ae2bcb866fd05 (127.0.99.129:38011)
I20240503 01:59:24.561620 2752 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:43599
W20240503 01:59:24.668828 2946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:24.669080 2946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:24.669129 2946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:24.672410 2946 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:24.672502 2946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:24.672590 2946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 01:59:24.676592 2946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43851
--never_fsync=true
--heap_profile_path=/tmp/kudu.2946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:24.677609 2946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:24.678773 2946 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:24.687165 2951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:24.687175 2952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:24.687188 2954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:24.687564 2946 server_base.cc:1034] running on GCE node
I20240503 01:59:24.688014 2946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:24.688676 2946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:24.689853 2946 hybrid_clock.cc:648] HybridClock initialized: now 1714701564689834 us; error 36 us; skew 500 ppm
I20240503 01:59:24.691972 2946 webserver.cc:460] Webserver started at http://127.0.99.130:39071/ using document root <none> and password file <none>
I20240503 01:59:24.692618 2946 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:24.692688 2946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:24.692876 2946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:24.694710 2946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "fb4fc23dca0c4de69aef5061d3a67b2f"
format_stamp: "Formatted at 2024-05-03 01:59:24 on dist-test-slave-pz5w"
I20240503 01:59:24.695283 2946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "fb4fc23dca0c4de69aef5061d3a67b2f"
format_stamp: "Formatted at 2024-05-03 01:59:24 on dist-test-slave-pz5w"
I20240503 01:59:24.699096 2946 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20240503 01:59:24.701756 2960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:24.703047 2946 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240503 01:59:24.703184 2946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "fb4fc23dca0c4de69aef5061d3a67b2f"
format_stamp: "Formatted at 2024-05-03 01:59:24 on dist-test-slave-pz5w"
I20240503 01:59:24.703286 2946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:24.714772 2946 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:24.715498 2946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:24.715655 2946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:24.716187 2946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:24.717159 2946 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:24.717207 2946 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:24.717248 2946 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:24.717269 2946 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:24.726279 2946 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:34163
I20240503 01:59:24.726295 3073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:34163 every 8 connection(s)
I20240503 01:59:24.727304 2946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 01:59:24.732964 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu as pid 2946
I20240503 01:59:24.733067 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 01:59:24.735422 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
/tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43851
--builtin_ntp_servers=127.0.99.148:35751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 01:59:24.736982 3074 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43851
I20240503 01:59:24.737288 3074 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:24.737979 3074 heartbeater.cc:511] Master 127.0.99.190:43851 requested a full tablet report, sending...
I20240503 01:59:24.739221 2752 ts_manager.cc:194] Registered new tserver with Master: fb4fc23dca0c4de69aef5061d3a67b2f (127.0.99.130:34163)
I20240503 01:59:24.739812 2752 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:43167
W20240503 01:59:24.848150 3077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:24.848479 3077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:24.848541 3077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:24.851940 3077 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:24.852033 3077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:24.852106 3077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 01:59:24.856098 3077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43851
--never_fsync=true
--heap_profile_path=/tmp/kudu.3077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:24.857108 3077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:24.858266 3077 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:24.866648 3082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:24.866794 3077 server_base.cc:1034] running on GCE node
W20240503 01:59:24.866634 3085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:24.866657 3083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:24.867390 3077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:24.868010 3077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:24.869185 3077 hybrid_clock.cc:648] HybridClock initialized: now 1714701564869163 us; error 36 us; skew 500 ppm
I20240503 01:59:24.871234 3077 webserver.cc:460] Webserver started at http://127.0.99.131:39689/ using document root <none> and password file <none>
I20240503 01:59:24.871805 3077 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:24.871866 3077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:24.872045 3077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:24.873934 3077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "ad0b1cf28c9c452db2230e4cd69363bc"
format_stamp: "Formatted at 2024-05-03 01:59:24 on dist-test-slave-pz5w"
I20240503 01:59:24.874476 3077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "ad0b1cf28c9c452db2230e4cd69363bc"
format_stamp: "Formatted at 2024-05-03 01:59:24 on dist-test-slave-pz5w"
I20240503 01:59:24.878244 3077 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 01:59:24.880792 3091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:24.881986 3077 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 01:59:24.882156 3077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "ad0b1cf28c9c452db2230e4cd69363bc"
format_stamp: "Formatted at 2024-05-03 01:59:24 on dist-test-slave-pz5w"
I20240503 01:59:24.882265 3077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:24.892772 3077 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:24.893457 3077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:24.893621 3077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:24.894167 3077 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:24.895148 3077 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:24.895205 3077 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:24.895262 3077 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:24.895287 3077 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:24.904398 3077 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:44037
I20240503 01:59:24.904415 3204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:44037 every 8 connection(s)
I20240503 01:59:24.905392 3077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 01:59:24.911850 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu as pid 3077
I20240503 01:59:24.911948 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 01:59:24.914238 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
/tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43851
--builtin_ntp_servers=127.0.99.148:35751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 01:59:24.914947 3205 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43851
I20240503 01:59:24.915235 3205 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:24.915982 3205 heartbeater.cc:511] Master 127.0.99.190:43851 requested a full tablet report, sending...
I20240503 01:59:24.917315 2752 ts_manager.cc:194] Registered new tserver with Master: ad0b1cf28c9c452db2230e4cd69363bc (127.0.99.131:44037)
I20240503 01:59:24.918109 2752 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:48449
W20240503 01:59:25.024605 3208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:25.024855 3208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:25.024900 3208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:25.030076 3208 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:25.030320 3208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:25.030483 3208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 01:59:25.036746 3208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43851
--never_fsync=true
--heap_profile_path=/tmp/kudu.3208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:25.037715 3208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:25.038875 3208 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:25.047164 3214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:25.047170 3216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:25.047169 3213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:25.047235 3208 server_base.cc:1034] running on GCE node
I20240503 01:59:25.047889 3208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:25.048569 3208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:25.049727 3208 hybrid_clock.cc:648] HybridClock initialized: now 1714701565049714 us; error 43 us; skew 500 ppm
I20240503 01:59:25.051853 3208 webserver.cc:460] Webserver started at http://127.0.99.132:34093/ using document root <none> and password file <none>
I20240503 01:59:25.052489 3208 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:25.052558 3208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:25.052750 3208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:25.054484 3208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "c8a3e298f9d441c09c064a7f5655980b"
format_stamp: "Formatted at 2024-05-03 01:59:25 on dist-test-slave-pz5w"
I20240503 01:59:25.054999 3208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "c8a3e298f9d441c09c064a7f5655980b"
format_stamp: "Formatted at 2024-05-03 01:59:25 on dist-test-slave-pz5w"
I20240503 01:59:25.058507 3208 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 01:59:25.060954 3222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:25.062114 3208 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 01:59:25.062271 3208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "c8a3e298f9d441c09c064a7f5655980b"
format_stamp: "Formatted at 2024-05-03 01:59:25 on dist-test-slave-pz5w"
I20240503 01:59:25.062381 3208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:25.077869 3208 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:25.078569 3208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:25.078742 3208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:25.079301 3208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:25.080219 3208 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:25.080268 3208 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:25.080324 3208 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:25.080353 3208 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:25.089561 3208 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:41327
I20240503 01:59:25.089596 3335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:41327 every 8 connection(s)
I20240503 01:59:25.090595 3208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 01:59:25.100003 3336 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43851
I20240503 01:59:25.100275 3336 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:25.100644 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu as pid 3208
I20240503 01:59:25.100750 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 01:59:25.101033 3336 heartbeater.cc:511] Master 127.0.99.190:43851 requested a full tablet report, sending...
I20240503 01:59:25.102191 2755 ts_manager.cc:194] Registered new tserver with Master: c8a3e298f9d441c09c064a7f5655980b (127.0.99.132:41327)
I20240503 01:59:25.102810 2755 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:42403
I20240503 01:59:25.102983 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
/tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43851
--builtin_ntp_servers=127.0.99.148:35751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 01:59:25.218850 3339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:25.219120 3339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:25.219185 3339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:25.222697 3339 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:25.222800 3339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:25.222895 3339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 01:59:25.227052 3339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43851
--never_fsync=true
--heap_profile_path=/tmp/kudu.3339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:25.228083 3339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:25.229408 3339 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:25.237989 3345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:25.238039 3339 server_base.cc:1034] running on GCE node
W20240503 01:59:25.238091 3347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:25.238127 3344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:25.238739 3339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:25.239475 3339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:25.240646 3339 hybrid_clock.cc:648] HybridClock initialized: now 1714701565240633 us; error 32 us; skew 500 ppm
I20240503 01:59:25.242798 3339 webserver.cc:460] Webserver started at http://127.0.99.133:42123/ using document root <none> and password file <none>
I20240503 01:59:25.243412 3339 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:25.243477 3339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:25.243669 3339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:25.245594 3339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "e296c081ab4a414294313ab9883cae84"
format_stamp: "Formatted at 2024-05-03 01:59:25 on dist-test-slave-pz5w"
I20240503 01:59:25.246116 3339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "e296c081ab4a414294313ab9883cae84"
format_stamp: "Formatted at 2024-05-03 01:59:25 on dist-test-slave-pz5w"
I20240503 01:59:25.249878 3339 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240503 01:59:25.252492 3353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:25.253669 3339 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 01:59:25.253800 3339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "e296c081ab4a414294313ab9883cae84"
format_stamp: "Formatted at 2024-05-03 01:59:25 on dist-test-slave-pz5w"
I20240503 01:59:25.253901 3339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:25.275009 3339 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:25.275756 3339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:25.275931 3339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:25.276530 3339 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:25.277473 3339 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:25.277523 3339 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:25.277578 3339 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:25.277604 3339 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:25.288229 3339 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:41953
I20240503 01:59:25.288262 3466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:41953 every 8 connection(s)
I20240503 01:59:25.289260 3339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 01:59:25.299521 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu as pid 3339
I20240503 01:59:25.299645 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 01:59:25.300403 3467 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43851
I20240503 01:59:25.300771 3467 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:25.301791 3467 heartbeater.cc:511] Master 127.0.99.190:43851 requested a full tablet report, sending...
I20240503 01:59:25.302757 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
/tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:0
--local_ip_for_outbound_sockets=127.0.99.134
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43851
--builtin_ntp_servers=127.0.99.148:35751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 01:59:25.303231 2755 ts_manager.cc:194] Registered new tserver with Master: e296c081ab4a414294313ab9883cae84 (127.0.99.133:41953)
I20240503 01:59:25.304116 2755 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:35467
W20240503 01:59:25.422556 3470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:25.422822 3470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:25.422868 3470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:25.426434 3470 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:25.426532 3470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:25.426605 3470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 01:59:25.430655 3470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43851
--never_fsync=true
--heap_profile_path=/tmp/kudu.3470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:25.431627 3470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:25.432813 3470 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:25.441437 3478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:25.441417 3476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:25.441426 3475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:25.441838 3470 server_base.cc:1034] running on GCE node
I20240503 01:59:25.442364 3470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:25.443024 3470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:25.444181 3470 hybrid_clock.cc:648] HybridClock initialized: now 1714701565444153 us; error 47 us; skew 500 ppm
I20240503 01:59:25.446358 3470 webserver.cc:460] Webserver started at http://127.0.99.134:46211/ using document root <none> and password file <none>
I20240503 01:59:25.446991 3470 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:25.447062 3470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:25.447250 3470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:25.449180 3470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "e4c1a8048e4647b6936035bc2fc22b66"
format_stamp: "Formatted at 2024-05-03 01:59:25 on dist-test-slave-pz5w"
I20240503 01:59:25.449697 3470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "e4c1a8048e4647b6936035bc2fc22b66"
format_stamp: "Formatted at 2024-05-03 01:59:25 on dist-test-slave-pz5w"
I20240503 01:59:25.453678 3470 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240503 01:59:25.456477 3484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:25.457774 3470 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240503 01:59:25.457923 3470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "e4c1a8048e4647b6936035bc2fc22b66"
format_stamp: "Formatted at 2024-05-03 01:59:25 on dist-test-slave-pz5w"
I20240503 01:59:25.458035 3470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:25.491976 3470 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:25.492751 3470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:25.492938 3470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:25.493534 3470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:25.494501 3470 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:25.494550 3470 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:25.494592 3470 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:25.494611 3470 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:25.503705 3470 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:46423
I20240503 01:59:25.503726 3597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:46423 every 8 connection(s)
I20240503 01:59:25.504740 3470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 01:59:25.511523 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu as pid 3470
I20240503 01:59:25.511648 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 01:59:25.514163 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
/tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:0
--local_ip_for_outbound_sockets=127.0.99.135
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43851
--builtin_ntp_servers=127.0.99.148:35751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 01:59:25.514685 3598 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43851
I20240503 01:59:25.515053 3598 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:25.515796 3598 heartbeater.cc:511] Master 127.0.99.190:43851 requested a full tablet report, sending...
I20240503 01:59:25.517108 2755 ts_manager.cc:194] Registered new tserver with Master: e4c1a8048e4647b6936035bc2fc22b66 (127.0.99.134:46423)
I20240503 01:59:25.517709 2755 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:57817
I20240503 01:59:25.564782 2943 heartbeater.cc:503] Master 127.0.99.190:43851 was elected leader, sending a full tablet report...
W20240503 01:59:25.630290 3601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:25.630535 3601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:25.630599 3601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:25.633967 3601 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:25.634074 3601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:25.634172 3601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 01:59:25.638300 3601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43851
--never_fsync=true
--heap_profile_path=/tmp/kudu.3601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:25.639330 3601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:25.640692 3601 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:25.649220 3609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:25.649220 3607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:25.649248 3606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:25.649502 3601 server_base.cc:1034] running on GCE node
I20240503 01:59:25.650055 3601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:25.650753 3601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:25.651947 3601 hybrid_clock.cc:648] HybridClock initialized: now 1714701565651938 us; error 31 us; skew 500 ppm
I20240503 01:59:25.654232 3601 webserver.cc:460] Webserver started at http://127.0.99.135:41415/ using document root <none> and password file <none>
I20240503 01:59:25.654929 3601 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:25.655021 3601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:25.655215 3601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:25.657519 3601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "6c8442cedca54ad589e0e050282afee3"
format_stamp: "Formatted at 2024-05-03 01:59:25 on dist-test-slave-pz5w"
I20240503 01:59:25.658396 3601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "6c8442cedca54ad589e0e050282afee3"
format_stamp: "Formatted at 2024-05-03 01:59:25 on dist-test-slave-pz5w"
I20240503 01:59:25.662997 3601 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 01:59:25.666049 3615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:25.667207 3601 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:25.667347 3601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "6c8442cedca54ad589e0e050282afee3"
format_stamp: "Formatted at 2024-05-03 01:59:25 on dist-test-slave-pz5w"
I20240503 01:59:25.667457 3601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:25.689347 3601 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:25.690073 3601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:25.690245 3601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:25.690805 3601 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:25.691785 3601 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:25.691835 3601 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:25.691877 3601 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:25.691890 3601 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:25.701848 3601 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:43573
I20240503 01:59:25.701933 3728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:43573 every 8 connection(s)
I20240503 01:59:25.703022 3601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 01:59:25.711443 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu as pid 3601
I20240503 01:59:25.711591 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 01:59:25.713687 3729 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43851
I20240503 01:59:25.714071 3729 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:25.714753 3729 heartbeater.cc:511] Master 127.0.99.190:43851 requested a full tablet report, sending...
I20240503 01:59:25.716056 2755 ts_manager.cc:194] Registered new tserver with Master: 6c8442cedca54ad589e0e050282afee3 (127.0.99.135:43573)
I20240503 01:59:25.716717 2755 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:48521
I20240503 01:59:25.726877 398 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240503 01:59:25.737349 2755 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38380:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 01:59:25.742619 3074 heartbeater.cc:503] Master 127.0.99.190:43851 was elected leader, sending a full tablet report...
I20240503 01:59:25.755056 2877 tablet_service.cc:1467] Processing CreateTablet for tablet 7fe4b04830cd42a292f72f27b788c75a (DEFAULT_TABLE table=TestTable [id=4e3c2b1187c5480abe84af1077137cc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:25.756075 2877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fe4b04830cd42a292f72f27b788c75a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:25.761797 3743 tablet_bootstrap.cc:492] T 7fe4b04830cd42a292f72f27b788c75a P 4d61ec80bf4c475ea86ae2bcb866fd05: Bootstrap starting.
I20240503 01:59:25.764292 3743 tablet_bootstrap.cc:654] T 7fe4b04830cd42a292f72f27b788c75a P 4d61ec80bf4c475ea86ae2bcb866fd05: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:25.765143 3743 log.cc:824] T 7fe4b04830cd42a292f72f27b788c75a P 4d61ec80bf4c475ea86ae2bcb866fd05: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:25.767043 3743 tablet_bootstrap.cc:492] T 7fe4b04830cd42a292f72f27b788c75a P 4d61ec80bf4c475ea86ae2bcb866fd05: No bootstrap required, opened a new log
I20240503 01:59:25.767233 3743 ts_tablet_manager.cc:1367] T 7fe4b04830cd42a292f72f27b788c75a P 4d61ec80bf4c475ea86ae2bcb866fd05: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 01:59:25.770341 3743 raft_consensus.cc:348] T 7fe4b04830cd42a292f72f27b788c75a P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } }
I20240503 01:59:25.770581 3743 raft_consensus.cc:374] T 7fe4b04830cd42a292f72f27b788c75a P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:25.770632 3743 raft_consensus.cc:729] T 7fe4b04830cd42a292f72f27b788c75a P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d61ec80bf4c475ea86ae2bcb866fd05, State: Initialized, Role: FOLLOWER
I20240503 01:59:25.771080 3743 consensus_queue.cc:260] T 7fe4b04830cd42a292f72f27b788c75a P 4d61ec80bf4c475ea86ae2bcb866fd05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } }
I20240503 01:59:25.771234 3743 raft_consensus.cc:388] T 7fe4b04830cd42a292f72f27b788c75a P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 01:59:25.771303 3743 raft_consensus.cc:482] T 7fe4b04830cd42a292f72f27b788c75a P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 01:59:25.771417 3743 raft_consensus.cc:3037] T 7fe4b04830cd42a292f72f27b788c75a P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:25.772478 3743 raft_consensus.cc:504] T 7fe4b04830cd42a292f72f27b788c75a P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } }
I20240503 01:59:25.772821 3743 leader_election.cc:302] T 7fe4b04830cd42a292f72f27b788c75a P 4d61ec80bf4c475ea86ae2bcb866fd05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d61ec80bf4c475ea86ae2bcb866fd05; no voters:
I20240503 01:59:25.773114 3743 leader_election.cc:288] T 7fe4b04830cd42a292f72f27b788c75a P 4d61ec80bf4c475ea86ae2bcb866fd05 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 01:59:25.773237 3745 raft_consensus.cc:2781] T 7fe4b04830cd42a292f72f27b788c75a P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:25.773463 3745 raft_consensus.cc:686] T 7fe4b04830cd42a292f72f27b788c75a P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 LEADER]: Becoming Leader. State: Replica: 4d61ec80bf4c475ea86ae2bcb866fd05, State: Running, Role: LEADER
I20240503 01:59:25.773823 3745 consensus_queue.cc:237] T 7fe4b04830cd42a292f72f27b788c75a P 4d61ec80bf4c475ea86ae2bcb866fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } }
I20240503 01:59:25.774013 3743 ts_tablet_manager.cc:1398] T 7fe4b04830cd42a292f72f27b788c75a P 4d61ec80bf4c475ea86ae2bcb866fd05: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20240503 01:59:25.777915 2755 catalog_manager.cc:5484] T 7fe4b04830cd42a292f72f27b788c75a P 4d61ec80bf4c475ea86ae2bcb866fd05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d61ec80bf4c475ea86ae2bcb866fd05 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:25.792125 398 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240503 01:59:25.793776 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 4d61ec80bf4c475ea86ae2bcb866fd05 to finish bootstrapping
I20240503 01:59:25.800009 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver fb4fc23dca0c4de69aef5061d3a67b2f to finish bootstrapping
I20240503 01:59:25.806695 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver ad0b1cf28c9c452db2230e4cd69363bc to finish bootstrapping
I20240503 01:59:25.813776 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver c8a3e298f9d441c09c064a7f5655980b to finish bootstrapping
I20240503 01:59:25.820268 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver e296c081ab4a414294313ab9883cae84 to finish bootstrapping
I20240503 01:59:25.826972 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver e4c1a8048e4647b6936035bc2fc22b66 to finish bootstrapping
I20240503 01:59:25.833544 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 6c8442cedca54ad589e0e050282afee3 to finish bootstrapping
I20240503 01:59:25.839888 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu with pid 3077
I20240503 01:59:25.844630 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu with pid 3208
I20240503 01:59:25.849315 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu with pid 3339
I20240503 01:59:25.854239 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu with pid 3470
I20240503 01:59:25.859125 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu with pid 3601
I20240503 01:59:29.615783 2755 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38380:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:29.625129 2877 tablet_service.cc:1467] Processing CreateTablet for tablet 0f440670fb6f4fccbc3ee8554dadb770 (DEFAULT_TABLE table=rebalance_test_table_0 [id=da03de56cae54136b05045054d83001e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:29.625490 2877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f440670fb6f4fccbc3ee8554dadb770. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:29.628077 3763 tablet_bootstrap.cc:492] T 0f440670fb6f4fccbc3ee8554dadb770 P 4d61ec80bf4c475ea86ae2bcb866fd05: Bootstrap starting.
I20240503 01:59:29.629212 3763 tablet_bootstrap.cc:654] T 0f440670fb6f4fccbc3ee8554dadb770 P 4d61ec80bf4c475ea86ae2bcb866fd05: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:29.630854 3763 tablet_bootstrap.cc:492] T 0f440670fb6f4fccbc3ee8554dadb770 P 4d61ec80bf4c475ea86ae2bcb866fd05: No bootstrap required, opened a new log
I20240503 01:59:29.630970 3763 ts_tablet_manager.cc:1367] T 0f440670fb6f4fccbc3ee8554dadb770 P 4d61ec80bf4c475ea86ae2bcb866fd05: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:29.630781 3008 tablet_service.cc:1467] Processing CreateTablet for tablet b2ba35f5c3084e08a42c6f3381aa1cb5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=da03de56cae54136b05045054d83001e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:29.630784 3007 tablet_service.cc:1467] Processing CreateTablet for tablet 1b96b32dada445ccbe9c4eaa0ddee050 (DEFAULT_TABLE table=rebalance_test_table_0 [id=da03de56cae54136b05045054d83001e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:29.631736 3763 raft_consensus.cc:348] T 0f440670fb6f4fccbc3ee8554dadb770 P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } }
I20240503 01:59:29.631891 3763 raft_consensus.cc:374] T 0f440670fb6f4fccbc3ee8554dadb770 P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:29.631927 3763 raft_consensus.cc:729] T 0f440670fb6f4fccbc3ee8554dadb770 P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d61ec80bf4c475ea86ae2bcb866fd05, State: Initialized, Role: FOLLOWER
I20240503 01:59:29.632019 3007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b96b32dada445ccbe9c4eaa0ddee050. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:29.632057 3763 consensus_queue.cc:260] T 0f440670fb6f4fccbc3ee8554dadb770 P 4d61ec80bf4c475ea86ae2bcb866fd05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } }
I20240503 01:59:29.632148 3763 raft_consensus.cc:388] T 0f440670fb6f4fccbc3ee8554dadb770 P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 01:59:29.632182 3763 raft_consensus.cc:482] T 0f440670fb6f4fccbc3ee8554dadb770 P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 01:59:29.632225 3763 raft_consensus.cc:3037] T 0f440670fb6f4fccbc3ee8554dadb770 P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:29.632457 3008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2ba35f5c3084e08a42c6f3381aa1cb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:29.633232 3763 raft_consensus.cc:504] T 0f440670fb6f4fccbc3ee8554dadb770 P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } }
I20240503 01:59:29.633390 3763 leader_election.cc:302] T 0f440670fb6f4fccbc3ee8554dadb770 P 4d61ec80bf4c475ea86ae2bcb866fd05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d61ec80bf4c475ea86ae2bcb866fd05; no voters:
I20240503 01:59:29.633576 3763 leader_election.cc:288] T 0f440670fb6f4fccbc3ee8554dadb770 P 4d61ec80bf4c475ea86ae2bcb866fd05 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 01:59:29.633718 3765 raft_consensus.cc:2781] T 0f440670fb6f4fccbc3ee8554dadb770 P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:29.633952 3763 ts_tablet_manager.cc:1398] T 0f440670fb6f4fccbc3ee8554dadb770 P 4d61ec80bf4c475ea86ae2bcb866fd05: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20240503 01:59:29.634224 3765 raft_consensus.cc:686] T 0f440670fb6f4fccbc3ee8554dadb770 P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 LEADER]: Becoming Leader. State: Replica: 4d61ec80bf4c475ea86ae2bcb866fd05, State: Running, Role: LEADER
I20240503 01:59:29.634467 3765 consensus_queue.cc:237] T 0f440670fb6f4fccbc3ee8554dadb770 P 4d61ec80bf4c475ea86ae2bcb866fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } }
I20240503 01:59:29.637068 2755 catalog_manager.cc:5484] T 0f440670fb6f4fccbc3ee8554dadb770 P 4d61ec80bf4c475ea86ae2bcb866fd05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d61ec80bf4c475ea86ae2bcb866fd05 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:29.640947 3770 tablet_bootstrap.cc:492] T 1b96b32dada445ccbe9c4eaa0ddee050 P fb4fc23dca0c4de69aef5061d3a67b2f: Bootstrap starting.
I20240503 01:59:29.643098 3770 tablet_bootstrap.cc:654] T 1b96b32dada445ccbe9c4eaa0ddee050 P fb4fc23dca0c4de69aef5061d3a67b2f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:29.643939 3770 log.cc:824] T 1b96b32dada445ccbe9c4eaa0ddee050 P fb4fc23dca0c4de69aef5061d3a67b2f: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:29.645828 3770 tablet_bootstrap.cc:492] T 1b96b32dada445ccbe9c4eaa0ddee050 P fb4fc23dca0c4de69aef5061d3a67b2f: No bootstrap required, opened a new log
I20240503 01:59:29.646049 3770 ts_tablet_manager.cc:1367] T 1b96b32dada445ccbe9c4eaa0ddee050 P fb4fc23dca0c4de69aef5061d3a67b2f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 01:59:29.649259 3770 raft_consensus.cc:348] T 1b96b32dada445ccbe9c4eaa0ddee050 P fb4fc23dca0c4de69aef5061d3a67b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } }
I20240503 01:59:29.649533 3770 raft_consensus.cc:374] T 1b96b32dada445ccbe9c4eaa0ddee050 P fb4fc23dca0c4de69aef5061d3a67b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:29.649600 3770 raft_consensus.cc:729] T 1b96b32dada445ccbe9c4eaa0ddee050 P fb4fc23dca0c4de69aef5061d3a67b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb4fc23dca0c4de69aef5061d3a67b2f, State: Initialized, Role: FOLLOWER
I20240503 01:59:29.650105 3770 consensus_queue.cc:260] T 1b96b32dada445ccbe9c4eaa0ddee050 P fb4fc23dca0c4de69aef5061d3a67b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } }
I20240503 01:59:29.650274 3770 raft_consensus.cc:388] T 1b96b32dada445ccbe9c4eaa0ddee050 P fb4fc23dca0c4de69aef5061d3a67b2f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 01:59:29.650378 3770 raft_consensus.cc:482] T 1b96b32dada445ccbe9c4eaa0ddee050 P fb4fc23dca0c4de69aef5061d3a67b2f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 01:59:29.650502 3770 raft_consensus.cc:3037] T 1b96b32dada445ccbe9c4eaa0ddee050 P fb4fc23dca0c4de69aef5061d3a67b2f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:29.651453 3770 raft_consensus.cc:504] T 1b96b32dada445ccbe9c4eaa0ddee050 P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } }
I20240503 01:59:29.651824 3770 leader_election.cc:302] T 1b96b32dada445ccbe9c4eaa0ddee050 P fb4fc23dca0c4de69aef5061d3a67b2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb4fc23dca0c4de69aef5061d3a67b2f; no voters:
I20240503 01:59:29.652149 3770 leader_election.cc:288] T 1b96b32dada445ccbe9c4eaa0ddee050 P fb4fc23dca0c4de69aef5061d3a67b2f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 01:59:29.652308 3772 raft_consensus.cc:2781] T 1b96b32dada445ccbe9c4eaa0ddee050 P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:29.652544 3772 raft_consensus.cc:686] T 1b96b32dada445ccbe9c4eaa0ddee050 P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 LEADER]: Becoming Leader. State: Replica: fb4fc23dca0c4de69aef5061d3a67b2f, State: Running, Role: LEADER
I20240503 01:59:29.652896 3772 consensus_queue.cc:237] T 1b96b32dada445ccbe9c4eaa0ddee050 P fb4fc23dca0c4de69aef5061d3a67b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } }
I20240503 01:59:29.653079 3770 ts_tablet_manager.cc:1398] T 1b96b32dada445ccbe9c4eaa0ddee050 P fb4fc23dca0c4de69aef5061d3a67b2f: Time spent starting tablet: real 0.007s user 0.000s sys 0.008s
I20240503 01:59:29.653326 3770 tablet_bootstrap.cc:492] T b2ba35f5c3084e08a42c6f3381aa1cb5 P fb4fc23dca0c4de69aef5061d3a67b2f: Bootstrap starting.
I20240503 01:59:29.654464 3770 tablet_bootstrap.cc:654] T b2ba35f5c3084e08a42c6f3381aa1cb5 P fb4fc23dca0c4de69aef5061d3a67b2f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:29.655992 3770 tablet_bootstrap.cc:492] T b2ba35f5c3084e08a42c6f3381aa1cb5 P fb4fc23dca0c4de69aef5061d3a67b2f: No bootstrap required, opened a new log
I20240503 01:59:29.656096 3770 ts_tablet_manager.cc:1367] T b2ba35f5c3084e08a42c6f3381aa1cb5 P fb4fc23dca0c4de69aef5061d3a67b2f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:29.656013 2755 catalog_manager.cc:5484] T 1b96b32dada445ccbe9c4eaa0ddee050 P fb4fc23dca0c4de69aef5061d3a67b2f reported cstate change: term changed from 0 to 1, leader changed from <none> to fb4fc23dca0c4de69aef5061d3a67b2f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:29.656741 3770 raft_consensus.cc:348] T b2ba35f5c3084e08a42c6f3381aa1cb5 P fb4fc23dca0c4de69aef5061d3a67b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } }
I20240503 01:59:29.656833 3770 raft_consensus.cc:374] T b2ba35f5c3084e08a42c6f3381aa1cb5 P fb4fc23dca0c4de69aef5061d3a67b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:29.656862 3770 raft_consensus.cc:729] T b2ba35f5c3084e08a42c6f3381aa1cb5 P fb4fc23dca0c4de69aef5061d3a67b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb4fc23dca0c4de69aef5061d3a67b2f, State: Initialized, Role: FOLLOWER
I20240503 01:59:29.656983 3770 consensus_queue.cc:260] T b2ba35f5c3084e08a42c6f3381aa1cb5 P fb4fc23dca0c4de69aef5061d3a67b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } }
I20240503 01:59:29.657066 3770 raft_consensus.cc:388] T b2ba35f5c3084e08a42c6f3381aa1cb5 P fb4fc23dca0c4de69aef5061d3a67b2f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 01:59:29.657105 3770 raft_consensus.cc:482] T b2ba35f5c3084e08a42c6f3381aa1cb5 P fb4fc23dca0c4de69aef5061d3a67b2f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 01:59:29.657151 3770 raft_consensus.cc:3037] T b2ba35f5c3084e08a42c6f3381aa1cb5 P fb4fc23dca0c4de69aef5061d3a67b2f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:29.658273 3770 raft_consensus.cc:504] T b2ba35f5c3084e08a42c6f3381aa1cb5 P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } }
I20240503 01:59:29.658440 3770 leader_election.cc:302] T b2ba35f5c3084e08a42c6f3381aa1cb5 P fb4fc23dca0c4de69aef5061d3a67b2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb4fc23dca0c4de69aef5061d3a67b2f; no voters:
I20240503 01:59:29.658541 3770 leader_election.cc:288] T b2ba35f5c3084e08a42c6f3381aa1cb5 P fb4fc23dca0c4de69aef5061d3a67b2f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 01:59:29.658777 3773 raft_consensus.cc:2781] T b2ba35f5c3084e08a42c6f3381aa1cb5 P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:29.658875 3770 ts_tablet_manager.cc:1398] T b2ba35f5c3084e08a42c6f3381aa1cb5 P fb4fc23dca0c4de69aef5061d3a67b2f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 01:59:29.658879 3773 raft_consensus.cc:686] T b2ba35f5c3084e08a42c6f3381aa1cb5 P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 LEADER]: Becoming Leader. State: Replica: fb4fc23dca0c4de69aef5061d3a67b2f, State: Running, Role: LEADER
I20240503 01:59:29.659060 3773 consensus_queue.cc:237] T b2ba35f5c3084e08a42c6f3381aa1cb5 P fb4fc23dca0c4de69aef5061d3a67b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } }
I20240503 01:59:29.660781 2755 catalog_manager.cc:5484] T b2ba35f5c3084e08a42c6f3381aa1cb5 P fb4fc23dca0c4de69aef5061d3a67b2f reported cstate change: term changed from 0 to 1, leader changed from <none> to fb4fc23dca0c4de69aef5061d3a67b2f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } health_report { overall_health: HEALTHY } } }
W20240503 01:59:29.794716 3779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:29.795032 3779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:29.798451 3779 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 71.8218 ms
time per row: 0.0359109 ms
I20240503 01:59:29.924937 2751 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38380:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:29.931058 2876 tablet_service.cc:1467] Processing CreateTablet for tablet 7056f7fb504545d19641ea2c58ebf6be (DEFAULT_TABLE table=rebalance_test_table_1 [id=c3c40e684e4449ba86b043703f35788b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:29.931063 3008 tablet_service.cc:1467] Processing CreateTablet for tablet bd3627afefd74c39b43a4ff26da65c39 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c3c40e684e4449ba86b043703f35788b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:29.931416 2876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7056f7fb504545d19641ea2c58ebf6be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:29.931421 3008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd3627afefd74c39b43a4ff26da65c39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:29.931058 2877 tablet_service.cc:1467] Processing CreateTablet for tablet b60c4ffc1dac4e5e8a2881cd5aa61547 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c3c40e684e4449ba86b043703f35788b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:29.933490 3770 tablet_bootstrap.cc:492] T bd3627afefd74c39b43a4ff26da65c39 P fb4fc23dca0c4de69aef5061d3a67b2f: Bootstrap starting.
I20240503 01:59:29.933564 2877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b60c4ffc1dac4e5e8a2881cd5aa61547. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:29.934293 3770 tablet_bootstrap.cc:654] T bd3627afefd74c39b43a4ff26da65c39 P fb4fc23dca0c4de69aef5061d3a67b2f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:29.933349 3763 tablet_bootstrap.cc:492] T 7056f7fb504545d19641ea2c58ebf6be P 4d61ec80bf4c475ea86ae2bcb866fd05: Bootstrap starting.
I20240503 01:59:29.936187 3763 tablet_bootstrap.cc:654] T 7056f7fb504545d19641ea2c58ebf6be P 4d61ec80bf4c475ea86ae2bcb866fd05: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:29.937409 3770 tablet_bootstrap.cc:492] T bd3627afefd74c39b43a4ff26da65c39 P fb4fc23dca0c4de69aef5061d3a67b2f: No bootstrap required, opened a new log
I20240503 01:59:29.937503 3770 ts_tablet_manager.cc:1367] T bd3627afefd74c39b43a4ff26da65c39 P fb4fc23dca0c4de69aef5061d3a67b2f: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 01:59:29.937945 3770 raft_consensus.cc:348] T bd3627afefd74c39b43a4ff26da65c39 P fb4fc23dca0c4de69aef5061d3a67b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } }
I20240503 01:59:29.938045 3770 raft_consensus.cc:374] T bd3627afefd74c39b43a4ff26da65c39 P fb4fc23dca0c4de69aef5061d3a67b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:29.938066 3770 raft_consensus.cc:729] T bd3627afefd74c39b43a4ff26da65c39 P fb4fc23dca0c4de69aef5061d3a67b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb4fc23dca0c4de69aef5061d3a67b2f, State: Initialized, Role: FOLLOWER
I20240503 01:59:29.938190 3770 consensus_queue.cc:260] T bd3627afefd74c39b43a4ff26da65c39 P fb4fc23dca0c4de69aef5061d3a67b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } }
I20240503 01:59:29.938257 3770 raft_consensus.cc:388] T bd3627afefd74c39b43a4ff26da65c39 P fb4fc23dca0c4de69aef5061d3a67b2f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 01:59:29.938297 3770 raft_consensus.cc:482] T bd3627afefd74c39b43a4ff26da65c39 P fb4fc23dca0c4de69aef5061d3a67b2f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 01:59:29.938340 3770 raft_consensus.cc:3037] T bd3627afefd74c39b43a4ff26da65c39 P fb4fc23dca0c4de69aef5061d3a67b2f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:29.939026 3770 raft_consensus.cc:504] T bd3627afefd74c39b43a4ff26da65c39 P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } }
I20240503 01:59:29.939144 3770 leader_election.cc:302] T bd3627afefd74c39b43a4ff26da65c39 P fb4fc23dca0c4de69aef5061d3a67b2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb4fc23dca0c4de69aef5061d3a67b2f; no voters:
I20240503 01:59:29.939245 3770 leader_election.cc:288] T bd3627afefd74c39b43a4ff26da65c39 P fb4fc23dca0c4de69aef5061d3a67b2f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 01:59:29.939325 3774 raft_consensus.cc:2781] T bd3627afefd74c39b43a4ff26da65c39 P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:29.939427 3774 raft_consensus.cc:686] T bd3627afefd74c39b43a4ff26da65c39 P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 LEADER]: Becoming Leader. State: Replica: fb4fc23dca0c4de69aef5061d3a67b2f, State: Running, Role: LEADER
I20240503 01:59:29.939473 3770 ts_tablet_manager.cc:1398] T bd3627afefd74c39b43a4ff26da65c39 P fb4fc23dca0c4de69aef5061d3a67b2f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 01:59:29.939559 3774 consensus_queue.cc:237] T bd3627afefd74c39b43a4ff26da65c39 P fb4fc23dca0c4de69aef5061d3a67b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } }
I20240503 01:59:29.940204 3763 tablet_bootstrap.cc:492] T 7056f7fb504545d19641ea2c58ebf6be P 4d61ec80bf4c475ea86ae2bcb866fd05: No bootstrap required, opened a new log
I20240503 01:59:29.940316 3763 ts_tablet_manager.cc:1367] T 7056f7fb504545d19641ea2c58ebf6be P 4d61ec80bf4c475ea86ae2bcb866fd05: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240503 01:59:29.941028 3763 raft_consensus.cc:348] T 7056f7fb504545d19641ea2c58ebf6be P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } }
I20240503 01:59:29.941144 3763 raft_consensus.cc:374] T 7056f7fb504545d19641ea2c58ebf6be P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:29.941184 3763 raft_consensus.cc:729] T 7056f7fb504545d19641ea2c58ebf6be P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d61ec80bf4c475ea86ae2bcb866fd05, State: Initialized, Role: FOLLOWER
I20240503 01:59:29.941309 3763 consensus_queue.cc:260] T 7056f7fb504545d19641ea2c58ebf6be P 4d61ec80bf4c475ea86ae2bcb866fd05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } }
I20240503 01:59:29.941382 3763 raft_consensus.cc:388] T 7056f7fb504545d19641ea2c58ebf6be P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 01:59:29.941411 3763 raft_consensus.cc:482] T 7056f7fb504545d19641ea2c58ebf6be P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 01:59:29.941471 3763 raft_consensus.cc:3037] T 7056f7fb504545d19641ea2c58ebf6be P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:29.942595 3763 raft_consensus.cc:504] T 7056f7fb504545d19641ea2c58ebf6be P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } }
I20240503 01:59:29.942823 3763 leader_election.cc:302] T 7056f7fb504545d19641ea2c58ebf6be P 4d61ec80bf4c475ea86ae2bcb866fd05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d61ec80bf4c475ea86ae2bcb866fd05; no voters:
I20240503 01:59:29.942953 3763 leader_election.cc:288] T 7056f7fb504545d19641ea2c58ebf6be P 4d61ec80bf4c475ea86ae2bcb866fd05 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 01:59:29.943078 3767 raft_consensus.cc:2781] T 7056f7fb504545d19641ea2c58ebf6be P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:29.943457 3767 raft_consensus.cc:686] T 7056f7fb504545d19641ea2c58ebf6be P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 LEADER]: Becoming Leader. State: Replica: 4d61ec80bf4c475ea86ae2bcb866fd05, State: Running, Role: LEADER
I20240503 01:59:29.943626 3767 consensus_queue.cc:237] T 7056f7fb504545d19641ea2c58ebf6be P 4d61ec80bf4c475ea86ae2bcb866fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } }
I20240503 01:59:29.943734 3763 ts_tablet_manager.cc:1398] T 7056f7fb504545d19641ea2c58ebf6be P 4d61ec80bf4c475ea86ae2bcb866fd05: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 01:59:29.943958 3763 tablet_bootstrap.cc:492] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d61ec80bf4c475ea86ae2bcb866fd05: Bootstrap starting.
I20240503 01:59:29.944063 2751 catalog_manager.cc:5484] T bd3627afefd74c39b43a4ff26da65c39 P fb4fc23dca0c4de69aef5061d3a67b2f reported cstate change: term changed from 0 to 1, leader changed from <none> to fb4fc23dca0c4de69aef5061d3a67b2f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:29.945247 3763 tablet_bootstrap.cc:654] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d61ec80bf4c475ea86ae2bcb866fd05: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:29.945278 2747 catalog_manager.cc:5484] T 7056f7fb504545d19641ea2c58ebf6be P 4d61ec80bf4c475ea86ae2bcb866fd05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d61ec80bf4c475ea86ae2bcb866fd05 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:29.946993 3763 tablet_bootstrap.cc:492] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d61ec80bf4c475ea86ae2bcb866fd05: No bootstrap required, opened a new log
I20240503 01:59:29.947082 3763 ts_tablet_manager.cc:1367] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d61ec80bf4c475ea86ae2bcb866fd05: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 01:59:29.947538 3763 raft_consensus.cc:348] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } }
I20240503 01:59:29.947651 3763 raft_consensus.cc:374] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:29.947680 3763 raft_consensus.cc:729] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d61ec80bf4c475ea86ae2bcb866fd05, State: Initialized, Role: FOLLOWER
I20240503 01:59:29.947818 3763 consensus_queue.cc:260] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d61ec80bf4c475ea86ae2bcb866fd05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } }
I20240503 01:59:29.947898 3763 raft_consensus.cc:388] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 01:59:29.947921 3763 raft_consensus.cc:482] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 01:59:29.947948 3763 raft_consensus.cc:3037] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:29.948891 3763 raft_consensus.cc:504] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } }
I20240503 01:59:29.949105 3763 leader_election.cc:302] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d61ec80bf4c475ea86ae2bcb866fd05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d61ec80bf4c475ea86ae2bcb866fd05; no voters:
I20240503 01:59:29.949240 3766 raft_consensus.cc:2781] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:29.949304 3766 raft_consensus.cc:686] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 LEADER]: Becoming Leader. State: Replica: 4d61ec80bf4c475ea86ae2bcb866fd05, State: Running, Role: LEADER
I20240503 01:59:29.949426 3766 consensus_queue.cc:237] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d61ec80bf4c475ea86ae2bcb866fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } }
I20240503 01:59:29.949949 3763 leader_election.cc:288] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d61ec80bf4c475ea86ae2bcb866fd05 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 01:59:29.950239 3763 ts_tablet_manager.cc:1398] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d61ec80bf4c475ea86ae2bcb866fd05: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240503 01:59:29.951398 2751 catalog_manager.cc:5484] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d61ec80bf4c475ea86ae2bcb866fd05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d61ec80bf4c475ea86ae2bcb866fd05 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } health_report { overall_health: HEALTHY } } }
W20240503 01:59:30.071101 3822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:30.071390 3822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:30.074702 3822 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
INSERT report
rows total: 2000
time total: 76.7858 ms
time per row: 0.0383929 ms
I20240503 01:59:30.196695 2747 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38380:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:30.203408 3007 tablet_service.cc:1467] Processing CreateTablet for tablet fc4a011058e447499e3d651c73f0b881 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c045b78f2dc741a4b00a4b13ac99530e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:30.203506 2877 tablet_service.cc:1467] Processing CreateTablet for tablet eefe13b91700485bbcc398b37cec51b4 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c045b78f2dc741a4b00a4b13ac99530e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:30.203408 3008 tablet_service.cc:1467] Processing CreateTablet for tablet c3c2ff70515f49cf9f51b8db5e130cba (DEFAULT_TABLE table=rebalance_test_table_2 [id=c045b78f2dc741a4b00a4b13ac99530e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:30.203917 3007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc4a011058e447499e3d651c73f0b881. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:30.203936 2877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eefe13b91700485bbcc398b37cec51b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:30.204072 3008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3c2ff70515f49cf9f51b8db5e130cba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:30.205904 3763 tablet_bootstrap.cc:492] T eefe13b91700485bbcc398b37cec51b4 P 4d61ec80bf4c475ea86ae2bcb866fd05: Bootstrap starting.
I20240503 01:59:30.206369 3770 tablet_bootstrap.cc:492] T c3c2ff70515f49cf9f51b8db5e130cba P fb4fc23dca0c4de69aef5061d3a67b2f: Bootstrap starting.
I20240503 01:59:30.206908 3763 tablet_bootstrap.cc:654] T eefe13b91700485bbcc398b37cec51b4 P 4d61ec80bf4c475ea86ae2bcb866fd05: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:30.207307 3770 tablet_bootstrap.cc:654] T c3c2ff70515f49cf9f51b8db5e130cba P fb4fc23dca0c4de69aef5061d3a67b2f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:30.208046 3763 tablet_bootstrap.cc:492] T eefe13b91700485bbcc398b37cec51b4 P 4d61ec80bf4c475ea86ae2bcb866fd05: No bootstrap required, opened a new log
I20240503 01:59:30.208148 3763 ts_tablet_manager.cc:1367] T eefe13b91700485bbcc398b37cec51b4 P 4d61ec80bf4c475ea86ae2bcb866fd05: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 01:59:30.208544 3770 tablet_bootstrap.cc:492] T c3c2ff70515f49cf9f51b8db5e130cba P fb4fc23dca0c4de69aef5061d3a67b2f: No bootstrap required, opened a new log
I20240503 01:59:30.208626 3770 ts_tablet_manager.cc:1367] T c3c2ff70515f49cf9f51b8db5e130cba P fb4fc23dca0c4de69aef5061d3a67b2f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 01:59:30.208616 3763 raft_consensus.cc:348] T eefe13b91700485bbcc398b37cec51b4 P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } }
I20240503 01:59:30.208731 3763 raft_consensus.cc:374] T eefe13b91700485bbcc398b37cec51b4 P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:30.208770 3763 raft_consensus.cc:729] T eefe13b91700485bbcc398b37cec51b4 P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d61ec80bf4c475ea86ae2bcb866fd05, State: Initialized, Role: FOLLOWER
I20240503 01:59:30.208904 3763 consensus_queue.cc:260] T eefe13b91700485bbcc398b37cec51b4 P 4d61ec80bf4c475ea86ae2bcb866fd05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } }
I20240503 01:59:30.208992 3763 raft_consensus.cc:388] T eefe13b91700485bbcc398b37cec51b4 P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 01:59:30.209021 3763 raft_consensus.cc:482] T eefe13b91700485bbcc398b37cec51b4 P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 01:59:30.209062 3763 raft_consensus.cc:3037] T eefe13b91700485bbcc398b37cec51b4 P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:30.209100 3770 raft_consensus.cc:348] T c3c2ff70515f49cf9f51b8db5e130cba P fb4fc23dca0c4de69aef5061d3a67b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } }
I20240503 01:59:30.209192 3770 raft_consensus.cc:374] T c3c2ff70515f49cf9f51b8db5e130cba P fb4fc23dca0c4de69aef5061d3a67b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:30.209219 3770 raft_consensus.cc:729] T c3c2ff70515f49cf9f51b8db5e130cba P fb4fc23dca0c4de69aef5061d3a67b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb4fc23dca0c4de69aef5061d3a67b2f, State: Initialized, Role: FOLLOWER
I20240503 01:59:30.209335 3770 consensus_queue.cc:260] T c3c2ff70515f49cf9f51b8db5e130cba P fb4fc23dca0c4de69aef5061d3a67b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } }
I20240503 01:59:30.209430 3770 raft_consensus.cc:388] T c3c2ff70515f49cf9f51b8db5e130cba P fb4fc23dca0c4de69aef5061d3a67b2f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 01:59:30.209466 3770 raft_consensus.cc:482] T c3c2ff70515f49cf9f51b8db5e130cba P fb4fc23dca0c4de69aef5061d3a67b2f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 01:59:30.209512 3770 raft_consensus.cc:3037] T c3c2ff70515f49cf9f51b8db5e130cba P fb4fc23dca0c4de69aef5061d3a67b2f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:30.209942 3763 raft_consensus.cc:504] T eefe13b91700485bbcc398b37cec51b4 P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } }
I20240503 01:59:30.210096 3763 leader_election.cc:302] T eefe13b91700485bbcc398b37cec51b4 P 4d61ec80bf4c475ea86ae2bcb866fd05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d61ec80bf4c475ea86ae2bcb866fd05; no voters:
I20240503 01:59:30.210179 3763 leader_election.cc:288] T eefe13b91700485bbcc398b37cec51b4 P 4d61ec80bf4c475ea86ae2bcb866fd05 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 01:59:30.210245 3767 raft_consensus.cc:2781] T eefe13b91700485bbcc398b37cec51b4 P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:30.210315 3767 raft_consensus.cc:686] T eefe13b91700485bbcc398b37cec51b4 P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 LEADER]: Becoming Leader. State: Replica: 4d61ec80bf4c475ea86ae2bcb866fd05, State: Running, Role: LEADER
I20240503 01:59:30.210435 3763 ts_tablet_manager.cc:1398] T eefe13b91700485bbcc398b37cec51b4 P 4d61ec80bf4c475ea86ae2bcb866fd05: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 01:59:30.210428 3767 consensus_queue.cc:237] T eefe13b91700485bbcc398b37cec51b4 P 4d61ec80bf4c475ea86ae2bcb866fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } }
I20240503 01:59:30.210661 3770 raft_consensus.cc:504] T c3c2ff70515f49cf9f51b8db5e130cba P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } }
I20240503 01:59:30.210798 3770 leader_election.cc:302] T c3c2ff70515f49cf9f51b8db5e130cba P fb4fc23dca0c4de69aef5061d3a67b2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb4fc23dca0c4de69aef5061d3a67b2f; no voters:
I20240503 01:59:30.210955 3774 raft_consensus.cc:2781] T c3c2ff70515f49cf9f51b8db5e130cba P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:30.211035 3774 raft_consensus.cc:686] T c3c2ff70515f49cf9f51b8db5e130cba P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 LEADER]: Becoming Leader. State: Replica: fb4fc23dca0c4de69aef5061d3a67b2f, State: Running, Role: LEADER
I20240503 01:59:30.211159 3774 consensus_queue.cc:237] T c3c2ff70515f49cf9f51b8db5e130cba P fb4fc23dca0c4de69aef5061d3a67b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } }
I20240503 01:59:30.211573 3770 leader_election.cc:288] T c3c2ff70515f49cf9f51b8db5e130cba P fb4fc23dca0c4de69aef5061d3a67b2f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 01:59:30.211870 3770 ts_tablet_manager.cc:1398] T c3c2ff70515f49cf9f51b8db5e130cba P fb4fc23dca0c4de69aef5061d3a67b2f: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 01:59:30.212029 3770 tablet_bootstrap.cc:492] T fc4a011058e447499e3d651c73f0b881 P fb4fc23dca0c4de69aef5061d3a67b2f: Bootstrap starting.
I20240503 01:59:30.212069 2747 catalog_manager.cc:5484] T eefe13b91700485bbcc398b37cec51b4 P 4d61ec80bf4c475ea86ae2bcb866fd05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d61ec80bf4c475ea86ae2bcb866fd05 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:30.212764 2750 catalog_manager.cc:5484] T c3c2ff70515f49cf9f51b8db5e130cba P fb4fc23dca0c4de69aef5061d3a67b2f reported cstate change: term changed from 0 to 1, leader changed from <none> to fb4fc23dca0c4de69aef5061d3a67b2f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:30.214115 3770 tablet_bootstrap.cc:654] T fc4a011058e447499e3d651c73f0b881 P fb4fc23dca0c4de69aef5061d3a67b2f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:30.216941 3770 tablet_bootstrap.cc:492] T fc4a011058e447499e3d651c73f0b881 P fb4fc23dca0c4de69aef5061d3a67b2f: No bootstrap required, opened a new log
I20240503 01:59:30.217067 3770 ts_tablet_manager.cc:1367] T fc4a011058e447499e3d651c73f0b881 P fb4fc23dca0c4de69aef5061d3a67b2f: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240503 01:59:30.217495 3770 raft_consensus.cc:348] T fc4a011058e447499e3d651c73f0b881 P fb4fc23dca0c4de69aef5061d3a67b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } }
I20240503 01:59:30.217607 3770 raft_consensus.cc:374] T fc4a011058e447499e3d651c73f0b881 P fb4fc23dca0c4de69aef5061d3a67b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:30.217635 3770 raft_consensus.cc:729] T fc4a011058e447499e3d651c73f0b881 P fb4fc23dca0c4de69aef5061d3a67b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb4fc23dca0c4de69aef5061d3a67b2f, State: Initialized, Role: FOLLOWER
I20240503 01:59:30.217752 3770 consensus_queue.cc:260] T fc4a011058e447499e3d651c73f0b881 P fb4fc23dca0c4de69aef5061d3a67b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } }
I20240503 01:59:30.217834 3770 raft_consensus.cc:388] T fc4a011058e447499e3d651c73f0b881 P fb4fc23dca0c4de69aef5061d3a67b2f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 01:59:30.217864 3770 raft_consensus.cc:482] T fc4a011058e447499e3d651c73f0b881 P fb4fc23dca0c4de69aef5061d3a67b2f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 01:59:30.217900 3770 raft_consensus.cc:3037] T fc4a011058e447499e3d651c73f0b881 P fb4fc23dca0c4de69aef5061d3a67b2f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:30.218787 3770 raft_consensus.cc:504] T fc4a011058e447499e3d651c73f0b881 P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } }
I20240503 01:59:30.218938 3770 leader_election.cc:302] T fc4a011058e447499e3d651c73f0b881 P fb4fc23dca0c4de69aef5061d3a67b2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb4fc23dca0c4de69aef5061d3a67b2f; no voters:
I20240503 01:59:30.219045 3770 leader_election.cc:288] T fc4a011058e447499e3d651c73f0b881 P fb4fc23dca0c4de69aef5061d3a67b2f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 01:59:30.219275 3770 ts_tablet_manager.cc:1398] T fc4a011058e447499e3d651c73f0b881 P fb4fc23dca0c4de69aef5061d3a67b2f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 01:59:30.219422 3772 raft_consensus.cc:2781] T fc4a011058e447499e3d651c73f0b881 P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:30.219492 3772 raft_consensus.cc:686] T fc4a011058e447499e3d651c73f0b881 P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 LEADER]: Becoming Leader. State: Replica: fb4fc23dca0c4de69aef5061d3a67b2f, State: Running, Role: LEADER
I20240503 01:59:30.219623 3772 consensus_queue.cc:237] T fc4a011058e447499e3d651c73f0b881 P fb4fc23dca0c4de69aef5061d3a67b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } }
I20240503 01:59:30.221350 2750 catalog_manager.cc:5484] T fc4a011058e447499e3d651c73f0b881 P fb4fc23dca0c4de69aef5061d3a67b2f reported cstate change: term changed from 0 to 1, leader changed from <none> to fb4fc23dca0c4de69aef5061d3a67b2f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } health_report { overall_health: HEALTHY } } }
W20240503 01:59:30.340298 3858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:30.340631 3858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:30.343796 3858 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
INSERT report
rows total: 2000
time total: 69.5054 ms
time per row: 0.0347527 ms
I20240503 01:59:30.459964 2749 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38380:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:30.466166 2877 tablet_service.cc:1467] Processing CreateTablet for tablet 365919d2a8d94985bc031ab16eefe0d8 (DEFAULT_TABLE table=rebalance_test_table_3 [id=dfaafa31db2c42809a098afd0da706a3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:30.466166 3008 tablet_service.cc:1467] Processing CreateTablet for tablet a6f5711f3c9c4f3ca60893103ee7be40 (DEFAULT_TABLE table=rebalance_test_table_3 [id=dfaafa31db2c42809a098afd0da706a3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:30.466193 2876 tablet_service.cc:1467] Processing CreateTablet for tablet 201397ae513b4c40bd2e53197616984e (DEFAULT_TABLE table=rebalance_test_table_3 [id=dfaafa31db2c42809a098afd0da706a3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:30.466543 3008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6f5711f3c9c4f3ca60893103ee7be40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:30.466725 2877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 365919d2a8d94985bc031ab16eefe0d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:30.466954 2876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 201397ae513b4c40bd2e53197616984e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:30.468678 3763 tablet_bootstrap.cc:492] T 365919d2a8d94985bc031ab16eefe0d8 P 4d61ec80bf4c475ea86ae2bcb866fd05: Bootstrap starting.
I20240503 01:59:30.468891 3770 tablet_bootstrap.cc:492] T a6f5711f3c9c4f3ca60893103ee7be40 P fb4fc23dca0c4de69aef5061d3a67b2f: Bootstrap starting.
I20240503 01:59:30.469699 3763 tablet_bootstrap.cc:654] T 365919d2a8d94985bc031ab16eefe0d8 P 4d61ec80bf4c475ea86ae2bcb866fd05: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:30.469787 3770 tablet_bootstrap.cc:654] T a6f5711f3c9c4f3ca60893103ee7be40 P fb4fc23dca0c4de69aef5061d3a67b2f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:30.471110 3763 tablet_bootstrap.cc:492] T 365919d2a8d94985bc031ab16eefe0d8 P 4d61ec80bf4c475ea86ae2bcb866fd05: No bootstrap required, opened a new log
I20240503 01:59:30.471117 3770 tablet_bootstrap.cc:492] T a6f5711f3c9c4f3ca60893103ee7be40 P fb4fc23dca0c4de69aef5061d3a67b2f: No bootstrap required, opened a new log
I20240503 01:59:30.471201 3763 ts_tablet_manager.cc:1367] T 365919d2a8d94985bc031ab16eefe0d8 P 4d61ec80bf4c475ea86ae2bcb866fd05: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 01:59:30.471205 3770 ts_tablet_manager.cc:1367] T a6f5711f3c9c4f3ca60893103ee7be40 P fb4fc23dca0c4de69aef5061d3a67b2f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 01:59:30.471621 3770 raft_consensus.cc:348] T a6f5711f3c9c4f3ca60893103ee7be40 P fb4fc23dca0c4de69aef5061d3a67b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } }
I20240503 01:59:30.471619 3763 raft_consensus.cc:348] T 365919d2a8d94985bc031ab16eefe0d8 P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } }
I20240503 01:59:30.471720 3763 raft_consensus.cc:374] T 365919d2a8d94985bc031ab16eefe0d8 P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:30.471721 3770 raft_consensus.cc:374] T a6f5711f3c9c4f3ca60893103ee7be40 P fb4fc23dca0c4de69aef5061d3a67b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:30.471740 3763 raft_consensus.cc:729] T 365919d2a8d94985bc031ab16eefe0d8 P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d61ec80bf4c475ea86ae2bcb866fd05, State: Initialized, Role: FOLLOWER
I20240503 01:59:30.471760 3770 raft_consensus.cc:729] T a6f5711f3c9c4f3ca60893103ee7be40 P fb4fc23dca0c4de69aef5061d3a67b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb4fc23dca0c4de69aef5061d3a67b2f, State: Initialized, Role: FOLLOWER
I20240503 01:59:30.471854 3763 consensus_queue.cc:260] T 365919d2a8d94985bc031ab16eefe0d8 P 4d61ec80bf4c475ea86ae2bcb866fd05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } }
I20240503 01:59:30.471946 3763 raft_consensus.cc:388] T 365919d2a8d94985bc031ab16eefe0d8 P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 01:59:30.471907 3770 consensus_queue.cc:260] T a6f5711f3c9c4f3ca60893103ee7be40 P fb4fc23dca0c4de69aef5061d3a67b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } }
I20240503 01:59:30.471987 3763 raft_consensus.cc:482] T 365919d2a8d94985bc031ab16eefe0d8 P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 01:59:30.472003 3770 raft_consensus.cc:388] T a6f5711f3c9c4f3ca60893103ee7be40 P fb4fc23dca0c4de69aef5061d3a67b2f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 01:59:30.472031 3763 raft_consensus.cc:3037] T 365919d2a8d94985bc031ab16eefe0d8 P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:30.472035 3770 raft_consensus.cc:482] T a6f5711f3c9c4f3ca60893103ee7be40 P fb4fc23dca0c4de69aef5061d3a67b2f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 01:59:30.472076 3770 raft_consensus.cc:3037] T a6f5711f3c9c4f3ca60893103ee7be40 P fb4fc23dca0c4de69aef5061d3a67b2f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:30.472959 3763 raft_consensus.cc:504] T 365919d2a8d94985bc031ab16eefe0d8 P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } }
I20240503 01:59:30.472975 3770 raft_consensus.cc:504] T a6f5711f3c9c4f3ca60893103ee7be40 P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } }
I20240503 01:59:30.473114 3763 leader_election.cc:302] T 365919d2a8d94985bc031ab16eefe0d8 P 4d61ec80bf4c475ea86ae2bcb866fd05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d61ec80bf4c475ea86ae2bcb866fd05; no voters:
I20240503 01:59:30.473114 3770 leader_election.cc:302] T a6f5711f3c9c4f3ca60893103ee7be40 P fb4fc23dca0c4de69aef5061d3a67b2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb4fc23dca0c4de69aef5061d3a67b2f; no voters:
I20240503 01:59:30.473215 3770 leader_election.cc:288] T a6f5711f3c9c4f3ca60893103ee7be40 P fb4fc23dca0c4de69aef5061d3a67b2f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 01:59:30.473302 3767 raft_consensus.cc:2781] T 365919d2a8d94985bc031ab16eefe0d8 P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:30.473399 3767 raft_consensus.cc:686] T 365919d2a8d94985bc031ab16eefe0d8 P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 LEADER]: Becoming Leader. State: Replica: 4d61ec80bf4c475ea86ae2bcb866fd05, State: Running, Role: LEADER
I20240503 01:59:30.473461 3770 ts_tablet_manager.cc:1398] T a6f5711f3c9c4f3ca60893103ee7be40 P fb4fc23dca0c4de69aef5061d3a67b2f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 01:59:30.473534 3767 consensus_queue.cc:237] T 365919d2a8d94985bc031ab16eefe0d8 P 4d61ec80bf4c475ea86ae2bcb866fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } }
I20240503 01:59:30.473659 3772 raft_consensus.cc:2781] T a6f5711f3c9c4f3ca60893103ee7be40 P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:30.473762 3772 raft_consensus.cc:686] T a6f5711f3c9c4f3ca60893103ee7be40 P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 LEADER]: Becoming Leader. State: Replica: fb4fc23dca0c4de69aef5061d3a67b2f, State: Running, Role: LEADER
I20240503 01:59:30.473886 3772 consensus_queue.cc:237] T a6f5711f3c9c4f3ca60893103ee7be40 P fb4fc23dca0c4de69aef5061d3a67b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } }
I20240503 01:59:30.474157 3763 leader_election.cc:288] T 365919d2a8d94985bc031ab16eefe0d8 P 4d61ec80bf4c475ea86ae2bcb866fd05 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 01:59:30.474474 3763 ts_tablet_manager.cc:1398] T 365919d2a8d94985bc031ab16eefe0d8 P 4d61ec80bf4c475ea86ae2bcb866fd05: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:30.474834 3763 tablet_bootstrap.cc:492] T 201397ae513b4c40bd2e53197616984e P 4d61ec80bf4c475ea86ae2bcb866fd05: Bootstrap starting.
I20240503 01:59:30.475462 2749 catalog_manager.cc:5484] T 365919d2a8d94985bc031ab16eefe0d8 P 4d61ec80bf4c475ea86ae2bcb866fd05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d61ec80bf4c475ea86ae2bcb866fd05 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:30.475940 3763 tablet_bootstrap.cc:654] T 201397ae513b4c40bd2e53197616984e P 4d61ec80bf4c475ea86ae2bcb866fd05: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:30.476548 2754 catalog_manager.cc:5484] T a6f5711f3c9c4f3ca60893103ee7be40 P fb4fc23dca0c4de69aef5061d3a67b2f reported cstate change: term changed from 0 to 1, leader changed from <none> to fb4fc23dca0c4de69aef5061d3a67b2f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:30.478261 3763 tablet_bootstrap.cc:492] T 201397ae513b4c40bd2e53197616984e P 4d61ec80bf4c475ea86ae2bcb866fd05: No bootstrap required, opened a new log
I20240503 01:59:30.478372 3763 ts_tablet_manager.cc:1367] T 201397ae513b4c40bd2e53197616984e P 4d61ec80bf4c475ea86ae2bcb866fd05: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:30.478878 3763 raft_consensus.cc:348] T 201397ae513b4c40bd2e53197616984e P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } }
I20240503 01:59:30.478981 3763 raft_consensus.cc:374] T 201397ae513b4c40bd2e53197616984e P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:30.479007 3763 raft_consensus.cc:729] T 201397ae513b4c40bd2e53197616984e P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d61ec80bf4c475ea86ae2bcb866fd05, State: Initialized, Role: FOLLOWER
I20240503 01:59:30.479147 3763 consensus_queue.cc:260] T 201397ae513b4c40bd2e53197616984e P 4d61ec80bf4c475ea86ae2bcb866fd05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } }
I20240503 01:59:30.479220 3763 raft_consensus.cc:388] T 201397ae513b4c40bd2e53197616984e P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 01:59:30.479249 3763 raft_consensus.cc:482] T 201397ae513b4c40bd2e53197616984e P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 01:59:30.479297 3763 raft_consensus.cc:3037] T 201397ae513b4c40bd2e53197616984e P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:30.480252 3763 raft_consensus.cc:504] T 201397ae513b4c40bd2e53197616984e P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } }
I20240503 01:59:30.480784 3763 leader_election.cc:302] T 201397ae513b4c40bd2e53197616984e P 4d61ec80bf4c475ea86ae2bcb866fd05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d61ec80bf4c475ea86ae2bcb866fd05; no voters:
I20240503 01:59:30.480954 3765 raft_consensus.cc:2781] T 201397ae513b4c40bd2e53197616984e P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:30.481029 3765 raft_consensus.cc:686] T 201397ae513b4c40bd2e53197616984e P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 LEADER]: Becoming Leader. State: Replica: 4d61ec80bf4c475ea86ae2bcb866fd05, State: Running, Role: LEADER
I20240503 01:59:30.481176 3765 consensus_queue.cc:237] T 201397ae513b4c40bd2e53197616984e P 4d61ec80bf4c475ea86ae2bcb866fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } }
I20240503 01:59:30.481632 3763 leader_election.cc:288] T 201397ae513b4c40bd2e53197616984e P 4d61ec80bf4c475ea86ae2bcb866fd05 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 01:59:30.481971 3763 ts_tablet_manager.cc:1398] T 201397ae513b4c40bd2e53197616984e P 4d61ec80bf4c475ea86ae2bcb866fd05: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:30.482678 2749 catalog_manager.cc:5484] T 201397ae513b4c40bd2e53197616984e P 4d61ec80bf4c475ea86ae2bcb866fd05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d61ec80bf4c475ea86ae2bcb866fd05 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } health_report { overall_health: HEALTHY } } }
W20240503 01:59:30.605629 3891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:30.605926 3891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:30.609822 3891 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
INSERT report
rows total: 2000
time total: 73.639 ms
time per row: 0.0368195 ms
I20240503 01:59:30.732286 2755 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38380:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:30.738435 3008 tablet_service.cc:1467] Processing CreateTablet for tablet 9bfd209000fd468385a66c8fa5b2edc8 (DEFAULT_TABLE table=rebalance_test_table_4 [id=8a70cfe3f9b84945b710ac326669a37e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:30.738780 3008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bfd209000fd468385a66c8fa5b2edc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:30.740665 3770 tablet_bootstrap.cc:492] T 9bfd209000fd468385a66c8fa5b2edc8 P fb4fc23dca0c4de69aef5061d3a67b2f: Bootstrap starting.
I20240503 01:59:30.740824 3007 tablet_service.cc:1467] Processing CreateTablet for tablet 87da933cf2034ccebc5a78c970acb563 (DEFAULT_TABLE table=rebalance_test_table_4 [id=8a70cfe3f9b84945b710ac326669a37e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:30.741094 3007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87da933cf2034ccebc5a78c970acb563. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:30.741473 3770 tablet_bootstrap.cc:654] T 9bfd209000fd468385a66c8fa5b2edc8 P fb4fc23dca0c4de69aef5061d3a67b2f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:30.742492 3770 tablet_bootstrap.cc:492] T 9bfd209000fd468385a66c8fa5b2edc8 P fb4fc23dca0c4de69aef5061d3a67b2f: No bootstrap required, opened a new log
I20240503 01:59:30.742586 3770 ts_tablet_manager.cc:1367] T 9bfd209000fd468385a66c8fa5b2edc8 P fb4fc23dca0c4de69aef5061d3a67b2f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:30.742990 3770 raft_consensus.cc:348] T 9bfd209000fd468385a66c8fa5b2edc8 P fb4fc23dca0c4de69aef5061d3a67b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } }
I20240503 01:59:30.743072 3770 raft_consensus.cc:374] T 9bfd209000fd468385a66c8fa5b2edc8 P fb4fc23dca0c4de69aef5061d3a67b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:30.743099 3770 raft_consensus.cc:729] T 9bfd209000fd468385a66c8fa5b2edc8 P fb4fc23dca0c4de69aef5061d3a67b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb4fc23dca0c4de69aef5061d3a67b2f, State: Initialized, Role: FOLLOWER
I20240503 01:59:30.743199 3770 consensus_queue.cc:260] T 9bfd209000fd468385a66c8fa5b2edc8 P fb4fc23dca0c4de69aef5061d3a67b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } }
I20240503 01:59:30.743270 3770 raft_consensus.cc:388] T 9bfd209000fd468385a66c8fa5b2edc8 P fb4fc23dca0c4de69aef5061d3a67b2f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 01:59:30.743297 3770 raft_consensus.cc:482] T 9bfd209000fd468385a66c8fa5b2edc8 P fb4fc23dca0c4de69aef5061d3a67b2f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 01:59:30.743332 3770 raft_consensus.cc:3037] T 9bfd209000fd468385a66c8fa5b2edc8 P fb4fc23dca0c4de69aef5061d3a67b2f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:30.743583 2876 tablet_service.cc:1467] Processing CreateTablet for tablet 0cbba6e3ce5b4de5b5ec9164e50f784a (DEFAULT_TABLE table=rebalance_test_table_4 [id=8a70cfe3f9b84945b710ac326669a37e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:30.743989 3770 raft_consensus.cc:504] T 9bfd209000fd468385a66c8fa5b2edc8 P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } }
I20240503 01:59:30.744100 3770 leader_election.cc:302] T 9bfd209000fd468385a66c8fa5b2edc8 P fb4fc23dca0c4de69aef5061d3a67b2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb4fc23dca0c4de69aef5061d3a67b2f; no voters:
I20240503 01:59:30.744123 2876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cbba6e3ce5b4de5b5ec9164e50f784a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:30.744176 3770 leader_election.cc:288] T 9bfd209000fd468385a66c8fa5b2edc8 P fb4fc23dca0c4de69aef5061d3a67b2f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 01:59:30.744258 3773 raft_consensus.cc:2781] T 9bfd209000fd468385a66c8fa5b2edc8 P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:30.744688 3770 ts_tablet_manager.cc:1398] T 9bfd209000fd468385a66c8fa5b2edc8 P fb4fc23dca0c4de69aef5061d3a67b2f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:30.744755 3773 raft_consensus.cc:686] T 9bfd209000fd468385a66c8fa5b2edc8 P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 LEADER]: Becoming Leader. State: Replica: fb4fc23dca0c4de69aef5061d3a67b2f, State: Running, Role: LEADER
I20240503 01:59:30.744935 3773 consensus_queue.cc:237] T 9bfd209000fd468385a66c8fa5b2edc8 P fb4fc23dca0c4de69aef5061d3a67b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } }
I20240503 01:59:30.746136 3770 tablet_bootstrap.cc:492] T 87da933cf2034ccebc5a78c970acb563 P fb4fc23dca0c4de69aef5061d3a67b2f: Bootstrap starting.
I20240503 01:59:30.746981 3770 tablet_bootstrap.cc:654] T 87da933cf2034ccebc5a78c970acb563 P fb4fc23dca0c4de69aef5061d3a67b2f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:30.747058 3763 tablet_bootstrap.cc:492] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 4d61ec80bf4c475ea86ae2bcb866fd05: Bootstrap starting.
I20240503 01:59:30.748071 3763 tablet_bootstrap.cc:654] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 4d61ec80bf4c475ea86ae2bcb866fd05: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:30.748198 2755 catalog_manager.cc:5484] T 9bfd209000fd468385a66c8fa5b2edc8 P fb4fc23dca0c4de69aef5061d3a67b2f reported cstate change: term changed from 0 to 1, leader changed from <none> to fb4fc23dca0c4de69aef5061d3a67b2f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:30.749594 3770 tablet_bootstrap.cc:492] T 87da933cf2034ccebc5a78c970acb563 P fb4fc23dca0c4de69aef5061d3a67b2f: No bootstrap required, opened a new log
I20240503 01:59:30.749698 3770 ts_tablet_manager.cc:1367] T 87da933cf2034ccebc5a78c970acb563 P fb4fc23dca0c4de69aef5061d3a67b2f: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 01:59:30.750362 3770 raft_consensus.cc:348] T 87da933cf2034ccebc5a78c970acb563 P fb4fc23dca0c4de69aef5061d3a67b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } }
I20240503 01:59:30.750501 3770 raft_consensus.cc:374] T 87da933cf2034ccebc5a78c970acb563 P fb4fc23dca0c4de69aef5061d3a67b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:30.750552 3770 raft_consensus.cc:729] T 87da933cf2034ccebc5a78c970acb563 P fb4fc23dca0c4de69aef5061d3a67b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb4fc23dca0c4de69aef5061d3a67b2f, State: Initialized, Role: FOLLOWER
I20240503 01:59:30.750697 3770 consensus_queue.cc:260] T 87da933cf2034ccebc5a78c970acb563 P fb4fc23dca0c4de69aef5061d3a67b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } }
I20240503 01:59:30.750794 3770 raft_consensus.cc:388] T 87da933cf2034ccebc5a78c970acb563 P fb4fc23dca0c4de69aef5061d3a67b2f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 01:59:30.750838 3770 raft_consensus.cc:482] T 87da933cf2034ccebc5a78c970acb563 P fb4fc23dca0c4de69aef5061d3a67b2f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 01:59:30.750892 3770 raft_consensus.cc:3037] T 87da933cf2034ccebc5a78c970acb563 P fb4fc23dca0c4de69aef5061d3a67b2f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:30.751876 3770 raft_consensus.cc:504] T 87da933cf2034ccebc5a78c970acb563 P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } }
I20240503 01:59:30.752069 3770 leader_election.cc:302] T 87da933cf2034ccebc5a78c970acb563 P fb4fc23dca0c4de69aef5061d3a67b2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb4fc23dca0c4de69aef5061d3a67b2f; no voters:
I20240503 01:59:30.752230 3770 leader_election.cc:288] T 87da933cf2034ccebc5a78c970acb563 P fb4fc23dca0c4de69aef5061d3a67b2f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 01:59:30.752298 3773 raft_consensus.cc:2781] T 87da933cf2034ccebc5a78c970acb563 P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:30.752445 3773 raft_consensus.cc:686] T 87da933cf2034ccebc5a78c970acb563 P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 LEADER]: Becoming Leader. State: Replica: fb4fc23dca0c4de69aef5061d3a67b2f, State: Running, Role: LEADER
I20240503 01:59:30.752564 3773 consensus_queue.cc:237] T 87da933cf2034ccebc5a78c970acb563 P fb4fc23dca0c4de69aef5061d3a67b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } }
I20240503 01:59:30.752619 3770 ts_tablet_manager.cc:1398] T 87da933cf2034ccebc5a78c970acb563 P fb4fc23dca0c4de69aef5061d3a67b2f: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240503 01:59:30.754381 2755 catalog_manager.cc:5484] T 87da933cf2034ccebc5a78c970acb563 P fb4fc23dca0c4de69aef5061d3a67b2f reported cstate change: term changed from 0 to 1, leader changed from <none> to fb4fc23dca0c4de69aef5061d3a67b2f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:30.758070 3763 tablet_bootstrap.cc:492] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 4d61ec80bf4c475ea86ae2bcb866fd05: No bootstrap required, opened a new log
I20240503 01:59:30.758177 3763 ts_tablet_manager.cc:1367] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 4d61ec80bf4c475ea86ae2bcb866fd05: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.002s
I20240503 01:59:30.758596 3763 raft_consensus.cc:348] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } }
I20240503 01:59:30.758723 3763 raft_consensus.cc:374] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:30.758745 3763 raft_consensus.cc:729] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d61ec80bf4c475ea86ae2bcb866fd05, State: Initialized, Role: FOLLOWER
I20240503 01:59:30.758870 3763 consensus_queue.cc:260] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 4d61ec80bf4c475ea86ae2bcb866fd05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } }
I20240503 01:59:30.758934 3763 raft_consensus.cc:388] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 01:59:30.758965 3763 raft_consensus.cc:482] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 01:59:30.759009 3763 raft_consensus.cc:3037] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:30.759769 3763 raft_consensus.cc:504] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } }
I20240503 01:59:30.759892 3763 leader_election.cc:302] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 4d61ec80bf4c475ea86ae2bcb866fd05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d61ec80bf4c475ea86ae2bcb866fd05; no voters:
I20240503 01:59:30.759992 3763 leader_election.cc:288] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 4d61ec80bf4c475ea86ae2bcb866fd05 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 01:59:30.760051 3767 raft_consensus.cc:2781] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:30.760236 3767 raft_consensus.cc:686] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 LEADER]: Becoming Leader. State: Replica: 4d61ec80bf4c475ea86ae2bcb866fd05, State: Running, Role: LEADER
I20240503 01:59:30.760242 3763 ts_tablet_manager.cc:1398] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 4d61ec80bf4c475ea86ae2bcb866fd05: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 01:59:30.760428 3767 consensus_queue.cc:237] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 4d61ec80bf4c475ea86ae2bcb866fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } }
I20240503 01:59:30.762013 2755 catalog_manager.cc:5484] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 4d61ec80bf4c475ea86ae2bcb866fd05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d61ec80bf4c475ea86ae2bcb866fd05 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } health_report { overall_health: HEALTHY } } }
W20240503 01:59:30.880985 3925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:30.881261 3925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:30.884443 3925 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
INSERT report
rows total: 2000
time total: 74.5457 ms
time per row: 0.0372729 ms
I20240503 01:59:31.001258 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
/tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:44037
--local_ip_for_outbound_sockets=127.0.99.131
--tserver_master_addrs=127.0.99.190:43851
--webserver_port=39689
--webserver_interface=127.0.99.131
--builtin_ntp_servers=127.0.99.148:35751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 01:59:31.111351 3955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:31.111603 3955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:31.111650 3955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:31.114928 3955 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:31.115012 3955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:31.115083 3955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 01:59:31.118942 3955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:44037
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=39689
--tserver_master_addrs=127.0.99.190:43851
--never_fsync=true
--heap_profile_path=/tmp/kudu.3955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:31.119918 3955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:31.121137 3955 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:31.129391 3961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:31.129448 3960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:31.129391 3963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:31.129766 3955 server_base.cc:1034] running on GCE node
I20240503 01:59:31.130190 3955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:31.130820 3955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:31.132001 3955 hybrid_clock.cc:648] HybridClock initialized: now 1714701571131979 us; error 38 us; skew 500 ppm
I20240503 01:59:31.134135 3955 webserver.cc:460] Webserver started at http://127.0.99.131:39689/ using document root <none> and password file <none>
I20240503 01:59:31.134730 3955 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:31.134804 3955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:31.138437 3955 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 01:59:31.140579 3969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:31.141600 3955 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:31.141712 3955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "ad0b1cf28c9c452db2230e4cd69363bc"
format_stamp: "Formatted at 2024-05-03 01:59:24 on dist-test-slave-pz5w"
I20240503 01:59:31.142163 3955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:31.153527 3955 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:31.154206 3955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:31.154366 3955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:31.154978 3955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:31.155884 3955 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:31.155932 3955 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:31.155987 3955 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:31.156015 3955 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:31.164232 3955 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:44037
I20240503 01:59:31.164283 4082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:44037 every 8 connection(s)
I20240503 01:59:31.165345 3955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 01:59:31.167093 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu as pid 3955
I20240503 01:59:31.167454 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
/tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:41327
--local_ip_for_outbound_sockets=127.0.99.132
--tserver_master_addrs=127.0.99.190:43851
--webserver_port=34093
--webserver_interface=127.0.99.132
--builtin_ntp_servers=127.0.99.148:35751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 01:59:31.174562 4083 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43851
I20240503 01:59:31.174827 4083 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:31.175402 4083 heartbeater.cc:511] Master 127.0.99.190:43851 requested a full tablet report, sending...
I20240503 01:59:31.176625 2748 ts_manager.cc:194] Re-registered known tserver with Master: ad0b1cf28c9c452db2230e4cd69363bc (127.0.99.131:44037)
I20240503 01:59:31.177240 2748 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:58079
W20240503 01:59:31.277978 4086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:31.278229 4086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:31.278275 4086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:31.281571 4086 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:31.281653 4086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:31.281724 4086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 01:59:31.285704 4086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:41327
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=34093
--tserver_master_addrs=127.0.99.190:43851
--never_fsync=true
--heap_profile_path=/tmp/kudu.4086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:31.286664 4086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:31.287808 4086 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:31.296032 4091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:31.296164 4092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:31.296294 4086 server_base.cc:1034] running on GCE node
W20240503 01:59:31.296032 4094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:31.296844 4086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:31.297412 4086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:31.298599 4086 hybrid_clock.cc:648] HybridClock initialized: now 1714701571298579 us; error 34 us; skew 500 ppm
I20240503 01:59:31.300704 4086 webserver.cc:460] Webserver started at http://127.0.99.132:34093/ using document root <none> and password file <none>
I20240503 01:59:31.301278 4086 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:31.301352 4086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:31.305018 4086 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 01:59:31.307091 4100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:31.308246 4086 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:31.308396 4086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "c8a3e298f9d441c09c064a7f5655980b"
format_stamp: "Formatted at 2024-05-03 01:59:25 on dist-test-slave-pz5w"
I20240503 01:59:31.308955 4086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:31.327678 4086 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:31.328366 4086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:31.328573 4086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:31.329172 4086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:31.330124 4086 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:31.330179 4086 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:31.330220 4086 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:31.330235 4086 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:31.338577 4086 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:41327
I20240503 01:59:31.338600 4213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:41327 every 8 connection(s)
I20240503 01:59:31.339607 4086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 01:59:31.343469 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu as pid 4086
I20240503 01:59:31.343914 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
/tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:41953
--local_ip_for_outbound_sockets=127.0.99.133
--tserver_master_addrs=127.0.99.190:43851
--webserver_port=42123
--webserver_interface=127.0.99.133
--builtin_ntp_servers=127.0.99.148:35751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 01:59:31.349225 4214 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43851
I20240503 01:59:31.349481 4214 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:31.350090 4214 heartbeater.cc:511] Master 127.0.99.190:43851 requested a full tablet report, sending...
I20240503 01:59:31.351238 2748 ts_manager.cc:194] Re-registered known tserver with Master: c8a3e298f9d441c09c064a7f5655980b (127.0.99.132:41327)
I20240503 01:59:31.351814 2748 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:54619
W20240503 01:59:31.453662 4217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:31.453913 4217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:31.453960 4217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:31.457350 4217 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:31.457437 4217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:31.457511 4217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 01:59:31.461369 4217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:41953
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=42123
--tserver_master_addrs=127.0.99.190:43851
--never_fsync=true
--heap_profile_path=/tmp/kudu.4217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:31.462321 4217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:31.463454 4217 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:31.471487 4223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:31.471537 4222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:31.471537 4225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:31.471880 4217 server_base.cc:1034] running on GCE node
I20240503 01:59:31.472257 4217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:31.472936 4217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:31.474120 4217 hybrid_clock.cc:648] HybridClock initialized: now 1714701571474096 us; error 41 us; skew 500 ppm
I20240503 01:59:31.476127 4217 webserver.cc:460] Webserver started at http://127.0.99.133:42123/ using document root <none> and password file <none>
I20240503 01:59:31.476742 4217 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:31.476822 4217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:31.480336 4217 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:31.482391 4231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:31.483457 4217 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:31.483588 4217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "e296c081ab4a414294313ab9883cae84"
format_stamp: "Formatted at 2024-05-03 01:59:25 on dist-test-slave-pz5w"
I20240503 01:59:31.484083 4217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:31.495513 4217 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:31.496168 4217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:31.496336 4217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:31.496891 4217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:31.497771 4217 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:31.497819 4217 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:31.497856 4217 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:31.497879 4217 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:31.505973 4217 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:41953
I20240503 01:59:31.506048 4344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:41953 every 8 connection(s)
I20240503 01:59:31.507035 4217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 01:59:31.510179 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu as pid 4217
I20240503 01:59:31.510597 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
/tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:46423
--local_ip_for_outbound_sockets=127.0.99.134
--tserver_master_addrs=127.0.99.190:43851
--webserver_port=46211
--webserver_interface=127.0.99.134
--builtin_ntp_servers=127.0.99.148:35751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 01:59:31.516561 4345 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43851
I20240503 01:59:31.516855 4345 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:31.517572 4345 heartbeater.cc:511] Master 127.0.99.190:43851 requested a full tablet report, sending...
I20240503 01:59:31.518736 2748 ts_manager.cc:194] Re-registered known tserver with Master: e296c081ab4a414294313ab9883cae84 (127.0.99.133:41953)
I20240503 01:59:31.519338 2748 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:45133
W20240503 01:59:31.622532 4348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:31.622784 4348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:31.622836 4348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:31.626091 4348 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:31.626174 4348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:31.626245 4348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 01:59:31.630209 4348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:46423
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=46211
--tserver_master_addrs=127.0.99.190:43851
--never_fsync=true
--heap_profile_path=/tmp/kudu.4348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:31.631143 4348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:31.632253 4348 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:31.640576 4356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:31.640587 4353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:31.640626 4354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:31.640807 4348 server_base.cc:1034] running on GCE node
I20240503 01:59:31.641223 4348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:31.641840 4348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:31.643033 4348 hybrid_clock.cc:648] HybridClock initialized: now 1714701571643017 us; error 25 us; skew 500 ppm
I20240503 01:59:31.645185 4348 webserver.cc:460] Webserver started at http://127.0.99.134:46211/ using document root <none> and password file <none>
I20240503 01:59:31.645758 4348 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:31.645826 4348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:31.649397 4348 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 01:59:31.651451 4362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:31.652559 4348 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 01:59:31.652680 4348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "e4c1a8048e4647b6936035bc2fc22b66"
format_stamp: "Formatted at 2024-05-03 01:59:25 on dist-test-slave-pz5w"
I20240503 01:59:31.653128 4348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:31.671701 4348 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:31.672521 4348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:31.672700 4348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:31.673277 4348 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:31.674240 4348 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:31.674293 4348 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:31.674350 4348 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:31.674401 4348 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:31.683002 4348 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:46423
I20240503 01:59:31.683039 4475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:46423 every 8 connection(s)
I20240503 01:59:31.684088 4348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 01:59:31.687004 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu as pid 4348
I20240503 01:59:31.687449 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
/tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:43573
--local_ip_for_outbound_sockets=127.0.99.135
--tserver_master_addrs=127.0.99.190:43851
--webserver_port=41415
--webserver_interface=127.0.99.135
--builtin_ntp_servers=127.0.99.148:35751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 01:59:31.694931 4476 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43851
I20240503 01:59:31.695211 4476 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:31.695876 4476 heartbeater.cc:511] Master 127.0.99.190:43851 requested a full tablet report, sending...
I20240503 01:59:31.697227 2748 ts_manager.cc:194] Re-registered known tserver with Master: e4c1a8048e4647b6936035bc2fc22b66 (127.0.99.134:46423)
I20240503 01:59:31.698006 2748 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:38105
W20240503 01:59:31.805419 4479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:31.805675 4479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:31.805717 4479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:31.809072 4479 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:31.809156 4479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:31.809227 4479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 01:59:31.813324 4479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:43573
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=41415
--tserver_master_addrs=127.0.99.190:43851
--never_fsync=true
--heap_profile_path=/tmp/kudu.4479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:31.814265 4479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:31.815414 4479 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:31.823783 4487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:31.823791 4485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:31.823949 4479 server_base.cc:1034] running on GCE node
W20240503 01:59:31.823782 4484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:31.824566 4479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:31.825174 4479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:31.826354 4479 hybrid_clock.cc:648] HybridClock initialized: now 1714701571826339 us; error 32 us; skew 500 ppm
I20240503 01:59:31.828408 4479 webserver.cc:460] Webserver started at http://127.0.99.135:41415/ using document root <none> and password file <none>
I20240503 01:59:31.828989 4479 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:31.829068 4479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:31.832780 4479 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 01:59:31.834815 4493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:31.835844 4479 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:31.835956 4479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "6c8442cedca54ad589e0e050282afee3"
format_stamp: "Formatted at 2024-05-03 01:59:25 on dist-test-slave-pz5w"
I20240503 01:59:31.836453 4479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:31.859431 4479 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:31.860157 4479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:31.860335 4479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:31.860947 4479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:31.861883 4479 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:31.861932 4479 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:31.861985 4479 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:31.862013 4479 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:31.871536 4479 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:43573
I20240503 01:59:31.871572 4606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:43573 every 8 connection(s)
I20240503 01:59:31.872692 4479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 01:59:31.873644 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu as pid 4479
I20240503 01:59:31.882059 4607 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43851
I20240503 01:59:31.882339 4607 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:31.883080 4607 heartbeater.cc:511] Master 127.0.99.190:43851 requested a full tablet report, sending...
I20240503 01:59:31.884251 2748 ts_manager.cc:194] Re-registered known tserver with Master: 6c8442cedca54ad589e0e050282afee3 (127.0.99.135:43573)
I20240503 01:59:31.884861 2748 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:46211
I20240503 01:59:32.084331 4017 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:32.085017 4148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:32.087731 4540 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:32.089074 3007 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:32.094923 4410 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:32.100314 4279 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:32.108711 2876 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:32.179257 4083 heartbeater.cc:503] Master 127.0.99.190:43851 was elected leader, sending a full tablet report...
I20240503 01:59:32.227877 2876 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:32.227912 4017 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:32.233103 4410 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:32.233741 3007 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:32.239584 4540 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:32.247296 4148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:32.250999 4279 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:32.290190 2897 consensus_queue.cc:237] T 365919d2a8d94985bc031ab16eefe0d8 P 4d61ec80bf4c475ea86ae2bcb866fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "ad0b1cf28c9c452db2230e4cd69363bc" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 44037 } attrs { promote: true } }
I20240503 01:59:32.291956 4697 raft_consensus.cc:2932] T 365919d2a8d94985bc031ab16eefe0d8 P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ad0b1cf28c9c452db2230e4cd69363bc (127.0.99.131) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "ad0b1cf28c9c452db2230e4cd69363bc" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 44037 } attrs { promote: true } } }
I20240503 01:59:32.295538 2749 catalog_manager.cc:5484] T 365919d2a8d94985bc031ab16eefe0d8 P 4d61ec80bf4c475ea86ae2bcb866fd05 reported cstate change: config changed from index -1 to 4, NON_VOTER ad0b1cf28c9c452db2230e4cd69363bc (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad0b1cf28c9c452db2230e4cd69363bc" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 44037 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 01:59:32.297607 2832 consensus_peers.cc:485] T 365919d2a8d94985bc031ab16eefe0d8 P 4d61ec80bf4c475ea86ae2bcb866fd05 -> Peer ad0b1cf28c9c452db2230e4cd69363bc (127.0.99.131:44037): Couldn't send request to peer ad0b1cf28c9c452db2230e4cd69363bc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 365919d2a8d94985bc031ab16eefe0d8. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:32.303920 3028 consensus_queue.cc:237] T c3c2ff70515f49cf9f51b8db5e130cba P fb4fc23dca0c4de69aef5061d3a67b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "6c8442cedca54ad589e0e050282afee3" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43573 } attrs { promote: true } }
I20240503 01:59:32.305459 4710 raft_consensus.cc:2932] T c3c2ff70515f49cf9f51b8db5e130cba P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6c8442cedca54ad589e0e050282afee3 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "6c8442cedca54ad589e0e050282afee3" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43573 } attrs { promote: true } } }
I20240503 01:59:32.308557 2749 catalog_manager.cc:5484] T c3c2ff70515f49cf9f51b8db5e130cba P fb4fc23dca0c4de69aef5061d3a67b2f reported cstate change: config changed from index -1 to 4, NON_VOTER 6c8442cedca54ad589e0e050282afee3 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c8442cedca54ad589e0e050282afee3" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43573 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 01:59:32.311405 2962 consensus_peers.cc:485] T c3c2ff70515f49cf9f51b8db5e130cba P fb4fc23dca0c4de69aef5061d3a67b2f -> Peer 6c8442cedca54ad589e0e050282afee3 (127.0.99.135:43573): Couldn't send request to peer 6c8442cedca54ad589e0e050282afee3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c3c2ff70515f49cf9f51b8db5e130cba. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:32.315798 2897 consensus_queue.cc:237] T 7fe4b04830cd42a292f72f27b788c75a P 4d61ec80bf4c475ea86ae2bcb866fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e296c081ab4a414294313ab9883cae84" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41953 } attrs { promote: true } }
I20240503 01:59:32.316835 4697 raft_consensus.cc:2932] T 7fe4b04830cd42a292f72f27b788c75a P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e296c081ab4a414294313ab9883cae84 (127.0.99.133) added. New config: { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e296c081ab4a414294313ab9883cae84" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41953 } attrs { promote: true } } }
I20240503 01:59:32.319437 2755 catalog_manager.cc:5484] T 7fe4b04830cd42a292f72f27b788c75a P 4d61ec80bf4c475ea86ae2bcb866fd05 reported cstate change: config changed from index -1 to 2, NON_VOTER e296c081ab4a414294313ab9883cae84 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" committed_config { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e296c081ab4a414294313ab9883cae84" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41953 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 01:59:32.321693 2830 consensus_peers.cc:485] T 7fe4b04830cd42a292f72f27b788c75a P 4d61ec80bf4c475ea86ae2bcb866fd05 -> Peer e296c081ab4a414294313ab9883cae84 (127.0.99.133:41953): Couldn't send request to peer e296c081ab4a414294313ab9883cae84. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7fe4b04830cd42a292f72f27b788c75a. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:32.326050 3028 consensus_queue.cc:237] T 9bfd209000fd468385a66c8fa5b2edc8 P fb4fc23dca0c4de69aef5061d3a67b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "c8a3e298f9d441c09c064a7f5655980b" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 41327 } attrs { promote: true } }
I20240503 01:59:32.327070 4710 raft_consensus.cc:2932] T 9bfd209000fd468385a66c8fa5b2edc8 P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8a3e298f9d441c09c064a7f5655980b (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "c8a3e298f9d441c09c064a7f5655980b" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 41327 } attrs { promote: true } } }
I20240503 01:59:32.329800 2755 catalog_manager.cc:5484] T 9bfd209000fd468385a66c8fa5b2edc8 P fb4fc23dca0c4de69aef5061d3a67b2f reported cstate change: config changed from index -1 to 4, NON_VOTER c8a3e298f9d441c09c064a7f5655980b (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8a3e298f9d441c09c064a7f5655980b" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 41327 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 01:59:32.331352 2962 consensus_peers.cc:485] T 9bfd209000fd468385a66c8fa5b2edc8 P fb4fc23dca0c4de69aef5061d3a67b2f -> Peer c8a3e298f9d441c09c064a7f5655980b (127.0.99.132:41327): Couldn't send request to peer c8a3e298f9d441c09c064a7f5655980b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9bfd209000fd468385a66c8fa5b2edc8. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:32.337348 2897 consensus_queue.cc:237] T 7056f7fb504545d19641ea2c58ebf6be P 4d61ec80bf4c475ea86ae2bcb866fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4c1a8048e4647b6936035bc2fc22b66" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46423 } attrs { promote: true } }
I20240503 01:59:32.338356 4697 raft_consensus.cc:2932] T 7056f7fb504545d19641ea2c58ebf6be P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e4c1a8048e4647b6936035bc2fc22b66 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4c1a8048e4647b6936035bc2fc22b66" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46423 } attrs { promote: true } } }
I20240503 01:59:32.340893 2755 catalog_manager.cc:5484] T 7056f7fb504545d19641ea2c58ebf6be P 4d61ec80bf4c475ea86ae2bcb866fd05 reported cstate change: config changed from index -1 to 4, NON_VOTER e4c1a8048e4647b6936035bc2fc22b66 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4c1a8048e4647b6936035bc2fc22b66" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46423 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 01:59:32.342482 2831 consensus_peers.cc:485] T 7056f7fb504545d19641ea2c58ebf6be P 4d61ec80bf4c475ea86ae2bcb866fd05 -> Peer e4c1a8048e4647b6936035bc2fc22b66 (127.0.99.134:46423): Couldn't send request to peer e4c1a8048e4647b6936035bc2fc22b66. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7056f7fb504545d19641ea2c58ebf6be. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:32.353960 4214 heartbeater.cc:503] Master 127.0.99.190:43851 was elected leader, sending a full tablet report...
I20240503 01:59:32.355810 3028 consensus_queue.cc:237] T fc4a011058e447499e3d651c73f0b881 P fb4fc23dca0c4de69aef5061d3a67b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "ad0b1cf28c9c452db2230e4cd69363bc" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 44037 } attrs { promote: true } }
I20240503 01:59:32.357046 4711 raft_consensus.cc:2932] T fc4a011058e447499e3d651c73f0b881 P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ad0b1cf28c9c452db2230e4cd69363bc (127.0.99.131) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "ad0b1cf28c9c452db2230e4cd69363bc" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 44037 } attrs { promote: true } } }
I20240503 01:59:32.359807 2749 catalog_manager.cc:5484] T fc4a011058e447499e3d651c73f0b881 P fb4fc23dca0c4de69aef5061d3a67b2f reported cstate change: config changed from index -1 to 4, NON_VOTER ad0b1cf28c9c452db2230e4cd69363bc (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad0b1cf28c9c452db2230e4cd69363bc" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 44037 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 01:59:32.361425 2963 consensus_peers.cc:485] T fc4a011058e447499e3d651c73f0b881 P fb4fc23dca0c4de69aef5061d3a67b2f -> Peer ad0b1cf28c9c452db2230e4cd69363bc (127.0.99.131:44037): Couldn't send request to peer ad0b1cf28c9c452db2230e4cd69363bc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc4a011058e447499e3d651c73f0b881. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:32.367704 2897 consensus_queue.cc:237] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 4d61ec80bf4c475ea86ae2bcb866fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "6c8442cedca54ad589e0e050282afee3" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43573 } attrs { promote: true } }
I20240503 01:59:32.368851 4698 raft_consensus.cc:2932] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6c8442cedca54ad589e0e050282afee3 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "6c8442cedca54ad589e0e050282afee3" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43573 } attrs { promote: true } } }
I20240503 01:59:32.371279 2749 catalog_manager.cc:5484] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 4d61ec80bf4c475ea86ae2bcb866fd05 reported cstate change: config changed from index -1 to 4, NON_VOTER 6c8442cedca54ad589e0e050282afee3 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c8442cedca54ad589e0e050282afee3" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43573 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 01:59:32.372818 2831 consensus_peers.cc:485] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 4d61ec80bf4c475ea86ae2bcb866fd05 -> Peer 6c8442cedca54ad589e0e050282afee3 (127.0.99.135:43573): Couldn't send request to peer 6c8442cedca54ad589e0e050282afee3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0cbba6e3ce5b4de5b5ec9164e50f784a. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:32.379071 3028 consensus_queue.cc:237] T b2ba35f5c3084e08a42c6f3381aa1cb5 P fb4fc23dca0c4de69aef5061d3a67b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "e296c081ab4a414294313ab9883cae84" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41953 } attrs { promote: true } }
I20240503 01:59:32.380195 4711 raft_consensus.cc:2932] T b2ba35f5c3084e08a42c6f3381aa1cb5 P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e296c081ab4a414294313ab9883cae84 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "e296c081ab4a414294313ab9883cae84" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41953 } attrs { promote: true } } }
I20240503 01:59:32.383100 2755 catalog_manager.cc:5484] T b2ba35f5c3084e08a42c6f3381aa1cb5 P fb4fc23dca0c4de69aef5061d3a67b2f reported cstate change: config changed from index -1 to 4, NON_VOTER e296c081ab4a414294313ab9883cae84 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e296c081ab4a414294313ab9883cae84" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41953 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 01:59:32.385679 2961 consensus_peers.cc:485] T b2ba35f5c3084e08a42c6f3381aa1cb5 P fb4fc23dca0c4de69aef5061d3a67b2f -> Peer e296c081ab4a414294313ab9883cae84 (127.0.99.133:41953): Couldn't send request to peer e296c081ab4a414294313ab9883cae84. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b2ba35f5c3084e08a42c6f3381aa1cb5. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:32.389776 3028 consensus_queue.cc:237] T 1b96b32dada445ccbe9c4eaa0ddee050 P fb4fc23dca0c4de69aef5061d3a67b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "c8a3e298f9d441c09c064a7f5655980b" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 41327 } attrs { promote: true } }
I20240503 01:59:32.390784 4711 raft_consensus.cc:2932] T 1b96b32dada445ccbe9c4eaa0ddee050 P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8a3e298f9d441c09c064a7f5655980b (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "c8a3e298f9d441c09c064a7f5655980b" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 41327 } attrs { promote: true } } }
W20240503 01:59:32.391309 2962 consensus_peers.cc:485] T 1b96b32dada445ccbe9c4eaa0ddee050 P fb4fc23dca0c4de69aef5061d3a67b2f -> Peer c8a3e298f9d441c09c064a7f5655980b (127.0.99.132:41327): Couldn't send request to peer c8a3e298f9d441c09c064a7f5655980b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1b96b32dada445ccbe9c4eaa0ddee050. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:32.393272 2755 catalog_manager.cc:5484] T 1b96b32dada445ccbe9c4eaa0ddee050 P fb4fc23dca0c4de69aef5061d3a67b2f reported cstate change: config changed from index -1 to 4, NON_VOTER c8a3e298f9d441c09c064a7f5655980b (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8a3e298f9d441c09c064a7f5655980b" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 41327 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:32.401351 2897 consensus_queue.cc:237] T 201397ae513b4c40bd2e53197616984e P 4d61ec80bf4c475ea86ae2bcb866fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4c1a8048e4647b6936035bc2fc22b66" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46423 } attrs { promote: true } }
I20240503 01:59:32.402473 4698 raft_consensus.cc:2932] T 201397ae513b4c40bd2e53197616984e P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e4c1a8048e4647b6936035bc2fc22b66 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4c1a8048e4647b6936035bc2fc22b66" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46423 } attrs { promote: true } } }
W20240503 01:59:32.402909 2831 consensus_peers.cc:485] T 201397ae513b4c40bd2e53197616984e P 4d61ec80bf4c475ea86ae2bcb866fd05 -> Peer e4c1a8048e4647b6936035bc2fc22b66 (127.0.99.134:46423): Couldn't send request to peer e4c1a8048e4647b6936035bc2fc22b66. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 201397ae513b4c40bd2e53197616984e. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:32.404878 2749 catalog_manager.cc:5484] T 201397ae513b4c40bd2e53197616984e P 4d61ec80bf4c475ea86ae2bcb866fd05 reported cstate change: config changed from index -1 to 4, NON_VOTER e4c1a8048e4647b6936035bc2fc22b66 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4c1a8048e4647b6936035bc2fc22b66" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46423 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:32.521854 4345 heartbeater.cc:503] Master 127.0.99.190:43851 was elected leader, sending a full tablet report...
I20240503 01:59:32.700285 4476 heartbeater.cc:503] Master 127.0.99.190:43851 was elected leader, sending a full tablet report...
I20240503 01:59:32.771143 4863 ts_tablet_manager.cc:902] T 9bfd209000fd468385a66c8fa5b2edc8 P c8a3e298f9d441c09c064a7f5655980b: Initiating tablet copy from peer fb4fc23dca0c4de69aef5061d3a67b2f (127.0.99.130:34163)
I20240503 01:59:32.771687 4864 ts_tablet_manager.cc:902] T 7fe4b04830cd42a292f72f27b788c75a P e296c081ab4a414294313ab9883cae84: Initiating tablet copy from peer 4d61ec80bf4c475ea86ae2bcb866fd05 (127.0.99.129:38011)
I20240503 01:59:32.771936 4863 tablet_copy_client.cc:308] T 9bfd209000fd468385a66c8fa5b2edc8 P c8a3e298f9d441c09c064a7f5655980b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:34163
I20240503 01:59:32.772404 4864 tablet_copy_client.cc:308] T 7fe4b04830cd42a292f72f27b788c75a P e296c081ab4a414294313ab9883cae84: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:38011
I20240503 01:59:32.775717 4875 ts_tablet_manager.cc:902] T c3c2ff70515f49cf9f51b8db5e130cba P 6c8442cedca54ad589e0e050282afee3: Initiating tablet copy from peer fb4fc23dca0c4de69aef5061d3a67b2f (127.0.99.130:34163)
I20240503 01:59:32.776062 3048 tablet_copy_service.cc:140] P fb4fc23dca0c4de69aef5061d3a67b2f: Received BeginTabletCopySession request for tablet 9bfd209000fd468385a66c8fa5b2edc8 from peer c8a3e298f9d441c09c064a7f5655980b ({username='slave'} at 127.0.99.132:38077)
I20240503 01:59:32.776204 3048 tablet_copy_service.cc:161] P fb4fc23dca0c4de69aef5061d3a67b2f: Beginning new tablet copy session on tablet 9bfd209000fd468385a66c8fa5b2edc8 from peer c8a3e298f9d441c09c064a7f5655980b at {username='slave'} at 127.0.99.132:38077: session id = c8a3e298f9d441c09c064a7f5655980b-9bfd209000fd468385a66c8fa5b2edc8
I20240503 01:59:32.776346 2917 tablet_copy_service.cc:140] P 4d61ec80bf4c475ea86ae2bcb866fd05: Received BeginTabletCopySession request for tablet 7fe4b04830cd42a292f72f27b788c75a from peer e296c081ab4a414294313ab9883cae84 ({username='slave'} at 127.0.99.133:51181)
I20240503 01:59:32.776510 2917 tablet_copy_service.cc:161] P 4d61ec80bf4c475ea86ae2bcb866fd05: Beginning new tablet copy session on tablet 7fe4b04830cd42a292f72f27b788c75a from peer e296c081ab4a414294313ab9883cae84 at {username='slave'} at 127.0.99.133:51181: session id = e296c081ab4a414294313ab9883cae84-7fe4b04830cd42a292f72f27b788c75a
I20240503 01:59:32.777091 4875 tablet_copy_client.cc:308] T c3c2ff70515f49cf9f51b8db5e130cba P 6c8442cedca54ad589e0e050282afee3: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:34163
I20240503 01:59:32.777943 2917 tablet_copy_source_session.cc:205] T 7fe4b04830cd42a292f72f27b788c75a P 4d61ec80bf4c475ea86ae2bcb866fd05: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:32.778146 3048 tablet_copy_source_session.cc:205] T 9bfd209000fd468385a66c8fa5b2edc8 P fb4fc23dca0c4de69aef5061d3a67b2f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:32.781464 4863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bfd209000fd468385a66c8fa5b2edc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.781464 4864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fe4b04830cd42a292f72f27b788c75a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.781610 3048 tablet_copy_service.cc:140] P fb4fc23dca0c4de69aef5061d3a67b2f: Received BeginTabletCopySession request for tablet c3c2ff70515f49cf9f51b8db5e130cba from peer 6c8442cedca54ad589e0e050282afee3 ({username='slave'} at 127.0.99.135:47285)
I20240503 01:59:32.781705 3048 tablet_copy_service.cc:161] P fb4fc23dca0c4de69aef5061d3a67b2f: Beginning new tablet copy session on tablet c3c2ff70515f49cf9f51b8db5e130cba from peer 6c8442cedca54ad589e0e050282afee3 at {username='slave'} at 127.0.99.135:47285: session id = 6c8442cedca54ad589e0e050282afee3-c3c2ff70515f49cf9f51b8db5e130cba
I20240503 01:59:32.782359 3048 tablet_copy_source_session.cc:205] T c3c2ff70515f49cf9f51b8db5e130cba P fb4fc23dca0c4de69aef5061d3a67b2f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:32.785179 4875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3c2ff70515f49cf9f51b8db5e130cba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.788237 4864 tablet_copy_client.cc:783] T 7fe4b04830cd42a292f72f27b788c75a P e296c081ab4a414294313ab9883cae84: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:32.788472 4864 tablet_copy_client.cc:647] T 7fe4b04830cd42a292f72f27b788c75a P e296c081ab4a414294313ab9883cae84: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:32.788523 4863 tablet_copy_client.cc:783] T 9bfd209000fd468385a66c8fa5b2edc8 P c8a3e298f9d441c09c064a7f5655980b: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:32.788692 4863 tablet_copy_client.cc:647] T 9bfd209000fd468385a66c8fa5b2edc8 P c8a3e298f9d441c09c064a7f5655980b: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:32.790455 4863 tablet_copy_client.cc:523] T 9bfd209000fd468385a66c8fa5b2edc8 P c8a3e298f9d441c09c064a7f5655980b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:32.790642 4864 tablet_copy_client.cc:523] T 7fe4b04830cd42a292f72f27b788c75a P e296c081ab4a414294313ab9883cae84: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:32.791548 4875 tablet_copy_client.cc:783] T c3c2ff70515f49cf9f51b8db5e130cba P 6c8442cedca54ad589e0e050282afee3: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:32.791749 4875 tablet_copy_client.cc:647] T c3c2ff70515f49cf9f51b8db5e130cba P 6c8442cedca54ad589e0e050282afee3: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:32.792209 4863 tablet_bootstrap.cc:492] T 9bfd209000fd468385a66c8fa5b2edc8 P c8a3e298f9d441c09c064a7f5655980b: Bootstrap starting.
I20240503 01:59:32.792635 4881 ts_tablet_manager.cc:902] T 365919d2a8d94985bc031ab16eefe0d8 P ad0b1cf28c9c452db2230e4cd69363bc: Initiating tablet copy from peer 4d61ec80bf4c475ea86ae2bcb866fd05 (127.0.99.129:38011)
I20240503 01:59:32.792706 4864 tablet_bootstrap.cc:492] T 7fe4b04830cd42a292f72f27b788c75a P e296c081ab4a414294313ab9883cae84: Bootstrap starting.
I20240503 01:59:32.793442 4875 tablet_copy_client.cc:523] T c3c2ff70515f49cf9f51b8db5e130cba P 6c8442cedca54ad589e0e050282afee3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:32.793603 4881 tablet_copy_client.cc:308] T 365919d2a8d94985bc031ab16eefe0d8 P ad0b1cf28c9c452db2230e4cd69363bc: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:38011
I20240503 01:59:32.795795 4875 tablet_bootstrap.cc:492] T c3c2ff70515f49cf9f51b8db5e130cba P 6c8442cedca54ad589e0e050282afee3: Bootstrap starting.
I20240503 01:59:32.798280 4864 log.cc:824] T 7fe4b04830cd42a292f72f27b788c75a P e296c081ab4a414294313ab9883cae84: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:32.799194 2917 tablet_copy_service.cc:140] P 4d61ec80bf4c475ea86ae2bcb866fd05: Received BeginTabletCopySession request for tablet 365919d2a8d94985bc031ab16eefe0d8 from peer ad0b1cf28c9c452db2230e4cd69363bc ({username='slave'} at 127.0.99.131:56457)
I20240503 01:59:32.799317 2917 tablet_copy_service.cc:161] P 4d61ec80bf4c475ea86ae2bcb866fd05: Beginning new tablet copy session on tablet 365919d2a8d94985bc031ab16eefe0d8 from peer ad0b1cf28c9c452db2230e4cd69363bc at {username='slave'} at 127.0.99.131:56457: session id = ad0b1cf28c9c452db2230e4cd69363bc-365919d2a8d94985bc031ab16eefe0d8
I20240503 01:59:32.799523 4863 log.cc:824] T 9bfd209000fd468385a66c8fa5b2edc8 P c8a3e298f9d441c09c064a7f5655980b: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:32.800004 4887 ts_tablet_manager.cc:902] T b2ba35f5c3084e08a42c6f3381aa1cb5 P e296c081ab4a414294313ab9883cae84: Initiating tablet copy from peer fb4fc23dca0c4de69aef5061d3a67b2f (127.0.99.130:34163)
I20240503 01:59:32.800238 2917 tablet_copy_source_session.cc:205] T 365919d2a8d94985bc031ab16eefe0d8 P 4d61ec80bf4c475ea86ae2bcb866fd05: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:32.800565 4887 tablet_copy_client.cc:308] T b2ba35f5c3084e08a42c6f3381aa1cb5 P e296c081ab4a414294313ab9883cae84: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:34163
I20240503 01:59:32.802090 4875 log.cc:824] T c3c2ff70515f49cf9f51b8db5e130cba P 6c8442cedca54ad589e0e050282afee3: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:32.803988 4881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 365919d2a8d94985bc031ab16eefe0d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.805414 3048 tablet_copy_service.cc:140] P fb4fc23dca0c4de69aef5061d3a67b2f: Received BeginTabletCopySession request for tablet b2ba35f5c3084e08a42c6f3381aa1cb5 from peer e296c081ab4a414294313ab9883cae84 ({username='slave'} at 127.0.99.133:40609)
I20240503 01:59:32.805534 3048 tablet_copy_service.cc:161] P fb4fc23dca0c4de69aef5061d3a67b2f: Beginning new tablet copy session on tablet b2ba35f5c3084e08a42c6f3381aa1cb5 from peer e296c081ab4a414294313ab9883cae84 at {username='slave'} at 127.0.99.133:40609: session id = e296c081ab4a414294313ab9883cae84-b2ba35f5c3084e08a42c6f3381aa1cb5
I20240503 01:59:32.806459 3048 tablet_copy_source_session.cc:205] T b2ba35f5c3084e08a42c6f3381aa1cb5 P fb4fc23dca0c4de69aef5061d3a67b2f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:32.807723 4887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2ba35f5c3084e08a42c6f3381aa1cb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.810891 4887 tablet_copy_client.cc:783] T b2ba35f5c3084e08a42c6f3381aa1cb5 P e296c081ab4a414294313ab9883cae84: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:32.811045 4887 tablet_copy_client.cc:647] T b2ba35f5c3084e08a42c6f3381aa1cb5 P e296c081ab4a414294313ab9883cae84: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:32.812346 4881 tablet_copy_client.cc:783] T 365919d2a8d94985bc031ab16eefe0d8 P ad0b1cf28c9c452db2230e4cd69363bc: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:32.812569 4881 tablet_copy_client.cc:647] T 365919d2a8d94985bc031ab16eefe0d8 P ad0b1cf28c9c452db2230e4cd69363bc: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:32.812608 4887 tablet_copy_client.cc:523] T b2ba35f5c3084e08a42c6f3381aa1cb5 P e296c081ab4a414294313ab9883cae84: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:32.814023 4887 tablet_bootstrap.cc:492] T b2ba35f5c3084e08a42c6f3381aa1cb5 P e296c081ab4a414294313ab9883cae84: Bootstrap starting.
I20240503 01:59:32.814210 4881 tablet_copy_client.cc:523] T 365919d2a8d94985bc031ab16eefe0d8 P ad0b1cf28c9c452db2230e4cd69363bc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:32.816294 4881 tablet_bootstrap.cc:492] T 365919d2a8d94985bc031ab16eefe0d8 P ad0b1cf28c9c452db2230e4cd69363bc: Bootstrap starting.
I20240503 01:59:32.816558 4864 tablet_bootstrap.cc:492] T 7fe4b04830cd42a292f72f27b788c75a P e296c081ab4a414294313ab9883cae84: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:32.816844 4864 tablet_bootstrap.cc:492] T 7fe4b04830cd42a292f72f27b788c75a P e296c081ab4a414294313ab9883cae84: Bootstrap complete.
I20240503 01:59:32.817137 4864 ts_tablet_manager.cc:1367] T 7fe4b04830cd42a292f72f27b788c75a P e296c081ab4a414294313ab9883cae84: Time spent bootstrapping tablet: real 0.025s user 0.008s sys 0.000s
I20240503 01:59:32.820758 4864 raft_consensus.cc:348] T 7fe4b04830cd42a292f72f27b788c75a P e296c081ab4a414294313ab9883cae84 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e296c081ab4a414294313ab9883cae84" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41953 } attrs { promote: true } }
I20240503 01:59:32.821156 4864 raft_consensus.cc:729] T 7fe4b04830cd42a292f72f27b788c75a P e296c081ab4a414294313ab9883cae84 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e296c081ab4a414294313ab9883cae84, State: Initialized, Role: LEARNER
I20240503 01:59:32.821559 4864 consensus_queue.cc:260] T 7fe4b04830cd42a292f72f27b788c75a P e296c081ab4a414294313ab9883cae84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e296c081ab4a414294313ab9883cae84" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41953 } attrs { promote: true } }
I20240503 01:59:32.822607 4864 ts_tablet_manager.cc:1398] T 7fe4b04830cd42a292f72f27b788c75a P e296c081ab4a414294313ab9883cae84: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240503 01:59:32.823837 4881 log.cc:824] T 365919d2a8d94985bc031ab16eefe0d8 P ad0b1cf28c9c452db2230e4cd69363bc: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:32.825004 2917 tablet_copy_service.cc:342] P 4d61ec80bf4c475ea86ae2bcb866fd05: Request end of tablet copy session e296c081ab4a414294313ab9883cae84-7fe4b04830cd42a292f72f27b788c75a received from {username='slave'} at 127.0.99.133:51181
I20240503 01:59:32.825112 2917 tablet_copy_service.cc:433] P 4d61ec80bf4c475ea86ae2bcb866fd05: ending tablet copy session e296c081ab4a414294313ab9883cae84-7fe4b04830cd42a292f72f27b788c75a on tablet 7fe4b04830cd42a292f72f27b788c75a with peer e296c081ab4a414294313ab9883cae84
I20240503 01:59:32.832159 4875 tablet_bootstrap.cc:492] T c3c2ff70515f49cf9f51b8db5e130cba P 6c8442cedca54ad589e0e050282afee3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:32.832603 4875 tablet_bootstrap.cc:492] T c3c2ff70515f49cf9f51b8db5e130cba P 6c8442cedca54ad589e0e050282afee3: Bootstrap complete.
I20240503 01:59:32.832938 4875 ts_tablet_manager.cc:1367] T c3c2ff70515f49cf9f51b8db5e130cba P 6c8442cedca54ad589e0e050282afee3: Time spent bootstrapping tablet: real 0.037s user 0.027s sys 0.008s
I20240503 01:59:32.835507 4875 raft_consensus.cc:348] T c3c2ff70515f49cf9f51b8db5e130cba P 6c8442cedca54ad589e0e050282afee3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "6c8442cedca54ad589e0e050282afee3" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43573 } attrs { promote: true } }
I20240503 01:59:32.835791 4875 raft_consensus.cc:729] T c3c2ff70515f49cf9f51b8db5e130cba P 6c8442cedca54ad589e0e050282afee3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6c8442cedca54ad589e0e050282afee3, State: Initialized, Role: LEARNER
I20240503 01:59:32.836163 4875 consensus_queue.cc:260] T c3c2ff70515f49cf9f51b8db5e130cba P 6c8442cedca54ad589e0e050282afee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "6c8442cedca54ad589e0e050282afee3" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43573 } attrs { promote: true } }
I20240503 01:59:32.837137 4875 ts_tablet_manager.cc:1398] T c3c2ff70515f49cf9f51b8db5e130cba P 6c8442cedca54ad589e0e050282afee3: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20240503 01:59:32.837528 4607 heartbeater.cc:503] Master 127.0.99.190:43851 was elected leader, sending a full tablet report...
I20240503 01:59:32.840473 3048 tablet_copy_service.cc:342] P fb4fc23dca0c4de69aef5061d3a67b2f: Request end of tablet copy session 6c8442cedca54ad589e0e050282afee3-c3c2ff70515f49cf9f51b8db5e130cba received from {username='slave'} at 127.0.99.135:47285
I20240503 01:59:32.840592 3048 tablet_copy_service.cc:433] P fb4fc23dca0c4de69aef5061d3a67b2f: ending tablet copy session 6c8442cedca54ad589e0e050282afee3-c3c2ff70515f49cf9f51b8db5e130cba on tablet c3c2ff70515f49cf9f51b8db5e130cba with peer 6c8442cedca54ad589e0e050282afee3
I20240503 01:59:32.846971 4863 tablet_bootstrap.cc:492] T 9bfd209000fd468385a66c8fa5b2edc8 P c8a3e298f9d441c09c064a7f5655980b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:32.847385 4875 ts_tablet_manager.cc:902] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 6c8442cedca54ad589e0e050282afee3: Initiating tablet copy from peer 4d61ec80bf4c475ea86ae2bcb866fd05 (127.0.99.129:38011)
I20240503 01:59:32.847908 4863 tablet_bootstrap.cc:492] T 9bfd209000fd468385a66c8fa5b2edc8 P c8a3e298f9d441c09c064a7f5655980b: Bootstrap complete.
I20240503 01:59:32.847965 4875 tablet_copy_client.cc:308] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 6c8442cedca54ad589e0e050282afee3: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:38011
I20240503 01:59:32.848106 4887 tablet_bootstrap.cc:492] T b2ba35f5c3084e08a42c6f3381aa1cb5 P e296c081ab4a414294313ab9883cae84: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:32.848392 4887 tablet_bootstrap.cc:492] T b2ba35f5c3084e08a42c6f3381aa1cb5 P e296c081ab4a414294313ab9883cae84: Bootstrap complete.
I20240503 01:59:32.848557 4887 ts_tablet_manager.cc:1367] T b2ba35f5c3084e08a42c6f3381aa1cb5 P e296c081ab4a414294313ab9883cae84: Time spent bootstrapping tablet: real 0.035s user 0.028s sys 0.005s
I20240503 01:59:32.848557 4863 ts_tablet_manager.cc:1367] T 9bfd209000fd468385a66c8fa5b2edc8 P c8a3e298f9d441c09c064a7f5655980b: Time spent bootstrapping tablet: real 0.056s user 0.031s sys 0.005s
I20240503 01:59:32.849166 4887 raft_consensus.cc:348] T b2ba35f5c3084e08a42c6f3381aa1cb5 P e296c081ab4a414294313ab9883cae84 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "e296c081ab4a414294313ab9883cae84" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41953 } attrs { promote: true } }
I20240503 01:59:32.849300 4887 raft_consensus.cc:729] T b2ba35f5c3084e08a42c6f3381aa1cb5 P e296c081ab4a414294313ab9883cae84 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e296c081ab4a414294313ab9883cae84, State: Initialized, Role: LEARNER
I20240503 01:59:32.849414 4887 consensus_queue.cc:260] T b2ba35f5c3084e08a42c6f3381aa1cb5 P e296c081ab4a414294313ab9883cae84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "e296c081ab4a414294313ab9883cae84" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41953 } attrs { promote: true } }
I20240503 01:59:32.851542 4863 raft_consensus.cc:348] T 9bfd209000fd468385a66c8fa5b2edc8 P c8a3e298f9d441c09c064a7f5655980b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "c8a3e298f9d441c09c064a7f5655980b" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 41327 } attrs { promote: true } }
I20240503 01:59:32.851814 4887 ts_tablet_manager.cc:1398] T b2ba35f5c3084e08a42c6f3381aa1cb5 P e296c081ab4a414294313ab9883cae84: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 01:59:32.851948 4863 raft_consensus.cc:729] T 9bfd209000fd468385a66c8fa5b2edc8 P c8a3e298f9d441c09c064a7f5655980b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c8a3e298f9d441c09c064a7f5655980b, State: Initialized, Role: LEARNER
I20240503 01:59:32.852448 4863 consensus_queue.cc:260] T 9bfd209000fd468385a66c8fa5b2edc8 P c8a3e298f9d441c09c064a7f5655980b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "c8a3e298f9d441c09c064a7f5655980b" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 41327 } attrs { promote: true } }
I20240503 01:59:32.853103 2917 tablet_copy_service.cc:140] P 4d61ec80bf4c475ea86ae2bcb866fd05: Received BeginTabletCopySession request for tablet 0cbba6e3ce5b4de5b5ec9164e50f784a from peer 6c8442cedca54ad589e0e050282afee3 ({username='slave'} at 127.0.99.135:53641)
I20240503 01:59:32.853236 2917 tablet_copy_service.cc:161] P 4d61ec80bf4c475ea86ae2bcb866fd05: Beginning new tablet copy session on tablet 0cbba6e3ce5b4de5b5ec9164e50f784a from peer 6c8442cedca54ad589e0e050282afee3 at {username='slave'} at 127.0.99.135:53641: session id = 6c8442cedca54ad589e0e050282afee3-0cbba6e3ce5b4de5b5ec9164e50f784a
I20240503 01:59:32.853736 4863 ts_tablet_manager.cc:1398] T 9bfd209000fd468385a66c8fa5b2edc8 P c8a3e298f9d441c09c064a7f5655980b: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20240503 01:59:32.854176 2917 tablet_copy_source_session.cc:205] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 4d61ec80bf4c475ea86ae2bcb866fd05: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:32.854329 3048 tablet_copy_service.cc:342] P fb4fc23dca0c4de69aef5061d3a67b2f: Request end of tablet copy session e296c081ab4a414294313ab9883cae84-b2ba35f5c3084e08a42c6f3381aa1cb5 received from {username='slave'} at 127.0.99.133:40609
I20240503 01:59:32.854409 3048 tablet_copy_service.cc:433] P fb4fc23dca0c4de69aef5061d3a67b2f: ending tablet copy session e296c081ab4a414294313ab9883cae84-b2ba35f5c3084e08a42c6f3381aa1cb5 on tablet b2ba35f5c3084e08a42c6f3381aa1cb5 with peer e296c081ab4a414294313ab9883cae84
I20240503 01:59:32.854751 3047 tablet_copy_service.cc:342] P fb4fc23dca0c4de69aef5061d3a67b2f: Request end of tablet copy session c8a3e298f9d441c09c064a7f5655980b-9bfd209000fd468385a66c8fa5b2edc8 received from {username='slave'} at 127.0.99.132:38077
I20240503 01:59:32.854825 3047 tablet_copy_service.cc:433] P fb4fc23dca0c4de69aef5061d3a67b2f: ending tablet copy session c8a3e298f9d441c09c064a7f5655980b-9bfd209000fd468385a66c8fa5b2edc8 on tablet 9bfd209000fd468385a66c8fa5b2edc8 with peer c8a3e298f9d441c09c064a7f5655980b
I20240503 01:59:32.855176 4875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cbba6e3ce5b4de5b5ec9164e50f784a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.857388 4875 tablet_copy_client.cc:783] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 6c8442cedca54ad589e0e050282afee3: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:32.857565 4875 tablet_copy_client.cc:647] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 6c8442cedca54ad589e0e050282afee3: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:32.858562 4881 tablet_bootstrap.cc:492] T 365919d2a8d94985bc031ab16eefe0d8 P ad0b1cf28c9c452db2230e4cd69363bc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:32.858896 4881 tablet_bootstrap.cc:492] T 365919d2a8d94985bc031ab16eefe0d8 P ad0b1cf28c9c452db2230e4cd69363bc: Bootstrap complete.
I20240503 01:59:32.858924 4875 tablet_copy_client.cc:523] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 6c8442cedca54ad589e0e050282afee3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:32.859205 4881 ts_tablet_manager.cc:1367] T 365919d2a8d94985bc031ab16eefe0d8 P ad0b1cf28c9c452db2230e4cd69363bc: Time spent bootstrapping tablet: real 0.043s user 0.037s sys 0.000s
I20240503 01:59:32.860073 4875 tablet_bootstrap.cc:492] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 6c8442cedca54ad589e0e050282afee3: Bootstrap starting.
I20240503 01:59:32.861394 4881 raft_consensus.cc:348] T 365919d2a8d94985bc031ab16eefe0d8 P ad0b1cf28c9c452db2230e4cd69363bc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "ad0b1cf28c9c452db2230e4cd69363bc" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 44037 } attrs { promote: true } }
I20240503 01:59:32.861624 4881 raft_consensus.cc:729] T 365919d2a8d94985bc031ab16eefe0d8 P ad0b1cf28c9c452db2230e4cd69363bc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ad0b1cf28c9c452db2230e4cd69363bc, State: Initialized, Role: LEARNER
I20240503 01:59:32.861955 4881 consensus_queue.cc:260] T 365919d2a8d94985bc031ab16eefe0d8 P ad0b1cf28c9c452db2230e4cd69363bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "ad0b1cf28c9c452db2230e4cd69363bc" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 44037 } attrs { promote: true } }
I20240503 01:59:32.862814 4881 ts_tablet_manager.cc:1398] T 365919d2a8d94985bc031ab16eefe0d8 P ad0b1cf28c9c452db2230e4cd69363bc: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20240503 01:59:32.863601 2917 tablet_copy_service.cc:342] P 4d61ec80bf4c475ea86ae2bcb866fd05: Request end of tablet copy session ad0b1cf28c9c452db2230e4cd69363bc-365919d2a8d94985bc031ab16eefe0d8 received from {username='slave'} at 127.0.99.131:56457
I20240503 01:59:32.863698 2917 tablet_copy_service.cc:433] P 4d61ec80bf4c475ea86ae2bcb866fd05: ending tablet copy session ad0b1cf28c9c452db2230e4cd69363bc-365919d2a8d94985bc031ab16eefe0d8 on tablet 365919d2a8d94985bc031ab16eefe0d8 with peer ad0b1cf28c9c452db2230e4cd69363bc
I20240503 01:59:32.878899 4875 tablet_bootstrap.cc:492] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 6c8442cedca54ad589e0e050282afee3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:32.879194 4875 tablet_bootstrap.cc:492] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 6c8442cedca54ad589e0e050282afee3: Bootstrap complete.
I20240503 01:59:32.879365 4875 ts_tablet_manager.cc:1367] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 6c8442cedca54ad589e0e050282afee3: Time spent bootstrapping tablet: real 0.019s user 0.017s sys 0.000s
I20240503 01:59:32.879858 4875 raft_consensus.cc:348] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 6c8442cedca54ad589e0e050282afee3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "6c8442cedca54ad589e0e050282afee3" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43573 } attrs { promote: true } }
I20240503 01:59:32.879963 4875 raft_consensus.cc:729] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 6c8442cedca54ad589e0e050282afee3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6c8442cedca54ad589e0e050282afee3, State: Initialized, Role: LEARNER
I20240503 01:59:32.880062 4875 consensus_queue.cc:260] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 6c8442cedca54ad589e0e050282afee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "6c8442cedca54ad589e0e050282afee3" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43573 } attrs { promote: true } }
I20240503 01:59:32.880307 4875 ts_tablet_manager.cc:1398] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 6c8442cedca54ad589e0e050282afee3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:32.880964 2917 tablet_copy_service.cc:342] P 4d61ec80bf4c475ea86ae2bcb866fd05: Request end of tablet copy session 6c8442cedca54ad589e0e050282afee3-0cbba6e3ce5b4de5b5ec9164e50f784a received from {username='slave'} at 127.0.99.135:53641
I20240503 01:59:32.881065 2917 tablet_copy_service.cc:433] P 4d61ec80bf4c475ea86ae2bcb866fd05: ending tablet copy session 6c8442cedca54ad589e0e050282afee3-0cbba6e3ce5b4de5b5ec9164e50f784a on tablet 0cbba6e3ce5b4de5b5ec9164e50f784a with peer 6c8442cedca54ad589e0e050282afee3
I20240503 01:59:32.892607 4906 ts_tablet_manager.cc:902] T 201397ae513b4c40bd2e53197616984e P e4c1a8048e4647b6936035bc2fc22b66: Initiating tablet copy from peer 4d61ec80bf4c475ea86ae2bcb866fd05 (127.0.99.129:38011)
I20240503 01:59:32.893309 4906 tablet_copy_client.cc:308] T 201397ae513b4c40bd2e53197616984e P e4c1a8048e4647b6936035bc2fc22b66: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:38011
I20240503 01:59:32.897123 2917 tablet_copy_service.cc:140] P 4d61ec80bf4c475ea86ae2bcb866fd05: Received BeginTabletCopySession request for tablet 201397ae513b4c40bd2e53197616984e from peer e4c1a8048e4647b6936035bc2fc22b66 ({username='slave'} at 127.0.99.134:55403)
I20240503 01:59:32.897245 2917 tablet_copy_service.cc:161] P 4d61ec80bf4c475ea86ae2bcb866fd05: Beginning new tablet copy session on tablet 201397ae513b4c40bd2e53197616984e from peer e4c1a8048e4647b6936035bc2fc22b66 at {username='slave'} at 127.0.99.134:55403: session id = e4c1a8048e4647b6936035bc2fc22b66-201397ae513b4c40bd2e53197616984e
I20240503 01:59:32.898121 2917 tablet_copy_source_session.cc:205] T 201397ae513b4c40bd2e53197616984e P 4d61ec80bf4c475ea86ae2bcb866fd05: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:32.898576 4912 ts_tablet_manager.cc:902] T 7056f7fb504545d19641ea2c58ebf6be P e4c1a8048e4647b6936035bc2fc22b66: Initiating tablet copy from peer 4d61ec80bf4c475ea86ae2bcb866fd05 (127.0.99.129:38011)
I20240503 01:59:32.899083 4912 tablet_copy_client.cc:308] T 7056f7fb504545d19641ea2c58ebf6be P e4c1a8048e4647b6936035bc2fc22b66: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:38011
I20240503 01:59:32.899693 2917 tablet_copy_service.cc:140] P 4d61ec80bf4c475ea86ae2bcb866fd05: Received BeginTabletCopySession request for tablet 7056f7fb504545d19641ea2c58ebf6be from peer e4c1a8048e4647b6936035bc2fc22b66 ({username='slave'} at 127.0.99.134:55403)
I20240503 01:59:32.899806 2917 tablet_copy_service.cc:161] P 4d61ec80bf4c475ea86ae2bcb866fd05: Beginning new tablet copy session on tablet 7056f7fb504545d19641ea2c58ebf6be from peer e4c1a8048e4647b6936035bc2fc22b66 at {username='slave'} at 127.0.99.134:55403: session id = e4c1a8048e4647b6936035bc2fc22b66-7056f7fb504545d19641ea2c58ebf6be
I20240503 01:59:32.900652 2917 tablet_copy_source_session.cc:205] T 7056f7fb504545d19641ea2c58ebf6be P 4d61ec80bf4c475ea86ae2bcb866fd05: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:32.901893 4912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7056f7fb504545d19641ea2c58ebf6be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.902299 4906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 201397ae513b4c40bd2e53197616984e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.908766 4912 tablet_copy_client.cc:783] T 7056f7fb504545d19641ea2c58ebf6be P e4c1a8048e4647b6936035bc2fc22b66: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:32.908993 4912 tablet_copy_client.cc:647] T 7056f7fb504545d19641ea2c58ebf6be P e4c1a8048e4647b6936035bc2fc22b66: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:32.908761 4906 tablet_copy_client.cc:783] T 201397ae513b4c40bd2e53197616984e P e4c1a8048e4647b6936035bc2fc22b66: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:32.909162 4906 tablet_copy_client.cc:647] T 201397ae513b4c40bd2e53197616984e P e4c1a8048e4647b6936035bc2fc22b66: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:32.910646 4912 tablet_copy_client.cc:523] T 7056f7fb504545d19641ea2c58ebf6be P e4c1a8048e4647b6936035bc2fc22b66: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:32.910646 4906 tablet_copy_client.cc:523] T 201397ae513b4c40bd2e53197616984e P e4c1a8048e4647b6936035bc2fc22b66: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:32.912776 4906 tablet_bootstrap.cc:492] T 201397ae513b4c40bd2e53197616984e P e4c1a8048e4647b6936035bc2fc22b66: Bootstrap starting.
I20240503 01:59:32.912776 4912 tablet_bootstrap.cc:492] T 7056f7fb504545d19641ea2c58ebf6be P e4c1a8048e4647b6936035bc2fc22b66: Bootstrap starting.
I20240503 01:59:32.920018 4906 log.cc:824] T 201397ae513b4c40bd2e53197616984e P e4c1a8048e4647b6936035bc2fc22b66: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:32.920018 4912 log.cc:824] T 7056f7fb504545d19641ea2c58ebf6be P e4c1a8048e4647b6936035bc2fc22b66: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:32.931493 4863 ts_tablet_manager.cc:902] T 1b96b32dada445ccbe9c4eaa0ddee050 P c8a3e298f9d441c09c064a7f5655980b: Initiating tablet copy from peer fb4fc23dca0c4de69aef5061d3a67b2f (127.0.99.130:34163)
I20240503 01:59:32.932428 4863 tablet_copy_client.cc:308] T 1b96b32dada445ccbe9c4eaa0ddee050 P c8a3e298f9d441c09c064a7f5655980b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:34163
I20240503 01:59:32.932963 3047 tablet_copy_service.cc:140] P fb4fc23dca0c4de69aef5061d3a67b2f: Received BeginTabletCopySession request for tablet 1b96b32dada445ccbe9c4eaa0ddee050 from peer c8a3e298f9d441c09c064a7f5655980b ({username='slave'} at 127.0.99.132:38077)
I20240503 01:59:32.933079 3047 tablet_copy_service.cc:161] P fb4fc23dca0c4de69aef5061d3a67b2f: Beginning new tablet copy session on tablet 1b96b32dada445ccbe9c4eaa0ddee050 from peer c8a3e298f9d441c09c064a7f5655980b at {username='slave'} at 127.0.99.132:38077: session id = c8a3e298f9d441c09c064a7f5655980b-1b96b32dada445ccbe9c4eaa0ddee050
I20240503 01:59:32.933857 3047 tablet_copy_source_session.cc:205] T 1b96b32dada445ccbe9c4eaa0ddee050 P fb4fc23dca0c4de69aef5061d3a67b2f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:32.934867 4863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b96b32dada445ccbe9c4eaa0ddee050. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.937325 4863 tablet_copy_client.cc:783] T 1b96b32dada445ccbe9c4eaa0ddee050 P c8a3e298f9d441c09c064a7f5655980b: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:32.937489 4863 tablet_copy_client.cc:647] T 1b96b32dada445ccbe9c4eaa0ddee050 P c8a3e298f9d441c09c064a7f5655980b: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:32.938771 4863 tablet_copy_client.cc:523] T 1b96b32dada445ccbe9c4eaa0ddee050 P c8a3e298f9d441c09c064a7f5655980b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:32.941030 4881 ts_tablet_manager.cc:902] T fc4a011058e447499e3d651c73f0b881 P ad0b1cf28c9c452db2230e4cd69363bc: Initiating tablet copy from peer fb4fc23dca0c4de69aef5061d3a67b2f (127.0.99.130:34163)
I20240503 01:59:32.941766 4881 tablet_copy_client.cc:308] T fc4a011058e447499e3d651c73f0b881 P ad0b1cf28c9c452db2230e4cd69363bc: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:34163
I20240503 01:59:32.947441 3047 tablet_copy_service.cc:140] P fb4fc23dca0c4de69aef5061d3a67b2f: Received BeginTabletCopySession request for tablet fc4a011058e447499e3d651c73f0b881 from peer ad0b1cf28c9c452db2230e4cd69363bc ({username='slave'} at 127.0.99.131:42613)
I20240503 01:59:32.947592 3047 tablet_copy_service.cc:161] P fb4fc23dca0c4de69aef5061d3a67b2f: Beginning new tablet copy session on tablet fc4a011058e447499e3d651c73f0b881 from peer ad0b1cf28c9c452db2230e4cd69363bc at {username='slave'} at 127.0.99.131:42613: session id = ad0b1cf28c9c452db2230e4cd69363bc-fc4a011058e447499e3d651c73f0b881
I20240503 01:59:32.948556 3047 tablet_copy_source_session.cc:205] T fc4a011058e447499e3d651c73f0b881 P fb4fc23dca0c4de69aef5061d3a67b2f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:32.949772 4881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc4a011058e447499e3d651c73f0b881. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.951710 4881 tablet_copy_client.cc:783] T fc4a011058e447499e3d651c73f0b881 P ad0b1cf28c9c452db2230e4cd69363bc: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:32.951874 4881 tablet_copy_client.cc:647] T fc4a011058e447499e3d651c73f0b881 P ad0b1cf28c9c452db2230e4cd69363bc: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:32.952654 4906 tablet_bootstrap.cc:492] T 201397ae513b4c40bd2e53197616984e P e4c1a8048e4647b6936035bc2fc22b66: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:32.952700 4912 tablet_bootstrap.cc:492] T 7056f7fb504545d19641ea2c58ebf6be P e4c1a8048e4647b6936035bc2fc22b66: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:32.953045 4906 tablet_bootstrap.cc:492] T 201397ae513b4c40bd2e53197616984e P e4c1a8048e4647b6936035bc2fc22b66: Bootstrap complete.
I20240503 01:59:32.953045 4912 tablet_bootstrap.cc:492] T 7056f7fb504545d19641ea2c58ebf6be P e4c1a8048e4647b6936035bc2fc22b66: Bootstrap complete.
I20240503 01:59:32.953159 4881 tablet_copy_client.cc:523] T fc4a011058e447499e3d651c73f0b881 P ad0b1cf28c9c452db2230e4cd69363bc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:32.953449 4906 ts_tablet_manager.cc:1367] T 201397ae513b4c40bd2e53197616984e P e4c1a8048e4647b6936035bc2fc22b66: Time spent bootstrapping tablet: real 0.041s user 0.029s sys 0.010s
I20240503 01:59:32.953449 4912 ts_tablet_manager.cc:1367] T 7056f7fb504545d19641ea2c58ebf6be P e4c1a8048e4647b6936035bc2fc22b66: Time spent bootstrapping tablet: real 0.041s user 0.027s sys 0.011s
I20240503 01:59:32.956305 4906 raft_consensus.cc:348] T 201397ae513b4c40bd2e53197616984e P e4c1a8048e4647b6936035bc2fc22b66 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4c1a8048e4647b6936035bc2fc22b66" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46423 } attrs { promote: true } }
I20240503 01:59:32.956602 4863 tablet_bootstrap.cc:492] T 1b96b32dada445ccbe9c4eaa0ddee050 P c8a3e298f9d441c09c064a7f5655980b: Bootstrap starting.
I20240503 01:59:32.956666 4906 raft_consensus.cc:729] T 201397ae513b4c40bd2e53197616984e P e4c1a8048e4647b6936035bc2fc22b66 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e4c1a8048e4647b6936035bc2fc22b66, State: Initialized, Role: LEARNER
I20240503 01:59:32.957053 4906 consensus_queue.cc:260] T 201397ae513b4c40bd2e53197616984e P e4c1a8048e4647b6936035bc2fc22b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4c1a8048e4647b6936035bc2fc22b66" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46423 } attrs { promote: true } }
I20240503 01:59:32.958094 4906 ts_tablet_manager.cc:1398] T 201397ae513b4c40bd2e53197616984e P e4c1a8048e4647b6936035bc2fc22b66: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240503 01:59:32.960073 2916 tablet_copy_service.cc:342] P 4d61ec80bf4c475ea86ae2bcb866fd05: Request end of tablet copy session e4c1a8048e4647b6936035bc2fc22b66-201397ae513b4c40bd2e53197616984e received from {username='slave'} at 127.0.99.134:55403
I20240503 01:59:32.960178 2916 tablet_copy_service.cc:433] P 4d61ec80bf4c475ea86ae2bcb866fd05: ending tablet copy session e4c1a8048e4647b6936035bc2fc22b66-201397ae513b4c40bd2e53197616984e on tablet 201397ae513b4c40bd2e53197616984e with peer e4c1a8048e4647b6936035bc2fc22b66
I20240503 01:59:32.961138 4881 tablet_bootstrap.cc:492] T fc4a011058e447499e3d651c73f0b881 P ad0b1cf28c9c452db2230e4cd69363bc: Bootstrap starting.
I20240503 01:59:32.963971 4912 raft_consensus.cc:348] T 7056f7fb504545d19641ea2c58ebf6be P e4c1a8048e4647b6936035bc2fc22b66 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4c1a8048e4647b6936035bc2fc22b66" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46423 } attrs { promote: true } }
I20240503 01:59:32.964113 4912 raft_consensus.cc:729] T 7056f7fb504545d19641ea2c58ebf6be P e4c1a8048e4647b6936035bc2fc22b66 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e4c1a8048e4647b6936035bc2fc22b66, State: Initialized, Role: LEARNER
I20240503 01:59:32.964226 4912 consensus_queue.cc:260] T 7056f7fb504545d19641ea2c58ebf6be P e4c1a8048e4647b6936035bc2fc22b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4c1a8048e4647b6936035bc2fc22b66" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46423 } attrs { promote: true } }
I20240503 01:59:32.964543 4912 ts_tablet_manager.cc:1398] T 7056f7fb504545d19641ea2c58ebf6be P e4c1a8048e4647b6936035bc2fc22b66: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20240503 01:59:32.965626 2916 tablet_copy_service.cc:342] P 4d61ec80bf4c475ea86ae2bcb866fd05: Request end of tablet copy session e4c1a8048e4647b6936035bc2fc22b66-7056f7fb504545d19641ea2c58ebf6be received from {username='slave'} at 127.0.99.134:55403
I20240503 01:59:32.965718 2916 tablet_copy_service.cc:433] P 4d61ec80bf4c475ea86ae2bcb866fd05: ending tablet copy session e4c1a8048e4647b6936035bc2fc22b66-7056f7fb504545d19641ea2c58ebf6be on tablet 7056f7fb504545d19641ea2c58ebf6be with peer e4c1a8048e4647b6936035bc2fc22b66
I20240503 01:59:32.987115 4881 tablet_bootstrap.cc:492] T fc4a011058e447499e3d651c73f0b881 P ad0b1cf28c9c452db2230e4cd69363bc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:32.987476 4881 tablet_bootstrap.cc:492] T fc4a011058e447499e3d651c73f0b881 P ad0b1cf28c9c452db2230e4cd69363bc: Bootstrap complete.
I20240503 01:59:32.987679 4881 ts_tablet_manager.cc:1367] T fc4a011058e447499e3d651c73f0b881 P ad0b1cf28c9c452db2230e4cd69363bc: Time spent bootstrapping tablet: real 0.027s user 0.020s sys 0.004s
I20240503 01:59:32.988355 4881 raft_consensus.cc:348] T fc4a011058e447499e3d651c73f0b881 P ad0b1cf28c9c452db2230e4cd69363bc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "ad0b1cf28c9c452db2230e4cd69363bc" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 44037 } attrs { promote: true } }
I20240503 01:59:32.988520 4881 raft_consensus.cc:729] T fc4a011058e447499e3d651c73f0b881 P ad0b1cf28c9c452db2230e4cd69363bc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ad0b1cf28c9c452db2230e4cd69363bc, State: Initialized, Role: LEARNER
I20240503 01:59:32.988631 4881 consensus_queue.cc:260] T fc4a011058e447499e3d651c73f0b881 P ad0b1cf28c9c452db2230e4cd69363bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "ad0b1cf28c9c452db2230e4cd69363bc" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 44037 } attrs { promote: true } }
I20240503 01:59:32.988780 4863 tablet_bootstrap.cc:492] T 1b96b32dada445ccbe9c4eaa0ddee050 P c8a3e298f9d441c09c064a7f5655980b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:32.988955 4881 ts_tablet_manager.cc:1398] T fc4a011058e447499e3d651c73f0b881 P ad0b1cf28c9c452db2230e4cd69363bc: Time spent starting tablet: real 0.001s user 0.000s sys 0.004s
I20240503 01:59:32.989038 4863 tablet_bootstrap.cc:492] T 1b96b32dada445ccbe9c4eaa0ddee050 P c8a3e298f9d441c09c064a7f5655980b: Bootstrap complete.
I20240503 01:59:32.989210 4863 ts_tablet_manager.cc:1367] T 1b96b32dada445ccbe9c4eaa0ddee050 P c8a3e298f9d441c09c064a7f5655980b: Time spent bootstrapping tablet: real 0.033s user 0.026s sys 0.000s
I20240503 01:59:32.989840 4863 raft_consensus.cc:348] T 1b96b32dada445ccbe9c4eaa0ddee050 P c8a3e298f9d441c09c064a7f5655980b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "c8a3e298f9d441c09c064a7f5655980b" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 41327 } attrs { promote: true } }
I20240503 01:59:32.989955 4863 raft_consensus.cc:729] T 1b96b32dada445ccbe9c4eaa0ddee050 P c8a3e298f9d441c09c064a7f5655980b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c8a3e298f9d441c09c064a7f5655980b, State: Initialized, Role: LEARNER
I20240503 01:59:32.990037 4863 consensus_queue.cc:260] T 1b96b32dada445ccbe9c4eaa0ddee050 P c8a3e298f9d441c09c064a7f5655980b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "c8a3e298f9d441c09c064a7f5655980b" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 41327 } attrs { promote: true } }
I20240503 01:59:32.990271 4863 ts_tablet_manager.cc:1398] T 1b96b32dada445ccbe9c4eaa0ddee050 P c8a3e298f9d441c09c064a7f5655980b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:32.990653 3047 tablet_copy_service.cc:342] P fb4fc23dca0c4de69aef5061d3a67b2f: Request end of tablet copy session ad0b1cf28c9c452db2230e4cd69363bc-fc4a011058e447499e3d651c73f0b881 received from {username='slave'} at 127.0.99.131:42613
I20240503 01:59:32.990741 3047 tablet_copy_service.cc:433] P fb4fc23dca0c4de69aef5061d3a67b2f: ending tablet copy session ad0b1cf28c9c452db2230e4cd69363bc-fc4a011058e447499e3d651c73f0b881 on tablet fc4a011058e447499e3d651c73f0b881 with peer ad0b1cf28c9c452db2230e4cd69363bc
I20240503 01:59:32.991806 3047 tablet_copy_service.cc:342] P fb4fc23dca0c4de69aef5061d3a67b2f: Request end of tablet copy session c8a3e298f9d441c09c064a7f5655980b-1b96b32dada445ccbe9c4eaa0ddee050 received from {username='slave'} at 127.0.99.132:38077
I20240503 01:59:32.991892 3047 tablet_copy_service.cc:433] P fb4fc23dca0c4de69aef5061d3a67b2f: ending tablet copy session c8a3e298f9d441c09c064a7f5655980b-1b96b32dada445ccbe9c4eaa0ddee050 on tablet 1b96b32dada445ccbe9c4eaa0ddee050 with peer c8a3e298f9d441c09c064a7f5655980b
I20240503 01:59:33.252948 4299 raft_consensus.cc:1203] T 7fe4b04830cd42a292f72f27b788c75a P e296c081ab4a414294313ab9883cae84 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 01:59:33.273653 4561 raft_consensus.cc:1203] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 6c8442cedca54ad589e0e050282afee3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:33.291903 4430 raft_consensus.cc:1203] T 7056f7fb504545d19641ea2c58ebf6be P e4c1a8048e4647b6936035bc2fc22b66 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:33.308342 4037 raft_consensus.cc:1203] T 365919d2a8d94985bc031ab16eefe0d8 P ad0b1cf28c9c452db2230e4cd69363bc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:33.310406 4299 raft_consensus.cc:1203] T b2ba35f5c3084e08a42c6f3381aa1cb5 P e296c081ab4a414294313ab9883cae84 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:33.317458 4168 raft_consensus.cc:1203] T 9bfd209000fd468385a66c8fa5b2edc8 P c8a3e298f9d441c09c064a7f5655980b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:33.323544 4561 raft_consensus.cc:1203] T c3c2ff70515f49cf9f51b8db5e130cba P 6c8442cedca54ad589e0e050282afee3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:33.407821 4037 raft_consensus.cc:1203] T fc4a011058e447499e3d651c73f0b881 P ad0b1cf28c9c452db2230e4cd69363bc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:33.435154 4430 raft_consensus.cc:1203] T 201397ae513b4c40bd2e53197616984e P e4c1a8048e4647b6936035bc2fc22b66 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:33.474648 4168 raft_consensus.cc:1203] T 1b96b32dada445ccbe9c4eaa0ddee050 P c8a3e298f9d441c09c064a7f5655980b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:33.638505 4762 raft_consensus.cc:1050] T 7fe4b04830cd42a292f72f27b788c75a P 4d61ec80bf4c475ea86ae2bcb866fd05: attempting to promote NON_VOTER e296c081ab4a414294313ab9883cae84 to VOTER
I20240503 01:59:33.639036 4762 consensus_queue.cc:237] T 7fe4b04830cd42a292f72f27b788c75a P 4d61ec80bf4c475ea86ae2bcb866fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e296c081ab4a414294313ab9883cae84" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41953 } attrs { promote: false } }
I20240503 01:59:33.640426 4299 raft_consensus.cc:1261] T 7fe4b04830cd42a292f72f27b788c75a P e296c081ab4a414294313ab9883cae84 [term 1 LEARNER]: Refusing update from remote peer 4d61ec80bf4c475ea86ae2bcb866fd05: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 01:59:33.641000 4762 consensus_queue.cc:1035] T 7fe4b04830cd42a292f72f27b788c75a P 4d61ec80bf4c475ea86ae2bcb866fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e296c081ab4a414294313ab9883cae84" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41953 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 01:59:33.643846 5002 raft_consensus.cc:2932] T 7fe4b04830cd42a292f72f27b788c75a P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e296c081ab4a414294313ab9883cae84 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e296c081ab4a414294313ab9883cae84" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41953 } attrs { promote: false } } }
I20240503 01:59:33.644479 4299 raft_consensus.cc:2932] T 7fe4b04830cd42a292f72f27b788c75a P e296c081ab4a414294313ab9883cae84 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e296c081ab4a414294313ab9883cae84 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e296c081ab4a414294313ab9883cae84" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41953 } attrs { promote: false } } }
I20240503 01:59:33.646354 2749 catalog_manager.cc:5484] T 7fe4b04830cd42a292f72f27b788c75a P 4d61ec80bf4c475ea86ae2bcb866fd05 reported cstate change: config changed from index 2 to 3, e296c081ab4a414294313ab9883cae84 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e296c081ab4a414294313ab9883cae84" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41953 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:33.697256 5001 raft_consensus.cc:1050] T 365919d2a8d94985bc031ab16eefe0d8 P 4d61ec80bf4c475ea86ae2bcb866fd05: attempting to promote NON_VOTER ad0b1cf28c9c452db2230e4cd69363bc to VOTER
I20240503 01:59:33.697610 5001 consensus_queue.cc:237] T 365919d2a8d94985bc031ab16eefe0d8 P 4d61ec80bf4c475ea86ae2bcb866fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "ad0b1cf28c9c452db2230e4cd69363bc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44037 } attrs { promote: false } }
I20240503 01:59:33.699193 4037 raft_consensus.cc:1261] T 365919d2a8d94985bc031ab16eefe0d8 P ad0b1cf28c9c452db2230e4cd69363bc [term 1 LEARNER]: Refusing update from remote peer 4d61ec80bf4c475ea86ae2bcb866fd05: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:33.699311 5002 raft_consensus.cc:1050] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 4d61ec80bf4c475ea86ae2bcb866fd05: attempting to promote NON_VOTER 6c8442cedca54ad589e0e050282afee3 to VOTER
I20240503 01:59:33.699602 5002 consensus_queue.cc:237] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 4d61ec80bf4c475ea86ae2bcb866fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "6c8442cedca54ad589e0e050282afee3" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43573 } attrs { promote: false } }
I20240503 01:59:33.699649 5001 consensus_queue.cc:1035] T 365919d2a8d94985bc031ab16eefe0d8 P 4d61ec80bf4c475ea86ae2bcb866fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad0b1cf28c9c452db2230e4cd69363bc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:33.700798 4561 raft_consensus.cc:1261] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 6c8442cedca54ad589e0e050282afee3 [term 1 LEARNER]: Refusing update from remote peer 4d61ec80bf4c475ea86ae2bcb866fd05: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:33.701246 5002 consensus_queue.cc:1035] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 4d61ec80bf4c475ea86ae2bcb866fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c8442cedca54ad589e0e050282afee3" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43573 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:33.702363 5001 raft_consensus.cc:2932] T 365919d2a8d94985bc031ab16eefe0d8 P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ad0b1cf28c9c452db2230e4cd69363bc (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "ad0b1cf28c9c452db2230e4cd69363bc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44037 } attrs { promote: false } } }
I20240503 01:59:33.702940 4037 raft_consensus.cc:2932] T 365919d2a8d94985bc031ab16eefe0d8 P ad0b1cf28c9c452db2230e4cd69363bc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ad0b1cf28c9c452db2230e4cd69363bc (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "ad0b1cf28c9c452db2230e4cd69363bc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44037 } attrs { promote: false } } }
I20240503 01:59:33.704528 5002 raft_consensus.cc:2932] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6c8442cedca54ad589e0e050282afee3 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "6c8442cedca54ad589e0e050282afee3" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43573 } attrs { promote: false } } }
I20240503 01:59:33.705020 2755 catalog_manager.cc:5484] T 365919d2a8d94985bc031ab16eefe0d8 P 4d61ec80bf4c475ea86ae2bcb866fd05 reported cstate change: config changed from index 4 to 5, ad0b1cf28c9c452db2230e4cd69363bc (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad0b1cf28c9c452db2230e4cd69363bc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44037 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:33.706377 4561 raft_consensus.cc:2932] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 6c8442cedca54ad589e0e050282afee3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6c8442cedca54ad589e0e050282afee3 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "6c8442cedca54ad589e0e050282afee3" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43573 } attrs { promote: false } } }
I20240503 01:59:33.709040 5001 raft_consensus.cc:1050] T 7056f7fb504545d19641ea2c58ebf6be P 4d61ec80bf4c475ea86ae2bcb866fd05: attempting to promote NON_VOTER e4c1a8048e4647b6936035bc2fc22b66 to VOTER
I20240503 01:59:33.708976 2749 catalog_manager.cc:5484] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 6c8442cedca54ad589e0e050282afee3 reported cstate change: config changed from index 4 to 5, 6c8442cedca54ad589e0e050282afee3 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "6c8442cedca54ad589e0e050282afee3" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43573 } attrs { promote: false } } }
I20240503 01:59:33.709331 5001 consensus_queue.cc:237] T 7056f7fb504545d19641ea2c58ebf6be P 4d61ec80bf4c475ea86ae2bcb866fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4c1a8048e4647b6936035bc2fc22b66" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46423 } attrs { promote: false } }
I20240503 01:59:33.710763 4430 raft_consensus.cc:1261] T 7056f7fb504545d19641ea2c58ebf6be P e4c1a8048e4647b6936035bc2fc22b66 [term 1 LEARNER]: Refusing update from remote peer 4d61ec80bf4c475ea86ae2bcb866fd05: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:33.711382 5001 consensus_queue.cc:1035] T 7056f7fb504545d19641ea2c58ebf6be P 4d61ec80bf4c475ea86ae2bcb866fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c1a8048e4647b6936035bc2fc22b66" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46423 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:33.714092 5002 raft_consensus.cc:2932] T 7056f7fb504545d19641ea2c58ebf6be P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e4c1a8048e4647b6936035bc2fc22b66 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4c1a8048e4647b6936035bc2fc22b66" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46423 } attrs { promote: false } } }
I20240503 01:59:33.714637 4430 raft_consensus.cc:2932] T 7056f7fb504545d19641ea2c58ebf6be P e4c1a8048e4647b6936035bc2fc22b66 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e4c1a8048e4647b6936035bc2fc22b66 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4c1a8048e4647b6936035bc2fc22b66" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46423 } attrs { promote: false } } }
I20240503 01:59:33.717065 2755 catalog_manager.cc:5484] T 7056f7fb504545d19641ea2c58ebf6be P e4c1a8048e4647b6936035bc2fc22b66 reported cstate change: config changed from index 4 to 5, e4c1a8048e4647b6936035bc2fc22b66 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4c1a8048e4647b6936035bc2fc22b66" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46423 } attrs { promote: false } } }
I20240503 01:59:33.784574 2897 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "365919d2a8d94985bc031ab16eefe0d8"
dest_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51332
I20240503 01:59:33.784762 2897 raft_consensus.cc:595] T 365919d2a8d94985bc031ab16eefe0d8 P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:33.802629 2897 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7fe4b04830cd42a292f72f27b788c75a"
dest_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51346
I20240503 01:59:33.802778 2897 raft_consensus.cc:595] T 7fe4b04830cd42a292f72f27b788c75a P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:33.810746 5033 raft_consensus.cc:1050] T b2ba35f5c3084e08a42c6f3381aa1cb5 P fb4fc23dca0c4de69aef5061d3a67b2f: attempting to promote NON_VOTER e296c081ab4a414294313ab9883cae84 to VOTER
I20240503 01:59:33.811329 5033 consensus_queue.cc:237] T b2ba35f5c3084e08a42c6f3381aa1cb5 P fb4fc23dca0c4de69aef5061d3a67b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "e296c081ab4a414294313ab9883cae84" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41953 } attrs { promote: false } }
I20240503 01:59:33.812546 4299 raft_consensus.cc:1261] T b2ba35f5c3084e08a42c6f3381aa1cb5 P e296c081ab4a414294313ab9883cae84 [term 1 LEARNER]: Refusing update from remote peer fb4fc23dca0c4de69aef5061d3a67b2f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:33.813017 5034 consensus_queue.cc:1035] T b2ba35f5c3084e08a42c6f3381aa1cb5 P fb4fc23dca0c4de69aef5061d3a67b2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e296c081ab4a414294313ab9883cae84" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41953 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:33.815219 2897 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0cbba6e3ce5b4de5b5ec9164e50f784a"
dest_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51374
I20240503 01:59:33.815371 2897 raft_consensus.cc:595] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:33.815471 5033 raft_consensus.cc:2932] T b2ba35f5c3084e08a42c6f3381aa1cb5 P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e296c081ab4a414294313ab9883cae84 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "e296c081ab4a414294313ab9883cae84" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41953 } attrs { promote: false } } }
I20240503 01:59:33.815960 4299 raft_consensus.cc:2932] T b2ba35f5c3084e08a42c6f3381aa1cb5 P e296c081ab4a414294313ab9883cae84 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e296c081ab4a414294313ab9883cae84 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "e296c081ab4a414294313ab9883cae84" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41953 } attrs { promote: false } } }
I20240503 01:59:33.818387 2755 catalog_manager.cc:5484] T b2ba35f5c3084e08a42c6f3381aa1cb5 P e296c081ab4a414294313ab9883cae84 reported cstate change: config changed from index 4 to 5, e296c081ab4a414294313ab9883cae84 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "e296c081ab4a414294313ab9883cae84" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41953 } attrs { promote: false } } }
I20240503 01:59:33.839239 2897 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7056f7fb504545d19641ea2c58ebf6be"
dest_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51390
I20240503 01:59:33.839382 2897 raft_consensus.cc:595] T 7056f7fb504545d19641ea2c58ebf6be P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:33.865937 5002 raft_consensus.cc:1050] T 201397ae513b4c40bd2e53197616984e P 4d61ec80bf4c475ea86ae2bcb866fd05: attempting to promote NON_VOTER e4c1a8048e4647b6936035bc2fc22b66 to VOTER
I20240503 01:59:33.866345 5002 consensus_queue.cc:237] T 201397ae513b4c40bd2e53197616984e P 4d61ec80bf4c475ea86ae2bcb866fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4c1a8048e4647b6936035bc2fc22b66" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46423 } attrs { promote: false } }
I20240503 01:59:33.867715 4430 raft_consensus.cc:1261] T 201397ae513b4c40bd2e53197616984e P e4c1a8048e4647b6936035bc2fc22b66 [term 1 LEARNER]: Refusing update from remote peer 4d61ec80bf4c475ea86ae2bcb866fd05: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:33.868148 5002 consensus_queue.cc:1035] T 201397ae513b4c40bd2e53197616984e P 4d61ec80bf4c475ea86ae2bcb866fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c1a8048e4647b6936035bc2fc22b66" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46423 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:33.870174 5001 raft_consensus.cc:2932] T 201397ae513b4c40bd2e53197616984e P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e4c1a8048e4647b6936035bc2fc22b66 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4c1a8048e4647b6936035bc2fc22b66" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46423 } attrs { promote: false } } }
I20240503 01:59:33.870683 4430 raft_consensus.cc:2932] T 201397ae513b4c40bd2e53197616984e P e4c1a8048e4647b6936035bc2fc22b66 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e4c1a8048e4647b6936035bc2fc22b66 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4c1a8048e4647b6936035bc2fc22b66" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46423 } attrs { promote: false } } }
I20240503 01:59:33.872808 2749 catalog_manager.cc:5484] T 201397ae513b4c40bd2e53197616984e P 4d61ec80bf4c475ea86ae2bcb866fd05 reported cstate change: config changed from index 4 to 5, e4c1a8048e4647b6936035bc2fc22b66 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4c1a8048e4647b6936035bc2fc22b66" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46423 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:33.891068 5034 raft_consensus.cc:1050] T 1b96b32dada445ccbe9c4eaa0ddee050 P fb4fc23dca0c4de69aef5061d3a67b2f: attempting to promote NON_VOTER c8a3e298f9d441c09c064a7f5655980b to VOTER
I20240503 01:59:33.891423 5034 consensus_queue.cc:237] T 1b96b32dada445ccbe9c4eaa0ddee050 P fb4fc23dca0c4de69aef5061d3a67b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "c8a3e298f9d441c09c064a7f5655980b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41327 } attrs { promote: false } }
I20240503 01:59:33.892740 4168 raft_consensus.cc:1261] T 1b96b32dada445ccbe9c4eaa0ddee050 P c8a3e298f9d441c09c064a7f5655980b [term 1 LEARNER]: Refusing update from remote peer fb4fc23dca0c4de69aef5061d3a67b2f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:33.893251 5033 consensus_queue.cc:1035] T 1b96b32dada445ccbe9c4eaa0ddee050 P fb4fc23dca0c4de69aef5061d3a67b2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8a3e298f9d441c09c064a7f5655980b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41327 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:33.894258 5033 raft_consensus.cc:1050] T c3c2ff70515f49cf9f51b8db5e130cba P fb4fc23dca0c4de69aef5061d3a67b2f: attempting to promote NON_VOTER 6c8442cedca54ad589e0e050282afee3 to VOTER
I20240503 01:59:33.894543 5033 consensus_queue.cc:237] T c3c2ff70515f49cf9f51b8db5e130cba P fb4fc23dca0c4de69aef5061d3a67b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "6c8442cedca54ad589e0e050282afee3" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43573 } attrs { promote: false } }
I20240503 01:59:33.895606 4561 raft_consensus.cc:1261] T c3c2ff70515f49cf9f51b8db5e130cba P 6c8442cedca54ad589e0e050282afee3 [term 1 LEARNER]: Refusing update from remote peer fb4fc23dca0c4de69aef5061d3a67b2f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:33.896014 5033 consensus_queue.cc:1035] T c3c2ff70515f49cf9f51b8db5e130cba P fb4fc23dca0c4de69aef5061d3a67b2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c8442cedca54ad589e0e050282afee3" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43573 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:33.896196 4755 raft_consensus.cc:2932] T 1b96b32dada445ccbe9c4eaa0ddee050 P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c8a3e298f9d441c09c064a7f5655980b (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "c8a3e298f9d441c09c064a7f5655980b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41327 } attrs { promote: false } } }
I20240503 01:59:33.896857 4168 raft_consensus.cc:2932] T 1b96b32dada445ccbe9c4eaa0ddee050 P c8a3e298f9d441c09c064a7f5655980b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c8a3e298f9d441c09c064a7f5655980b (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "c8a3e298f9d441c09c064a7f5655980b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41327 } attrs { promote: false } } }
I20240503 01:59:33.897923 5034 raft_consensus.cc:2932] T c3c2ff70515f49cf9f51b8db5e130cba P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6c8442cedca54ad589e0e050282afee3 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "6c8442cedca54ad589e0e050282afee3" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43573 } attrs { promote: false } } }
I20240503 01:59:33.899263 2755 catalog_manager.cc:5484] T 1b96b32dada445ccbe9c4eaa0ddee050 P fb4fc23dca0c4de69aef5061d3a67b2f reported cstate change: config changed from index 4 to 5, c8a3e298f9d441c09c064a7f5655980b (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8a3e298f9d441c09c064a7f5655980b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41327 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:33.899682 4561 raft_consensus.cc:2932] T c3c2ff70515f49cf9f51b8db5e130cba P 6c8442cedca54ad589e0e050282afee3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6c8442cedca54ad589e0e050282afee3 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "6c8442cedca54ad589e0e050282afee3" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43573 } attrs { promote: false } } }
I20240503 01:59:33.901618 2748 catalog_manager.cc:5484] T c3c2ff70515f49cf9f51b8db5e130cba P 6c8442cedca54ad589e0e050282afee3 reported cstate change: config changed from index 4 to 5, 6c8442cedca54ad589e0e050282afee3 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "6c8442cedca54ad589e0e050282afee3" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43573 } attrs { promote: false } } }
I20240503 01:59:33.913029 5034 raft_consensus.cc:1050] T 9bfd209000fd468385a66c8fa5b2edc8 P fb4fc23dca0c4de69aef5061d3a67b2f: attempting to promote NON_VOTER c8a3e298f9d441c09c064a7f5655980b to VOTER
I20240503 01:59:33.913290 5033 raft_consensus.cc:1050] T fc4a011058e447499e3d651c73f0b881 P fb4fc23dca0c4de69aef5061d3a67b2f: attempting to promote NON_VOTER ad0b1cf28c9c452db2230e4cd69363bc to VOTER
I20240503 01:59:33.913380 5034 consensus_queue.cc:237] T 9bfd209000fd468385a66c8fa5b2edc8 P fb4fc23dca0c4de69aef5061d3a67b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "c8a3e298f9d441c09c064a7f5655980b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41327 } attrs { promote: false } }
I20240503 01:59:33.913563 5033 consensus_queue.cc:237] T fc4a011058e447499e3d651c73f0b881 P fb4fc23dca0c4de69aef5061d3a67b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "ad0b1cf28c9c452db2230e4cd69363bc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44037 } attrs { promote: false } }
I20240503 01:59:33.914458 4168 raft_consensus.cc:1261] T 9bfd209000fd468385a66c8fa5b2edc8 P c8a3e298f9d441c09c064a7f5655980b [term 1 LEARNER]: Refusing update from remote peer fb4fc23dca0c4de69aef5061d3a67b2f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:33.914628 4037 raft_consensus.cc:1261] T fc4a011058e447499e3d651c73f0b881 P ad0b1cf28c9c452db2230e4cd69363bc [term 1 LEARNER]: Refusing update from remote peer fb4fc23dca0c4de69aef5061d3a67b2f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:33.914794 4755 consensus_queue.cc:1035] T 9bfd209000fd468385a66c8fa5b2edc8 P fb4fc23dca0c4de69aef5061d3a67b2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8a3e298f9d441c09c064a7f5655980b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41327 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:33.914978 5033 consensus_queue.cc:1035] T fc4a011058e447499e3d651c73f0b881 P fb4fc23dca0c4de69aef5061d3a67b2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad0b1cf28c9c452db2230e4cd69363bc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:33.916586 5034 raft_consensus.cc:2932] T 9bfd209000fd468385a66c8fa5b2edc8 P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c8a3e298f9d441c09c064a7f5655980b (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "c8a3e298f9d441c09c064a7f5655980b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41327 } attrs { promote: false } } }
I20240503 01:59:33.916676 5033 raft_consensus.cc:2932] T fc4a011058e447499e3d651c73f0b881 P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ad0b1cf28c9c452db2230e4cd69363bc (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "ad0b1cf28c9c452db2230e4cd69363bc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44037 } attrs { promote: false } } }
I20240503 01:59:33.917071 4168 raft_consensus.cc:2932] T 9bfd209000fd468385a66c8fa5b2edc8 P c8a3e298f9d441c09c064a7f5655980b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c8a3e298f9d441c09c064a7f5655980b (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "c8a3e298f9d441c09c064a7f5655980b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41327 } attrs { promote: false } } }
I20240503 01:59:33.917075 4037 raft_consensus.cc:2932] T fc4a011058e447499e3d651c73f0b881 P ad0b1cf28c9c452db2230e4cd69363bc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ad0b1cf28c9c452db2230e4cd69363bc (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "ad0b1cf28c9c452db2230e4cd69363bc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44037 } attrs { promote: false } } }
I20240503 01:59:33.919406 2749 catalog_manager.cc:5484] T fc4a011058e447499e3d651c73f0b881 P ad0b1cf28c9c452db2230e4cd69363bc reported cstate change: config changed from index 4 to 5, ad0b1cf28c9c452db2230e4cd69363bc (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "ad0b1cf28c9c452db2230e4cd69363bc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44037 } attrs { promote: false } } }
I20240503 01:59:33.919548 2755 catalog_manager.cc:5484] T 9bfd209000fd468385a66c8fa5b2edc8 P c8a3e298f9d441c09c064a7f5655980b reported cstate change: config changed from index 4 to 5, c8a3e298f9d441c09c064a7f5655980b (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "c8a3e298f9d441c09c064a7f5655980b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41327 } attrs { promote: false } } }
I20240503 01:59:34.051337 3028 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1b96b32dada445ccbe9c4eaa0ddee050"
dest_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59576
I20240503 01:59:34.051518 3028 raft_consensus.cc:595] T 1b96b32dada445ccbe9c4eaa0ddee050 P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:34.062000 3028 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fc4a011058e447499e3d651c73f0b881"
dest_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59600
I20240503 01:59:34.062161 3028 raft_consensus.cc:595] T fc4a011058e447499e3d651c73f0b881 P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:34.072603 2897 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "201397ae513b4c40bd2e53197616984e"
dest_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51408
I20240503 01:59:34.072767 2897 raft_consensus.cc:595] T 201397ae513b4c40bd2e53197616984e P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:34.084316 2897 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "365919d2a8d94985bc031ab16eefe0d8"
dest_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51434
I20240503 01:59:34.084523 2897 raft_consensus.cc:595] T 365919d2a8d94985bc031ab16eefe0d8 P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:34.085413 5002 raft_consensus.cc:979] T 7fe4b04830cd42a292f72f27b788c75a P 4d61ec80bf4c475ea86ae2bcb866fd05: : Instructing follower e296c081ab4a414294313ab9883cae84 to start an election
I20240503 01:59:34.085517 5002 raft_consensus.cc:1067] T 7fe4b04830cd42a292f72f27b788c75a P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 LEADER]: Signalling peer e296c081ab4a414294313ab9883cae84 to start an election
I20240503 01:59:34.086114 4299 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7fe4b04830cd42a292f72f27b788c75a"
dest_uuid: "e296c081ab4a414294313ab9883cae84"
from {username='slave'} at 127.0.99.129:59807
I20240503 01:59:34.086309 4299 raft_consensus.cc:482] T 7fe4b04830cd42a292f72f27b788c75a P e296c081ab4a414294313ab9883cae84 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 01:59:34.086419 4299 raft_consensus.cc:3037] T 7fe4b04830cd42a292f72f27b788c75a P e296c081ab4a414294313ab9883cae84 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:34.087352 4299 raft_consensus.cc:504] T 7fe4b04830cd42a292f72f27b788c75a P e296c081ab4a414294313ab9883cae84 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e296c081ab4a414294313ab9883cae84" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41953 } attrs { promote: false } }
I20240503 01:59:34.088411 4299 leader_election.cc:288] T 7fe4b04830cd42a292f72f27b788c75a P e296c081ab4a414294313ab9883cae84 [CANDIDATE]: Term 2 election: Requested vote from peers 4d61ec80bf4c475ea86ae2bcb866fd05 (127.0.99.129:38011)
I20240503 01:59:34.089007 2897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fe4b04830cd42a292f72f27b788c75a" candidate_uuid: "e296c081ab4a414294313ab9883cae84" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05"
I20240503 01:59:34.089171 2897 raft_consensus.cc:3032] T 7fe4b04830cd42a292f72f27b788c75a P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 01:59:34.089217 2897 raft_consensus.cc:729] T 7fe4b04830cd42a292f72f27b788c75a P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4d61ec80bf4c475ea86ae2bcb866fd05, State: Running, Role: LEADER
I20240503 01:59:34.089367 2897 consensus_queue.cc:260] T 7fe4b04830cd42a292f72f27b788c75a P 4d61ec80bf4c475ea86ae2bcb866fd05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e296c081ab4a414294313ab9883cae84" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41953 } attrs { promote: false } }
I20240503 01:59:34.089491 2897 raft_consensus.cc:3037] T 7fe4b04830cd42a292f72f27b788c75a P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:34.090363 2897 raft_consensus.cc:2454] T 7fe4b04830cd42a292f72f27b788c75a P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e296c081ab4a414294313ab9883cae84 in term 2.
I20240503 01:59:34.090765 4232 leader_election.cc:302] T 7fe4b04830cd42a292f72f27b788c75a P e296c081ab4a414294313ab9883cae84 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4d61ec80bf4c475ea86ae2bcb866fd05, e296c081ab4a414294313ab9883cae84; no voters:
I20240503 01:59:34.090991 4896 raft_consensus.cc:2781] T 7fe4b04830cd42a292f72f27b788c75a P e296c081ab4a414294313ab9883cae84 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 01:59:34.091177 4896 raft_consensus.cc:686] T 7fe4b04830cd42a292f72f27b788c75a P e296c081ab4a414294313ab9883cae84 [term 2 LEADER]: Becoming Leader. State: Replica: e296c081ab4a414294313ab9883cae84, State: Running, Role: LEADER
I20240503 01:59:34.091481 4896 consensus_queue.cc:237] T 7fe4b04830cd42a292f72f27b788c75a P e296c081ab4a414294313ab9883cae84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e296c081ab4a414294313ab9883cae84" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41953 } attrs { promote: false } }
I20240503 01:59:34.093861 2748 catalog_manager.cc:5484] T 7fe4b04830cd42a292f72f27b788c75a P e296c081ab4a414294313ab9883cae84 reported cstate change: term changed from 1 to 2, leader changed from 4d61ec80bf4c475ea86ae2bcb866fd05 (127.0.99.129) to e296c081ab4a414294313ab9883cae84 (127.0.99.133). New cstate: current_term: 2 leader_uuid: "e296c081ab4a414294313ab9883cae84" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e296c081ab4a414294313ab9883cae84" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41953 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:34.098013 3028 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b2ba35f5c3084e08a42c6f3381aa1cb5"
dest_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59614
I20240503 01:59:34.098153 3028 raft_consensus.cc:595] T b2ba35f5c3084e08a42c6f3381aa1cb5 P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 LEADER]: Received request to transfer leadership
W20240503 01:59:34.103463 2739 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 7fe4b04830cd42a292f72f27b788c75a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader e296c081ab4a414294313ab9883cae84 (127.0.99.133:41953) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20240503 01:59:34.115569 5001 raft_consensus.cc:979] T 7056f7fb504545d19641ea2c58ebf6be P 4d61ec80bf4c475ea86ae2bcb866fd05: : Instructing follower e4c1a8048e4647b6936035bc2fc22b66 to start an election
I20240503 01:59:34.115706 4762 raft_consensus.cc:1067] T 7056f7fb504545d19641ea2c58ebf6be P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 LEADER]: Signalling peer e4c1a8048e4647b6936035bc2fc22b66 to start an election
I20240503 01:59:34.116312 4430 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7056f7fb504545d19641ea2c58ebf6be"
dest_uuid: "e4c1a8048e4647b6936035bc2fc22b66"
from {username='slave'} at 127.0.99.129:43121
I20240503 01:59:34.116580 4430 raft_consensus.cc:482] T 7056f7fb504545d19641ea2c58ebf6be P e4c1a8048e4647b6936035bc2fc22b66 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 01:59:34.116696 4430 raft_consensus.cc:3037] T 7056f7fb504545d19641ea2c58ebf6be P e4c1a8048e4647b6936035bc2fc22b66 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:34.117794 4430 raft_consensus.cc:504] T 7056f7fb504545d19641ea2c58ebf6be P e4c1a8048e4647b6936035bc2fc22b66 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4c1a8048e4647b6936035bc2fc22b66" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46423 } attrs { promote: false } }
I20240503 01:59:34.118924 4430 leader_election.cc:288] T 7056f7fb504545d19641ea2c58ebf6be P e4c1a8048e4647b6936035bc2fc22b66 [CANDIDATE]: Term 2 election: Requested vote from peers 4d61ec80bf4c475ea86ae2bcb866fd05 (127.0.99.129:38011)
I20240503 01:59:34.119333 2897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7056f7fb504545d19641ea2c58ebf6be" candidate_uuid: "e4c1a8048e4647b6936035bc2fc22b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05"
I20240503 01:59:34.119462 2897 raft_consensus.cc:3032] T 7056f7fb504545d19641ea2c58ebf6be P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 01:59:34.119510 2897 raft_consensus.cc:729] T 7056f7fb504545d19641ea2c58ebf6be P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4d61ec80bf4c475ea86ae2bcb866fd05, State: Running, Role: LEADER
I20240503 01:59:34.119671 2897 consensus_queue.cc:260] T 7056f7fb504545d19641ea2c58ebf6be P 4d61ec80bf4c475ea86ae2bcb866fd05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4c1a8048e4647b6936035bc2fc22b66" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46423 } attrs { promote: false } }
I20240503 01:59:34.119846 2897 raft_consensus.cc:3037] T 7056f7fb504545d19641ea2c58ebf6be P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:34.120762 2897 raft_consensus.cc:2454] T 7056f7fb504545d19641ea2c58ebf6be P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4c1a8048e4647b6936035bc2fc22b66 in term 2.
I20240503 01:59:34.121201 4363 leader_election.cc:302] T 7056f7fb504545d19641ea2c58ebf6be P e4c1a8048e4647b6936035bc2fc22b66 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4d61ec80bf4c475ea86ae2bcb866fd05, e4c1a8048e4647b6936035bc2fc22b66; no voters:
I20240503 01:59:34.121599 4938 raft_consensus.cc:2781] T 7056f7fb504545d19641ea2c58ebf6be P e4c1a8048e4647b6936035bc2fc22b66 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 01:59:34.121801 4938 raft_consensus.cc:686] T 7056f7fb504545d19641ea2c58ebf6be P e4c1a8048e4647b6936035bc2fc22b66 [term 2 LEADER]: Becoming Leader. State: Replica: e4c1a8048e4647b6936035bc2fc22b66, State: Running, Role: LEADER
I20240503 01:59:34.122407 4938 consensus_queue.cc:237] T 7056f7fb504545d19641ea2c58ebf6be P e4c1a8048e4647b6936035bc2fc22b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4c1a8048e4647b6936035bc2fc22b66" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46423 } attrs { promote: false } }
I20240503 01:59:34.123505 2897 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0cbba6e3ce5b4de5b5ec9164e50f784a"
dest_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51458
I20240503 01:59:34.123618 2897 raft_consensus.cc:595] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:34.125149 2748 catalog_manager.cc:5484] T 7056f7fb504545d19641ea2c58ebf6be P e4c1a8048e4647b6936035bc2fc22b66 reported cstate change: term changed from 1 to 2, leader changed from 4d61ec80bf4c475ea86ae2bcb866fd05 (127.0.99.129) to e4c1a8048e4647b6936035bc2fc22b66 (127.0.99.134). New cstate: current_term: 2 leader_uuid: "e4c1a8048e4647b6936035bc2fc22b66" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4c1a8048e4647b6936035bc2fc22b66" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46423 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:34.137985 3028 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9bfd209000fd468385a66c8fa5b2edc8"
dest_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59630
I20240503 01:59:34.138142 3028 raft_consensus.cc:595] T 9bfd209000fd468385a66c8fa5b2edc8 P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:34.150681 3028 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c3c2ff70515f49cf9f51b8db5e130cba"
dest_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59654
I20240503 01:59:34.150831 3028 raft_consensus.cc:595] T c3c2ff70515f49cf9f51b8db5e130cba P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:34.165220 4762 raft_consensus.cc:979] T 365919d2a8d94985bc031ab16eefe0d8 P 4d61ec80bf4c475ea86ae2bcb866fd05: : Instructing follower ad0b1cf28c9c452db2230e4cd69363bc to start an election
I20240503 01:59:34.165345 4762 raft_consensus.cc:1067] T 365919d2a8d94985bc031ab16eefe0d8 P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 LEADER]: Signalling peer ad0b1cf28c9c452db2230e4cd69363bc to start an election
I20240503 01:59:34.165887 4037 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "365919d2a8d94985bc031ab16eefe0d8"
dest_uuid: "ad0b1cf28c9c452db2230e4cd69363bc"
from {username='slave'} at 127.0.99.129:37743
I20240503 01:59:34.166113 4037 raft_consensus.cc:482] T 365919d2a8d94985bc031ab16eefe0d8 P ad0b1cf28c9c452db2230e4cd69363bc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 01:59:34.166219 4037 raft_consensus.cc:3037] T 365919d2a8d94985bc031ab16eefe0d8 P ad0b1cf28c9c452db2230e4cd69363bc [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:34.167217 4037 raft_consensus.cc:504] T 365919d2a8d94985bc031ab16eefe0d8 P ad0b1cf28c9c452db2230e4cd69363bc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "ad0b1cf28c9c452db2230e4cd69363bc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44037 } attrs { promote: false } }
I20240503 01:59:34.168247 4037 leader_election.cc:288] T 365919d2a8d94985bc031ab16eefe0d8 P ad0b1cf28c9c452db2230e4cd69363bc [CANDIDATE]: Term 2 election: Requested vote from peers 4d61ec80bf4c475ea86ae2bcb866fd05 (127.0.99.129:38011)
I20240503 01:59:34.168831 2897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "365919d2a8d94985bc031ab16eefe0d8" candidate_uuid: "ad0b1cf28c9c452db2230e4cd69363bc" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05"
I20240503 01:59:34.168972 2897 raft_consensus.cc:3032] T 365919d2a8d94985bc031ab16eefe0d8 P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 01:59:34.169023 2897 raft_consensus.cc:729] T 365919d2a8d94985bc031ab16eefe0d8 P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4d61ec80bf4c475ea86ae2bcb866fd05, State: Running, Role: LEADER
I20240503 01:59:34.169180 2897 consensus_queue.cc:260] T 365919d2a8d94985bc031ab16eefe0d8 P 4d61ec80bf4c475ea86ae2bcb866fd05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "ad0b1cf28c9c452db2230e4cd69363bc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44037 } attrs { promote: false } }
I20240503 01:59:34.169319 2897 raft_consensus.cc:3037] T 365919d2a8d94985bc031ab16eefe0d8 P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:34.170323 2897 raft_consensus.cc:2454] T 365919d2a8d94985bc031ab16eefe0d8 P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad0b1cf28c9c452db2230e4cd69363bc in term 2.
I20240503 01:59:34.170775 3970 leader_election.cc:302] T 365919d2a8d94985bc031ab16eefe0d8 P ad0b1cf28c9c452db2230e4cd69363bc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4d61ec80bf4c475ea86ae2bcb866fd05, ad0b1cf28c9c452db2230e4cd69363bc; no voters:
I20240503 01:59:34.171034 4904 raft_consensus.cc:2781] T 365919d2a8d94985bc031ab16eefe0d8 P ad0b1cf28c9c452db2230e4cd69363bc [term 2 FOLLOWER]: Leader election won for term 2
I20240503 01:59:34.171241 4904 raft_consensus.cc:686] T 365919d2a8d94985bc031ab16eefe0d8 P ad0b1cf28c9c452db2230e4cd69363bc [term 2 LEADER]: Becoming Leader. State: Replica: ad0b1cf28c9c452db2230e4cd69363bc, State: Running, Role: LEADER
I20240503 01:59:34.171555 4904 consensus_queue.cc:237] T 365919d2a8d94985bc031ab16eefe0d8 P ad0b1cf28c9c452db2230e4cd69363bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "ad0b1cf28c9c452db2230e4cd69363bc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44037 } attrs { promote: false } }
I20240503 01:59:34.174399 2748 catalog_manager.cc:5484] T 365919d2a8d94985bc031ab16eefe0d8 P ad0b1cf28c9c452db2230e4cd69363bc reported cstate change: term changed from 1 to 2, leader changed from 4d61ec80bf4c475ea86ae2bcb866fd05 (127.0.99.129) to ad0b1cf28c9c452db2230e4cd69363bc (127.0.99.131). New cstate: current_term: 2 leader_uuid: "ad0b1cf28c9c452db2230e4cd69363bc" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad0b1cf28c9c452db2230e4cd69363bc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44037 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:34.256896 5001 raft_consensus.cc:979] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 4d61ec80bf4c475ea86ae2bcb866fd05: : Instructing follower 6c8442cedca54ad589e0e050282afee3 to start an election
I20240503 01:59:34.257017 5001 raft_consensus.cc:1067] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 LEADER]: Signalling peer 6c8442cedca54ad589e0e050282afee3 to start an election
I20240503 01:59:34.257501 4561 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0cbba6e3ce5b4de5b5ec9164e50f784a"
dest_uuid: "6c8442cedca54ad589e0e050282afee3"
from {username='slave'} at 127.0.99.129:46301
I20240503 01:59:34.257696 4561 raft_consensus.cc:482] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 6c8442cedca54ad589e0e050282afee3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 01:59:34.257820 4561 raft_consensus.cc:3037] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 6c8442cedca54ad589e0e050282afee3 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:34.258695 4561 raft_consensus.cc:504] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 6c8442cedca54ad589e0e050282afee3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "6c8442cedca54ad589e0e050282afee3" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43573 } attrs { promote: false } }
I20240503 01:59:34.259577 4561 leader_election.cc:288] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 6c8442cedca54ad589e0e050282afee3 [CANDIDATE]: Term 2 election: Requested vote from peers 4d61ec80bf4c475ea86ae2bcb866fd05 (127.0.99.129:38011)
I20240503 01:59:34.260000 2897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cbba6e3ce5b4de5b5ec9164e50f784a" candidate_uuid: "6c8442cedca54ad589e0e050282afee3" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05"
I20240503 01:59:34.260126 2897 raft_consensus.cc:3032] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 01:59:34.260174 2897 raft_consensus.cc:729] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4d61ec80bf4c475ea86ae2bcb866fd05, State: Running, Role: LEADER
I20240503 01:59:34.260330 2897 consensus_queue.cc:260] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 4d61ec80bf4c475ea86ae2bcb866fd05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "6c8442cedca54ad589e0e050282afee3" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43573 } attrs { promote: false } }
I20240503 01:59:34.260519 2897 raft_consensus.cc:3037] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:34.261483 2897 raft_consensus.cc:2454] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c8442cedca54ad589e0e050282afee3 in term 2.
I20240503 01:59:34.262154 4494 leader_election.cc:302] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 6c8442cedca54ad589e0e050282afee3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4d61ec80bf4c475ea86ae2bcb866fd05, 6c8442cedca54ad589e0e050282afee3; no voters:
I20240503 01:59:34.262452 4898 raft_consensus.cc:2781] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 6c8442cedca54ad589e0e050282afee3 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 01:59:34.262643 4898 raft_consensus.cc:686] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 6c8442cedca54ad589e0e050282afee3 [term 2 LEADER]: Becoming Leader. State: Replica: 6c8442cedca54ad589e0e050282afee3, State: Running, Role: LEADER
I20240503 01:59:34.262977 4898 consensus_queue.cc:237] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 6c8442cedca54ad589e0e050282afee3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "6c8442cedca54ad589e0e050282afee3" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43573 } attrs { promote: false } }
I20240503 01:59:34.265166 2748 catalog_manager.cc:5484] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 6c8442cedca54ad589e0e050282afee3 reported cstate change: term changed from 1 to 2, leader changed from 4d61ec80bf4c475ea86ae2bcb866fd05 (127.0.99.129) to 6c8442cedca54ad589e0e050282afee3 (127.0.99.135). New cstate: current_term: 2 leader_uuid: "6c8442cedca54ad589e0e050282afee3" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c8442cedca54ad589e0e050282afee3" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43573 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:34.358533 4762 raft_consensus.cc:979] T 201397ae513b4c40bd2e53197616984e P 4d61ec80bf4c475ea86ae2bcb866fd05: : Instructing follower e4c1a8048e4647b6936035bc2fc22b66 to start an election
I20240503 01:59:34.358673 4762 raft_consensus.cc:1067] T 201397ae513b4c40bd2e53197616984e P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 LEADER]: Signalling peer e4c1a8048e4647b6936035bc2fc22b66 to start an election
I20240503 01:59:34.359172 4430 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "201397ae513b4c40bd2e53197616984e"
dest_uuid: "e4c1a8048e4647b6936035bc2fc22b66"
from {username='slave'} at 127.0.99.129:43121
I20240503 01:59:34.359320 4430 raft_consensus.cc:482] T 201397ae513b4c40bd2e53197616984e P e4c1a8048e4647b6936035bc2fc22b66 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 01:59:34.359381 4430 raft_consensus.cc:3037] T 201397ae513b4c40bd2e53197616984e P e4c1a8048e4647b6936035bc2fc22b66 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:34.360328 4430 raft_consensus.cc:504] T 201397ae513b4c40bd2e53197616984e P e4c1a8048e4647b6936035bc2fc22b66 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4c1a8048e4647b6936035bc2fc22b66" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46423 } attrs { promote: false } }
I20240503 01:59:34.360724 4430 leader_election.cc:288] T 201397ae513b4c40bd2e53197616984e P e4c1a8048e4647b6936035bc2fc22b66 [CANDIDATE]: Term 2 election: Requested vote from peers 4d61ec80bf4c475ea86ae2bcb866fd05 (127.0.99.129:38011)
I20240503 01:59:34.361111 2897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "201397ae513b4c40bd2e53197616984e" candidate_uuid: "e4c1a8048e4647b6936035bc2fc22b66" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05"
I20240503 01:59:34.361224 2897 raft_consensus.cc:3032] T 201397ae513b4c40bd2e53197616984e P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 01:59:34.361267 2897 raft_consensus.cc:729] T 201397ae513b4c40bd2e53197616984e P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4d61ec80bf4c475ea86ae2bcb866fd05, State: Running, Role: LEADER
I20240503 01:59:34.361420 2897 consensus_queue.cc:260] T 201397ae513b4c40bd2e53197616984e P 4d61ec80bf4c475ea86ae2bcb866fd05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4c1a8048e4647b6936035bc2fc22b66" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46423 } attrs { promote: false } }
I20240503 01:59:34.361588 2897 raft_consensus.cc:3037] T 201397ae513b4c40bd2e53197616984e P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:34.362560 2897 raft_consensus.cc:2454] T 201397ae513b4c40bd2e53197616984e P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4c1a8048e4647b6936035bc2fc22b66 in term 2.
I20240503 01:59:34.362874 4363 leader_election.cc:302] T 201397ae513b4c40bd2e53197616984e P e4c1a8048e4647b6936035bc2fc22b66 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4d61ec80bf4c475ea86ae2bcb866fd05, e4c1a8048e4647b6936035bc2fc22b66; no voters:
I20240503 01:59:34.363111 5249 raft_consensus.cc:2781] T 201397ae513b4c40bd2e53197616984e P e4c1a8048e4647b6936035bc2fc22b66 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 01:59:34.363211 5249 raft_consensus.cc:686] T 201397ae513b4c40bd2e53197616984e P e4c1a8048e4647b6936035bc2fc22b66 [term 2 LEADER]: Becoming Leader. State: Replica: e4c1a8048e4647b6936035bc2fc22b66, State: Running, Role: LEADER
I20240503 01:59:34.363359 5249 consensus_queue.cc:237] T 201397ae513b4c40bd2e53197616984e P e4c1a8048e4647b6936035bc2fc22b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4c1a8048e4647b6936035bc2fc22b66" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46423 } attrs { promote: false } }
I20240503 01:59:34.364955 2748 catalog_manager.cc:5484] T 201397ae513b4c40bd2e53197616984e P e4c1a8048e4647b6936035bc2fc22b66 reported cstate change: term changed from 1 to 2, leader changed from 4d61ec80bf4c475ea86ae2bcb866fd05 (127.0.99.129) to e4c1a8048e4647b6936035bc2fc22b66 (127.0.99.134). New cstate: current_term: 2 leader_uuid: "e4c1a8048e4647b6936035bc2fc22b66" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4c1a8048e4647b6936035bc2fc22b66" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46423 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:34.371337 3028 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1b96b32dada445ccbe9c4eaa0ddee050"
dest_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59672
I20240503 01:59:34.371487 3028 raft_consensus.cc:595] T 1b96b32dada445ccbe9c4eaa0ddee050 P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:34.382513 3028 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fc4a011058e447499e3d651c73f0b881"
dest_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59690
I20240503 01:59:34.382704 3028 raft_consensus.cc:595] T fc4a011058e447499e3d651c73f0b881 P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:34.383762 5034 raft_consensus.cc:979] T b2ba35f5c3084e08a42c6f3381aa1cb5 P fb4fc23dca0c4de69aef5061d3a67b2f: : Instructing follower e296c081ab4a414294313ab9883cae84 to start an election
I20240503 01:59:34.383937 5034 raft_consensus.cc:1067] T b2ba35f5c3084e08a42c6f3381aa1cb5 P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 LEADER]: Signalling peer e296c081ab4a414294313ab9883cae84 to start an election
I20240503 01:59:34.384492 4299 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b2ba35f5c3084e08a42c6f3381aa1cb5"
dest_uuid: "e296c081ab4a414294313ab9883cae84"
from {username='slave'} at 127.0.99.130:35005
I20240503 01:59:34.384627 4299 raft_consensus.cc:482] T b2ba35f5c3084e08a42c6f3381aa1cb5 P e296c081ab4a414294313ab9883cae84 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 01:59:34.384693 4299 raft_consensus.cc:3037] T b2ba35f5c3084e08a42c6f3381aa1cb5 P e296c081ab4a414294313ab9883cae84 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:34.385464 4299 raft_consensus.cc:504] T b2ba35f5c3084e08a42c6f3381aa1cb5 P e296c081ab4a414294313ab9883cae84 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "e296c081ab4a414294313ab9883cae84" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41953 } attrs { promote: false } }
I20240503 01:59:34.385809 4299 leader_election.cc:288] T b2ba35f5c3084e08a42c6f3381aa1cb5 P e296c081ab4a414294313ab9883cae84 [CANDIDATE]: Term 2 election: Requested vote from peers fb4fc23dca0c4de69aef5061d3a67b2f (127.0.99.130:34163)
I20240503 01:59:34.386200 3028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2ba35f5c3084e08a42c6f3381aa1cb5" candidate_uuid: "e296c081ab4a414294313ab9883cae84" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f"
I20240503 01:59:34.386364 3028 raft_consensus.cc:3032] T b2ba35f5c3084e08a42c6f3381aa1cb5 P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 LEADER]: Stepping down as leader of term 1
I20240503 01:59:34.386406 3028 raft_consensus.cc:729] T b2ba35f5c3084e08a42c6f3381aa1cb5 P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fb4fc23dca0c4de69aef5061d3a67b2f, State: Running, Role: LEADER
I20240503 01:59:34.386567 3028 consensus_queue.cc:260] T b2ba35f5c3084e08a42c6f3381aa1cb5 P fb4fc23dca0c4de69aef5061d3a67b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "e296c081ab4a414294313ab9883cae84" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41953 } attrs { promote: false } }
I20240503 01:59:34.386713 3028 raft_consensus.cc:3037] T b2ba35f5c3084e08a42c6f3381aa1cb5 P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:34.387666 3028 raft_consensus.cc:2454] T b2ba35f5c3084e08a42c6f3381aa1cb5 P fb4fc23dca0c4de69aef5061d3a67b2f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e296c081ab4a414294313ab9883cae84 in term 2.
I20240503 01:59:34.387996 4233 leader_election.cc:302] T b2ba35f5c3084e08a42c6f3381aa1cb5 P e296c081ab4a414294313ab9883cae84 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: e296c081ab4a414294313ab9883cae84, fb4fc23dca0c4de69aef5061d3a67b2f; no voters:
I20240503 01:59:34.388185 4896 raft_consensus.cc:2781] T b2ba35f5c3084e08a42c6f3381aa1cb5 P e296c081ab4a414294313ab9883cae84 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 01:59:34.388270 4896 raft_consensus.cc:686] T b2ba35f5c3084e08a42c6f3381aa1cb5 P e296c081ab4a414294313ab9883cae84 [term 2 LEADER]: Becoming Leader. State: Replica: e296c081ab4a414294313ab9883cae84, State: Running, Role: LEADER
I20240503 01:59:34.388432 4896 consensus_queue.cc:237] T b2ba35f5c3084e08a42c6f3381aa1cb5 P e296c081ab4a414294313ab9883cae84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "e296c081ab4a414294313ab9883cae84" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41953 } attrs { promote: false } }
I20240503 01:59:34.389945 2748 catalog_manager.cc:5484] T b2ba35f5c3084e08a42c6f3381aa1cb5 P e296c081ab4a414294313ab9883cae84 reported cstate change: term changed from 1 to 2, leader changed from fb4fc23dca0c4de69aef5061d3a67b2f (127.0.99.130) to e296c081ab4a414294313ab9883cae84 (127.0.99.133). New cstate: current_term: 2 leader_uuid: "e296c081ab4a414294313ab9883cae84" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e296c081ab4a414294313ab9883cae84" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41953 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:34.394340 5034 raft_consensus.cc:979] T 9bfd209000fd468385a66c8fa5b2edc8 P fb4fc23dca0c4de69aef5061d3a67b2f: : Instructing follower c8a3e298f9d441c09c064a7f5655980b to start an election
I20240503 01:59:34.394479 5034 raft_consensus.cc:1067] T 9bfd209000fd468385a66c8fa5b2edc8 P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 LEADER]: Signalling peer c8a3e298f9d441c09c064a7f5655980b to start an election
I20240503 01:59:34.394992 4168 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9bfd209000fd468385a66c8fa5b2edc8"
dest_uuid: "c8a3e298f9d441c09c064a7f5655980b"
from {username='slave'} at 127.0.99.130:38231
I20240503 01:59:34.395198 4168 raft_consensus.cc:482] T 9bfd209000fd468385a66c8fa5b2edc8 P c8a3e298f9d441c09c064a7f5655980b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 01:59:34.395294 4168 raft_consensus.cc:3037] T 9bfd209000fd468385a66c8fa5b2edc8 P c8a3e298f9d441c09c064a7f5655980b [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:34.396108 4168 raft_consensus.cc:504] T 9bfd209000fd468385a66c8fa5b2edc8 P c8a3e298f9d441c09c064a7f5655980b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "c8a3e298f9d441c09c064a7f5655980b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41327 } attrs { promote: false } }
I20240503 01:59:34.397097 4168 leader_election.cc:288] T 9bfd209000fd468385a66c8fa5b2edc8 P c8a3e298f9d441c09c064a7f5655980b [CANDIDATE]: Term 2 election: Requested vote from peers fb4fc23dca0c4de69aef5061d3a67b2f (127.0.99.130:34163)
I20240503 01:59:34.399236 3028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bfd209000fd468385a66c8fa5b2edc8" candidate_uuid: "c8a3e298f9d441c09c064a7f5655980b" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f"
I20240503 01:59:34.399372 3028 raft_consensus.cc:3032] T 9bfd209000fd468385a66c8fa5b2edc8 P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 LEADER]: Stepping down as leader of term 1
I20240503 01:59:34.399412 3028 raft_consensus.cc:729] T 9bfd209000fd468385a66c8fa5b2edc8 P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fb4fc23dca0c4de69aef5061d3a67b2f, State: Running, Role: LEADER
I20240503 01:59:34.399574 3028 consensus_queue.cc:260] T 9bfd209000fd468385a66c8fa5b2edc8 P fb4fc23dca0c4de69aef5061d3a67b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "c8a3e298f9d441c09c064a7f5655980b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41327 } attrs { promote: false } }
I20240503 01:59:34.399715 3028 raft_consensus.cc:3037] T 9bfd209000fd468385a66c8fa5b2edc8 P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:34.400789 3028 raft_consensus.cc:2454] T 9bfd209000fd468385a66c8fa5b2edc8 P fb4fc23dca0c4de69aef5061d3a67b2f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8a3e298f9d441c09c064a7f5655980b in term 2.
I20240503 01:59:34.401156 4102 leader_election.cc:302] T 9bfd209000fd468385a66c8fa5b2edc8 P c8a3e298f9d441c09c064a7f5655980b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: c8a3e298f9d441c09c064a7f5655980b, fb4fc23dca0c4de69aef5061d3a67b2f; no voters:
I20240503 01:59:34.401669 4903 raft_consensus.cc:2781] T 9bfd209000fd468385a66c8fa5b2edc8 P c8a3e298f9d441c09c064a7f5655980b [term 2 FOLLOWER]: Leader election won for term 2
I20240503 01:59:34.401850 4903 raft_consensus.cc:686] T 9bfd209000fd468385a66c8fa5b2edc8 P c8a3e298f9d441c09c064a7f5655980b [term 2 LEADER]: Becoming Leader. State: Replica: c8a3e298f9d441c09c064a7f5655980b, State: Running, Role: LEADER
I20240503 01:59:34.402187 4903 consensus_queue.cc:237] T 9bfd209000fd468385a66c8fa5b2edc8 P c8a3e298f9d441c09c064a7f5655980b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "c8a3e298f9d441c09c064a7f5655980b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41327 } attrs { promote: false } }
I20240503 01:59:34.405236 2748 catalog_manager.cc:5484] T 9bfd209000fd468385a66c8fa5b2edc8 P c8a3e298f9d441c09c064a7f5655980b reported cstate change: term changed from 1 to 2, leader changed from fb4fc23dca0c4de69aef5061d3a67b2f (127.0.99.130) to c8a3e298f9d441c09c064a7f5655980b (127.0.99.132). New cstate: current_term: 2 leader_uuid: "c8a3e298f9d441c09c064a7f5655980b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8a3e298f9d441c09c064a7f5655980b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41327 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:34.434985 5034 raft_consensus.cc:979] T c3c2ff70515f49cf9f51b8db5e130cba P fb4fc23dca0c4de69aef5061d3a67b2f: : Instructing follower 6c8442cedca54ad589e0e050282afee3 to start an election
I20240503 01:59:34.435117 5034 raft_consensus.cc:1067] T c3c2ff70515f49cf9f51b8db5e130cba P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 LEADER]: Signalling peer 6c8442cedca54ad589e0e050282afee3 to start an election
I20240503 01:59:34.435676 4561 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c3c2ff70515f49cf9f51b8db5e130cba"
dest_uuid: "6c8442cedca54ad589e0e050282afee3"
from {username='slave'} at 127.0.99.130:37153
I20240503 01:59:34.435825 4561 raft_consensus.cc:482] T c3c2ff70515f49cf9f51b8db5e130cba P 6c8442cedca54ad589e0e050282afee3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 01:59:34.435895 4561 raft_consensus.cc:3037] T c3c2ff70515f49cf9f51b8db5e130cba P 6c8442cedca54ad589e0e050282afee3 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:34.436841 4561 raft_consensus.cc:504] T c3c2ff70515f49cf9f51b8db5e130cba P 6c8442cedca54ad589e0e050282afee3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "6c8442cedca54ad589e0e050282afee3" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43573 } attrs { promote: false } }
I20240503 01:59:34.437209 3028 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c3c2ff70515f49cf9f51b8db5e130cba"
dest_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59704
I20240503 01:59:34.437256 4561 leader_election.cc:288] T c3c2ff70515f49cf9f51b8db5e130cba P 6c8442cedca54ad589e0e050282afee3 [CANDIDATE]: Term 2 election: Requested vote from peers fb4fc23dca0c4de69aef5061d3a67b2f (127.0.99.130:34163)
I20240503 01:59:34.437343 3028 raft_consensus.cc:595] T c3c2ff70515f49cf9f51b8db5e130cba P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:34.437669 3028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3c2ff70515f49cf9f51b8db5e130cba" candidate_uuid: "6c8442cedca54ad589e0e050282afee3" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f"
I20240503 01:59:34.437774 3028 raft_consensus.cc:3032] T c3c2ff70515f49cf9f51b8db5e130cba P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 LEADER]: Stepping down as leader of term 1
I20240503 01:59:34.437822 3028 raft_consensus.cc:729] T c3c2ff70515f49cf9f51b8db5e130cba P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fb4fc23dca0c4de69aef5061d3a67b2f, State: Running, Role: LEADER
I20240503 01:59:34.438030 3028 consensus_queue.cc:260] T c3c2ff70515f49cf9f51b8db5e130cba P fb4fc23dca0c4de69aef5061d3a67b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "6c8442cedca54ad589e0e050282afee3" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43573 } attrs { promote: false } }
I20240503 01:59:34.438177 3028 raft_consensus.cc:3037] T c3c2ff70515f49cf9f51b8db5e130cba P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:34.439052 3028 raft_consensus.cc:2454] T c3c2ff70515f49cf9f51b8db5e130cba P fb4fc23dca0c4de69aef5061d3a67b2f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c8442cedca54ad589e0e050282afee3 in term 2.
I20240503 01:59:34.439332 4495 leader_election.cc:302] T c3c2ff70515f49cf9f51b8db5e130cba P 6c8442cedca54ad589e0e050282afee3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6c8442cedca54ad589e0e050282afee3, fb4fc23dca0c4de69aef5061d3a67b2f; no voters:
I20240503 01:59:34.439536 4898 raft_consensus.cc:2781] T c3c2ff70515f49cf9f51b8db5e130cba P 6c8442cedca54ad589e0e050282afee3 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 01:59:34.439673 4898 raft_consensus.cc:686] T c3c2ff70515f49cf9f51b8db5e130cba P 6c8442cedca54ad589e0e050282afee3 [term 2 LEADER]: Becoming Leader. State: Replica: 6c8442cedca54ad589e0e050282afee3, State: Running, Role: LEADER
I20240503 01:59:34.439857 4898 consensus_queue.cc:237] T c3c2ff70515f49cf9f51b8db5e130cba P 6c8442cedca54ad589e0e050282afee3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "6c8442cedca54ad589e0e050282afee3" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43573 } attrs { promote: false } }
I20240503 01:59:34.441427 2748 catalog_manager.cc:5484] T c3c2ff70515f49cf9f51b8db5e130cba P 6c8442cedca54ad589e0e050282afee3 reported cstate change: term changed from 1 to 2, leader changed from fb4fc23dca0c4de69aef5061d3a67b2f (127.0.99.130) to 6c8442cedca54ad589e0e050282afee3 (127.0.99.135). New cstate: current_term: 2 leader_uuid: "6c8442cedca54ad589e0e050282afee3" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c8442cedca54ad589e0e050282afee3" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43573 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:34.469923 4755 raft_consensus.cc:979] T 1b96b32dada445ccbe9c4eaa0ddee050 P fb4fc23dca0c4de69aef5061d3a67b2f: : Instructing follower c8a3e298f9d441c09c064a7f5655980b to start an election
I20240503 01:59:34.470039 4755 raft_consensus.cc:1067] T 1b96b32dada445ccbe9c4eaa0ddee050 P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 LEADER]: Signalling peer c8a3e298f9d441c09c064a7f5655980b to start an election
I20240503 01:59:34.470620 4168 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1b96b32dada445ccbe9c4eaa0ddee050"
dest_uuid: "c8a3e298f9d441c09c064a7f5655980b"
from {username='slave'} at 127.0.99.130:38231
I20240503 01:59:34.470769 4168 raft_consensus.cc:482] T 1b96b32dada445ccbe9c4eaa0ddee050 P c8a3e298f9d441c09c064a7f5655980b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 01:59:34.470822 4168 raft_consensus.cc:3037] T 1b96b32dada445ccbe9c4eaa0ddee050 P c8a3e298f9d441c09c064a7f5655980b [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:34.471589 4168 raft_consensus.cc:504] T 1b96b32dada445ccbe9c4eaa0ddee050 P c8a3e298f9d441c09c064a7f5655980b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "c8a3e298f9d441c09c064a7f5655980b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41327 } attrs { promote: false } }
I20240503 01:59:34.471887 4168 leader_election.cc:288] T 1b96b32dada445ccbe9c4eaa0ddee050 P c8a3e298f9d441c09c064a7f5655980b [CANDIDATE]: Term 2 election: Requested vote from peers fb4fc23dca0c4de69aef5061d3a67b2f (127.0.99.130:34163)
I20240503 01:59:34.472498 3028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b96b32dada445ccbe9c4eaa0ddee050" candidate_uuid: "c8a3e298f9d441c09c064a7f5655980b" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f"
I20240503 01:59:34.472635 3028 raft_consensus.cc:3032] T 1b96b32dada445ccbe9c4eaa0ddee050 P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 LEADER]: Stepping down as leader of term 1
I20240503 01:59:34.472683 3028 raft_consensus.cc:729] T 1b96b32dada445ccbe9c4eaa0ddee050 P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fb4fc23dca0c4de69aef5061d3a67b2f, State: Running, Role: LEADER
I20240503 01:59:34.472846 3028 consensus_queue.cc:260] T 1b96b32dada445ccbe9c4eaa0ddee050 P fb4fc23dca0c4de69aef5061d3a67b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "c8a3e298f9d441c09c064a7f5655980b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41327 } attrs { promote: false } }
I20240503 01:59:34.473009 3028 raft_consensus.cc:3037] T 1b96b32dada445ccbe9c4eaa0ddee050 P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:34.474028 3028 raft_consensus.cc:2454] T 1b96b32dada445ccbe9c4eaa0ddee050 P fb4fc23dca0c4de69aef5061d3a67b2f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8a3e298f9d441c09c064a7f5655980b in term 2.
I20240503 01:59:34.474346 4102 leader_election.cc:302] T 1b96b32dada445ccbe9c4eaa0ddee050 P c8a3e298f9d441c09c064a7f5655980b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: c8a3e298f9d441c09c064a7f5655980b, fb4fc23dca0c4de69aef5061d3a67b2f; no voters:
I20240503 01:59:34.474534 4903 raft_consensus.cc:2781] T 1b96b32dada445ccbe9c4eaa0ddee050 P c8a3e298f9d441c09c064a7f5655980b [term 2 FOLLOWER]: Leader election won for term 2
I20240503 01:59:34.474622 4903 raft_consensus.cc:686] T 1b96b32dada445ccbe9c4eaa0ddee050 P c8a3e298f9d441c09c064a7f5655980b [term 2 LEADER]: Becoming Leader. State: Replica: c8a3e298f9d441c09c064a7f5655980b, State: Running, Role: LEADER
I20240503 01:59:34.474764 4903 consensus_queue.cc:237] T 1b96b32dada445ccbe9c4eaa0ddee050 P c8a3e298f9d441c09c064a7f5655980b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "c8a3e298f9d441c09c064a7f5655980b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41327 } attrs { promote: false } }
I20240503 01:59:34.476145 2748 catalog_manager.cc:5484] T 1b96b32dada445ccbe9c4eaa0ddee050 P c8a3e298f9d441c09c064a7f5655980b reported cstate change: term changed from 1 to 2, leader changed from fb4fc23dca0c4de69aef5061d3a67b2f (127.0.99.130) to c8a3e298f9d441c09c064a7f5655980b (127.0.99.132). New cstate: current_term: 2 leader_uuid: "c8a3e298f9d441c09c064a7f5655980b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8a3e298f9d441c09c064a7f5655980b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41327 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:34.527221 5034 raft_consensus.cc:979] T fc4a011058e447499e3d651c73f0b881 P fb4fc23dca0c4de69aef5061d3a67b2f: : Instructing follower ad0b1cf28c9c452db2230e4cd69363bc to start an election
I20240503 01:59:34.527333 5034 raft_consensus.cc:1067] T fc4a011058e447499e3d651c73f0b881 P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 LEADER]: Signalling peer ad0b1cf28c9c452db2230e4cd69363bc to start an election
I20240503 01:59:34.527853 4037 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fc4a011058e447499e3d651c73f0b881"
dest_uuid: "ad0b1cf28c9c452db2230e4cd69363bc"
from {username='slave'} at 127.0.99.130:42343
I20240503 01:59:34.527989 4037 raft_consensus.cc:482] T fc4a011058e447499e3d651c73f0b881 P ad0b1cf28c9c452db2230e4cd69363bc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 01:59:34.528035 4037 raft_consensus.cc:3037] T fc4a011058e447499e3d651c73f0b881 P ad0b1cf28c9c452db2230e4cd69363bc [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:34.528931 4037 raft_consensus.cc:504] T fc4a011058e447499e3d651c73f0b881 P ad0b1cf28c9c452db2230e4cd69363bc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "ad0b1cf28c9c452db2230e4cd69363bc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44037 } attrs { promote: false } }
I20240503 01:59:34.529321 4037 leader_election.cc:288] T fc4a011058e447499e3d651c73f0b881 P ad0b1cf28c9c452db2230e4cd69363bc [CANDIDATE]: Term 2 election: Requested vote from peers fb4fc23dca0c4de69aef5061d3a67b2f (127.0.99.130:34163)
I20240503 01:59:34.529790 3028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc4a011058e447499e3d651c73f0b881" candidate_uuid: "ad0b1cf28c9c452db2230e4cd69363bc" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f"
I20240503 01:59:34.529903 3028 raft_consensus.cc:3032] T fc4a011058e447499e3d651c73f0b881 P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 LEADER]: Stepping down as leader of term 1
I20240503 01:59:34.529940 3028 raft_consensus.cc:729] T fc4a011058e447499e3d651c73f0b881 P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fb4fc23dca0c4de69aef5061d3a67b2f, State: Running, Role: LEADER
I20240503 01:59:34.530081 3028 consensus_queue.cc:260] T fc4a011058e447499e3d651c73f0b881 P fb4fc23dca0c4de69aef5061d3a67b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "ad0b1cf28c9c452db2230e4cd69363bc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44037 } attrs { promote: false } }
I20240503 01:59:34.530236 3028 raft_consensus.cc:3037] T fc4a011058e447499e3d651c73f0b881 P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:34.531008 3028 raft_consensus.cc:2454] T fc4a011058e447499e3d651c73f0b881 P fb4fc23dca0c4de69aef5061d3a67b2f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad0b1cf28c9c452db2230e4cd69363bc in term 2.
I20240503 01:59:34.531327 3971 leader_election.cc:302] T fc4a011058e447499e3d651c73f0b881 P ad0b1cf28c9c452db2230e4cd69363bc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: ad0b1cf28c9c452db2230e4cd69363bc, fb4fc23dca0c4de69aef5061d3a67b2f; no voters:
I20240503 01:59:34.531533 4904 raft_consensus.cc:2781] T fc4a011058e447499e3d651c73f0b881 P ad0b1cf28c9c452db2230e4cd69363bc [term 2 FOLLOWER]: Leader election won for term 2
I20240503 01:59:34.531625 4904 raft_consensus.cc:686] T fc4a011058e447499e3d651c73f0b881 P ad0b1cf28c9c452db2230e4cd69363bc [term 2 LEADER]: Becoming Leader. State: Replica: ad0b1cf28c9c452db2230e4cd69363bc, State: Running, Role: LEADER
I20240503 01:59:34.531798 4904 consensus_queue.cc:237] T fc4a011058e447499e3d651c73f0b881 P ad0b1cf28c9c452db2230e4cd69363bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "ad0b1cf28c9c452db2230e4cd69363bc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44037 } attrs { promote: false } }
I20240503 01:59:34.533325 2748 catalog_manager.cc:5484] T fc4a011058e447499e3d651c73f0b881 P ad0b1cf28c9c452db2230e4cd69363bc reported cstate change: term changed from 1 to 2, leader changed from fb4fc23dca0c4de69aef5061d3a67b2f (127.0.99.130) to ad0b1cf28c9c452db2230e4cd69363bc (127.0.99.131). New cstate: current_term: 2 leader_uuid: "ad0b1cf28c9c452db2230e4cd69363bc" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad0b1cf28c9c452db2230e4cd69363bc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44037 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:34.620060 2897 raft_consensus.cc:1261] T 365919d2a8d94985bc031ab16eefe0d8 P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 2 FOLLOWER]: Refusing update from remote peer ad0b1cf28c9c452db2230e4cd69363bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 01:59:34.620887 4904 consensus_queue.cc:1035] T 365919d2a8d94985bc031ab16eefe0d8 P ad0b1cf28c9c452db2230e4cd69363bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:34.654356 2897 raft_consensus.cc:1261] T 7fe4b04830cd42a292f72f27b788c75a P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 2 FOLLOWER]: Refusing update from remote peer e296c081ab4a414294313ab9883cae84: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 01:59:34.654897 4896 consensus_queue.cc:1035] T 7fe4b04830cd42a292f72f27b788c75a P e296c081ab4a414294313ab9883cae84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:34.703400 2897 raft_consensus.cc:1261] T 7056f7fb504545d19641ea2c58ebf6be P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 2 FOLLOWER]: Refusing update from remote peer e4c1a8048e4647b6936035bc2fc22b66: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 01:59:34.704075 5249 consensus_queue.cc:1035] T 7056f7fb504545d19641ea2c58ebf6be P e4c1a8048e4647b6936035bc2fc22b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:34.724630 4299 consensus_queue.cc:237] T 7fe4b04830cd42a292f72f27b788c75a P e296c081ab4a414294313ab9883cae84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e296c081ab4a414294313ab9883cae84" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41953 } attrs { promote: false } }
I20240503 01:59:34.725311 5226 raft_consensus.cc:2932] T 7fe4b04830cd42a292f72f27b788c75a P e296c081ab4a414294313ab9883cae84 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 4d61ec80bf4c475ea86ae2bcb866fd05 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e296c081ab4a414294313ab9883cae84" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41953 } attrs { promote: false } } }
I20240503 01:59:34.726608 2739 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7fe4b04830cd42a292f72f27b788c75a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 01:59:34.727586 2748 catalog_manager.cc:5484] T 7fe4b04830cd42a292f72f27b788c75a P e296c081ab4a414294313ab9883cae84 reported cstate change: config changed from index 3 to 5, VOTER 4d61ec80bf4c475ea86ae2bcb866fd05 (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "e296c081ab4a414294313ab9883cae84" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e296c081ab4a414294313ab9883cae84" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41953 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:34.730312 2876 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fe4b04830cd42a292f72f27b788c75a with delete_type TABLET_DATA_TOMBSTONED (TS 4d61ec80bf4c475ea86ae2bcb866fd05 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:33814
I20240503 01:59:34.730834 5402 tablet_replica.cc:330] T 7fe4b04830cd42a292f72f27b788c75a P 4d61ec80bf4c475ea86ae2bcb866fd05: stopping tablet replica
I20240503 01:59:34.731132 5402 raft_consensus.cc:2229] T 7fe4b04830cd42a292f72f27b788c75a P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:34.731294 5402 raft_consensus.cc:2258] T 7fe4b04830cd42a292f72f27b788c75a P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:34.732437 5402 ts_tablet_manager.cc:1875] T 7fe4b04830cd42a292f72f27b788c75a P 4d61ec80bf4c475ea86ae2bcb866fd05: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:34.734459 5402 ts_tablet_manager.cc:1888] T 7fe4b04830cd42a292f72f27b788c75a P 4d61ec80bf4c475ea86ae2bcb866fd05: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240503 01:59:34.734544 5402 log.cc:1196] T 7fe4b04830cd42a292f72f27b788c75a P 4d61ec80bf4c475ea86ae2bcb866fd05: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/7fe4b04830cd42a292f72f27b788c75a
I20240503 01:59:34.735229 2739 catalog_manager.cc:4830] TS 4d61ec80bf4c475ea86ae2bcb866fd05 (127.0.99.129:38011): tablet 7fe4b04830cd42a292f72f27b788c75a (table TestTable [id=4e3c2b1187c5480abe84af1077137cc9]) successfully deleted
I20240503 01:59:34.751137 2897 raft_consensus.cc:1261] T 201397ae513b4c40bd2e53197616984e P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 2 FOLLOWER]: Refusing update from remote peer e4c1a8048e4647b6936035bc2fc22b66: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 01:59:34.751631 5249 consensus_queue.cc:1035] T 201397ae513b4c40bd2e53197616984e P e4c1a8048e4647b6936035bc2fc22b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:34.798938 4429 consensus_queue.cc:237] T 7056f7fb504545d19641ea2c58ebf6be P e4c1a8048e4647b6936035bc2fc22b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e4c1a8048e4647b6936035bc2fc22b66" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46423 } attrs { promote: false } }
I20240503 01:59:34.799670 5249 raft_consensus.cc:2932] T 7056f7fb504545d19641ea2c58ebf6be P e4c1a8048e4647b6936035bc2fc22b66 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 4d61ec80bf4c475ea86ae2bcb866fd05 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e4c1a8048e4647b6936035bc2fc22b66" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46423 } attrs { promote: false } } }
I20240503 01:59:34.801116 2740 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7056f7fb504545d19641ea2c58ebf6be with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 01:59:34.802101 2749 catalog_manager.cc:5484] T 7056f7fb504545d19641ea2c58ebf6be P e4c1a8048e4647b6936035bc2fc22b66 reported cstate change: config changed from index 5 to 7, VOTER 4d61ec80bf4c475ea86ae2bcb866fd05 (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "e4c1a8048e4647b6936035bc2fc22b66" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e4c1a8048e4647b6936035bc2fc22b66" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46423 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:34.804772 2876 tablet_service.cc:1514] Processing DeleteTablet for tablet 7056f7fb504545d19641ea2c58ebf6be with delete_type TABLET_DATA_TOMBSTONED (TS 4d61ec80bf4c475ea86ae2bcb866fd05 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:33814
I20240503 01:59:34.805058 5402 tablet_replica.cc:330] T 7056f7fb504545d19641ea2c58ebf6be P 4d61ec80bf4c475ea86ae2bcb866fd05: stopping tablet replica
I20240503 01:59:34.805213 5402 raft_consensus.cc:2229] T 7056f7fb504545d19641ea2c58ebf6be P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:34.805460 5402 raft_consensus.cc:2258] T 7056f7fb504545d19641ea2c58ebf6be P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:34.806113 5402 ts_tablet_manager.cc:1875] T 7056f7fb504545d19641ea2c58ebf6be P 4d61ec80bf4c475ea86ae2bcb866fd05: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:34.808099 5402 ts_tablet_manager.cc:1888] T 7056f7fb504545d19641ea2c58ebf6be P 4d61ec80bf4c475ea86ae2bcb866fd05: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 01:59:34.808174 5402 log.cc:1196] T 7056f7fb504545d19641ea2c58ebf6be P 4d61ec80bf4c475ea86ae2bcb866fd05: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/7056f7fb504545d19641ea2c58ebf6be
I20240503 01:59:34.808789 2739 catalog_manager.cc:4830] TS 4d61ec80bf4c475ea86ae2bcb866fd05 (127.0.99.129:38011): tablet 7056f7fb504545d19641ea2c58ebf6be (table rebalance_test_table_1 [id=c3c40e684e4449ba86b043703f35788b]) successfully deleted
I20240503 01:59:34.816109 2897 raft_consensus.cc:1261] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 2 FOLLOWER]: Refusing update from remote peer 6c8442cedca54ad589e0e050282afee3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 01:59:34.816658 4898 consensus_queue.cc:1035] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 6c8442cedca54ad589e0e050282afee3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:34.828233 4037 consensus_queue.cc:237] T 365919d2a8d94985bc031ab16eefe0d8 P ad0b1cf28c9c452db2230e4cd69363bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ad0b1cf28c9c452db2230e4cd69363bc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44037 } attrs { promote: false } }
I20240503 01:59:34.828999 5276 raft_consensus.cc:2932] T 365919d2a8d94985bc031ab16eefe0d8 P ad0b1cf28c9c452db2230e4cd69363bc [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 4d61ec80bf4c475ea86ae2bcb866fd05 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ad0b1cf28c9c452db2230e4cd69363bc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44037 } attrs { promote: false } } }
I20240503 01:59:34.830323 2741 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 365919d2a8d94985bc031ab16eefe0d8 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 01:59:34.831313 2748 catalog_manager.cc:5484] T 365919d2a8d94985bc031ab16eefe0d8 P ad0b1cf28c9c452db2230e4cd69363bc reported cstate change: config changed from index 5 to 7, VOTER 4d61ec80bf4c475ea86ae2bcb866fd05 (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "ad0b1cf28c9c452db2230e4cd69363bc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ad0b1cf28c9c452db2230e4cd69363bc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44037 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:34.833608 2876 tablet_service.cc:1514] Processing DeleteTablet for tablet 365919d2a8d94985bc031ab16eefe0d8 with delete_type TABLET_DATA_TOMBSTONED (TS 4d61ec80bf4c475ea86ae2bcb866fd05 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:33814
I20240503 01:59:34.833833 5402 tablet_replica.cc:330] T 365919d2a8d94985bc031ab16eefe0d8 P 4d61ec80bf4c475ea86ae2bcb866fd05: stopping tablet replica
I20240503 01:59:34.833940 5402 raft_consensus.cc:2229] T 365919d2a8d94985bc031ab16eefe0d8 P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:34.834033 5402 raft_consensus.cc:2258] T 365919d2a8d94985bc031ab16eefe0d8 P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:34.834573 5402 ts_tablet_manager.cc:1875] T 365919d2a8d94985bc031ab16eefe0d8 P 4d61ec80bf4c475ea86ae2bcb866fd05: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:34.836266 5402 ts_tablet_manager.cc:1888] T 365919d2a8d94985bc031ab16eefe0d8 P 4d61ec80bf4c475ea86ae2bcb866fd05: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 01:59:34.836328 5402 log.cc:1196] T 365919d2a8d94985bc031ab16eefe0d8 P 4d61ec80bf4c475ea86ae2bcb866fd05: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/365919d2a8d94985bc031ab16eefe0d8
I20240503 01:59:34.836809 2739 catalog_manager.cc:4830] TS 4d61ec80bf4c475ea86ae2bcb866fd05 (127.0.99.129:38011): tablet 365919d2a8d94985bc031ab16eefe0d8 (table rebalance_test_table_3 [id=dfaafa31db2c42809a098afd0da706a3]) successfully deleted
I20240503 01:59:34.888684 3028 raft_consensus.cc:1261] T c3c2ff70515f49cf9f51b8db5e130cba P fb4fc23dca0c4de69aef5061d3a67b2f [term 2 FOLLOWER]: Refusing update from remote peer 6c8442cedca54ad589e0e050282afee3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 01:59:34.889310 5278 consensus_queue.cc:1035] T c3c2ff70515f49cf9f51b8db5e130cba P 6c8442cedca54ad589e0e050282afee3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:34.895931 4561 consensus_queue.cc:237] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 6c8442cedca54ad589e0e050282afee3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c8442cedca54ad589e0e050282afee3" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43573 } attrs { promote: false } }
I20240503 01:59:34.896571 5278 raft_consensus.cc:2932] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 6c8442cedca54ad589e0e050282afee3 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 4d61ec80bf4c475ea86ae2bcb866fd05 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c8442cedca54ad589e0e050282afee3" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43573 } attrs { promote: false } } }
I20240503 01:59:34.897889 2740 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0cbba6e3ce5b4de5b5ec9164e50f784a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 01:59:34.898741 2748 catalog_manager.cc:5484] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 6c8442cedca54ad589e0e050282afee3 reported cstate change: config changed from index 5 to 7, VOTER 4d61ec80bf4c475ea86ae2bcb866fd05 (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "6c8442cedca54ad589e0e050282afee3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c8442cedca54ad589e0e050282afee3" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43573 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:34.901106 2876 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cbba6e3ce5b4de5b5ec9164e50f784a with delete_type TABLET_DATA_TOMBSTONED (TS 4d61ec80bf4c475ea86ae2bcb866fd05 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:33814
I20240503 01:59:34.901331 5402 tablet_replica.cc:330] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 4d61ec80bf4c475ea86ae2bcb866fd05: stopping tablet replica
I20240503 01:59:34.901463 5402 raft_consensus.cc:2229] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:34.901579 5402 raft_consensus.cc:2258] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:34.902246 5402 ts_tablet_manager.cc:1875] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 4d61ec80bf4c475ea86ae2bcb866fd05: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:34.904299 5402 ts_tablet_manager.cc:1888] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 4d61ec80bf4c475ea86ae2bcb866fd05: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 01:59:34.904369 5402 log.cc:1196] T 0cbba6e3ce5b4de5b5ec9164e50f784a P 4d61ec80bf4c475ea86ae2bcb866fd05: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/0cbba6e3ce5b4de5b5ec9164e50f784a
I20240503 01:59:34.904969 2739 catalog_manager.cc:4830] TS 4d61ec80bf4c475ea86ae2bcb866fd05 (127.0.99.129:38011): tablet 0cbba6e3ce5b4de5b5ec9164e50f784a (table rebalance_test_table_4 [id=8a70cfe3f9b84945b710ac326669a37e]) successfully deleted
I20240503 01:59:34.970554 3028 raft_consensus.cc:1261] T b2ba35f5c3084e08a42c6f3381aa1cb5 P fb4fc23dca0c4de69aef5061d3a67b2f [term 2 FOLLOWER]: Refusing update from remote peer e296c081ab4a414294313ab9883cae84: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 01:59:34.970993 5226 consensus_queue.cc:1035] T b2ba35f5c3084e08a42c6f3381aa1cb5 P e296c081ab4a414294313ab9883cae84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:34.976081 3028 raft_consensus.cc:1261] T 9bfd209000fd468385a66c8fa5b2edc8 P fb4fc23dca0c4de69aef5061d3a67b2f [term 2 FOLLOWER]: Refusing update from remote peer c8a3e298f9d441c09c064a7f5655980b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 01:59:34.976538 4903 consensus_queue.cc:1035] T 9bfd209000fd468385a66c8fa5b2edc8 P c8a3e298f9d441c09c064a7f5655980b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:34.992206 4429 consensus_queue.cc:237] T 201397ae513b4c40bd2e53197616984e P e4c1a8048e4647b6936035bc2fc22b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e4c1a8048e4647b6936035bc2fc22b66" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46423 } attrs { promote: false } }
I20240503 01:59:34.992852 4938 raft_consensus.cc:2932] T 201397ae513b4c40bd2e53197616984e P e4c1a8048e4647b6936035bc2fc22b66 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 4d61ec80bf4c475ea86ae2bcb866fd05 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e4c1a8048e4647b6936035bc2fc22b66" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46423 } attrs { promote: false } } }
I20240503 01:59:34.994197 2740 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 201397ae513b4c40bd2e53197616984e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 01:59:34.995174 2748 catalog_manager.cc:5484] T 201397ae513b4c40bd2e53197616984e P e4c1a8048e4647b6936035bc2fc22b66 reported cstate change: config changed from index 5 to 7, VOTER 4d61ec80bf4c475ea86ae2bcb866fd05 (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "e4c1a8048e4647b6936035bc2fc22b66" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e4c1a8048e4647b6936035bc2fc22b66" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46423 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:34.997519 2876 tablet_service.cc:1514] Processing DeleteTablet for tablet 201397ae513b4c40bd2e53197616984e with delete_type TABLET_DATA_TOMBSTONED (TS 4d61ec80bf4c475ea86ae2bcb866fd05 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:33814
I20240503 01:59:34.997768 5402 tablet_replica.cc:330] T 201397ae513b4c40bd2e53197616984e P 4d61ec80bf4c475ea86ae2bcb866fd05: stopping tablet replica
I20240503 01:59:34.997907 5402 raft_consensus.cc:2229] T 201397ae513b4c40bd2e53197616984e P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:34.998015 5402 raft_consensus.cc:2258] T 201397ae513b4c40bd2e53197616984e P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:34.998793 5402 ts_tablet_manager.cc:1875] T 201397ae513b4c40bd2e53197616984e P 4d61ec80bf4c475ea86ae2bcb866fd05: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:35.000918 5402 ts_tablet_manager.cc:1888] T 201397ae513b4c40bd2e53197616984e P 4d61ec80bf4c475ea86ae2bcb866fd05: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 01:59:35.001006 5402 log.cc:1196] T 201397ae513b4c40bd2e53197616984e P 4d61ec80bf4c475ea86ae2bcb866fd05: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/201397ae513b4c40bd2e53197616984e
I20240503 01:59:35.001482 2739 catalog_manager.cc:4830] TS 4d61ec80bf4c475ea86ae2bcb866fd05 (127.0.99.129:38011): tablet 201397ae513b4c40bd2e53197616984e (table rebalance_test_table_3 [id=dfaafa31db2c42809a098afd0da706a3]) successfully deleted
I20240503 01:59:35.007081 3028 raft_consensus.cc:1261] T fc4a011058e447499e3d651c73f0b881 P fb4fc23dca0c4de69aef5061d3a67b2f [term 2 FOLLOWER]: Refusing update from remote peer ad0b1cf28c9c452db2230e4cd69363bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 01:59:35.007463 4904 consensus_queue.cc:1035] T fc4a011058e447499e3d651c73f0b881 P ad0b1cf28c9c452db2230e4cd69363bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:35.018396 3028 raft_consensus.cc:1261] T 1b96b32dada445ccbe9c4eaa0ddee050 P fb4fc23dca0c4de69aef5061d3a67b2f [term 2 FOLLOWER]: Refusing update from remote peer c8a3e298f9d441c09c064a7f5655980b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 01:59:35.018644 4299 consensus_queue.cc:237] T b2ba35f5c3084e08a42c6f3381aa1cb5 P e296c081ab4a414294313ab9883cae84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e296c081ab4a414294313ab9883cae84" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41953 } attrs { promote: false } }
I20240503 01:59:35.019222 5226 raft_consensus.cc:2932] T b2ba35f5c3084e08a42c6f3381aa1cb5 P e296c081ab4a414294313ab9883cae84 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER fb4fc23dca0c4de69aef5061d3a67b2f (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e296c081ab4a414294313ab9883cae84" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41953 } attrs { promote: false } } }
I20240503 01:59:35.019455 5453 consensus_queue.cc:1035] T 1b96b32dada445ccbe9c4eaa0ddee050 P c8a3e298f9d441c09c064a7f5655980b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:35.020642 2739 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b2ba35f5c3084e08a42c6f3381aa1cb5 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 01:59:35.021518 2748 catalog_manager.cc:5484] T b2ba35f5c3084e08a42c6f3381aa1cb5 P e296c081ab4a414294313ab9883cae84 reported cstate change: config changed from index 5 to 7, VOTER fb4fc23dca0c4de69aef5061d3a67b2f (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "e296c081ab4a414294313ab9883cae84" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e296c081ab4a414294313ab9883cae84" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41953 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:35.024597 3007 tablet_service.cc:1514] Processing DeleteTablet for tablet b2ba35f5c3084e08a42c6f3381aa1cb5 with delete_type TABLET_DATA_TOMBSTONED (TS fb4fc23dca0c4de69aef5061d3a67b2f not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:59142
I20240503 01:59:35.025077 5497 tablet_replica.cc:330] T b2ba35f5c3084e08a42c6f3381aa1cb5 P fb4fc23dca0c4de69aef5061d3a67b2f: stopping tablet replica
I20240503 01:59:35.025349 5497 raft_consensus.cc:2229] T b2ba35f5c3084e08a42c6f3381aa1cb5 P fb4fc23dca0c4de69aef5061d3a67b2f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:35.025501 5497 raft_consensus.cc:2258] T b2ba35f5c3084e08a42c6f3381aa1cb5 P fb4fc23dca0c4de69aef5061d3a67b2f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:35.026640 5497 ts_tablet_manager.cc:1875] T b2ba35f5c3084e08a42c6f3381aa1cb5 P fb4fc23dca0c4de69aef5061d3a67b2f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:35.028870 5497 ts_tablet_manager.cc:1888] T b2ba35f5c3084e08a42c6f3381aa1cb5 P fb4fc23dca0c4de69aef5061d3a67b2f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 01:59:35.028971 5497 log.cc:1196] T b2ba35f5c3084e08a42c6f3381aa1cb5 P fb4fc23dca0c4de69aef5061d3a67b2f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/b2ba35f5c3084e08a42c6f3381aa1cb5
I20240503 01:59:35.029491 2740 catalog_manager.cc:4830] TS fb4fc23dca0c4de69aef5061d3a67b2f (127.0.99.130:34163): tablet b2ba35f5c3084e08a42c6f3381aa1cb5 (table rebalance_test_table_0 [id=da03de56cae54136b05045054d83001e]) successfully deleted
I20240503 01:59:35.043915 4168 consensus_queue.cc:237] T 9bfd209000fd468385a66c8fa5b2edc8 P c8a3e298f9d441c09c064a7f5655980b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c8a3e298f9d441c09c064a7f5655980b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41327 } attrs { promote: false } }
I20240503 01:59:35.044867 5453 raft_consensus.cc:2932] T 9bfd209000fd468385a66c8fa5b2edc8 P c8a3e298f9d441c09c064a7f5655980b [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER fb4fc23dca0c4de69aef5061d3a67b2f (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c8a3e298f9d441c09c064a7f5655980b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41327 } attrs { promote: false } } }
I20240503 01:59:35.046142 2740 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9bfd209000fd468385a66c8fa5b2edc8 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 01:59:35.047298 2755 catalog_manager.cc:5484] T 9bfd209000fd468385a66c8fa5b2edc8 P c8a3e298f9d441c09c064a7f5655980b reported cstate change: config changed from index 5 to 7, VOTER fb4fc23dca0c4de69aef5061d3a67b2f (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "c8a3e298f9d441c09c064a7f5655980b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c8a3e298f9d441c09c064a7f5655980b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41327 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:35.052309 3007 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bfd209000fd468385a66c8fa5b2edc8 with delete_type TABLET_DATA_TOMBSTONED (TS fb4fc23dca0c4de69aef5061d3a67b2f not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:59142
I20240503 01:59:35.052431 4561 consensus_queue.cc:237] T c3c2ff70515f49cf9f51b8db5e130cba P 6c8442cedca54ad589e0e050282afee3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c8442cedca54ad589e0e050282afee3" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43573 } attrs { promote: false } }
I20240503 01:59:35.052613 5497 tablet_replica.cc:330] T 9bfd209000fd468385a66c8fa5b2edc8 P fb4fc23dca0c4de69aef5061d3a67b2f: stopping tablet replica
I20240503 01:59:35.052762 5497 raft_consensus.cc:2229] T 9bfd209000fd468385a66c8fa5b2edc8 P fb4fc23dca0c4de69aef5061d3a67b2f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:35.052877 5497 raft_consensus.cc:2258] T 9bfd209000fd468385a66c8fa5b2edc8 P fb4fc23dca0c4de69aef5061d3a67b2f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:35.053042 5278 raft_consensus.cc:2932] T c3c2ff70515f49cf9f51b8db5e130cba P 6c8442cedca54ad589e0e050282afee3 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER fb4fc23dca0c4de69aef5061d3a67b2f (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c8442cedca54ad589e0e050282afee3" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43573 } attrs { promote: false } } }
I20240503 01:59:35.053694 5497 ts_tablet_manager.cc:1875] T 9bfd209000fd468385a66c8fa5b2edc8 P fb4fc23dca0c4de69aef5061d3a67b2f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:35.054333 2740 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c3c2ff70515f49cf9f51b8db5e130cba with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 01:59:35.055136 2755 catalog_manager.cc:5484] T c3c2ff70515f49cf9f51b8db5e130cba P 6c8442cedca54ad589e0e050282afee3 reported cstate change: config changed from index 5 to 7, VOTER fb4fc23dca0c4de69aef5061d3a67b2f (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "6c8442cedca54ad589e0e050282afee3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c8442cedca54ad589e0e050282afee3" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43573 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:35.056129 5497 ts_tablet_manager.cc:1888] T 9bfd209000fd468385a66c8fa5b2edc8 P fb4fc23dca0c4de69aef5061d3a67b2f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 01:59:35.056210 5497 log.cc:1196] T 9bfd209000fd468385a66c8fa5b2edc8 P fb4fc23dca0c4de69aef5061d3a67b2f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/9bfd209000fd468385a66c8fa5b2edc8
I20240503 01:59:35.056870 2740 catalog_manager.cc:4830] TS fb4fc23dca0c4de69aef5061d3a67b2f (127.0.99.130:34163): tablet 9bfd209000fd468385a66c8fa5b2edc8 (table rebalance_test_table_4 [id=8a70cfe3f9b84945b710ac326669a37e]) successfully deleted
I20240503 01:59:35.057981 3007 tablet_service.cc:1514] Processing DeleteTablet for tablet c3c2ff70515f49cf9f51b8db5e130cba with delete_type TABLET_DATA_TOMBSTONED (TS fb4fc23dca0c4de69aef5061d3a67b2f not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:59142
I20240503 01:59:35.058171 5497 tablet_replica.cc:330] T c3c2ff70515f49cf9f51b8db5e130cba P fb4fc23dca0c4de69aef5061d3a67b2f: stopping tablet replica
I20240503 01:59:35.058300 5497 raft_consensus.cc:2229] T c3c2ff70515f49cf9f51b8db5e130cba P fb4fc23dca0c4de69aef5061d3a67b2f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:35.058414 5497 raft_consensus.cc:2258] T c3c2ff70515f49cf9f51b8db5e130cba P fb4fc23dca0c4de69aef5061d3a67b2f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:35.059165 5497 ts_tablet_manager.cc:1875] T c3c2ff70515f49cf9f51b8db5e130cba P fb4fc23dca0c4de69aef5061d3a67b2f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:35.061033 5497 ts_tablet_manager.cc:1888] T c3c2ff70515f49cf9f51b8db5e130cba P fb4fc23dca0c4de69aef5061d3a67b2f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 01:59:35.061102 5497 log.cc:1196] T c3c2ff70515f49cf9f51b8db5e130cba P fb4fc23dca0c4de69aef5061d3a67b2f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/c3c2ff70515f49cf9f51b8db5e130cba
I20240503 01:59:35.061605 2740 catalog_manager.cc:4830] TS fb4fc23dca0c4de69aef5061d3a67b2f (127.0.99.130:34163): tablet c3c2ff70515f49cf9f51b8db5e130cba (table rebalance_test_table_2 [id=c045b78f2dc741a4b00a4b13ac99530e]) successfully deleted
I20240503 01:59:35.078959 4168 consensus_queue.cc:237] T 1b96b32dada445ccbe9c4eaa0ddee050 P c8a3e298f9d441c09c064a7f5655980b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c8a3e298f9d441c09c064a7f5655980b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41327 } attrs { promote: false } }
I20240503 01:59:35.079650 4903 raft_consensus.cc:2932] T 1b96b32dada445ccbe9c4eaa0ddee050 P c8a3e298f9d441c09c064a7f5655980b [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER fb4fc23dca0c4de69aef5061d3a67b2f (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c8a3e298f9d441c09c064a7f5655980b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41327 } attrs { promote: false } } }
I20240503 01:59:35.081022 2740 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1b96b32dada445ccbe9c4eaa0ddee050 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 01:59:35.081916 2755 catalog_manager.cc:5484] T 1b96b32dada445ccbe9c4eaa0ddee050 P c8a3e298f9d441c09c064a7f5655980b reported cstate change: config changed from index 5 to 7, VOTER fb4fc23dca0c4de69aef5061d3a67b2f (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "c8a3e298f9d441c09c064a7f5655980b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c8a3e298f9d441c09c064a7f5655980b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41327 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:35.084138 3007 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b96b32dada445ccbe9c4eaa0ddee050 with delete_type TABLET_DATA_TOMBSTONED (TS fb4fc23dca0c4de69aef5061d3a67b2f not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:59142
I20240503 01:59:35.084364 5497 tablet_replica.cc:330] T 1b96b32dada445ccbe9c4eaa0ddee050 P fb4fc23dca0c4de69aef5061d3a67b2f: stopping tablet replica
I20240503 01:59:35.084504 5497 raft_consensus.cc:2229] T 1b96b32dada445ccbe9c4eaa0ddee050 P fb4fc23dca0c4de69aef5061d3a67b2f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:35.084599 5497 raft_consensus.cc:2258] T 1b96b32dada445ccbe9c4eaa0ddee050 P fb4fc23dca0c4de69aef5061d3a67b2f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:35.085244 5497 ts_tablet_manager.cc:1875] T 1b96b32dada445ccbe9c4eaa0ddee050 P fb4fc23dca0c4de69aef5061d3a67b2f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:35.087218 5497 ts_tablet_manager.cc:1888] T 1b96b32dada445ccbe9c4eaa0ddee050 P fb4fc23dca0c4de69aef5061d3a67b2f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 01:59:35.087294 5497 log.cc:1196] T 1b96b32dada445ccbe9c4eaa0ddee050 P fb4fc23dca0c4de69aef5061d3a67b2f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/1b96b32dada445ccbe9c4eaa0ddee050
I20240503 01:59:35.087800 2740 catalog_manager.cc:4830] TS fb4fc23dca0c4de69aef5061d3a67b2f (127.0.99.130:34163): tablet 1b96b32dada445ccbe9c4eaa0ddee050 (table rebalance_test_table_0 [id=da03de56cae54136b05045054d83001e]) successfully deleted
I20240503 01:59:35.186108 4037 consensus_queue.cc:237] T fc4a011058e447499e3d651c73f0b881 P ad0b1cf28c9c452db2230e4cd69363bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ad0b1cf28c9c452db2230e4cd69363bc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44037 } attrs { promote: false } }
I20240503 01:59:35.186816 4904 raft_consensus.cc:2932] T fc4a011058e447499e3d651c73f0b881 P ad0b1cf28c9c452db2230e4cd69363bc [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER fb4fc23dca0c4de69aef5061d3a67b2f (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ad0b1cf28c9c452db2230e4cd69363bc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44037 } attrs { promote: false } } }
I20240503 01:59:35.188230 2741 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fc4a011058e447499e3d651c73f0b881 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 01:59:35.189229 2755 catalog_manager.cc:5484] T fc4a011058e447499e3d651c73f0b881 P ad0b1cf28c9c452db2230e4cd69363bc reported cstate change: config changed from index 5 to 7, VOTER fb4fc23dca0c4de69aef5061d3a67b2f (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "ad0b1cf28c9c452db2230e4cd69363bc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ad0b1cf28c9c452db2230e4cd69363bc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44037 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:35.191648 3007 tablet_service.cc:1514] Processing DeleteTablet for tablet fc4a011058e447499e3d651c73f0b881 with delete_type TABLET_DATA_TOMBSTONED (TS fb4fc23dca0c4de69aef5061d3a67b2f not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:59142
I20240503 01:59:35.191897 5497 tablet_replica.cc:330] T fc4a011058e447499e3d651c73f0b881 P fb4fc23dca0c4de69aef5061d3a67b2f: stopping tablet replica
I20240503 01:59:35.192014 5497 raft_consensus.cc:2229] T fc4a011058e447499e3d651c73f0b881 P fb4fc23dca0c4de69aef5061d3a67b2f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:35.192124 5497 raft_consensus.cc:2258] T fc4a011058e447499e3d651c73f0b881 P fb4fc23dca0c4de69aef5061d3a67b2f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:35.192773 5497 ts_tablet_manager.cc:1875] T fc4a011058e447499e3d651c73f0b881 P fb4fc23dca0c4de69aef5061d3a67b2f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:35.194572 5497 ts_tablet_manager.cc:1888] T fc4a011058e447499e3d651c73f0b881 P fb4fc23dca0c4de69aef5061d3a67b2f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 01:59:35.194644 5497 log.cc:1196] T fc4a011058e447499e3d651c73f0b881 P fb4fc23dca0c4de69aef5061d3a67b2f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/fc4a011058e447499e3d651c73f0b881
I20240503 01:59:35.195216 2740 catalog_manager.cc:4830] TS fb4fc23dca0c4de69aef5061d3a67b2f (127.0.99.130:34163): tablet fc4a011058e447499e3d651c73f0b881 (table rebalance_test_table_2 [id=c045b78f2dc741a4b00a4b13ac99530e]) successfully deleted
I20240503 01:59:35.346375 4017 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:35.347110 4540 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:35.348065 2876 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:35.351622 4279 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:35.358609 4148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:35.365214 4410 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:35.383787 3007 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:35.456732 4017 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:35.459426 4148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:35.459631 4540 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:35.462085 4410 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:35.465122 2876 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:35.475800 3007 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:35.476318 4279 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:35.729065 4540 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:35.729852 4148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:35.733258 2876 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:35.736364 4279 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:35.742437 4017 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:35.758237 4410 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:35.771158 3007 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:35.879251 4017 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:35.879873 2876 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:35.883307 4279 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:35.890130 4148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:35.896571 4540 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:35.901669 3007 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:35.903062 4410 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:36.003931 4540 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:36.005641 4148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:36.006820 2876 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:36.007442 4279 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:36.016764 4017 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:36.017096 3007 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:36.032858 4410 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
667273152a48411eb72e6817f2e34b09 | 127.0.99.190:43851 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:35751 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
4d61ec80bf4c475ea86ae2bcb866fd05 | 127.0.99.129:38011 | HEALTHY | <none> | 3 | 0
6c8442cedca54ad589e0e050282afee3 | 127.0.99.135:43573 | HEALTHY | <none> | 2 | 0
ad0b1cf28c9c452db2230e4cd69363bc | 127.0.99.131:44037 | HEALTHY | <none> | 2 | 0
c8a3e298f9d441c09c064a7f5655980b | 127.0.99.132:41327 | HEALTHY | <none> | 2 | 0
e296c081ab4a414294313ab9883cae84 | 127.0.99.133:41953 | HEALTHY | <none> | 2 | 0
e4c1a8048e4647b6936035bc2fc22b66 | 127.0.99.134:46423 | HEALTHY | <none> | 2 | 0
fb4fc23dca0c4de69aef5061d3a67b2f | 127.0.99.130:34163 | HEALTHY | <none> | 3 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.0.99.129 | experimental | 127.0.99.129:38011
local_ip_for_outbound_sockets | 127.0.99.130 | experimental | 127.0.99.130:34163
local_ip_for_outbound_sockets | 127.0.99.131 | experimental | 127.0.99.131:44037
local_ip_for_outbound_sockets | 127.0.99.132 | experimental | 127.0.99.132:41327
local_ip_for_outbound_sockets | 127.0.99.133 | experimental | 127.0.99.133:41953
local_ip_for_outbound_sockets | 127.0.99.134 | experimental | 127.0.99.134:46423
local_ip_for_outbound_sockets | 127.0.99.135 | experimental | 127.0.99.135:43573
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.129:38011
server_dump_info_path | /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.99.130:34163
server_dump_info_path | /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.131:44037
server_dump_info_path | /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.0.99.132:41327
server_dump_info_path | /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.133:41953
server_dump_info_path | /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.134:46423
server_dump_info_path | /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.0.99.135:43573
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:35751 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 01:59:36.057979 398 log_verifier.cc:126] Checking tablet 0cbba6e3ce5b4de5b5ec9164e50f784a
I20240503 01:59:36.069136 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 0cbba6e3ce5b4de5b5ec9164e50f784a
I20240503 01:59:36.069240 398 log_verifier.cc:126] Checking tablet 0f440670fb6f4fccbc3ee8554dadb770
I20240503 01:59:36.077852 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 0f440670fb6f4fccbc3ee8554dadb770
I20240503 01:59:36.077919 398 log_verifier.cc:126] Checking tablet 1b96b32dada445ccbe9c4eaa0ddee050
I20240503 01:59:36.087177 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 1b96b32dada445ccbe9c4eaa0ddee050
I20240503 01:59:36.087241 398 log_verifier.cc:126] Checking tablet 201397ae513b4c40bd2e53197616984e
I20240503 01:59:36.096554 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 201397ae513b4c40bd2e53197616984e
I20240503 01:59:36.096618 398 log_verifier.cc:126] Checking tablet 365919d2a8d94985bc031ab16eefe0d8
I20240503 01:59:36.105695 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 365919d2a8d94985bc031ab16eefe0d8
I20240503 01:59:36.105755 398 log_verifier.cc:126] Checking tablet 7056f7fb504545d19641ea2c58ebf6be
I20240503 01:59:36.114655 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 7056f7fb504545d19641ea2c58ebf6be
I20240503 01:59:36.114722 398 log_verifier.cc:126] Checking tablet 7fe4b04830cd42a292f72f27b788c75a
I20240503 01:59:36.122802 398 log_verifier.cc:177] Verified matching terms for 5 ops in tablet 7fe4b04830cd42a292f72f27b788c75a
I20240503 01:59:36.122866 398 log_verifier.cc:126] Checking tablet 87da933cf2034ccebc5a78c970acb563
I20240503 01:59:36.131474 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 87da933cf2034ccebc5a78c970acb563
I20240503 01:59:36.131537 398 log_verifier.cc:126] Checking tablet 9bfd209000fd468385a66c8fa5b2edc8
I20240503 01:59:36.140403 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 9bfd209000fd468385a66c8fa5b2edc8
I20240503 01:59:36.140470 398 log_verifier.cc:126] Checking tablet a6f5711f3c9c4f3ca60893103ee7be40
I20240503 01:59:36.149048 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet a6f5711f3c9c4f3ca60893103ee7be40
I20240503 01:59:36.149111 398 log_verifier.cc:126] Checking tablet b2ba35f5c3084e08a42c6f3381aa1cb5
I20240503 01:59:36.158164 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet b2ba35f5c3084e08a42c6f3381aa1cb5
I20240503 01:59:36.158224 398 log_verifier.cc:126] Checking tablet b60c4ffc1dac4e5e8a2881cd5aa61547
I20240503 01:59:36.167254 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet b60c4ffc1dac4e5e8a2881cd5aa61547
I20240503 01:59:36.167330 398 log_verifier.cc:126] Checking tablet bd3627afefd74c39b43a4ff26da65c39
I20240503 01:59:36.177392 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet bd3627afefd74c39b43a4ff26da65c39
I20240503 01:59:36.177459 398 log_verifier.cc:126] Checking tablet c3c2ff70515f49cf9f51b8db5e130cba
I20240503 01:59:36.186329 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet c3c2ff70515f49cf9f51b8db5e130cba
I20240503 01:59:36.186417 398 log_verifier.cc:126] Checking tablet eefe13b91700485bbcc398b37cec51b4
I20240503 01:59:36.195588 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet eefe13b91700485bbcc398b37cec51b4
I20240503 01:59:36.195645 398 log_verifier.cc:126] Checking tablet fc4a011058e447499e3d651c73f0b881
I20240503 01:59:36.204428 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet fc4a011058e447499e3d651c73f0b881
I20240503 01:59:36.206265 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
/tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:0
--local_ip_for_outbound_sockets=127.0.99.136
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43851
--builtin_ntp_servers=127.0.99.148:35751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 01:59:36.316406 5711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:36.316638 5711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:36.316706 5711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:36.320009 5711 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:36.320091 5711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:36.320179 5711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240503 01:59:36.324025 5711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43851
--never_fsync=true
--heap_profile_path=/tmp/kudu.5711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:36.325019 5711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:36.326133 5711 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:36.334441 5716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:36.334460 5717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:36.334493 5719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:36.334837 5711 server_base.cc:1034] running on GCE node
I20240503 01:59:36.335232 5711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:36.335835 5711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:36.336997 5711 hybrid_clock.cc:648] HybridClock initialized: now 1714701576336978 us; error 35 us; skew 500 ppm
I20240503 01:59:36.338842 5711 webserver.cc:460] Webserver started at http://127.0.99.136:34875/ using document root <none> and password file <none>
I20240503 01:59:36.339403 5711 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:36.339476 5711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:36.339668 5711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:36.341740 5711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "4d3846546549444596d290ff0e21932e"
format_stamp: "Formatted at 2024-05-03 01:59:36 on dist-test-slave-pz5w"
I20240503 01:59:36.342255 5711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "4d3846546549444596d290ff0e21932e"
format_stamp: "Formatted at 2024-05-03 01:59:36 on dist-test-slave-pz5w"
I20240503 01:59:36.346292 5711 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 01:59:36.348932 5725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:36.350023 5711 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 01:59:36.350136 5711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "4d3846546549444596d290ff0e21932e"
format_stamp: "Formatted at 2024-05-03 01:59:36 on dist-test-slave-pz5w"
I20240503 01:59:36.350232 5711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:36.363479 5711 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:36.364179 5711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:36.364344 5711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:36.364907 5711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:36.365813 5711 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:36.365861 5711 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:36.365916 5711 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:36.365947 5711 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:36.374944 5711 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:43135
I20240503 01:59:36.374969 5838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:43135 every 8 connection(s)
I20240503 01:59:36.375906 5711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 01:59:36.382009 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu as pid 5711
I20240503 01:59:36.382122 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 01:59:36.384975 5839 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43851
I20240503 01:59:36.385200 5839 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:36.385763 5839 heartbeater.cc:511] Master 127.0.99.190:43851 requested a full tablet report, sending...
I20240503 01:59:36.386804 2752 ts_manager.cc:194] Registered new tserver with Master: 4d3846546549444596d290ff0e21932e (127.0.99.136:43135)
I20240503 01:59:36.387583 2752 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:54769
I20240503 01:59:36.577512 4017 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:36.577577 4540 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:36.586855 5770 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:36.589423 4279 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:36.592758 3007 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:36.592859 4410 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:36.598245 4148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:36.598409 2876 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:36.710130 5770 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:36.710341 4148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:36.715642 4017 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:36.717201 4540 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:36.728034 4410 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:36.737629 4279 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:36.737689 3007 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:36.745780 2876 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:36.778539 3028 consensus_queue.cc:237] T 87da933cf2034ccebc5a78c970acb563 P fb4fc23dca0c4de69aef5061d3a67b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "4d3846546549444596d290ff0e21932e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43135 } attrs { promote: true } }
I20240503 01:59:36.779521 5932 raft_consensus.cc:2932] T 87da933cf2034ccebc5a78c970acb563 P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4d3846546549444596d290ff0e21932e (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "4d3846546549444596d290ff0e21932e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43135 } attrs { promote: true } } }
I20240503 01:59:36.782006 2752 catalog_manager.cc:5484] T 87da933cf2034ccebc5a78c970acb563 P fb4fc23dca0c4de69aef5061d3a67b2f reported cstate change: config changed from index -1 to 4, NON_VOTER 4d3846546549444596d290ff0e21932e (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d3846546549444596d290ff0e21932e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43135 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 01:59:36.784334 2963 consensus_peers.cc:485] T 87da933cf2034ccebc5a78c970acb563 P fb4fc23dca0c4de69aef5061d3a67b2f -> Peer 4d3846546549444596d290ff0e21932e (127.0.99.136:43135): Couldn't send request to peer 4d3846546549444596d290ff0e21932e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 87da933cf2034ccebc5a78c970acb563. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:36.789723 2897 consensus_queue.cc:237] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d61ec80bf4c475ea86ae2bcb866fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "4d3846546549444596d290ff0e21932e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43135 } attrs { promote: true } }
I20240503 01:59:36.790717 5945 raft_consensus.cc:2932] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4d3846546549444596d290ff0e21932e (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "4d3846546549444596d290ff0e21932e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43135 } attrs { promote: true } } }
I20240503 01:59:36.793651 2750 catalog_manager.cc:5484] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d61ec80bf4c475ea86ae2bcb866fd05 reported cstate change: config changed from index -1 to 4, NON_VOTER 4d3846546549444596d290ff0e21932e (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d3846546549444596d290ff0e21932e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43135 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 01:59:36.795527 2832 consensus_peers.cc:485] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d61ec80bf4c475ea86ae2bcb866fd05 -> Peer 4d3846546549444596d290ff0e21932e (127.0.99.136:43135): Couldn't send request to peer 4d3846546549444596d290ff0e21932e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b60c4ffc1dac4e5e8a2881cd5aa61547. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.193733 5968 ts_tablet_manager.cc:902] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d3846546549444596d290ff0e21932e: Initiating tablet copy from peer 4d61ec80bf4c475ea86ae2bcb866fd05 (127.0.99.129:38011)
I20240503 01:59:37.194523 5968 tablet_copy_client.cc:308] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d3846546549444596d290ff0e21932e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:38011
I20240503 01:59:37.198760 2916 tablet_copy_service.cc:140] P 4d61ec80bf4c475ea86ae2bcb866fd05: Received BeginTabletCopySession request for tablet b60c4ffc1dac4e5e8a2881cd5aa61547 from peer 4d3846546549444596d290ff0e21932e ({username='slave'} at 127.0.99.136:39483)
I20240503 01:59:37.198899 2916 tablet_copy_service.cc:161] P 4d61ec80bf4c475ea86ae2bcb866fd05: Beginning new tablet copy session on tablet b60c4ffc1dac4e5e8a2881cd5aa61547 from peer 4d3846546549444596d290ff0e21932e at {username='slave'} at 127.0.99.136:39483: session id = 4d3846546549444596d290ff0e21932e-b60c4ffc1dac4e5e8a2881cd5aa61547
I20240503 01:59:37.199759 2916 tablet_copy_source_session.cc:205] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d61ec80bf4c475ea86ae2bcb866fd05: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:37.203279 5968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b60c4ffc1dac4e5e8a2881cd5aa61547. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:37.208930 5968 tablet_copy_client.cc:783] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d3846546549444596d290ff0e21932e: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:37.209162 5968 tablet_copy_client.cc:647] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d3846546549444596d290ff0e21932e: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:37.211192 5968 tablet_copy_client.cc:523] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d3846546549444596d290ff0e21932e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:37.213001 5968 tablet_bootstrap.cc:492] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d3846546549444596d290ff0e21932e: Bootstrap starting.
I20240503 01:59:37.218801 5968 log.cc:824] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d3846546549444596d290ff0e21932e: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:37.241384 5968 tablet_bootstrap.cc:492] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d3846546549444596d290ff0e21932e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:37.241722 5968 tablet_bootstrap.cc:492] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d3846546549444596d290ff0e21932e: Bootstrap complete.
I20240503 01:59:37.242019 5968 ts_tablet_manager.cc:1367] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d3846546549444596d290ff0e21932e: Time spent bootstrapping tablet: real 0.029s user 0.028s sys 0.000s
I20240503 01:59:37.244015 5968 raft_consensus.cc:348] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d3846546549444596d290ff0e21932e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "4d3846546549444596d290ff0e21932e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43135 } attrs { promote: true } }
I20240503 01:59:37.244236 5968 raft_consensus.cc:729] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d3846546549444596d290ff0e21932e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4d3846546549444596d290ff0e21932e, State: Initialized, Role: LEARNER
I20240503 01:59:37.244568 5968 consensus_queue.cc:260] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d3846546549444596d290ff0e21932e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "4d3846546549444596d290ff0e21932e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43135 } attrs { promote: true } }
I20240503 01:59:37.245337 5839 heartbeater.cc:503] Master 127.0.99.190:43851 was elected leader, sending a full tablet report...
I20240503 01:59:37.245393 5968 ts_tablet_manager.cc:1398] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d3846546549444596d290ff0e21932e: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20240503 01:59:37.246203 2916 tablet_copy_service.cc:342] P 4d61ec80bf4c475ea86ae2bcb866fd05: Request end of tablet copy session 4d3846546549444596d290ff0e21932e-b60c4ffc1dac4e5e8a2881cd5aa61547 received from {username='slave'} at 127.0.99.136:39483
I20240503 01:59:37.246287 2916 tablet_copy_service.cc:433] P 4d61ec80bf4c475ea86ae2bcb866fd05: ending tablet copy session 4d3846546549444596d290ff0e21932e-b60c4ffc1dac4e5e8a2881cd5aa61547 on tablet b60c4ffc1dac4e5e8a2881cd5aa61547 with peer 4d3846546549444596d290ff0e21932e
I20240503 01:59:37.405606 5968 ts_tablet_manager.cc:902] T 87da933cf2034ccebc5a78c970acb563 P 4d3846546549444596d290ff0e21932e: Initiating tablet copy from peer fb4fc23dca0c4de69aef5061d3a67b2f (127.0.99.130:34163)
I20240503 01:59:37.406070 5968 tablet_copy_client.cc:308] T 87da933cf2034ccebc5a78c970acb563 P 4d3846546549444596d290ff0e21932e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:34163
I20240503 01:59:37.409423 3047 tablet_copy_service.cc:140] P fb4fc23dca0c4de69aef5061d3a67b2f: Received BeginTabletCopySession request for tablet 87da933cf2034ccebc5a78c970acb563 from peer 4d3846546549444596d290ff0e21932e ({username='slave'} at 127.0.99.136:38963)
I20240503 01:59:37.409547 3047 tablet_copy_service.cc:161] P fb4fc23dca0c4de69aef5061d3a67b2f: Beginning new tablet copy session on tablet 87da933cf2034ccebc5a78c970acb563 from peer 4d3846546549444596d290ff0e21932e at {username='slave'} at 127.0.99.136:38963: session id = 4d3846546549444596d290ff0e21932e-87da933cf2034ccebc5a78c970acb563
I20240503 01:59:37.410291 3047 tablet_copy_source_session.cc:205] T 87da933cf2034ccebc5a78c970acb563 P fb4fc23dca0c4de69aef5061d3a67b2f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:37.411219 5968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87da933cf2034ccebc5a78c970acb563. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:37.413296 5968 tablet_copy_client.cc:783] T 87da933cf2034ccebc5a78c970acb563 P 4d3846546549444596d290ff0e21932e: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:37.413465 5968 tablet_copy_client.cc:647] T 87da933cf2034ccebc5a78c970acb563 P 4d3846546549444596d290ff0e21932e: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:37.414733 5968 tablet_copy_client.cc:523] T 87da933cf2034ccebc5a78c970acb563 P 4d3846546549444596d290ff0e21932e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:37.415895 5968 tablet_bootstrap.cc:492] T 87da933cf2034ccebc5a78c970acb563 P 4d3846546549444596d290ff0e21932e: Bootstrap starting.
I20240503 01:59:37.433283 5968 tablet_bootstrap.cc:492] T 87da933cf2034ccebc5a78c970acb563 P 4d3846546549444596d290ff0e21932e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:37.433594 5968 tablet_bootstrap.cc:492] T 87da933cf2034ccebc5a78c970acb563 P 4d3846546549444596d290ff0e21932e: Bootstrap complete.
I20240503 01:59:37.433795 5968 ts_tablet_manager.cc:1367] T 87da933cf2034ccebc5a78c970acb563 P 4d3846546549444596d290ff0e21932e: Time spent bootstrapping tablet: real 0.018s user 0.018s sys 0.000s
I20240503 01:59:37.434367 5968 raft_consensus.cc:348] T 87da933cf2034ccebc5a78c970acb563 P 4d3846546549444596d290ff0e21932e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "4d3846546549444596d290ff0e21932e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43135 } attrs { promote: true } }
I20240503 01:59:37.434495 5968 raft_consensus.cc:729] T 87da933cf2034ccebc5a78c970acb563 P 4d3846546549444596d290ff0e21932e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4d3846546549444596d290ff0e21932e, State: Initialized, Role: LEARNER
I20240503 01:59:37.434604 5968 consensus_queue.cc:260] T 87da933cf2034ccebc5a78c970acb563 P 4d3846546549444596d290ff0e21932e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "4d3846546549444596d290ff0e21932e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43135 } attrs { promote: true } }
I20240503 01:59:37.434863 5968 ts_tablet_manager.cc:1398] T 87da933cf2034ccebc5a78c970acb563 P 4d3846546549444596d290ff0e21932e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:37.435633 3047 tablet_copy_service.cc:342] P fb4fc23dca0c4de69aef5061d3a67b2f: Request end of tablet copy session 4d3846546549444596d290ff0e21932e-87da933cf2034ccebc5a78c970acb563 received from {username='slave'} at 127.0.99.136:38963
I20240503 01:59:37.435710 3047 tablet_copy_service.cc:433] P fb4fc23dca0c4de69aef5061d3a67b2f: ending tablet copy session 4d3846546549444596d290ff0e21932e-87da933cf2034ccebc5a78c970acb563 on tablet 87da933cf2034ccebc5a78c970acb563 with peer 4d3846546549444596d290ff0e21932e
I20240503 01:59:37.789005 5793 raft_consensus.cc:1203] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d3846546549444596d290ff0e21932e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:38.014945 5793 raft_consensus.cc:1203] T 87da933cf2034ccebc5a78c970acb563 P 4d3846546549444596d290ff0e21932e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:38.340055 6048 raft_consensus.cc:1050] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d61ec80bf4c475ea86ae2bcb866fd05: attempting to promote NON_VOTER 4d3846546549444596d290ff0e21932e to VOTER
I20240503 01:59:38.340579 6048 consensus_queue.cc:237] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d61ec80bf4c475ea86ae2bcb866fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "4d3846546549444596d290ff0e21932e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43135 } attrs { promote: false } }
I20240503 01:59:38.342264 5793 raft_consensus.cc:1261] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d3846546549444596d290ff0e21932e [term 1 LEARNER]: Refusing update from remote peer 4d61ec80bf4c475ea86ae2bcb866fd05: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:38.342819 6049 consensus_queue.cc:1035] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d61ec80bf4c475ea86ae2bcb866fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3846546549444596d290ff0e21932e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43135 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:38.346300 6048 raft_consensus.cc:2932] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4d3846546549444596d290ff0e21932e (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "4d3846546549444596d290ff0e21932e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43135 } attrs { promote: false } } }
I20240503 01:59:38.347007 5793 raft_consensus.cc:2932] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d3846546549444596d290ff0e21932e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4d3846546549444596d290ff0e21932e (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "4d3846546549444596d290ff0e21932e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43135 } attrs { promote: false } } }
I20240503 01:59:38.361475 2750 catalog_manager.cc:5484] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d61ec80bf4c475ea86ae2bcb866fd05 reported cstate change: config changed from index 4 to 5, 4d3846546549444596d290ff0e21932e (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d3846546549444596d290ff0e21932e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43135 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:38.396024 6027 raft_consensus.cc:1050] T 87da933cf2034ccebc5a78c970acb563 P fb4fc23dca0c4de69aef5061d3a67b2f: attempting to promote NON_VOTER 4d3846546549444596d290ff0e21932e to VOTER
I20240503 01:59:38.396489 6027 consensus_queue.cc:237] T 87da933cf2034ccebc5a78c970acb563 P fb4fc23dca0c4de69aef5061d3a67b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "4d3846546549444596d290ff0e21932e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43135 } attrs { promote: false } }
I20240503 01:59:38.397779 5793 raft_consensus.cc:1261] T 87da933cf2034ccebc5a78c970acb563 P 4d3846546549444596d290ff0e21932e [term 1 LEARNER]: Refusing update from remote peer fb4fc23dca0c4de69aef5061d3a67b2f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:38.398196 6025 consensus_queue.cc:1035] T 87da933cf2034ccebc5a78c970acb563 P fb4fc23dca0c4de69aef5061d3a67b2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3846546549444596d290ff0e21932e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43135 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:38.401495 5793 raft_consensus.cc:2932] T 87da933cf2034ccebc5a78c970acb563 P 4d3846546549444596d290ff0e21932e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4d3846546549444596d290ff0e21932e (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "4d3846546549444596d290ff0e21932e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43135 } attrs { promote: false } } }
I20240503 01:59:38.400739 6027 raft_consensus.cc:2932] T 87da933cf2034ccebc5a78c970acb563 P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4d3846546549444596d290ff0e21932e (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "4d3846546549444596d290ff0e21932e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43135 } attrs { promote: false } } }
I20240503 01:59:38.458200 2752 catalog_manager.cc:5484] T 87da933cf2034ccebc5a78c970acb563 P fb4fc23dca0c4de69aef5061d3a67b2f reported cstate change: config changed from index 4 to 5, 4d3846546549444596d290ff0e21932e (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d3846546549444596d290ff0e21932e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43135 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:38.509583 2897 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b60c4ffc1dac4e5e8a2881cd5aa61547"
dest_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51648
I20240503 01:59:38.509734 2897 raft_consensus.cc:595] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:38.521952 3028 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "87da933cf2034ccebc5a78c970acb563"
dest_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59872
I20240503 01:59:38.522109 3028 raft_consensus.cc:595] T 87da933cf2034ccebc5a78c970acb563 P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:38.734388 2897 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b60c4ffc1dac4e5e8a2881cd5aa61547"
dest_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43486
I20240503 01:59:38.734539 2897 raft_consensus.cc:595] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:38.738080 6048 raft_consensus.cc:979] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d61ec80bf4c475ea86ae2bcb866fd05: : Instructing follower 4d3846546549444596d290ff0e21932e to start an election
I20240503 01:59:38.738186 6048 raft_consensus.cc:1067] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 LEADER]: Signalling peer 4d3846546549444596d290ff0e21932e to start an election
I20240503 01:59:38.738688 5793 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b60c4ffc1dac4e5e8a2881cd5aa61547"
dest_uuid: "4d3846546549444596d290ff0e21932e"
from {username='slave'} at 127.0.99.129:48365
I20240503 01:59:38.738876 5793 raft_consensus.cc:482] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d3846546549444596d290ff0e21932e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 01:59:38.738986 5793 raft_consensus.cc:3037] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d3846546549444596d290ff0e21932e [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:38.739890 5793 raft_consensus.cc:504] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d3846546549444596d290ff0e21932e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "4d3846546549444596d290ff0e21932e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43135 } attrs { promote: false } }
I20240503 01:59:38.740931 5793 leader_election.cc:288] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d3846546549444596d290ff0e21932e [CANDIDATE]: Term 2 election: Requested vote from peers 4d61ec80bf4c475ea86ae2bcb866fd05 (127.0.99.129:38011)
I20240503 01:59:38.741376 2897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b60c4ffc1dac4e5e8a2881cd5aa61547" candidate_uuid: "4d3846546549444596d290ff0e21932e" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05"
I20240503 01:59:38.741497 2897 raft_consensus.cc:3032] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 01:59:38.741542 2897 raft_consensus.cc:729] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4d61ec80bf4c475ea86ae2bcb866fd05, State: Running, Role: LEADER
I20240503 01:59:38.741708 2897 consensus_queue.cc:260] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d61ec80bf4c475ea86ae2bcb866fd05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "4d3846546549444596d290ff0e21932e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43135 } attrs { promote: false } }
I20240503 01:59:38.741838 2897 raft_consensus.cc:3037] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:38.742578 2897 raft_consensus.cc:2454] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3846546549444596d290ff0e21932e in term 2.
I20240503 01:59:38.742955 5726 leader_election.cc:302] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d3846546549444596d290ff0e21932e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4d3846546549444596d290ff0e21932e, 4d61ec80bf4c475ea86ae2bcb866fd05; no voters:
I20240503 01:59:38.743180 5985 raft_consensus.cc:2781] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d3846546549444596d290ff0e21932e [term 2 FOLLOWER]: Leader election won for term 2
I20240503 01:59:38.743361 5985 raft_consensus.cc:686] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d3846546549444596d290ff0e21932e [term 2 LEADER]: Becoming Leader. State: Replica: 4d3846546549444596d290ff0e21932e, State: Running, Role: LEADER
I20240503 01:59:38.743660 5985 consensus_queue.cc:237] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d3846546549444596d290ff0e21932e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "4d3846546549444596d290ff0e21932e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43135 } attrs { promote: false } }
I20240503 01:59:38.746630 2750 catalog_manager.cc:5484] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d3846546549444596d290ff0e21932e reported cstate change: term changed from 1 to 2, leader changed from 4d61ec80bf4c475ea86ae2bcb866fd05 (127.0.99.129) to 4d3846546549444596d290ff0e21932e (127.0.99.136). New cstate: current_term: 2 leader_uuid: "4d3846546549444596d290ff0e21932e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d3846546549444596d290ff0e21932e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43135 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:38.748055 3028 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "87da933cf2034ccebc5a78c970acb563"
dest_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45824
I20240503 01:59:38.748199 3028 raft_consensus.cc:595] T 87da933cf2034ccebc5a78c970acb563 P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 LEADER]: Received request to transfer leadership
W20240503 01:59:38.753610 2741 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet b60c4ffc1dac4e5e8a2881cd5aa61547 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 4d3846546549444596d290ff0e21932e (127.0.99.136:43135) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20240503 01:59:38.812510 6026 raft_consensus.cc:979] T 87da933cf2034ccebc5a78c970acb563 P fb4fc23dca0c4de69aef5061d3a67b2f: : Instructing follower 4d3846546549444596d290ff0e21932e to start an election
I20240503 01:59:38.812626 6026 raft_consensus.cc:1067] T 87da933cf2034ccebc5a78c970acb563 P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 LEADER]: Signalling peer 4d3846546549444596d290ff0e21932e to start an election
I20240503 01:59:38.813107 5793 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "87da933cf2034ccebc5a78c970acb563"
dest_uuid: "4d3846546549444596d290ff0e21932e"
from {username='slave'} at 127.0.99.130:43701
I20240503 01:59:38.813274 5793 raft_consensus.cc:482] T 87da933cf2034ccebc5a78c970acb563 P 4d3846546549444596d290ff0e21932e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 01:59:38.813341 5793 raft_consensus.cc:3037] T 87da933cf2034ccebc5a78c970acb563 P 4d3846546549444596d290ff0e21932e [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:38.814172 5793 raft_consensus.cc:504] T 87da933cf2034ccebc5a78c970acb563 P 4d3846546549444596d290ff0e21932e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "4d3846546549444596d290ff0e21932e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43135 } attrs { promote: false } }
I20240503 01:59:38.814525 5793 leader_election.cc:288] T 87da933cf2034ccebc5a78c970acb563 P 4d3846546549444596d290ff0e21932e [CANDIDATE]: Term 2 election: Requested vote from peers fb4fc23dca0c4de69aef5061d3a67b2f (127.0.99.130:34163)
I20240503 01:59:38.814920 3028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87da933cf2034ccebc5a78c970acb563" candidate_uuid: "4d3846546549444596d290ff0e21932e" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f"
I20240503 01:59:38.815048 3028 raft_consensus.cc:3032] T 87da933cf2034ccebc5a78c970acb563 P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 LEADER]: Stepping down as leader of term 1
I20240503 01:59:38.815078 3028 raft_consensus.cc:729] T 87da933cf2034ccebc5a78c970acb563 P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fb4fc23dca0c4de69aef5061d3a67b2f, State: Running, Role: LEADER
I20240503 01:59:38.815279 3028 consensus_queue.cc:260] T 87da933cf2034ccebc5a78c970acb563 P fb4fc23dca0c4de69aef5061d3a67b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "4d3846546549444596d290ff0e21932e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43135 } attrs { promote: false } }
I20240503 01:59:38.815407 3028 raft_consensus.cc:3037] T 87da933cf2034ccebc5a78c970acb563 P fb4fc23dca0c4de69aef5061d3a67b2f [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:38.816150 3028 raft_consensus.cc:2454] T 87da933cf2034ccebc5a78c970acb563 P fb4fc23dca0c4de69aef5061d3a67b2f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3846546549444596d290ff0e21932e in term 2.
I20240503 01:59:38.816498 5727 leader_election.cc:302] T 87da933cf2034ccebc5a78c970acb563 P 4d3846546549444596d290ff0e21932e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4d3846546549444596d290ff0e21932e, fb4fc23dca0c4de69aef5061d3a67b2f; no voters:
I20240503 01:59:38.816689 5985 raft_consensus.cc:2781] T 87da933cf2034ccebc5a78c970acb563 P 4d3846546549444596d290ff0e21932e [term 2 FOLLOWER]: Leader election won for term 2
I20240503 01:59:38.816781 5985 raft_consensus.cc:686] T 87da933cf2034ccebc5a78c970acb563 P 4d3846546549444596d290ff0e21932e [term 2 LEADER]: Becoming Leader. State: Replica: 4d3846546549444596d290ff0e21932e, State: Running, Role: LEADER
I20240503 01:59:38.816927 5985 consensus_queue.cc:237] T 87da933cf2034ccebc5a78c970acb563 P 4d3846546549444596d290ff0e21932e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } } peers { permanent_uuid: "4d3846546549444596d290ff0e21932e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43135 } attrs { promote: false } }
I20240503 01:59:38.818406 2750 catalog_manager.cc:5484] T 87da933cf2034ccebc5a78c970acb563 P 4d3846546549444596d290ff0e21932e reported cstate change: term changed from 1 to 2, leader changed from fb4fc23dca0c4de69aef5061d3a67b2f (127.0.99.130) to 4d3846546549444596d290ff0e21932e (127.0.99.136). New cstate: current_term: 2 leader_uuid: "4d3846546549444596d290ff0e21932e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d3846546549444596d290ff0e21932e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43135 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:39.242565 2897 raft_consensus.cc:1261] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 2 FOLLOWER]: Refusing update from remote peer 4d3846546549444596d290ff0e21932e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 01:59:39.243157 5985 consensus_queue.cc:1035] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d3846546549444596d290ff0e21932e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d61ec80bf4c475ea86ae2bcb866fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38011 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:39.399999 5793 consensus_queue.cc:237] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d3846546549444596d290ff0e21932e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4d3846546549444596d290ff0e21932e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43135 } attrs { promote: false } }
I20240503 01:59:39.400774 5985 raft_consensus.cc:2932] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d3846546549444596d290ff0e21932e [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 4d61ec80bf4c475ea86ae2bcb866fd05 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4d3846546549444596d290ff0e21932e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43135 } attrs { promote: false } } }
I20240503 01:59:39.402158 2741 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b60c4ffc1dac4e5e8a2881cd5aa61547 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 01:59:39.402957 2752 catalog_manager.cc:5484] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d3846546549444596d290ff0e21932e reported cstate change: config changed from index 5 to 7, VOTER 4d61ec80bf4c475ea86ae2bcb866fd05 (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "4d3846546549444596d290ff0e21932e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4d3846546549444596d290ff0e21932e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43135 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:39.405663 2876 tablet_service.cc:1514] Processing DeleteTablet for tablet b60c4ffc1dac4e5e8a2881cd5aa61547 with delete_type TABLET_DATA_TOMBSTONED (TS 4d61ec80bf4c475ea86ae2bcb866fd05 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:33814
I20240503 01:59:39.406008 6138 tablet_replica.cc:330] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d61ec80bf4c475ea86ae2bcb866fd05: stopping tablet replica
I20240503 01:59:39.406133 6138 raft_consensus.cc:2229] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:39.406241 6138 raft_consensus.cc:2258] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d61ec80bf4c475ea86ae2bcb866fd05 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:39.407384 6138 ts_tablet_manager.cc:1875] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d61ec80bf4c475ea86ae2bcb866fd05: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:39.409271 6138 ts_tablet_manager.cc:1888] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d61ec80bf4c475ea86ae2bcb866fd05: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 01:59:39.409348 6138 log.cc:1196] T b60c4ffc1dac4e5e8a2881cd5aa61547 P 4d61ec80bf4c475ea86ae2bcb866fd05: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/b60c4ffc1dac4e5e8a2881cd5aa61547
I20240503 01:59:39.409934 2739 catalog_manager.cc:4830] TS 4d61ec80bf4c475ea86ae2bcb866fd05 (127.0.99.129:38011): tablet b60c4ffc1dac4e5e8a2881cd5aa61547 (table rebalance_test_table_1 [id=c3c40e684e4449ba86b043703f35788b]) successfully deleted
I20240503 01:59:39.429265 3028 raft_consensus.cc:1261] T 87da933cf2034ccebc5a78c970acb563 P fb4fc23dca0c4de69aef5061d3a67b2f [term 2 FOLLOWER]: Refusing update from remote peer 4d3846546549444596d290ff0e21932e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 01:59:39.429742 5985 consensus_queue.cc:1035] T 87da933cf2034ccebc5a78c970acb563 P 4d3846546549444596d290ff0e21932e [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb4fc23dca0c4de69aef5061d3a67b2f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34163 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:39.962904 5793 consensus_queue.cc:237] T 87da933cf2034ccebc5a78c970acb563 P 4d3846546549444596d290ff0e21932e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4d3846546549444596d290ff0e21932e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43135 } attrs { promote: false } }
I20240503 01:59:39.963665 6094 raft_consensus.cc:2932] T 87da933cf2034ccebc5a78c970acb563 P 4d3846546549444596d290ff0e21932e [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER fb4fc23dca0c4de69aef5061d3a67b2f (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4d3846546549444596d290ff0e21932e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43135 } attrs { promote: false } } }
I20240503 01:59:39.965325 2741 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 87da933cf2034ccebc5a78c970acb563 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 7)
I20240503 01:59:39.966138 2752 catalog_manager.cc:5484] T 87da933cf2034ccebc5a78c970acb563 P 4d3846546549444596d290ff0e21932e reported cstate change: config changed from index 5 to 7, VOTER fb4fc23dca0c4de69aef5061d3a67b2f (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "4d3846546549444596d290ff0e21932e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4d3846546549444596d290ff0e21932e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43135 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:39.968820 3007 tablet_service.cc:1514] Processing DeleteTablet for tablet 87da933cf2034ccebc5a78c970acb563 with delete_type TABLET_DATA_TOMBSTONED (TS fb4fc23dca0c4de69aef5061d3a67b2f not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:59142
I20240503 01:59:39.969180 6165 tablet_replica.cc:330] T 87da933cf2034ccebc5a78c970acb563 P fb4fc23dca0c4de69aef5061d3a67b2f: stopping tablet replica
I20240503 01:59:39.969341 6165 raft_consensus.cc:2229] T 87da933cf2034ccebc5a78c970acb563 P fb4fc23dca0c4de69aef5061d3a67b2f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:39.969449 6165 raft_consensus.cc:2258] T 87da933cf2034ccebc5a78c970acb563 P fb4fc23dca0c4de69aef5061d3a67b2f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:39.970518 6165 ts_tablet_manager.cc:1875] T 87da933cf2034ccebc5a78c970acb563 P fb4fc23dca0c4de69aef5061d3a67b2f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:39.972352 6165 ts_tablet_manager.cc:1888] T 87da933cf2034ccebc5a78c970acb563 P fb4fc23dca0c4de69aef5061d3a67b2f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 01:59:39.972455 6165 log.cc:1196] T 87da933cf2034ccebc5a78c970acb563 P fb4fc23dca0c4de69aef5061d3a67b2f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/87da933cf2034ccebc5a78c970acb563
I20240503 01:59:39.973037 2740 catalog_manager.cc:4830] TS fb4fc23dca0c4de69aef5061d3a67b2f (127.0.99.130:34163): tablet 87da933cf2034ccebc5a78c970acb563 (table rebalance_test_table_4 [id=8a70cfe3f9b84945b710ac326669a37e]) successfully deleted
I20240503 01:59:40.086050 2876 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:40.087265 5770 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:40.092988 4410 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:40.094273 4279 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:40.104862 4148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:40.105186 4017 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:40.109695 4540 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:40.128708 3007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:40.209241 5770 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:40.213743 4540 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:40.216037 4017 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:40.223064 4279 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:40.223271 2876 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:40.223311 4410 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:40.224023 3007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:40.224439 4148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:40.337535 5770 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:40.342005 4017 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:40.346992 2876 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:40.349067 4540 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:40.353534 3007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:40.356355 4279 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:40.362751 4148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:40.363454 4410 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
667273152a48411eb72e6817f2e34b09 | 127.0.99.190:43851 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:35751 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
4d3846546549444596d290ff0e21932e | 127.0.99.136:43135 | HEALTHY | <none> | 2 | 0
4d61ec80bf4c475ea86ae2bcb866fd05 | 127.0.99.129:38011 | HEALTHY | <none> | 2 | 0
6c8442cedca54ad589e0e050282afee3 | 127.0.99.135:43573 | HEALTHY | <none> | 2 | 0
ad0b1cf28c9c452db2230e4cd69363bc | 127.0.99.131:44037 | HEALTHY | <none> | 2 | 0
c8a3e298f9d441c09c064a7f5655980b | 127.0.99.132:41327 | HEALTHY | <none> | 2 | 0
e296c081ab4a414294313ab9883cae84 | 127.0.99.133:41953 | HEALTHY | <none> | 2 | 0
e4c1a8048e4647b6936035bc2fc22b66 | 127.0.99.134:46423 | HEALTHY | <none> | 2 | 0
fb4fc23dca0c4de69aef5061d3a67b2f | 127.0.99.130:34163 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.0.99.129 | experimental | 127.0.99.129:38011
local_ip_for_outbound_sockets | 127.0.99.130 | experimental | 127.0.99.130:34163
local_ip_for_outbound_sockets | 127.0.99.131 | experimental | 127.0.99.131:44037
local_ip_for_outbound_sockets | 127.0.99.132 | experimental | 127.0.99.132:41327
local_ip_for_outbound_sockets | 127.0.99.133 | experimental | 127.0.99.133:41953
local_ip_for_outbound_sockets | 127.0.99.134 | experimental | 127.0.99.134:46423
local_ip_for_outbound_sockets | 127.0.99.135 | experimental | 127.0.99.135:43573
local_ip_for_outbound_sockets | 127.0.99.136 | experimental | 127.0.99.136:43135
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.129:38011
server_dump_info_path | /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.99.130:34163
server_dump_info_path | /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.131:44037
server_dump_info_path | /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.0.99.132:41327
server_dump_info_path | /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.133:41953
server_dump_info_path | /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.134:46423
server_dump_info_path | /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.0.99.135:43573
server_dump_info_path | /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.0.99.136:43135
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:35751 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 2
Maximum | 2
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 01:59:40.405476 398 log_verifier.cc:126] Checking tablet 0cbba6e3ce5b4de5b5ec9164e50f784a
I20240503 01:59:40.411818 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 0cbba6e3ce5b4de5b5ec9164e50f784a
I20240503 01:59:40.411890 398 log_verifier.cc:126] Checking tablet 0f440670fb6f4fccbc3ee8554dadb770
I20240503 01:59:40.417475 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 0f440670fb6f4fccbc3ee8554dadb770
I20240503 01:59:40.417543 398 log_verifier.cc:126] Checking tablet 1b96b32dada445ccbe9c4eaa0ddee050
I20240503 01:59:40.423241 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 1b96b32dada445ccbe9c4eaa0ddee050
I20240503 01:59:40.423310 398 log_verifier.cc:126] Checking tablet 201397ae513b4c40bd2e53197616984e
I20240503 01:59:40.429100 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 201397ae513b4c40bd2e53197616984e
I20240503 01:59:40.429162 398 log_verifier.cc:126] Checking tablet 365919d2a8d94985bc031ab16eefe0d8
I20240503 01:59:40.435146 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 365919d2a8d94985bc031ab16eefe0d8
I20240503 01:59:40.435220 398 log_verifier.cc:126] Checking tablet 7056f7fb504545d19641ea2c58ebf6be
I20240503 01:59:40.441120 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 7056f7fb504545d19641ea2c58ebf6be
I20240503 01:59:40.441183 398 log_verifier.cc:126] Checking tablet 7fe4b04830cd42a292f72f27b788c75a
I20240503 01:59:40.445974 398 log_verifier.cc:177] Verified matching terms for 5 ops in tablet 7fe4b04830cd42a292f72f27b788c75a
I20240503 01:59:40.446041 398 log_verifier.cc:126] Checking tablet 87da933cf2034ccebc5a78c970acb563
I20240503 01:59:40.454486 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 87da933cf2034ccebc5a78c970acb563
I20240503 01:59:40.454556 398 log_verifier.cc:126] Checking tablet 9bfd209000fd468385a66c8fa5b2edc8
I20240503 01:59:40.460289 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 9bfd209000fd468385a66c8fa5b2edc8
I20240503 01:59:40.460357 398 log_verifier.cc:126] Checking tablet a6f5711f3c9c4f3ca60893103ee7be40
I20240503 01:59:40.465929 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet a6f5711f3c9c4f3ca60893103ee7be40
I20240503 01:59:40.465991 398 log_verifier.cc:126] Checking tablet b2ba35f5c3084e08a42c6f3381aa1cb5
I20240503 01:59:40.471737 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet b2ba35f5c3084e08a42c6f3381aa1cb5
I20240503 01:59:40.471802 398 log_verifier.cc:126] Checking tablet b60c4ffc1dac4e5e8a2881cd5aa61547
I20240503 01:59:40.480010 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet b60c4ffc1dac4e5e8a2881cd5aa61547
I20240503 01:59:40.480091 398 log_verifier.cc:126] Checking tablet bd3627afefd74c39b43a4ff26da65c39
I20240503 01:59:40.485539 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet bd3627afefd74c39b43a4ff26da65c39
I20240503 01:59:40.485606 398 log_verifier.cc:126] Checking tablet c3c2ff70515f49cf9f51b8db5e130cba
I20240503 01:59:40.491350 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet c3c2ff70515f49cf9f51b8db5e130cba
I20240503 01:59:40.491421 398 log_verifier.cc:126] Checking tablet eefe13b91700485bbcc398b37cec51b4
I20240503 01:59:40.497584 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet eefe13b91700485bbcc398b37cec51b4
I20240503 01:59:40.497661 398 log_verifier.cc:126] Checking tablet fc4a011058e447499e3d651c73f0b881
I20240503 01:59:40.503443 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet fc4a011058e447499e3d651c73f0b881
I20240503 01:59:40.503947 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu with pid 2815
I20240503 01:59:40.511274 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu with pid 2946
I20240503 01:59:40.517554 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu with pid 3955
I20240503 01:59:40.522807 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu with pid 4086
I20240503 01:59:40.527771 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu with pid 4217
I20240503 01:59:40.532792 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu with pid 4348
I20240503 01:59:40.537635 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu with pid 4479
I20240503 01:59:40.543007 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu with pid 5711
I20240503 01:59:40.547943 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu with pid 2724
2024-05-03T01:59:40Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/1 (16489 ms)
[----------] 1 test from RebalanceParamTest (16489 ms total)
[----------] 1 test from DDLDuringRebalancingTest
[ RUN ] DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing/1
I20240503 01:59:40.583804 398 test_util.cc:268] Using random seed: 312165236
I20240503 01:59:40.584703 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 01:59:40.584750 398 ts_itest-base.cc:113] --------------
I20240503 01:59:40.584764 398 ts_itest-base.cc:114] 8 tablet servers
I20240503 01:59:40.584779 398 ts_itest-base.cc:115] 3 replicas per TS
I20240503 01:59:40.584790 398 ts_itest-base.cc:116] --------------
2024-05-03T01:59:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T01:59:40Z Disabled control of system clock
I20240503 01:59:40.593807 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
/tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:35423
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:40175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:35423
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240503 01:59:40.700603 6285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:40.700866 6285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:40.700908 6285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:40.704002 6285 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:40.704090 6285 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 01:59:40.704113 6285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:40.704129 6285 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 01:59:40.704145 6285 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 01:59:40.707971 6285 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:40175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:35423
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:35423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.6285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:40.708995 6285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:40.710096 6285 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:40.718863 6293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:40.718864 6290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:40.719377 6285 server_base.cc:1034] running on GCE node
W20240503 01:59:40.720026 6291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:40.720664 6285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:40.721788 6285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:40.722962 6285 hybrid_clock.cc:648] HybridClock initialized: now 1714701580722934 us; error 44 us; skew 500 ppm
I20240503 01:59:40.724982 6285 webserver.cc:460] Webserver started at http://127.0.99.190:46319/ using document root <none> and password file <none>
I20240503 01:59:40.725531 6285 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:40.725589 6285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:40.725761 6285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:40.727404 6285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "953f1c0a125d48d39bdd77e2ab9f058a"
format_stamp: "Formatted at 2024-05-03 01:59:40 on dist-test-slave-pz5w"
I20240503 01:59:40.727866 6285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "953f1c0a125d48d39bdd77e2ab9f058a"
format_stamp: "Formatted at 2024-05-03 01:59:40 on dist-test-slave-pz5w"
I20240503 01:59:40.731176 6285 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 01:59:40.733358 6299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:40.734503 6285 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 01:59:40.734625 6285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "953f1c0a125d48d39bdd77e2ab9f058a"
format_stamp: "Formatted at 2024-05-03 01:59:40 on dist-test-slave-pz5w"
I20240503 01:59:40.734728 6285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:40.751060 6285 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:40.751714 6285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:40.751870 6285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:40.758643 6285 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:35423
I20240503 01:59:40.758663 6351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:35423 every 8 connection(s)
I20240503 01:59:40.759650 6285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 01:59:40.760658 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu as pid 6285
I20240503 01:59:40.760795 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 01:59:40.763610 6352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:40.768534 6352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Bootstrap starting.
I20240503 01:59:40.770617 6352 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:40.771494 6352 log.cc:824] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:40.773278 6352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: No bootstrap required, opened a new log
I20240503 01:59:40.775878 6352 raft_consensus.cc:348] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "953f1c0a125d48d39bdd77e2ab9f058a" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 35423 } }
I20240503 01:59:40.776090 6352 raft_consensus.cc:374] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:40.776139 6352 raft_consensus.cc:729] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 953f1c0a125d48d39bdd77e2ab9f058a, State: Initialized, Role: FOLLOWER
I20240503 01:59:40.776635 6352 consensus_queue.cc:260] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "953f1c0a125d48d39bdd77e2ab9f058a" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 35423 } }
I20240503 01:59:40.776772 6352 raft_consensus.cc:388] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 01:59:40.776818 6352 raft_consensus.cc:482] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 01:59:40.776926 6352 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:40.777709 6352 raft_consensus.cc:504] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "953f1c0a125d48d39bdd77e2ab9f058a" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 35423 } }
I20240503 01:59:40.778036 6352 leader_election.cc:302] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 953f1c0a125d48d39bdd77e2ab9f058a; no voters:
I20240503 01:59:40.778327 6352 leader_election.cc:288] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 01:59:40.778587 6357 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:40.778791 6357 raft_consensus.cc:686] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a [term 1 LEADER]: Becoming Leader. State: Replica: 953f1c0a125d48d39bdd77e2ab9f058a, State: Running, Role: LEADER
I20240503 01:59:40.779091 6357 consensus_queue.cc:237] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "953f1c0a125d48d39bdd77e2ab9f058a" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 35423 } }
I20240503 01:59:40.779424 6352 sys_catalog.cc:564] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a [sys.catalog]: configured and running, proceeding with master startup.
I20240503 01:59:40.780625 6358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "953f1c0a125d48d39bdd77e2ab9f058a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "953f1c0a125d48d39bdd77e2ab9f058a" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 35423 } } }
I20240503 01:59:40.780733 6358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a [sys.catalog]: This master's current role is: LEADER
I20240503 01:59:40.780932 6359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 953f1c0a125d48d39bdd77e2ab9f058a. Latest consensus state: current_term: 1 leader_uuid: "953f1c0a125d48d39bdd77e2ab9f058a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "953f1c0a125d48d39bdd77e2ab9f058a" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 35423 } } }
I20240503 01:59:40.781126 6359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a [sys.catalog]: This master's current role is: LEADER
W20240503 01:59:40.784174 6372 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 01:59:40.784413 6372 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 01:59:40.784507 6366 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 01:59:40.785364 6366 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 01:59:40.790221 6366 catalog_manager.cc:1347] Generated new cluster ID: 63d003ad449347f8b7eb4dde012c291b
I20240503 01:59:40.790313 6366 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 01:59:40.800036 6366 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 01:59:40.801038 6366 catalog_manager.cc:1504] Loading token signing keys...
I20240503 01:59:40.809257 6366 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Generated new TSK 0
I20240503 01:59:40.809993 6366 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 01:59:40.821344 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
/tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35423
--builtin_ntp_servers=127.0.99.148:40175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 01:59:40.941810 6376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:40.942028 6376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:40.942076 6376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:40.945137 6376 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:40.945214 6376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:40.945293 6376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 01:59:40.948953 6376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:40175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35423
--never_fsync=true
--heap_profile_path=/tmp/kudu.6376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:40.949859 6376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:40.950968 6376 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:40.959241 6382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:40.959277 6384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:40.959321 6376 server_base.cc:1034] running on GCE node
W20240503 01:59:40.959676 6381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:40.960237 6376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:40.960901 6376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:40.962102 6376 hybrid_clock.cc:648] HybridClock initialized: now 1714701580962084 us; error 36 us; skew 500 ppm
I20240503 01:59:40.964210 6376 webserver.cc:460] Webserver started at http://127.0.99.129:35161/ using document root <none> and password file <none>
I20240503 01:59:40.964823 6376 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:40.964882 6376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:40.965054 6376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:40.966778 6376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "dec3e4a04d604f7eaaa3f077d081218a"
format_stamp: "Formatted at 2024-05-03 01:59:40 on dist-test-slave-pz5w"
I20240503 01:59:40.967332 6376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "dec3e4a04d604f7eaaa3f077d081218a"
format_stamp: "Formatted at 2024-05-03 01:59:40 on dist-test-slave-pz5w"
I20240503 01:59:40.970978 6376 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20240503 01:59:40.973582 6390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:40.974746 6376 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 01:59:40.974875 6376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "dec3e4a04d604f7eaaa3f077d081218a"
format_stamp: "Formatted at 2024-05-03 01:59:40 on dist-test-slave-pz5w"
I20240503 01:59:40.974979 6376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:41.003280 6376 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:41.003890 6376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:41.004058 6376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:41.004621 6376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:41.005520 6376 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:41.005568 6376 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:41.005623 6376 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:41.005650 6376 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:41.013530 6376 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:33093
I20240503 01:59:41.013576 6503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:33093 every 8 connection(s)
I20240503 01:59:41.014514 6376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 01:59:41.018014 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu as pid 6376
I20240503 01:59:41.018160 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 01:59:41.020306 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
/tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35423
--builtin_ntp_servers=127.0.99.148:40175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 01:59:41.023841 6504 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35423
I20240503 01:59:41.024076 6504 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:41.024672 6504 heartbeater.cc:511] Master 127.0.99.190:35423 requested a full tablet report, sending...
I20240503 01:59:41.026273 6316 ts_manager.cc:194] Registered new tserver with Master: dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 01:59:41.027912 6316 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:34993
W20240503 01:59:41.131533 6507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:41.131759 6507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:41.131800 6507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:41.134793 6507 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:41.134860 6507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:41.134927 6507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 01:59:41.138517 6507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:40175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35423
--never_fsync=true
--heap_profile_path=/tmp/kudu.6507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:41.139354 6507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:41.140456 6507 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:41.148366 6512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:41.148362 6513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:41.148639 6515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:41.148910 6507 server_base.cc:1034] running on GCE node
I20240503 01:59:41.149322 6507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:41.149930 6507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:41.151105 6507 hybrid_clock.cc:648] HybridClock initialized: now 1714701581151088 us; error 31 us; skew 500 ppm
I20240503 01:59:41.153180 6507 webserver.cc:460] Webserver started at http://127.0.99.130:34153/ using document root <none> and password file <none>
I20240503 01:59:41.153754 6507 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:41.153816 6507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:41.153995 6507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:41.155711 6507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "80fd18ef4d604c86b909d5c008a46b4f"
format_stamp: "Formatted at 2024-05-03 01:59:41 on dist-test-slave-pz5w"
I20240503 01:59:41.156265 6507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "80fd18ef4d604c86b909d5c008a46b4f"
format_stamp: "Formatted at 2024-05-03 01:59:41 on dist-test-slave-pz5w"
I20240503 01:59:41.159780 6507 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20240503 01:59:41.162142 6521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:41.163192 6507 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20240503 01:59:41.163309 6507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "80fd18ef4d604c86b909d5c008a46b4f"
format_stamp: "Formatted at 2024-05-03 01:59:41 on dist-test-slave-pz5w"
I20240503 01:59:41.163408 6507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:41.174670 6507 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:41.175287 6507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:41.175439 6507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:41.176070 6507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:41.177026 6507 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:41.177073 6507 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:41.177124 6507 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:41.177153 6507 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:41.185408 6507 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:43863
I20240503 01:59:41.185433 6634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:43863 every 8 connection(s)
I20240503 01:59:41.186399 6507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 01:59:41.186800 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu as pid 6507
I20240503 01:59:41.186920 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 01:59:41.189296 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
/tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35423
--builtin_ntp_servers=127.0.99.148:40175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 01:59:41.196280 6635 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35423
I20240503 01:59:41.196566 6635 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:41.197130 6635 heartbeater.cc:511] Master 127.0.99.190:35423 requested a full tablet report, sending...
I20240503 01:59:41.198184 6316 ts_manager.cc:194] Registered new tserver with Master: 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 01:59:41.198738 6316 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:38679
W20240503 01:59:41.296967 6638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:41.297222 6638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:41.297282 6638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:41.300438 6638 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:41.300534 6638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:41.300622 6638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 01:59:41.304718 6638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:40175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35423
--never_fsync=true
--heap_profile_path=/tmp/kudu.6638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:41.305698 6638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:41.306854 6638 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:41.315323 6644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:41.315323 6643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:41.315452 6638 server_base.cc:1034] running on GCE node
W20240503 01:59:41.315582 6646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:41.316005 6638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:41.316643 6638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:41.317875 6638 hybrid_clock.cc:648] HybridClock initialized: now 1714701581317849 us; error 39 us; skew 500 ppm
I20240503 01:59:41.319913 6638 webserver.cc:460] Webserver started at http://127.0.99.131:45339/ using document root <none> and password file <none>
I20240503 01:59:41.320519 6638 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:41.320583 6638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:41.320771 6638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:41.322441 6638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
format_stamp: "Formatted at 2024-05-03 01:59:41 on dist-test-slave-pz5w"
I20240503 01:59:41.322923 6638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
format_stamp: "Formatted at 2024-05-03 01:59:41 on dist-test-slave-pz5w"
I20240503 01:59:41.326361 6638 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 01:59:41.328600 6652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:41.329638 6638 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 01:59:41.329747 6638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
format_stamp: "Formatted at 2024-05-03 01:59:41 on dist-test-slave-pz5w"
I20240503 01:59:41.329841 6638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:41.356364 6638 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:41.357116 6638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:41.357251 6638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:41.357740 6638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:41.358611 6638 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:41.358657 6638 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:41.358696 6638 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:41.358716 6638 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:41.367295 6638 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:42655
I20240503 01:59:41.367312 6765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:42655 every 8 connection(s)
I20240503 01:59:41.368319 6638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 01:59:41.375609 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu as pid 6638
I20240503 01:59:41.375723 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 01:59:41.378163 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
/tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35423
--builtin_ntp_servers=127.0.99.148:40175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 01:59:41.378193 6766 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35423
I20240503 01:59:41.378453 6766 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:41.379128 6766 heartbeater.cc:511] Master 127.0.99.190:35423 requested a full tablet report, sending...
I20240503 01:59:41.380326 6316 ts_manager.cc:194] Registered new tserver with Master: c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 01:59:41.380954 6316 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:35431
W20240503 01:59:41.483448 6769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:41.483683 6769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:41.483734 6769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:41.486871 6769 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:41.486958 6769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:41.487038 6769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 01:59:41.491057 6769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:40175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35423
--never_fsync=true
--heap_profile_path=/tmp/kudu.6769
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:41.491956 6769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:41.493096 6769 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:41.500631 6775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:41.500667 6774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:41.500802 6769 server_base.cc:1034] running on GCE node
W20240503 01:59:41.501006 6777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:41.501372 6769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:41.501925 6769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:41.503069 6769 hybrid_clock.cc:648] HybridClock initialized: now 1714701581503058 us; error 26 us; skew 500 ppm
I20240503 01:59:41.505040 6769 webserver.cc:460] Webserver started at http://127.0.99.132:40379/ using document root <none> and password file <none>
I20240503 01:59:41.505580 6769 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:41.505640 6769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:41.505823 6769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:41.507427 6769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "492448570cc94ed1879feb2ec3b9051d"
format_stamp: "Formatted at 2024-05-03 01:59:41 on dist-test-slave-pz5w"
I20240503 01:59:41.507877 6769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "492448570cc94ed1879feb2ec3b9051d"
format_stamp: "Formatted at 2024-05-03 01:59:41 on dist-test-slave-pz5w"
I20240503 01:59:41.511166 6769 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 01:59:41.513423 6783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:41.514420 6769 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:41.514529 6769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "492448570cc94ed1879feb2ec3b9051d"
format_stamp: "Formatted at 2024-05-03 01:59:41 on dist-test-slave-pz5w"
I20240503 01:59:41.514631 6769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:41.531066 6769 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:41.531656 6769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:41.531807 6769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:41.532306 6769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:41.533267 6769 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:41.533316 6769 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:41.533368 6769 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:41.533393 6769 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:41.541185 6769 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:42719
I20240503 01:59:41.541211 6896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:42719 every 8 connection(s)
I20240503 01:59:41.542065 6769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 01:59:41.543927 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu as pid 6769
I20240503 01:59:41.544051 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 01:59:41.546377 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
/tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35423
--builtin_ntp_servers=127.0.99.148:40175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 01:59:41.550923 6897 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35423
I20240503 01:59:41.551198 6897 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:41.551913 6897 heartbeater.cc:511] Master 127.0.99.190:35423 requested a full tablet report, sending...
I20240503 01:59:41.552997 6316 ts_manager.cc:194] Registered new tserver with Master: 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 01:59:41.553534 6316 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:37321
W20240503 01:59:41.653745 6900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:41.653972 6900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:41.654017 6900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:41.657066 6900 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:41.657140 6900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:41.657209 6900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 01:59:41.660828 6900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:40175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35423
--never_fsync=true
--heap_profile_path=/tmp/kudu.6900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:41.661737 6900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:41.662837 6900 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:41.670871 6906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:41.671028 6900 server_base.cc:1034] running on GCE node
W20240503 01:59:41.670871 6908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:41.671319 6905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:41.671720 6900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:41.672305 6900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:41.673450 6900 hybrid_clock.cc:648] HybridClock initialized: now 1714701581673429 us; error 38 us; skew 500 ppm
I20240503 01:59:41.675436 6900 webserver.cc:460] Webserver started at http://127.0.99.133:41957/ using document root <none> and password file <none>
I20240503 01:59:41.676008 6900 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:41.676069 6900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:41.676242 6900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:41.677960 6900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "9942306855404cfd891adf227dd65eb2"
format_stamp: "Formatted at 2024-05-03 01:59:41 on dist-test-slave-pz5w"
I20240503 01:59:41.678463 6900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "9942306855404cfd891adf227dd65eb2"
format_stamp: "Formatted at 2024-05-03 01:59:41 on dist-test-slave-pz5w"
I20240503 01:59:41.682075 6900 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 01:59:41.684303 6914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:41.685385 6900 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:41.685482 6900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "9942306855404cfd891adf227dd65eb2"
format_stamp: "Formatted at 2024-05-03 01:59:41 on dist-test-slave-pz5w"
I20240503 01:59:41.685564 6900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:41.704116 6900 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:41.704748 6900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:41.704887 6900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:41.705386 6900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:41.706234 6900 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:41.706280 6900 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:41.706318 6900 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:41.706331 6900 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:41.714017 6900 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:33211
I20240503 01:59:41.714057 7027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:33211 every 8 connection(s)
I20240503 01:59:41.714931 6900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 01:59:41.722080 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu as pid 6900
I20240503 01:59:41.722209 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 01:59:41.724025 7028 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35423
I20240503 01:59:41.724267 7028 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:41.724851 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
/tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:0
--local_ip_for_outbound_sockets=127.0.99.134
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35423
--builtin_ntp_servers=127.0.99.148:40175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 01:59:41.724936 7028 heartbeater.cc:511] Master 127.0.99.190:35423 requested a full tablet report, sending...
I20240503 01:59:41.725924 6316 ts_manager.cc:194] Registered new tserver with Master: 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 01:59:41.726667 6316 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:36425
W20240503 01:59:41.832849 7031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:41.833073 7031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:41.833117 7031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:41.836114 7031 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:41.836189 7031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:41.836256 7031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 01:59:41.839819 7031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:40175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35423
--never_fsync=true
--heap_profile_path=/tmp/kudu.7031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:41.840700 7031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:41.841740 7031 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:41.849622 7036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:41.849732 7031 server_base.cc:1034] running on GCE node
W20240503 01:59:41.849763 7037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:41.849622 7039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:41.850296 7031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:41.850870 7031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:41.852025 7031 hybrid_clock.cc:648] HybridClock initialized: now 1714701581852009 us; error 30 us; skew 500 ppm
I20240503 01:59:41.853986 7031 webserver.cc:460] Webserver started at http://127.0.99.134:42917/ using document root <none> and password file <none>
I20240503 01:59:41.854527 7031 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:41.854585 7031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:41.854760 7031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:41.856436 7031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "99ead02915954b4fab23b033ad6d546b"
format_stamp: "Formatted at 2024-05-03 01:59:41 on dist-test-slave-pz5w"
I20240503 01:59:41.856909 7031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "99ead02915954b4fab23b033ad6d546b"
format_stamp: "Formatted at 2024-05-03 01:59:41 on dist-test-slave-pz5w"
I20240503 01:59:41.860261 7031 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20240503 01:59:41.862421 7045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:41.863389 7031 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:41.863492 7031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "99ead02915954b4fab23b033ad6d546b"
format_stamp: "Formatted at 2024-05-03 01:59:41 on dist-test-slave-pz5w"
I20240503 01:59:41.863577 7031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:41.878281 7031 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:41.878911 7031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:41.879056 7031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:41.879575 7031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:41.880514 7031 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:41.880561 7031 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:41.880620 7031 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:41.880649 7031 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:41.888473 7031 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:46157
I20240503 01:59:41.888500 7158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:46157 every 8 connection(s)
I20240503 01:59:41.889395 7031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 01:59:41.890201 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu as pid 7031
I20240503 01:59:41.890309 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 01:59:41.892575 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
/tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:0
--local_ip_for_outbound_sockets=127.0.99.135
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35423
--builtin_ntp_servers=127.0.99.148:40175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 01:59:41.898084 7159 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35423
I20240503 01:59:41.898303 7159 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:41.898813 7159 heartbeater.cc:511] Master 127.0.99.190:35423 requested a full tablet report, sending...
I20240503 01:59:41.899757 6316 ts_manager.cc:194] Registered new tserver with Master: 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 01:59:41.900278 6316 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:39823
W20240503 01:59:41.997994 7162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:41.998219 7162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:41.998265 7162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:42.001305 7162 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:42.001372 7162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:42.001452 7162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 01:59:42.005146 7162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:40175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35423
--never_fsync=true
--heap_profile_path=/tmp/kudu.7162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:42.006059 7162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:42.007158 7162 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:42.014884 7168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:42.014884 7170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:42.014977 7167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:42.015516 7162 server_base.cc:1034] running on GCE node
I20240503 01:59:42.015899 7162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:42.016542 7162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:42.017688 7162 hybrid_clock.cc:648] HybridClock initialized: now 1714701582017670 us; error 30 us; skew 500 ppm
I20240503 01:59:42.019477 7162 webserver.cc:460] Webserver started at http://127.0.99.135:36139/ using document root <none> and password file <none>
I20240503 01:59:42.019971 7162 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:42.020020 7162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:42.020175 7162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:42.021749 7162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "2a01ace80baf4057b6bf1995cf88468b"
format_stamp: "Formatted at 2024-05-03 01:59:42 on dist-test-slave-pz5w"
I20240503 01:59:42.022154 7162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "2a01ace80baf4057b6bf1995cf88468b"
format_stamp: "Formatted at 2024-05-03 01:59:42 on dist-test-slave-pz5w"
I20240503 01:59:42.025218 7162 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20240503 01:59:42.027212 7176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:42.028191 7162 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:42.028278 7162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "2a01ace80baf4057b6bf1995cf88468b"
format_stamp: "Formatted at 2024-05-03 01:59:42 on dist-test-slave-pz5w"
I20240503 01:59:42.028349 7162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:42.030282 6504 heartbeater.cc:503] Master 127.0.99.190:35423 was elected leader, sending a full tablet report...
I20240503 01:59:42.056773 7162 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:42.057469 7162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:42.057636 7162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:42.058187 7162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:42.059091 7162 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:42.059145 7162 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:42.059204 7162 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:42.059233 7162 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:42.067363 7162 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:34079
I20240503 01:59:42.067396 7289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:34079 every 8 connection(s)
I20240503 01:59:42.068344 7162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 01:59:42.077191 7290 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35423
I20240503 01:59:42.077420 7290 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:42.077965 7290 heartbeater.cc:511] Master 127.0.99.190:35423 requested a full tablet report, sending...
I20240503 01:59:42.078186 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu as pid 7162
I20240503 01:59:42.078300 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 01:59:42.079141 6316 ts_manager.cc:194] Registered new tserver with Master: 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 01:59:42.079739 6316 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:37779
I20240503 01:59:42.080698 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
/tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:0
--local_ip_for_outbound_sockets=127.0.99.136
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35423
--builtin_ntp_servers=127.0.99.148:40175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 01:59:42.190339 7293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:42.190554 7293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:42.190598 7293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:42.193774 7293 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:42.193854 7293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:42.193931 7293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240503 01:59:42.197695 7293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:40175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35423
--never_fsync=true
--heap_profile_path=/tmp/kudu.7293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:42.198582 7293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:42.199669 7293 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 01:59:42.200764 6635 heartbeater.cc:503] Master 127.0.99.190:35423 was elected leader, sending a full tablet report...
W20240503 01:59:42.208060 7301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:42.208076 7298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:42.208122 7299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:42.208230 7293 server_base.cc:1034] running on GCE node
I20240503 01:59:42.208727 7293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:42.209291 7293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:42.210458 7293 hybrid_clock.cc:648] HybridClock initialized: now 1714701582210439 us; error 30 us; skew 500 ppm
I20240503 01:59:42.212337 7293 webserver.cc:460] Webserver started at http://127.0.99.136:45791/ using document root <none> and password file <none>
I20240503 01:59:42.212898 7293 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:42.212956 7293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:42.213119 7293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:42.214725 7293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023"
format_stamp: "Formatted at 2024-05-03 01:59:42 on dist-test-slave-pz5w"
I20240503 01:59:42.215174 7293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023"
format_stamp: "Formatted at 2024-05-03 01:59:42 on dist-test-slave-pz5w"
I20240503 01:59:42.218430 7293 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20240503 01:59:42.220531 7307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:42.221500 7293 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240503 01:59:42.221602 7293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023"
format_stamp: "Formatted at 2024-05-03 01:59:42 on dist-test-slave-pz5w"
I20240503 01:59:42.221681 7293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:42.240090 7293 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:42.240705 7293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:42.240854 7293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:42.241340 7293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:42.242172 7293 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:42.242218 7293 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:42.242269 7293 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:42.242297 7293 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:42.250890 7293 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:45311
I20240503 01:59:42.250943 7420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:45311 every 8 connection(s)
I20240503 01:59:42.251828 7293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 01:59:42.256179 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu as pid 7293
I20240503 01:59:42.256285 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 01:59:42.260869 7421 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35423
I20240503 01:59:42.261142 7421 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:42.261721 7421 heartbeater.cc:511] Master 127.0.99.190:35423 requested a full tablet report, sending...
I20240503 01:59:42.262781 6316 ts_manager.cc:194] Registered new tserver with Master: e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311)
I20240503 01:59:42.263341 6316 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:33033
I20240503 01:59:42.270490 398 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 01:59:42.280479 398 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 01:59:42.280571 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu with pid 6900
I20240503 01:59:42.285181 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu with pid 7031
I20240503 01:59:42.290323 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu with pid 7162
I20240503 01:59:42.295159 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu with pid 7293
I20240503 01:59:42.383071 6766 heartbeater.cc:503] Master 127.0.99.190:35423 was elected leader, sending a full tablet report...
I20240503 01:59:42.555970 6897 heartbeater.cc:503] Master 127.0.99.190:35423 was elected leader, sending a full tablet report...
I20240503 01:59:46.050999 6316 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:46.071869 6830 tablet_service.cc:1467] Processing CreateTablet for tablet 481e1968fa634172a4312352e1a1b6ec (DEFAULT_TABLE table=rebalance_test_table_0 [id=cce4a4befe484b7f83338cb366b6106d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:46.071868 6438 tablet_service.cc:1467] Processing CreateTablet for tablet 771f069dd030449489008509f9794d7e (DEFAULT_TABLE table=rebalance_test_table_0 [id=cce4a4befe484b7f83338cb366b6106d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:46.071862 6700 tablet_service.cc:1467] Processing CreateTablet for tablet 771f069dd030449489008509f9794d7e (DEFAULT_TABLE table=rebalance_test_table_0 [id=cce4a4befe484b7f83338cb366b6106d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:46.072804 6830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 481e1968fa634172a4312352e1a1b6ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:46.072808 6438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 771f069dd030449489008509f9794d7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:46.072808 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 771f069dd030449489008509f9794d7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:46.076632 6831 tablet_service.cc:1467] Processing CreateTablet for tablet 771f069dd030449489008509f9794d7e (DEFAULT_TABLE table=rebalance_test_table_0 [id=cce4a4befe484b7f83338cb366b6106d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:46.076740 6437 tablet_service.cc:1467] Processing CreateTablet for tablet 37b626b17ee242f6ac73689212b7474b (DEFAULT_TABLE table=rebalance_test_table_0 [id=cce4a4befe484b7f83338cb366b6106d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:46.076933 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 771f069dd030449489008509f9794d7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:46.077008 6437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37b626b17ee242f6ac73689212b7474b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:46.077353 6699 tablet_service.cc:1467] Processing CreateTablet for tablet 37b626b17ee242f6ac73689212b7474b (DEFAULT_TABLE table=rebalance_test_table_0 [id=cce4a4befe484b7f83338cb366b6106d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:46.077601 6699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37b626b17ee242f6ac73689212b7474b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:46.071869 6436 tablet_service.cc:1467] Processing CreateTablet for tablet 481e1968fa634172a4312352e1a1b6ec (DEFAULT_TABLE table=rebalance_test_table_0 [id=cce4a4befe484b7f83338cb366b6106d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:46.079587 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 481e1968fa634172a4312352e1a1b6ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:46.081864 7442 tablet_bootstrap.cc:492] T 37b626b17ee242f6ac73689212b7474b P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 01:59:46.082485 6569 tablet_service.cc:1467] Processing CreateTablet for tablet 37b626b17ee242f6ac73689212b7474b (DEFAULT_TABLE table=rebalance_test_table_0 [id=cce4a4befe484b7f83338cb366b6106d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:46.083074 7443 tablet_bootstrap.cc:492] T 37b626b17ee242f6ac73689212b7474b P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 01:59:46.082485 6568 tablet_service.cc:1467] Processing CreateTablet for tablet 481e1968fa634172a4312352e1a1b6ec (DEFAULT_TABLE table=rebalance_test_table_0 [id=cce4a4befe484b7f83338cb366b6106d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:46.083467 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37b626b17ee242f6ac73689212b7474b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:46.084303 7442 tablet_bootstrap.cc:654] T 37b626b17ee242f6ac73689212b7474b P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:46.084919 7441 tablet_bootstrap.cc:492] T 481e1968fa634172a4312352e1a1b6ec P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 01:59:46.085163 7442 log.cc:824] T 37b626b17ee242f6ac73689212b7474b P dec3e4a04d604f7eaaa3f077d081218a: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:46.085561 7443 tablet_bootstrap.cc:654] T 37b626b17ee242f6ac73689212b7474b P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:46.085602 6568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 481e1968fa634172a4312352e1a1b6ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:46.086376 7443 log.cc:824] T 37b626b17ee242f6ac73689212b7474b P c2ac2eb7cc264047b9ab32fe3a4fc025: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:46.086871 7441 tablet_bootstrap.cc:654] T 481e1968fa634172a4312352e1a1b6ec P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:46.087548 7441 log.cc:824] T 481e1968fa634172a4312352e1a1b6ec P 492448570cc94ed1879feb2ec3b9051d: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:46.087858 7442 tablet_bootstrap.cc:492] T 37b626b17ee242f6ac73689212b7474b P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 01:59:46.088050 7442 ts_tablet_manager.cc:1367] T 37b626b17ee242f6ac73689212b7474b P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240503 01:59:46.089098 7443 tablet_bootstrap.cc:492] T 37b626b17ee242f6ac73689212b7474b P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 01:59:46.089274 7443 ts_tablet_manager.cc:1367] T 37b626b17ee242f6ac73689212b7474b P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240503 01:59:46.090291 7441 tablet_bootstrap.cc:492] T 481e1968fa634172a4312352e1a1b6ec P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 01:59:46.090499 7441 ts_tablet_manager.cc:1367] T 481e1968fa634172a4312352e1a1b6ec P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 01:59:46.092001 7442 raft_consensus.cc:348] T 37b626b17ee242f6ac73689212b7474b P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:46.092253 7442 raft_consensus.cc:374] T 37b626b17ee242f6ac73689212b7474b P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:46.092310 7442 raft_consensus.cc:729] T 37b626b17ee242f6ac73689212b7474b P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 01:59:46.092813 7442 consensus_queue.cc:260] T 37b626b17ee242f6ac73689212b7474b P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:46.092782 7443 raft_consensus.cc:348] T 37b626b17ee242f6ac73689212b7474b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:46.092972 7443 raft_consensus.cc:374] T 37b626b17ee242f6ac73689212b7474b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:46.093021 7443 raft_consensus.cc:729] T 37b626b17ee242f6ac73689212b7474b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 01:59:46.093478 7443 consensus_queue.cc:260] T 37b626b17ee242f6ac73689212b7474b P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:46.093932 7442 ts_tablet_manager.cc:1398] T 37b626b17ee242f6ac73689212b7474b P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240503 01:59:46.094002 7447 tablet_bootstrap.cc:492] T 37b626b17ee242f6ac73689212b7474b P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 01:59:46.094269 7443 ts_tablet_manager.cc:1398] T 37b626b17ee242f6ac73689212b7474b P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240503 01:59:46.094301 7442 tablet_bootstrap.cc:492] T 481e1968fa634172a4312352e1a1b6ec P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 01:59:46.094763 7441 raft_consensus.cc:348] T 481e1968fa634172a4312352e1a1b6ec P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:46.095026 7441 raft_consensus.cc:374] T 481e1968fa634172a4312352e1a1b6ec P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:46.095120 7441 raft_consensus.cc:729] T 481e1968fa634172a4312352e1a1b6ec P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 01:59:46.095273 7442 tablet_bootstrap.cc:654] T 481e1968fa634172a4312352e1a1b6ec P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:46.095431 7443 tablet_bootstrap.cc:492] T 771f069dd030449489008509f9794d7e P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 01:59:46.095655 7441 consensus_queue.cc:260] T 481e1968fa634172a4312352e1a1b6ec P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:46.096334 7443 tablet_bootstrap.cc:654] T 771f069dd030449489008509f9794d7e P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:46.096594 7441 ts_tablet_manager.cc:1398] T 481e1968fa634172a4312352e1a1b6ec P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240503 01:59:46.096982 7441 tablet_bootstrap.cc:492] T 771f069dd030449489008509f9794d7e P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 01:59:46.097788 7442 tablet_bootstrap.cc:492] T 481e1968fa634172a4312352e1a1b6ec P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 01:59:46.097884 7442 ts_tablet_manager.cc:1367] T 481e1968fa634172a4312352e1a1b6ec P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:46.098418 7442 raft_consensus.cc:348] T 481e1968fa634172a4312352e1a1b6ec P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:46.098531 7442 raft_consensus.cc:374] T 481e1968fa634172a4312352e1a1b6ec P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:46.098563 7442 raft_consensus.cc:729] T 481e1968fa634172a4312352e1a1b6ec P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 01:59:46.098711 7442 consensus_queue.cc:260] T 481e1968fa634172a4312352e1a1b6ec P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:46.099463 7447 tablet_bootstrap.cc:654] T 37b626b17ee242f6ac73689212b7474b P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:46.099050 7442 ts_tablet_manager.cc:1398] T 481e1968fa634172a4312352e1a1b6ec P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:46.099937 7442 tablet_bootstrap.cc:492] T 771f069dd030449489008509f9794d7e P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 01:59:46.100117 7441 tablet_bootstrap.cc:654] T 771f069dd030449489008509f9794d7e P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:46.100531 7447 log.cc:824] T 37b626b17ee242f6ac73689212b7474b P 80fd18ef4d604c86b909d5c008a46b4f: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:46.100816 7442 tablet_bootstrap.cc:654] T 771f069dd030449489008509f9794d7e P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:46.101394 7443 tablet_bootstrap.cc:492] T 771f069dd030449489008509f9794d7e P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 01:59:46.101506 7443 ts_tablet_manager.cc:1367] T 771f069dd030449489008509f9794d7e P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 01:59:46.102150 7441 tablet_bootstrap.cc:492] T 771f069dd030449489008509f9794d7e P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 01:59:46.102097 7443 raft_consensus.cc:348] T 771f069dd030449489008509f9794d7e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:46.102216 7443 raft_consensus.cc:374] T 771f069dd030449489008509f9794d7e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:46.102232 7441 ts_tablet_manager.cc:1367] T 771f069dd030449489008509f9794d7e P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 01:59:46.102257 7443 raft_consensus.cc:729] T 771f069dd030449489008509f9794d7e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 01:59:46.102386 7443 consensus_queue.cc:260] T 771f069dd030449489008509f9794d7e P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:46.102681 7443 ts_tablet_manager.cc:1398] T 771f069dd030449489008509f9794d7e P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:46.102746 7441 raft_consensus.cc:348] T 771f069dd030449489008509f9794d7e P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:46.102851 7441 raft_consensus.cc:374] T 771f069dd030449489008509f9794d7e P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:46.102885 7441 raft_consensus.cc:729] T 771f069dd030449489008509f9794d7e P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 01:59:46.103003 7441 consensus_queue.cc:260] T 771f069dd030449489008509f9794d7e P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:46.103277 7441 ts_tablet_manager.cc:1398] T 771f069dd030449489008509f9794d7e P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:46.104426 7442 tablet_bootstrap.cc:492] T 771f069dd030449489008509f9794d7e P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 01:59:46.104516 7442 ts_tablet_manager.cc:1367] T 771f069dd030449489008509f9794d7e P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 01:59:46.104902 7447 tablet_bootstrap.cc:492] T 37b626b17ee242f6ac73689212b7474b P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 01:59:46.104936 7442 raft_consensus.cc:348] T 771f069dd030449489008509f9794d7e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:46.105039 7442 raft_consensus.cc:374] T 771f069dd030449489008509f9794d7e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:46.105070 7442 raft_consensus.cc:729] T 771f069dd030449489008509f9794d7e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 01:59:46.105098 7447 ts_tablet_manager.cc:1367] T 37b626b17ee242f6ac73689212b7474b P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240503 01:59:46.105177 7442 consensus_queue.cc:260] T 771f069dd030449489008509f9794d7e P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:46.106638 7442 ts_tablet_manager.cc:1398] T 771f069dd030449489008509f9794d7e P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:46.108729 7447 raft_consensus.cc:348] T 37b626b17ee242f6ac73689212b7474b P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:46.108908 7447 raft_consensus.cc:374] T 37b626b17ee242f6ac73689212b7474b P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:46.108958 7447 raft_consensus.cc:729] T 37b626b17ee242f6ac73689212b7474b P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 01:59:46.109339 7447 consensus_queue.cc:260] T 37b626b17ee242f6ac73689212b7474b P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:46.110124 7447 ts_tablet_manager.cc:1398] T 37b626b17ee242f6ac73689212b7474b P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240503 01:59:46.110368 7447 tablet_bootstrap.cc:492] T 481e1968fa634172a4312352e1a1b6ec P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 01:59:46.111289 7447 tablet_bootstrap.cc:654] T 481e1968fa634172a4312352e1a1b6ec P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:46.112435 7447 tablet_bootstrap.cc:492] T 481e1968fa634172a4312352e1a1b6ec P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 01:59:46.112520 7447 ts_tablet_manager.cc:1367] T 481e1968fa634172a4312352e1a1b6ec P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:46.113096 7447 raft_consensus.cc:348] T 481e1968fa634172a4312352e1a1b6ec P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:46.113200 7447 raft_consensus.cc:374] T 481e1968fa634172a4312352e1a1b6ec P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:46.113230 7447 raft_consensus.cc:729] T 481e1968fa634172a4312352e1a1b6ec P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 01:59:46.113353 7447 consensus_queue.cc:260] T 481e1968fa634172a4312352e1a1b6ec P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:46.113623 7447 ts_tablet_manager.cc:1398] T 481e1968fa634172a4312352e1a1b6ec P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20240503 01:59:46.122737 6767 tablet.cc:2369] T 37b626b17ee242f6ac73689212b7474b P c2ac2eb7cc264047b9ab32fe3a4fc025: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 01:59:46.160972 7451 raft_consensus.cc:482] T 481e1968fa634172a4312352e1a1b6ec P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:46.161226 7451 raft_consensus.cc:504] T 481e1968fa634172a4312352e1a1b6ec P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:46.162422 7451 leader_election.cc:288] T 481e1968fa634172a4312352e1a1b6ec P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 01:59:46.162418 7448 raft_consensus.cc:482] T 771f069dd030449489008509f9794d7e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:46.162590 7448 raft_consensus.cc:504] T 771f069dd030449489008509f9794d7e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:46.163800 7448 leader_election.cc:288] T 771f069dd030449489008509f9794d7e P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 01:59:46.167618 6582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481e1968fa634172a4312352e1a1b6ec" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 01:59:46.167971 6582 raft_consensus.cc:2454] T 481e1968fa634172a4312352e1a1b6ec P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 01:59:46.168359 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481e1968fa634172a4312352e1a1b6ec" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 01:59:46.168511 6786 leader_election.cc:302] T 481e1968fa634172a4312352e1a1b6ec P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 80fd18ef4d604c86b909d5c008a46b4f; no voters:
I20240503 01:59:46.168603 6458 raft_consensus.cc:2454] T 481e1968fa634172a4312352e1a1b6ec P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 01:59:46.168807 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771f069dd030449489008509f9794d7e" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 01:59:46.168819 7451 raft_consensus.cc:2781] T 481e1968fa634172a4312352e1a1b6ec P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:46.168928 7451 raft_consensus.cc:482] T 481e1968fa634172a4312352e1a1b6ec P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:46.168967 7451 raft_consensus.cc:3037] T 481e1968fa634172a4312352e1a1b6ec P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:46.168978 6851 raft_consensus.cc:2454] T 771f069dd030449489008509f9794d7e P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 01:59:46.169260 6720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771f069dd030449489008509f9794d7e" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 01:59:46.169342 6394 leader_election.cc:302] T 771f069dd030449489008509f9794d7e P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:46.169498 6720 raft_consensus.cc:2454] T 771f069dd030449489008509f9794d7e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 01:59:46.169590 7448 raft_consensus.cc:2781] T 771f069dd030449489008509f9794d7e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:46.169667 7448 raft_consensus.cc:482] T 771f069dd030449489008509f9794d7e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:46.169705 7448 raft_consensus.cc:3037] T 771f069dd030449489008509f9794d7e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:46.170027 7451 raft_consensus.cc:504] T 481e1968fa634172a4312352e1a1b6ec P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:46.170433 7451 leader_election.cc:288] T 481e1968fa634172a4312352e1a1b6ec P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Requested vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 01:59:46.170715 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481e1968fa634172a4312352e1a1b6ec" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 01:59:46.170688 7448 raft_consensus.cc:504] T 771f069dd030449489008509f9794d7e P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:46.170823 6458 raft_consensus.cc:3037] T 481e1968fa634172a4312352e1a1b6ec P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:46.170826 6582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481e1968fa634172a4312352e1a1b6ec" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 01:59:46.170956 6582 raft_consensus.cc:3037] T 481e1968fa634172a4312352e1a1b6ec P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:46.171101 7448 leader_election.cc:288] T 771f069dd030449489008509f9794d7e P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Requested vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 01:59:46.171520 6720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771f069dd030449489008509f9794d7e" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 01:59:46.171554 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771f069dd030449489008509f9794d7e" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 01:59:46.171665 6851 raft_consensus.cc:3037] T 771f069dd030449489008509f9794d7e P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:46.171653 6458 raft_consensus.cc:2454] T 481e1968fa634172a4312352e1a1b6ec P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 01:59:46.171638 6720 raft_consensus.cc:3037] T 771f069dd030449489008509f9794d7e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:46.172019 6582 raft_consensus.cc:2454] T 481e1968fa634172a4312352e1a1b6ec P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 01:59:46.172070 6784 leader_election.cc:302] T 481e1968fa634172a4312352e1a1b6ec P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:46.172340 7451 raft_consensus.cc:2781] T 481e1968fa634172a4312352e1a1b6ec P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:46.172569 7451 raft_consensus.cc:686] T 481e1968fa634172a4312352e1a1b6ec P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Becoming Leader. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Running, Role: LEADER
I20240503 01:59:46.172731 6720 raft_consensus.cc:2454] T 771f069dd030449489008509f9794d7e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 01:59:46.172899 7451 consensus_queue.cc:237] T 481e1968fa634172a4312352e1a1b6ec P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:46.173022 6851 raft_consensus.cc:2454] T 771f069dd030449489008509f9794d7e P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 01:59:46.173424 6394 leader_election.cc:302] T 771f069dd030449489008509f9794d7e P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:46.173575 7448 raft_consensus.cc:2781] T 771f069dd030449489008509f9794d7e P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:46.173779 7448 raft_consensus.cc:686] T 771f069dd030449489008509f9794d7e P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Becoming Leader. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Running, Role: LEADER
I20240503 01:59:46.174055 7448 consensus_queue.cc:237] T 771f069dd030449489008509f9794d7e P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:46.174888 7449 raft_consensus.cc:482] T 37b626b17ee242f6ac73689212b7474b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:46.174986 7449 raft_consensus.cc:504] T 37b626b17ee242f6ac73689212b7474b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:46.176034 7449 leader_election.cc:288] T 37b626b17ee242f6ac73689212b7474b P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 01:59:46.176229 6311 catalog_manager.cc:5484] T 481e1968fa634172a4312352e1a1b6ec P 492448570cc94ed1879feb2ec3b9051d reported cstate change: term changed from 0 to 1, leader changed from <none> to 492448570cc94ed1879feb2ec3b9051d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:46.176921 6316 catalog_manager.cc:5484] T 771f069dd030449489008509f9794d7e P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: term changed from 0 to 1, leader changed from <none> to dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129). New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:46.181200 6582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37b626b17ee242f6ac73689212b7474b" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 01:59:46.181361 6582 raft_consensus.cc:2454] T 37b626b17ee242f6ac73689212b7474b P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 01:59:46.181564 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37b626b17ee242f6ac73689212b7474b" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 01:59:46.181680 6458 raft_consensus.cc:2454] T 37b626b17ee242f6ac73689212b7474b P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 01:59:46.181684 6655 leader_election.cc:302] T 37b626b17ee242f6ac73689212b7474b P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 01:59:46.181910 7449 raft_consensus.cc:2781] T 37b626b17ee242f6ac73689212b7474b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:46.181979 7449 raft_consensus.cc:482] T 37b626b17ee242f6ac73689212b7474b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:46.182024 7449 raft_consensus.cc:3037] T 37b626b17ee242f6ac73689212b7474b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:46.182828 7449 raft_consensus.cc:504] T 37b626b17ee242f6ac73689212b7474b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:46.183135 7449 leader_election.cc:288] T 37b626b17ee242f6ac73689212b7474b P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Requested vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 01:59:46.183468 6582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37b626b17ee242f6ac73689212b7474b" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 01:59:46.183470 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37b626b17ee242f6ac73689212b7474b" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 01:59:46.183584 6582 raft_consensus.cc:3037] T 37b626b17ee242f6ac73689212b7474b P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:46.183583 6458 raft_consensus.cc:3037] T 37b626b17ee242f6ac73689212b7474b P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:46.184504 6582 raft_consensus.cc:2454] T 37b626b17ee242f6ac73689212b7474b P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 01:59:46.184504 6458 raft_consensus.cc:2454] T 37b626b17ee242f6ac73689212b7474b P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 01:59:46.184777 6653 leader_election.cc:302] T 37b626b17ee242f6ac73689212b7474b P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2ac2eb7cc264047b9ab32fe3a4fc025, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:46.184932 7449 raft_consensus.cc:2781] T 37b626b17ee242f6ac73689212b7474b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:46.185156 7449 raft_consensus.cc:686] T 37b626b17ee242f6ac73689212b7474b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Becoming Leader. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Running, Role: LEADER
I20240503 01:59:46.185528 7449 consensus_queue.cc:237] T 37b626b17ee242f6ac73689212b7474b P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:46.188305 6316 catalog_manager.cc:5484] T 37b626b17ee242f6ac73689212b7474b P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } }
W20240503 01:59:46.190800 6636 tablet.cc:2369] T 37b626b17ee242f6ac73689212b7474b P 80fd18ef4d604c86b909d5c008a46b4f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 01:59:46.268930 6505 tablet.cc:2369] T 37b626b17ee242f6ac73689212b7474b P dec3e4a04d604f7eaaa3f077d081218a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 01:59:46.296535 6898 tablet.cc:2369] T 481e1968fa634172a4312352e1a1b6ec P 492448570cc94ed1879feb2ec3b9051d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 01:59:46.305845 7471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:46.306133 7471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:46.309264 7471 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240503 01:59:46.404742 6851 raft_consensus.cc:1261] T 771f069dd030449489008509f9794d7e P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:46.405774 6720 raft_consensus.cc:1261] T 771f069dd030449489008509f9794d7e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:46.405791 7463 consensus_queue.cc:1035] T 771f069dd030449489008509f9794d7e P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:46.406572 6458 raft_consensus.cc:1261] T 481e1968fa634172a4312352e1a1b6ec P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:46.409090 6582 raft_consensus.cc:1261] T 481e1968fa634172a4312352e1a1b6ec P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:46.409076 7451 consensus_queue.cc:1035] T 481e1968fa634172a4312352e1a1b6ec P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:46.411572 7463 consensus_queue.cc:1035] T 771f069dd030449489008509f9794d7e P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:46.414579 6457 raft_consensus.cc:1261] T 37b626b17ee242f6ac73689212b7474b P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:46.415051 7469 consensus_queue.cc:1035] T 37b626b17ee242f6ac73689212b7474b P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:46.415143 7462 consensus_queue.cc:1035] T 481e1968fa634172a4312352e1a1b6ec P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:46.417845 6582 raft_consensus.cc:1261] T 37b626b17ee242f6ac73689212b7474b P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:46.418318 7469 consensus_queue.cc:1035] T 37b626b17ee242f6ac73689212b7474b P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:46.425622 7500 mvcc.cc:205] Tried to move back new op lower bound from 7023417697946636288 to 7023417697018605568. Current Snapshot: MvccSnapshot[applied={T|T < 7023417697946636288}]
I20240503 01:59:46.427717 7499 mvcc.cc:205] Tried to move back new op lower bound from 7023417697906659328 to 7023417696967557120. Current Snapshot: MvccSnapshot[applied={T|T < 7023417697906659328}]
I20240503 01:59:46.429035 7498 mvcc.cc:205] Tried to move back new op lower bound from 7023417697946636288 to 7023417697018605568. Current Snapshot: MvccSnapshot[applied={T|T < 7023417697946636288}]
I20240503 01:59:46.429582 7510 mvcc.cc:205] Tried to move back new op lower bound from 7023417697906659328 to 7023417696967557120. Current Snapshot: MvccSnapshot[applied={T|T < 7023417697906659328}]
INSERT report
rows total: 2000
time total: 121.219 ms
time per row: 0.0606094 ms
I20240503 01:59:46.477841 6313 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:46.483974 6699 tablet_service.cc:1467] Processing CreateTablet for tablet 74ebf5d0fb9a42e6a7ec743a98960e13 (DEFAULT_TABLE table=rebalance_test_table_1 [id=6e24d367bc1d434584d8a7363cf538b6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:46.484313 6699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74ebf5d0fb9a42e6a7ec743a98960e13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:46.484297 6568 tablet_service.cc:1467] Processing CreateTablet for tablet 74ebf5d0fb9a42e6a7ec743a98960e13 (DEFAULT_TABLE table=rebalance_test_table_1 [id=6e24d367bc1d434584d8a7363cf538b6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:46.484692 6568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74ebf5d0fb9a42e6a7ec743a98960e13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:46.484920 6438 tablet_service.cc:1467] Processing CreateTablet for tablet 7a44b6d65bfa43e291b06b0dfd38fab3 (DEFAULT_TABLE table=rebalance_test_table_1 [id=6e24d367bc1d434584d8a7363cf538b6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:46.485203 6438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a44b6d65bfa43e291b06b0dfd38fab3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:46.486471 7447 tablet_bootstrap.cc:492] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 01:59:46.486702 6569 tablet_service.cc:1467] Processing CreateTablet for tablet 1fa274c98fc147f29344ed537e2ccf17 (DEFAULT_TABLE table=rebalance_test_table_1 [id=6e24d367bc1d434584d8a7363cf538b6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:46.486927 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fa274c98fc147f29344ed537e2ccf17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:46.487484 7447 tablet_bootstrap.cc:654] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:46.487507 6831 tablet_service.cc:1467] Processing CreateTablet for tablet 1fa274c98fc147f29344ed537e2ccf17 (DEFAULT_TABLE table=rebalance_test_table_1 [id=6e24d367bc1d434584d8a7363cf538b6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:46.487792 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fa274c98fc147f29344ed537e2ccf17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:46.487967 6700 tablet_service.cc:1467] Processing CreateTablet for tablet 7a44b6d65bfa43e291b06b0dfd38fab3 (DEFAULT_TABLE table=rebalance_test_table_1 [id=6e24d367bc1d434584d8a7363cf538b6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:46.488107 7443 tablet_bootstrap.cc:492] T 74ebf5d0fb9a42e6a7ec743a98960e13 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 01:59:46.488194 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a44b6d65bfa43e291b06b0dfd38fab3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:46.488507 7442 tablet_bootstrap.cc:492] T 7a44b6d65bfa43e291b06b0dfd38fab3 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 01:59:46.489285 7442 tablet_bootstrap.cc:654] T 7a44b6d65bfa43e291b06b0dfd38fab3 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:46.489285 7443 tablet_bootstrap.cc:654] T 74ebf5d0fb9a42e6a7ec743a98960e13 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:46.487507 6830 tablet_service.cc:1467] Processing CreateTablet for tablet 74ebf5d0fb9a42e6a7ec743a98960e13 (DEFAULT_TABLE table=rebalance_test_table_1 [id=6e24d367bc1d434584d8a7363cf538b6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:46.490129 7441 tablet_bootstrap.cc:492] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 01:59:46.490303 6830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74ebf5d0fb9a42e6a7ec743a98960e13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:46.490728 7447 tablet_bootstrap.cc:492] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 01:59:46.490821 7447 ts_tablet_manager.cc:1367] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:46.490978 7441 tablet_bootstrap.cc:654] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:46.491267 7447 raft_consensus.cc:348] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:46.491367 7447 raft_consensus.cc:374] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:46.491396 7447 raft_consensus.cc:729] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 01:59:46.491505 7447 consensus_queue.cc:260] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:46.491637 6568 tablet_service.cc:1467] Processing CreateTablet for tablet 7a44b6d65bfa43e291b06b0dfd38fab3 (DEFAULT_TABLE table=rebalance_test_table_1 [id=6e24d367bc1d434584d8a7363cf538b6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:46.491756 7447 ts_tablet_manager.cc:1398] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:46.491861 6568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a44b6d65bfa43e291b06b0dfd38fab3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:46.491892 7447 tablet_bootstrap.cc:492] T 1fa274c98fc147f29344ed537e2ccf17 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 01:59:46.492755 7447 tablet_bootstrap.cc:654] T 1fa274c98fc147f29344ed537e2ccf17 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:46.492926 6699 tablet_service.cc:1467] Processing CreateTablet for tablet 1fa274c98fc147f29344ed537e2ccf17 (DEFAULT_TABLE table=rebalance_test_table_1 [id=6e24d367bc1d434584d8a7363cf538b6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:46.493167 6699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fa274c98fc147f29344ed537e2ccf17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:46.493897 7447 tablet_bootstrap.cc:492] T 1fa274c98fc147f29344ed537e2ccf17 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 01:59:46.493978 7447 ts_tablet_manager.cc:1367] T 1fa274c98fc147f29344ed537e2ccf17 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:46.494397 7447 raft_consensus.cc:348] T 1fa274c98fc147f29344ed537e2ccf17 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:46.494513 7447 raft_consensus.cc:374] T 1fa274c98fc147f29344ed537e2ccf17 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:46.494544 7447 raft_consensus.cc:729] T 1fa274c98fc147f29344ed537e2ccf17 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 01:59:46.494678 7447 consensus_queue.cc:260] T 1fa274c98fc147f29344ed537e2ccf17 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:46.494997 7447 ts_tablet_manager.cc:1398] T 1fa274c98fc147f29344ed537e2ccf17 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:46.495142 7447 tablet_bootstrap.cc:492] T 7a44b6d65bfa43e291b06b0dfd38fab3 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 01:59:46.495204 7442 tablet_bootstrap.cc:492] T 7a44b6d65bfa43e291b06b0dfd38fab3 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 01:59:46.495283 7442 ts_tablet_manager.cc:1367] T 7a44b6d65bfa43e291b06b0dfd38fab3 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 01:59:46.495731 7442 raft_consensus.cc:348] T 7a44b6d65bfa43e291b06b0dfd38fab3 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:46.495858 7442 raft_consensus.cc:374] T 7a44b6d65bfa43e291b06b0dfd38fab3 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:46.495890 7442 raft_consensus.cc:729] T 7a44b6d65bfa43e291b06b0dfd38fab3 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 01:59:46.495929 7443 tablet_bootstrap.cc:492] T 74ebf5d0fb9a42e6a7ec743a98960e13 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 01:59:46.496018 7443 ts_tablet_manager.cc:1367] T 74ebf5d0fb9a42e6a7ec743a98960e13 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 01:59:46.496017 7442 consensus_queue.cc:260] T 7a44b6d65bfa43e291b06b0dfd38fab3 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:46.496273 7442 ts_tablet_manager.cc:1398] T 7a44b6d65bfa43e291b06b0dfd38fab3 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:46.496351 7447 tablet_bootstrap.cc:654] T 7a44b6d65bfa43e291b06b0dfd38fab3 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:46.496598 7443 raft_consensus.cc:348] T 74ebf5d0fb9a42e6a7ec743a98960e13 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:46.496692 7443 raft_consensus.cc:374] T 74ebf5d0fb9a42e6a7ec743a98960e13 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:46.496721 7443 raft_consensus.cc:729] T 74ebf5d0fb9a42e6a7ec743a98960e13 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 01:59:46.496842 7443 consensus_queue.cc:260] T 74ebf5d0fb9a42e6a7ec743a98960e13 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:46.497092 7443 ts_tablet_manager.cc:1398] T 74ebf5d0fb9a42e6a7ec743a98960e13 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:46.497248 7443 tablet_bootstrap.cc:492] T 7a44b6d65bfa43e291b06b0dfd38fab3 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 01:59:46.497429 7447 tablet_bootstrap.cc:492] T 7a44b6d65bfa43e291b06b0dfd38fab3 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 01:59:46.497516 7447 ts_tablet_manager.cc:1367] T 7a44b6d65bfa43e291b06b0dfd38fab3 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 01:59:46.497910 7447 raft_consensus.cc:348] T 7a44b6d65bfa43e291b06b0dfd38fab3 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:46.498001 7447 raft_consensus.cc:374] T 7a44b6d65bfa43e291b06b0dfd38fab3 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:46.498032 7447 raft_consensus.cc:729] T 7a44b6d65bfa43e291b06b0dfd38fab3 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 01:59:46.498042 7443 tablet_bootstrap.cc:654] T 7a44b6d65bfa43e291b06b0dfd38fab3 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:46.498129 7447 consensus_queue.cc:260] T 7a44b6d65bfa43e291b06b0dfd38fab3 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:46.498440 7447 ts_tablet_manager.cc:1398] T 7a44b6d65bfa43e291b06b0dfd38fab3 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:46.500360 7443 tablet_bootstrap.cc:492] T 7a44b6d65bfa43e291b06b0dfd38fab3 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 01:59:46.500468 7443 ts_tablet_manager.cc:1367] T 7a44b6d65bfa43e291b06b0dfd38fab3 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 01:59:46.500739 7441 tablet_bootstrap.cc:492] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 01:59:46.500838 7441 ts_tablet_manager.cc:1367] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 01:59:46.500823 7443 raft_consensus.cc:348] T 7a44b6d65bfa43e291b06b0dfd38fab3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:46.500898 7443 raft_consensus.cc:374] T 7a44b6d65bfa43e291b06b0dfd38fab3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:46.500931 7443 raft_consensus.cc:729] T 7a44b6d65bfa43e291b06b0dfd38fab3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 01:59:46.501041 7443 consensus_queue.cc:260] T 7a44b6d65bfa43e291b06b0dfd38fab3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:46.501248 7443 ts_tablet_manager.cc:1398] T 7a44b6d65bfa43e291b06b0dfd38fab3 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:46.501281 7441 raft_consensus.cc:348] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:46.501386 7441 raft_consensus.cc:374] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:46.501386 7443 tablet_bootstrap.cc:492] T 1fa274c98fc147f29344ed537e2ccf17 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 01:59:46.501420 7441 raft_consensus.cc:729] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 01:59:46.501539 7441 consensus_queue.cc:260] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:46.501773 7441 ts_tablet_manager.cc:1398] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:46.501935 7441 tablet_bootstrap.cc:492] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 01:59:46.502235 7443 tablet_bootstrap.cc:654] T 1fa274c98fc147f29344ed537e2ccf17 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:46.503137 7441 tablet_bootstrap.cc:654] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:46.503170 7443 tablet_bootstrap.cc:492] T 1fa274c98fc147f29344ed537e2ccf17 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 01:59:46.503252 7443 ts_tablet_manager.cc:1367] T 1fa274c98fc147f29344ed537e2ccf17 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 01:59:46.503610 7443 raft_consensus.cc:348] T 1fa274c98fc147f29344ed537e2ccf17 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:46.503717 7443 raft_consensus.cc:374] T 1fa274c98fc147f29344ed537e2ccf17 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:46.503752 7443 raft_consensus.cc:729] T 1fa274c98fc147f29344ed537e2ccf17 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 01:59:46.503854 7443 consensus_queue.cc:260] T 1fa274c98fc147f29344ed537e2ccf17 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:46.504107 7443 ts_tablet_manager.cc:1398] T 1fa274c98fc147f29344ed537e2ccf17 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:46.504770 7441 tablet_bootstrap.cc:492] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 01:59:46.504861 7441 ts_tablet_manager.cc:1367] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:46.505234 7441 raft_consensus.cc:348] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:46.505365 7441 raft_consensus.cc:374] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:46.505402 7441 raft_consensus.cc:729] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 01:59:46.505530 7441 consensus_queue.cc:260] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:46.505800 7441 ts_tablet_manager.cc:1398] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:46.588780 7506 raft_consensus.cc:482] T 7a44b6d65bfa43e291b06b0dfd38fab3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:46.588918 7506 raft_consensus.cc:504] T 7a44b6d65bfa43e291b06b0dfd38fab3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:46.589365 7506 leader_election.cc:288] T 7a44b6d65bfa43e291b06b0dfd38fab3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 01:59:46.589840 6582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a44b6d65bfa43e291b06b0dfd38fab3" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 01:59:46.589840 6457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a44b6d65bfa43e291b06b0dfd38fab3" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 01:59:46.589999 6457 raft_consensus.cc:2454] T 7a44b6d65bfa43e291b06b0dfd38fab3 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 01:59:46.589999 6582 raft_consensus.cc:2454] T 7a44b6d65bfa43e291b06b0dfd38fab3 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 01:59:46.590323 6653 leader_election.cc:302] T 7a44b6d65bfa43e291b06b0dfd38fab3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2ac2eb7cc264047b9ab32fe3a4fc025, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:46.590495 7506 raft_consensus.cc:2781] T 7a44b6d65bfa43e291b06b0dfd38fab3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:46.590565 7506 raft_consensus.cc:482] T 7a44b6d65bfa43e291b06b0dfd38fab3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:46.590618 7506 raft_consensus.cc:3037] T 7a44b6d65bfa43e291b06b0dfd38fab3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:46.591459 7506 raft_consensus.cc:504] T 7a44b6d65bfa43e291b06b0dfd38fab3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:46.591811 7506 leader_election.cc:288] T 7a44b6d65bfa43e291b06b0dfd38fab3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Requested vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 01:59:46.592188 6582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a44b6d65bfa43e291b06b0dfd38fab3" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 01:59:46.592196 6457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a44b6d65bfa43e291b06b0dfd38fab3" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 01:59:46.592322 6457 raft_consensus.cc:3037] T 7a44b6d65bfa43e291b06b0dfd38fab3 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:46.592321 6582 raft_consensus.cc:3037] T 7a44b6d65bfa43e291b06b0dfd38fab3 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:46.593271 6457 raft_consensus.cc:2454] T 7a44b6d65bfa43e291b06b0dfd38fab3 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 01:59:46.593271 6582 raft_consensus.cc:2454] T 7a44b6d65bfa43e291b06b0dfd38fab3 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 01:59:46.593624 6653 leader_election.cc:302] T 7a44b6d65bfa43e291b06b0dfd38fab3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2ac2eb7cc264047b9ab32fe3a4fc025, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:46.593796 7506 raft_consensus.cc:2781] T 7a44b6d65bfa43e291b06b0dfd38fab3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:46.593870 7506 raft_consensus.cc:686] T 7a44b6d65bfa43e291b06b0dfd38fab3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Becoming Leader. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Running, Role: LEADER
I20240503 01:59:46.594013 7506 consensus_queue.cc:237] T 7a44b6d65bfa43e291b06b0dfd38fab3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:46.595808 6313 catalog_manager.cc:5484] T 7a44b6d65bfa43e291b06b0dfd38fab3 P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:46.730288 7462 raft_consensus.cc:482] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:46.730429 7462 raft_consensus.cc:504] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:46.730940 7462 leader_election.cc:288] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 01:59:46.731400 6582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fa274c98fc147f29344ed537e2ccf17" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 01:59:46.731550 6582 raft_consensus.cc:2454] T 1fa274c98fc147f29344ed537e2ccf17 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 01:59:46.731829 6786 leader_election.cc:302] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 80fd18ef4d604c86b909d5c008a46b4f; no voters:
I20240503 01:59:46.732005 7462 raft_consensus.cc:2781] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:46.732065 7462 raft_consensus.cc:482] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:46.732103 7462 raft_consensus.cc:3037] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:46.733124 7462 raft_consensus.cc:504] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:46.733491 7462 leader_election.cc:288] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Requested vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 01:59:46.733814 6582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fa274c98fc147f29344ed537e2ccf17" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 01:59:46.733911 6582 raft_consensus.cc:3037] T 1fa274c98fc147f29344ed537e2ccf17 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:46.734735 6582 raft_consensus.cc:2454] T 1fa274c98fc147f29344ed537e2ccf17 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 01:59:46.734970 6719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fa274c98fc147f29344ed537e2ccf17" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 01:59:46.735025 6786 leader_election.cc:302] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 80fd18ef4d604c86b909d5c008a46b4f; no voters:
I20240503 01:59:46.734970 6720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fa274c98fc147f29344ed537e2ccf17" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 01:59:46.735103 6719 raft_consensus.cc:3037] T 1fa274c98fc147f29344ed537e2ccf17 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:46.735206 7462 raft_consensus.cc:2781] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:46.735301 7462 raft_consensus.cc:686] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Becoming Leader. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Running, Role: LEADER
I20240503 01:59:46.735433 7462 consensus_queue.cc:237] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:46.736128 6719 raft_consensus.cc:2454] T 1fa274c98fc147f29344ed537e2ccf17 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 01:59:46.737809 6313 catalog_manager.cc:5484] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d reported cstate change: term changed from 0 to 1, leader changed from <none> to 492448570cc94ed1879feb2ec3b9051d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:46.861881 7456 raft_consensus.cc:482] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:46.862017 7456 raft_consensus.cc:504] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:46.863032 7456 leader_election.cc:288] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 01:59:46.866664 6719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ebf5d0fb9a42e6a7ec743a98960e13" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 01:59:46.866668 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ebf5d0fb9a42e6a7ec743a98960e13" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 01:59:46.866843 6851 raft_consensus.cc:2454] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 01:59:46.866843 6719 raft_consensus.cc:2454] T 74ebf5d0fb9a42e6a7ec743a98960e13 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 01:59:46.867210 6525 leader_election.cc:302] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 01:59:46.867441 7456 raft_consensus.cc:2781] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:46.867503 7456 raft_consensus.cc:482] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:46.867537 7456 raft_consensus.cc:3037] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:46.868423 7456 raft_consensus.cc:504] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:46.868773 7456 leader_election.cc:288] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Requested vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 01:59:46.869110 6719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ebf5d0fb9a42e6a7ec743a98960e13" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 01:59:46.869174 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ebf5d0fb9a42e6a7ec743a98960e13" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 01:59:46.869251 6719 raft_consensus.cc:3037] T 74ebf5d0fb9a42e6a7ec743a98960e13 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:46.869294 6851 raft_consensus.cc:3037] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:46.870020 6719 raft_consensus.cc:2454] T 74ebf5d0fb9a42e6a7ec743a98960e13 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 01:59:46.870194 6851 raft_consensus.cc:2454] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 01:59:46.870342 6525 leader_election.cc:302] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 01:59:46.870517 7456 raft_consensus.cc:2781] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:46.870707 7456 raft_consensus.cc:686] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Becoming Leader. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Running, Role: LEADER
I20240503 01:59:46.870997 7456 consensus_queue.cc:237] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:46.873519 6313 catalog_manager.cc:5484] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:46.989737 7506 consensus_queue.cc:1035] T 7a44b6d65bfa43e291b06b0dfd38fab3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:46.993353 7506 consensus_queue.cc:1035] T 7a44b6d65bfa43e291b06b0dfd38fab3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 01:59:47.060813 7547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:47.061105 7547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:47.064880 7547 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240503 01:59:47.126363 7451 consensus_queue.cc:1035] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:47.129887 7462 consensus_queue.cc:1035] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:47.173337 6720 raft_consensus.cc:1261] T 74ebf5d0fb9a42e6a7ec743a98960e13 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:47.173915 6851 raft_consensus.cc:1261] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:47.176752 7545 consensus_queue.cc:1035] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:47.177235 7545 consensus_queue.cc:1035] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 118.923 ms
time per row: 0.0594614 ms
I20240503 01:59:47.232656 6307 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:47.237965 6438 tablet_service.cc:1467] Processing CreateTablet for tablet ba8b8fdffeeb423a9d89dca999b25904 (DEFAULT_TABLE table=rebalance_test_table_2 [id=5a03be9f319a4f58a441327c9bbe6ad9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:47.237972 6568 tablet_service.cc:1467] Processing CreateTablet for tablet ba8b8fdffeeb423a9d89dca999b25904 (DEFAULT_TABLE table=rebalance_test_table_2 [id=5a03be9f319a4f58a441327c9bbe6ad9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:47.238273 6568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba8b8fdffeeb423a9d89dca999b25904. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:47.238274 6438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba8b8fdffeeb423a9d89dca999b25904. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:47.238289 6830 tablet_service.cc:1467] Processing CreateTablet for tablet ba8b8fdffeeb423a9d89dca999b25904 (DEFAULT_TABLE table=rebalance_test_table_2 [id=5a03be9f319a4f58a441327c9bbe6ad9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:47.238441 6831 tablet_service.cc:1467] Processing CreateTablet for tablet 714c076813ff4c4f84cb16d256f7b28c (DEFAULT_TABLE table=rebalance_test_table_2 [id=5a03be9f319a4f58a441327c9bbe6ad9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:47.238516 6830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba8b8fdffeeb423a9d89dca999b25904. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:47.238649 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 714c076813ff4c4f84cb16d256f7b28c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:47.240510 7602 tablet_bootstrap.cc:492] T ba8b8fdffeeb423a9d89dca999b25904 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 01:59:47.240644 7603 tablet_bootstrap.cc:492] T ba8b8fdffeeb423a9d89dca999b25904 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 01:59:47.240975 6438 tablet_service.cc:1467] Processing CreateTablet for tablet 714c076813ff4c4f84cb16d256f7b28c (DEFAULT_TABLE table=rebalance_test_table_2 [id=5a03be9f319a4f58a441327c9bbe6ad9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:47.241087 6568 tablet_service.cc:1467] Processing CreateTablet for tablet 7a44d033ab014466bd2100c0ca250ec8 (DEFAULT_TABLE table=rebalance_test_table_2 [id=5a03be9f319a4f58a441327c9bbe6ad9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:47.241194 6438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 714c076813ff4c4f84cb16d256f7b28c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:47.241267 6568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a44d033ab014466bd2100c0ca250ec8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:47.241345 7602 tablet_bootstrap.cc:654] T ba8b8fdffeeb423a9d89dca999b25904 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:47.241518 6699 tablet_service.cc:1467] Processing CreateTablet for tablet 714c076813ff4c4f84cb16d256f7b28c (DEFAULT_TABLE table=rebalance_test_table_2 [id=5a03be9f319a4f58a441327c9bbe6ad9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:47.241782 6699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 714c076813ff4c4f84cb16d256f7b28c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:47.243041 6436 tablet_service.cc:1467] Processing CreateTablet for tablet 7a44d033ab014466bd2100c0ca250ec8 (DEFAULT_TABLE table=rebalance_test_table_2 [id=5a03be9f319a4f58a441327c9bbe6ad9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:47.243260 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a44d033ab014466bd2100c0ca250ec8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:47.243389 7606 tablet_bootstrap.cc:492] T 714c076813ff4c4f84cb16d256f7b28c P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 01:59:47.243494 7603 tablet_bootstrap.cc:654] T ba8b8fdffeeb423a9d89dca999b25904 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:47.243750 7604 tablet_bootstrap.cc:492] T 714c076813ff4c4f84cb16d256f7b28c P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 01:59:47.244216 7606 tablet_bootstrap.cc:654] T 714c076813ff4c4f84cb16d256f7b28c P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:47.244647 7604 tablet_bootstrap.cc:654] T 714c076813ff4c4f84cb16d256f7b28c P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:47.245091 6831 tablet_service.cc:1467] Processing CreateTablet for tablet 7a44d033ab014466bd2100c0ca250ec8 (DEFAULT_TABLE table=rebalance_test_table_2 [id=5a03be9f319a4f58a441327c9bbe6ad9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:47.245306 7606 tablet_bootstrap.cc:492] T 714c076813ff4c4f84cb16d256f7b28c P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 01:59:47.245316 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a44d033ab014466bd2100c0ca250ec8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:47.245379 7606 ts_tablet_manager.cc:1367] T 714c076813ff4c4f84cb16d256f7b28c P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:47.245795 7606 raft_consensus.cc:348] T 714c076813ff4c4f84cb16d256f7b28c P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:47.245898 7606 raft_consensus.cc:374] T 714c076813ff4c4f84cb16d256f7b28c P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:47.245930 7606 raft_consensus.cc:729] T 714c076813ff4c4f84cb16d256f7b28c P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 01:59:47.246062 7606 consensus_queue.cc:260] T 714c076813ff4c4f84cb16d256f7b28c P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:47.246335 7606 ts_tablet_manager.cc:1398] T 714c076813ff4c4f84cb16d256f7b28c P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:47.246698 7603 tablet_bootstrap.cc:492] T ba8b8fdffeeb423a9d89dca999b25904 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 01:59:47.246800 7602 tablet_bootstrap.cc:492] T ba8b8fdffeeb423a9d89dca999b25904 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 01:59:47.246831 7603 ts_tablet_manager.cc:1367] T ba8b8fdffeeb423a9d89dca999b25904 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 01:59:47.246901 7602 ts_tablet_manager.cc:1367] T ba8b8fdffeeb423a9d89dca999b25904 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 01:59:47.247085 7604 tablet_bootstrap.cc:492] T 714c076813ff4c4f84cb16d256f7b28c P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 01:59:47.247174 7604 ts_tablet_manager.cc:1367] T 714c076813ff4c4f84cb16d256f7b28c P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:47.247315 7602 raft_consensus.cc:348] T ba8b8fdffeeb423a9d89dca999b25904 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:47.247417 7602 raft_consensus.cc:374] T ba8b8fdffeeb423a9d89dca999b25904 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:47.247450 7602 raft_consensus.cc:729] T ba8b8fdffeeb423a9d89dca999b25904 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 01:59:47.247584 7602 consensus_queue.cc:260] T ba8b8fdffeeb423a9d89dca999b25904 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:47.247596 7604 raft_consensus.cc:348] T 714c076813ff4c4f84cb16d256f7b28c P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:47.247697 7604 raft_consensus.cc:374] T 714c076813ff4c4f84cb16d256f7b28c P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:47.247736 7604 raft_consensus.cc:729] T 714c076813ff4c4f84cb16d256f7b28c P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 01:59:47.247833 7602 ts_tablet_manager.cc:1398] T ba8b8fdffeeb423a9d89dca999b25904 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:47.247859 7604 consensus_queue.cc:260] T 714c076813ff4c4f84cb16d256f7b28c P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:47.248009 7602 tablet_bootstrap.cc:492] T 714c076813ff4c4f84cb16d256f7b28c P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 01:59:47.248121 7604 ts_tablet_manager.cc:1398] T 714c076813ff4c4f84cb16d256f7b28c P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:47.248287 7604 tablet_bootstrap.cc:492] T ba8b8fdffeeb423a9d89dca999b25904 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 01:59:47.248819 7602 tablet_bootstrap.cc:654] T 714c076813ff4c4f84cb16d256f7b28c P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:47.249136 7604 tablet_bootstrap.cc:654] T ba8b8fdffeeb423a9d89dca999b25904 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:47.247888 7603 raft_consensus.cc:348] T ba8b8fdffeeb423a9d89dca999b25904 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:47.250007 7603 raft_consensus.cc:374] T ba8b8fdffeeb423a9d89dca999b25904 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:47.250057 7603 raft_consensus.cc:729] T ba8b8fdffeeb423a9d89dca999b25904 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 01:59:47.250236 7603 consensus_queue.cc:260] T ba8b8fdffeeb423a9d89dca999b25904 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:47.250490 7602 tablet_bootstrap.cc:492] T 714c076813ff4c4f84cb16d256f7b28c P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 01:59:47.250564 7602 ts_tablet_manager.cc:1367] T 714c076813ff4c4f84cb16d256f7b28c P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 01:59:47.250594 7603 ts_tablet_manager.cc:1398] T ba8b8fdffeeb423a9d89dca999b25904 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 01:59:47.250763 7603 tablet_bootstrap.cc:492] T 7a44d033ab014466bd2100c0ca250ec8 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 01:59:47.250984 7602 raft_consensus.cc:348] T 714c076813ff4c4f84cb16d256f7b28c P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:47.251073 7602 raft_consensus.cc:374] T 714c076813ff4c4f84cb16d256f7b28c P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:47.251103 7602 raft_consensus.cc:729] T 714c076813ff4c4f84cb16d256f7b28c P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 01:59:47.251246 7602 consensus_queue.cc:260] T 714c076813ff4c4f84cb16d256f7b28c P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:47.251556 7602 ts_tablet_manager.cc:1398] T 714c076813ff4c4f84cb16d256f7b28c P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:47.251729 7602 tablet_bootstrap.cc:492] T 7a44d033ab014466bd2100c0ca250ec8 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 01:59:47.252583 7602 tablet_bootstrap.cc:654] T 7a44d033ab014466bd2100c0ca250ec8 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:47.252774 7603 tablet_bootstrap.cc:654] T 7a44d033ab014466bd2100c0ca250ec8 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:47.253659 7602 tablet_bootstrap.cc:492] T 7a44d033ab014466bd2100c0ca250ec8 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 01:59:47.253747 7602 ts_tablet_manager.cc:1367] T 7a44d033ab014466bd2100c0ca250ec8 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:47.253881 7603 tablet_bootstrap.cc:492] T 7a44d033ab014466bd2100c0ca250ec8 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 01:59:47.253890 7604 tablet_bootstrap.cc:492] T ba8b8fdffeeb423a9d89dca999b25904 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 01:59:47.253978 7604 ts_tablet_manager.cc:1367] T ba8b8fdffeeb423a9d89dca999b25904 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 01:59:47.253996 7603 ts_tablet_manager.cc:1367] T 7a44d033ab014466bd2100c0ca250ec8 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 01:59:47.254186 7602 raft_consensus.cc:348] T 7a44d033ab014466bd2100c0ca250ec8 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:47.254294 7602 raft_consensus.cc:374] T 7a44d033ab014466bd2100c0ca250ec8 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:47.254324 7602 raft_consensus.cc:729] T 7a44d033ab014466bd2100c0ca250ec8 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 01:59:47.254400 7604 raft_consensus.cc:348] T ba8b8fdffeeb423a9d89dca999b25904 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:47.254436 7602 consensus_queue.cc:260] T 7a44d033ab014466bd2100c0ca250ec8 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:47.254508 7604 raft_consensus.cc:374] T ba8b8fdffeeb423a9d89dca999b25904 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:47.254477 7603 raft_consensus.cc:348] T 7a44d033ab014466bd2100c0ca250ec8 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:47.254544 7604 raft_consensus.cc:729] T ba8b8fdffeeb423a9d89dca999b25904 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 01:59:47.254559 7603 raft_consensus.cc:374] T 7a44d033ab014466bd2100c0ca250ec8 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:47.254586 7603 raft_consensus.cc:729] T 7a44d033ab014466bd2100c0ca250ec8 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 01:59:47.254657 7604 consensus_queue.cc:260] T ba8b8fdffeeb423a9d89dca999b25904 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:47.254701 7602 ts_tablet_manager.cc:1398] T 7a44d033ab014466bd2100c0ca250ec8 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:47.254760 7603 consensus_queue.cc:260] T 7a44d033ab014466bd2100c0ca250ec8 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:47.254909 7604 ts_tablet_manager.cc:1398] T ba8b8fdffeeb423a9d89dca999b25904 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:47.255062 7604 tablet_bootstrap.cc:492] T 7a44d033ab014466bd2100c0ca250ec8 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 01:59:47.255506 7603 ts_tablet_manager.cc:1398] T 7a44d033ab014466bd2100c0ca250ec8 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:47.255965 7604 tablet_bootstrap.cc:654] T 7a44d033ab014466bd2100c0ca250ec8 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:47.257068 7604 tablet_bootstrap.cc:492] T 7a44d033ab014466bd2100c0ca250ec8 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 01:59:47.257161 7604 ts_tablet_manager.cc:1367] T 7a44d033ab014466bd2100c0ca250ec8 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:47.257555 7604 raft_consensus.cc:348] T 7a44d033ab014466bd2100c0ca250ec8 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:47.257661 7604 raft_consensus.cc:374] T 7a44d033ab014466bd2100c0ca250ec8 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:47.257704 7604 raft_consensus.cc:729] T 7a44d033ab014466bd2100c0ca250ec8 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 01:59:47.257819 7604 consensus_queue.cc:260] T 7a44d033ab014466bd2100c0ca250ec8 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:47.258082 7604 ts_tablet_manager.cc:1398] T 7a44d033ab014466bd2100c0ca250ec8 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:47.268744 7462 raft_consensus.cc:482] T 714c076813ff4c4f84cb16d256f7b28c P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:47.268903 7462 raft_consensus.cc:504] T 714c076813ff4c4f84cb16d256f7b28c P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:47.269270 7462 leader_election.cc:288] T 714c076813ff4c4f84cb16d256f7b28c P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 01:59:47.269665 6457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "714c076813ff4c4f84cb16d256f7b28c" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 01:59:47.269804 6457 raft_consensus.cc:2454] T 714c076813ff4c4f84cb16d256f7b28c P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 01:59:47.269939 6719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "714c076813ff4c4f84cb16d256f7b28c" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 01:59:47.270090 6719 raft_consensus.cc:2454] T 714c076813ff4c4f84cb16d256f7b28c P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 01:59:47.270119 6784 leader_election.cc:302] T 714c076813ff4c4f84cb16d256f7b28c P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:47.270298 7462 raft_consensus.cc:2781] T 714c076813ff4c4f84cb16d256f7b28c P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:47.270380 7462 raft_consensus.cc:482] T 714c076813ff4c4f84cb16d256f7b28c P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:47.270418 7462 raft_consensus.cc:3037] T 714c076813ff4c4f84cb16d256f7b28c P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:47.271265 7462 raft_consensus.cc:504] T 714c076813ff4c4f84cb16d256f7b28c P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:47.271610 7462 leader_election.cc:288] T 714c076813ff4c4f84cb16d256f7b28c P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Requested vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 01:59:47.271574 7553 raft_consensus.cc:482] T 7a44d033ab014466bd2100c0ca250ec8 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:47.271695 7553 raft_consensus.cc:504] T 7a44d033ab014466bd2100c0ca250ec8 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:47.272018 6457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "714c076813ff4c4f84cb16d256f7b28c" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 01:59:47.272114 6457 raft_consensus.cc:3037] T 714c076813ff4c4f84cb16d256f7b28c P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:47.272071 6719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "714c076813ff4c4f84cb16d256f7b28c" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 01:59:47.272208 6719 raft_consensus.cc:3037] T 714c076813ff4c4f84cb16d256f7b28c P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:47.272218 7553 leader_election.cc:288] T 7a44d033ab014466bd2100c0ca250ec8 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 01:59:47.273065 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a44d033ab014466bd2100c0ca250ec8" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 01:59:47.273104 6457 raft_consensus.cc:2454] T 714c076813ff4c4f84cb16d256f7b28c P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 01:59:47.273196 6851 raft_consensus.cc:2454] T 7a44d033ab014466bd2100c0ca250ec8 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 01:59:47.273195 6719 raft_consensus.cc:2454] T 714c076813ff4c4f84cb16d256f7b28c P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 01:59:47.273474 6784 leader_election.cc:302] T 714c076813ff4c4f84cb16d256f7b28c P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:47.273557 6394 leader_election.cc:302] T 7a44d033ab014466bd2100c0ca250ec8 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:47.273725 7553 raft_consensus.cc:2781] T 7a44d033ab014466bd2100c0ca250ec8 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:47.273737 7462 raft_consensus.cc:2781] T 714c076813ff4c4f84cb16d256f7b28c P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:47.273785 7553 raft_consensus.cc:482] T 7a44d033ab014466bd2100c0ca250ec8 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:47.273830 7553 raft_consensus.cc:3037] T 7a44d033ab014466bd2100c0ca250ec8 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:47.273834 7462 raft_consensus.cc:686] T 714c076813ff4c4f84cb16d256f7b28c P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Becoming Leader. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Running, Role: LEADER
I20240503 01:59:47.273972 7462 consensus_queue.cc:237] T 714c076813ff4c4f84cb16d256f7b28c P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:47.274662 7553 raft_consensus.cc:504] T 7a44d033ab014466bd2100c0ca250ec8 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:47.275010 7553 leader_election.cc:288] T 7a44d033ab014466bd2100c0ca250ec8 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Requested vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 01:59:47.275393 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a44d033ab014466bd2100c0ca250ec8" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 01:59:47.275492 6851 raft_consensus.cc:3037] T 7a44d033ab014466bd2100c0ca250ec8 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:47.276177 6851 raft_consensus.cc:2454] T 7a44d033ab014466bd2100c0ca250ec8 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 01:59:47.276458 6394 leader_election.cc:302] T 7a44d033ab014466bd2100c0ca250ec8 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:47.276648 7553 raft_consensus.cc:2781] T 7a44d033ab014466bd2100c0ca250ec8 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:47.276736 7553 raft_consensus.cc:686] T 7a44d033ab014466bd2100c0ca250ec8 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Becoming Leader. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Running, Role: LEADER
I20240503 01:59:47.276867 7553 consensus_queue.cc:237] T 7a44d033ab014466bd2100c0ca250ec8 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:47.277235 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a44d033ab014466bd2100c0ca250ec8" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 01:59:47.277201 6582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a44d033ab014466bd2100c0ca250ec8" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 01:59:47.277344 6589 raft_consensus.cc:3037] T 7a44d033ab014466bd2100c0ca250ec8 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:47.278025 6307 catalog_manager.cc:5484] T 714c076813ff4c4f84cb16d256f7b28c P 492448570cc94ed1879feb2ec3b9051d reported cstate change: term changed from 0 to 1, leader changed from <none> to 492448570cc94ed1879feb2ec3b9051d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:47.278242 6589 raft_consensus.cc:2454] T 7a44d033ab014466bd2100c0ca250ec8 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 01:59:47.278550 6313 catalog_manager.cc:5484] T 7a44d033ab014466bd2100c0ca250ec8 P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: term changed from 0 to 1, leader changed from <none> to dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129). New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:47.455832 7553 raft_consensus.cc:482] T ba8b8fdffeeb423a9d89dca999b25904 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:47.455989 7553 raft_consensus.cc:504] T ba8b8fdffeeb423a9d89dca999b25904 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:47.456444 7553 leader_election.cc:288] T ba8b8fdffeeb423a9d89dca999b25904 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 01:59:47.456921 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8b8fdffeeb423a9d89dca999b25904" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 01:59:47.456930 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8b8fdffeeb423a9d89dca999b25904" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 01:59:47.457052 6589 raft_consensus.cc:2454] T ba8b8fdffeeb423a9d89dca999b25904 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 01:59:47.457078 6851 raft_consensus.cc:2454] T ba8b8fdffeeb423a9d89dca999b25904 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 01:59:47.457374 6393 leader_election.cc:302] T ba8b8fdffeeb423a9d89dca999b25904 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:47.457541 7553 raft_consensus.cc:2781] T ba8b8fdffeeb423a9d89dca999b25904 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:47.457607 7553 raft_consensus.cc:482] T ba8b8fdffeeb423a9d89dca999b25904 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:47.457659 7553 raft_consensus.cc:3037] T ba8b8fdffeeb423a9d89dca999b25904 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:47.458431 7553 raft_consensus.cc:504] T ba8b8fdffeeb423a9d89dca999b25904 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:47.458770 7553 leader_election.cc:288] T ba8b8fdffeeb423a9d89dca999b25904 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Requested vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 01:59:47.459116 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8b8fdffeeb423a9d89dca999b25904" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 01:59:47.459127 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8b8fdffeeb423a9d89dca999b25904" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 01:59:47.459232 6589 raft_consensus.cc:3037] T ba8b8fdffeeb423a9d89dca999b25904 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:47.459239 6851 raft_consensus.cc:3037] T ba8b8fdffeeb423a9d89dca999b25904 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:47.459947 6589 raft_consensus.cc:2454] T ba8b8fdffeeb423a9d89dca999b25904 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 01:59:47.459969 6851 raft_consensus.cc:2454] T ba8b8fdffeeb423a9d89dca999b25904 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 01:59:47.460220 6393 leader_election.cc:302] T ba8b8fdffeeb423a9d89dca999b25904 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:47.460394 7553 raft_consensus.cc:2781] T ba8b8fdffeeb423a9d89dca999b25904 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:47.460481 7553 raft_consensus.cc:686] T ba8b8fdffeeb423a9d89dca999b25904 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Becoming Leader. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Running, Role: LEADER
I20240503 01:59:47.460618 7553 consensus_queue.cc:237] T ba8b8fdffeeb423a9d89dca999b25904 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:47.462270 6313 catalog_manager.cc:5484] T ba8b8fdffeeb423a9d89dca999b25904 P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: term changed from 0 to 1, leader changed from <none> to dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129). New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } }
W20240503 01:59:47.589689 7620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:47.589949 7620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:47.593164 7620 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240503 01:59:47.671476 6850 raft_consensus.cc:1261] T 7a44d033ab014466bd2100c0ca250ec8 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:47.672019 6851 raft_consensus.cc:1261] T ba8b8fdffeeb423a9d89dca999b25904 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:47.672052 6589 raft_consensus.cc:1261] T ba8b8fdffeeb423a9d89dca999b25904 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:47.672503 7448 consensus_queue.cc:1035] T ba8b8fdffeeb423a9d89dca999b25904 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:47.672618 7553 consensus_queue.cc:1035] T 7a44d033ab014466bd2100c0ca250ec8 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:47.672806 7448 consensus_queue.cc:1035] T ba8b8fdffeeb423a9d89dca999b25904 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:47.672930 6719 raft_consensus.cc:1261] T 714c076813ff4c4f84cb16d256f7b28c P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:47.673302 6457 raft_consensus.cc:1261] T 714c076813ff4c4f84cb16d256f7b28c P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:47.673707 7462 consensus_queue.cc:1035] T 714c076813ff4c4f84cb16d256f7b28c P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:47.674434 6582 raft_consensus.cc:1261] T 7a44d033ab014466bd2100c0ca250ec8 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:47.675968 7448 consensus_queue.cc:1035] T 7a44d033ab014466bd2100c0ca250ec8 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:47.680635 7462 consensus_queue.cc:1035] T 714c076813ff4c4f84cb16d256f7b28c P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 100.944 ms
time per row: 0.050472 ms
I20240503 01:59:47.735862 6311 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:47.741575 6568 tablet_service.cc:1467] Processing CreateTablet for tablet ac7fa0c3722a461c91fb237fd85794a3 (DEFAULT_TABLE table=rebalance_test_table_3 [id=72f29547da0b4a95974018c2782cb359]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:47.741744 6699 tablet_service.cc:1467] Processing CreateTablet for tablet ac7fa0c3722a461c91fb237fd85794a3 (DEFAULT_TABLE table=rebalance_test_table_3 [id=72f29547da0b4a95974018c2782cb359]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:47.741833 6436 tablet_service.cc:1467] Processing CreateTablet for tablet ac7fa0c3722a461c91fb237fd85794a3 (DEFAULT_TABLE table=rebalance_test_table_3 [id=72f29547da0b4a95974018c2782cb359]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:47.741904 6568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac7fa0c3722a461c91fb237fd85794a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:47.742022 6699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac7fa0c3722a461c91fb237fd85794a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:47.742075 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac7fa0c3722a461c91fb237fd85794a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:47.743623 7606 tablet_bootstrap.cc:492] T ac7fa0c3722a461c91fb237fd85794a3 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 01:59:47.743953 7603 tablet_bootstrap.cc:492] T ac7fa0c3722a461c91fb237fd85794a3 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 01:59:47.744513 7602 tablet_bootstrap.cc:492] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 01:59:47.744516 7606 tablet_bootstrap.cc:654] T ac7fa0c3722a461c91fb237fd85794a3 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:47.745306 7602 tablet_bootstrap.cc:654] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:47.745484 6436 tablet_service.cc:1467] Processing CreateTablet for tablet 4392cd97a4334d77ac6403002d991758 (DEFAULT_TABLE table=rebalance_test_table_3 [id=72f29547da0b4a95974018c2782cb359]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:47.745705 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4392cd97a4334d77ac6403002d991758. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:47.745734 6700 tablet_service.cc:1467] Processing CreateTablet for tablet 4392cd97a4334d77ac6403002d991758 (DEFAULT_TABLE table=rebalance_test_table_3 [id=72f29547da0b4a95974018c2782cb359]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:47.745771 6831 tablet_service.cc:1467] Processing CreateTablet for tablet 915028b3478341d588ed6031e657c259 (DEFAULT_TABLE table=rebalance_test_table_3 [id=72f29547da0b4a95974018c2782cb359]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:47.745954 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4392cd97a4334d77ac6403002d991758. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:47.746037 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 915028b3478341d588ed6031e657c259. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:47.746419 7603 tablet_bootstrap.cc:654] T ac7fa0c3722a461c91fb237fd85794a3 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:47.747498 7603 tablet_bootstrap.cc:492] T ac7fa0c3722a461c91fb237fd85794a3 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 01:59:47.747591 7603 ts_tablet_manager.cc:1367] T ac7fa0c3722a461c91fb237fd85794a3 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240503 01:59:47.748050 7603 raft_consensus.cc:348] T ac7fa0c3722a461c91fb237fd85794a3 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:47.748160 7603 raft_consensus.cc:374] T ac7fa0c3722a461c91fb237fd85794a3 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:47.748194 7603 raft_consensus.cc:729] T ac7fa0c3722a461c91fb237fd85794a3 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 01:59:47.748147 6830 tablet_service.cc:1467] Processing CreateTablet for tablet 4392cd97a4334d77ac6403002d991758 (DEFAULT_TABLE table=rebalance_test_table_3 [id=72f29547da0b4a95974018c2782cb359]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:47.748268 7606 tablet_bootstrap.cc:492] T ac7fa0c3722a461c91fb237fd85794a3 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 01:59:47.748344 7606 ts_tablet_manager.cc:1367] T ac7fa0c3722a461c91fb237fd85794a3 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240503 01:59:47.748370 6830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4392cd97a4334d77ac6403002d991758. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:47.748317 7603 consensus_queue.cc:260] T ac7fa0c3722a461c91fb237fd85794a3 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:47.748481 6699 tablet_service.cc:1467] Processing CreateTablet for tablet 915028b3478341d588ed6031e657c259 (DEFAULT_TABLE table=rebalance_test_table_3 [id=72f29547da0b4a95974018c2782cb359]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:47.748661 7603 ts_tablet_manager.cc:1398] T ac7fa0c3722a461c91fb237fd85794a3 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:47.748693 6699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 915028b3478341d588ed6031e657c259. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:47.748793 7606 raft_consensus.cc:348] T ac7fa0c3722a461c91fb237fd85794a3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:47.748890 7606 raft_consensus.cc:374] T ac7fa0c3722a461c91fb237fd85794a3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:47.748916 7606 raft_consensus.cc:729] T ac7fa0c3722a461c91fb237fd85794a3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 01:59:47.749022 7606 consensus_queue.cc:260] T ac7fa0c3722a461c91fb237fd85794a3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:47.749280 7606 ts_tablet_manager.cc:1398] T ac7fa0c3722a461c91fb237fd85794a3 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:47.749444 7606 tablet_bootstrap.cc:492] T 4392cd97a4334d77ac6403002d991758 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 01:59:47.750208 7604 tablet_bootstrap.cc:492] T 915028b3478341d588ed6031e657c259 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 01:59:47.750404 7606 tablet_bootstrap.cc:654] T 4392cd97a4334d77ac6403002d991758 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:47.751013 7604 tablet_bootstrap.cc:654] T 915028b3478341d588ed6031e657c259 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:47.751955 7604 tablet_bootstrap.cc:492] T 915028b3478341d588ed6031e657c259 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 01:59:47.752034 7604 ts_tablet_manager.cc:1367] T 915028b3478341d588ed6031e657c259 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:47.752146 6568 tablet_service.cc:1467] Processing CreateTablet for tablet 915028b3478341d588ed6031e657c259 (DEFAULT_TABLE table=rebalance_test_table_3 [id=72f29547da0b4a95974018c2782cb359]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:47.752424 6568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 915028b3478341d588ed6031e657c259. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:47.752502 7604 raft_consensus.cc:348] T 915028b3478341d588ed6031e657c259 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:47.752597 7604 raft_consensus.cc:374] T 915028b3478341d588ed6031e657c259 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:47.752627 7604 raft_consensus.cc:729] T 915028b3478341d588ed6031e657c259 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 01:59:47.752753 7604 consensus_queue.cc:260] T 915028b3478341d588ed6031e657c259 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:47.753003 7604 ts_tablet_manager.cc:1398] T 915028b3478341d588ed6031e657c259 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:47.753087 7602 tablet_bootstrap.cc:492] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 01:59:47.753161 7604 tablet_bootstrap.cc:492] T 4392cd97a4334d77ac6403002d991758 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 01:59:47.753211 7602 ts_tablet_manager.cc:1367] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 01:59:47.753738 7602 raft_consensus.cc:348] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:47.753845 7602 raft_consensus.cc:374] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:47.753876 7602 raft_consensus.cc:729] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 01:59:47.754017 7604 tablet_bootstrap.cc:654] T 4392cd97a4334d77ac6403002d991758 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:47.753988 7602 consensus_queue.cc:260] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:47.754231 7602 ts_tablet_manager.cc:1398] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:47.754391 7602 tablet_bootstrap.cc:492] T 4392cd97a4334d77ac6403002d991758 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 01:59:47.754547 7603 tablet_bootstrap.cc:492] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 01:59:47.755218 7606 tablet_bootstrap.cc:492] T 4392cd97a4334d77ac6403002d991758 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 01:59:47.755239 7602 tablet_bootstrap.cc:654] T 4392cd97a4334d77ac6403002d991758 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:47.755286 7606 ts_tablet_manager.cc:1367] T 4392cd97a4334d77ac6403002d991758 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 01:59:47.755352 7603 tablet_bootstrap.cc:654] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:47.755600 7606 raft_consensus.cc:348] T 4392cd97a4334d77ac6403002d991758 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:47.755673 7606 raft_consensus.cc:374] T 4392cd97a4334d77ac6403002d991758 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:47.755698 7606 raft_consensus.cc:729] T 4392cd97a4334d77ac6403002d991758 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 01:59:47.755808 7606 consensus_queue.cc:260] T 4392cd97a4334d77ac6403002d991758 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:47.756070 7606 ts_tablet_manager.cc:1398] T 4392cd97a4334d77ac6403002d991758 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:47.756253 7606 tablet_bootstrap.cc:492] T 915028b3478341d588ed6031e657c259 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 01:59:47.756512 7602 tablet_bootstrap.cc:492] T 4392cd97a4334d77ac6403002d991758 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 01:59:47.756585 7602 ts_tablet_manager.cc:1367] T 4392cd97a4334d77ac6403002d991758 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:47.756613 7603 tablet_bootstrap.cc:492] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 01:59:47.756695 7603 ts_tablet_manager.cc:1367] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:47.756984 7602 raft_consensus.cc:348] T 4392cd97a4334d77ac6403002d991758 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:47.757077 7602 raft_consensus.cc:374] T 4392cd97a4334d77ac6403002d991758 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:47.757107 7602 raft_consensus.cc:729] T 4392cd97a4334d77ac6403002d991758 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 01:59:47.757123 7606 tablet_bootstrap.cc:654] T 915028b3478341d588ed6031e657c259 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:47.757107 7603 raft_consensus.cc:348] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:47.757205 7603 raft_consensus.cc:374] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:47.757236 7603 raft_consensus.cc:729] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 01:59:47.757221 7602 consensus_queue.cc:260] T 4392cd97a4334d77ac6403002d991758 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:47.757346 7603 consensus_queue.cc:260] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:47.757560 7602 ts_tablet_manager.cc:1398] T 4392cd97a4334d77ac6403002d991758 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:47.757601 7603 ts_tablet_manager.cc:1398] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:47.758059 7606 tablet_bootstrap.cc:492] T 915028b3478341d588ed6031e657c259 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 01:59:47.758144 7606 ts_tablet_manager.cc:1367] T 915028b3478341d588ed6031e657c259 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:47.758644 7606 raft_consensus.cc:348] T 915028b3478341d588ed6031e657c259 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:47.758729 7606 raft_consensus.cc:374] T 915028b3478341d588ed6031e657c259 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:47.758756 7606 raft_consensus.cc:729] T 915028b3478341d588ed6031e657c259 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 01:59:47.758865 7606 consensus_queue.cc:260] T 915028b3478341d588ed6031e657c259 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:47.759094 7606 ts_tablet_manager.cc:1398] T 915028b3478341d588ed6031e657c259 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:47.762441 7604 tablet_bootstrap.cc:492] T 4392cd97a4334d77ac6403002d991758 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 01:59:47.762545 7604 ts_tablet_manager.cc:1367] T 4392cd97a4334d77ac6403002d991758 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240503 01:59:47.762980 7604 raft_consensus.cc:348] T 4392cd97a4334d77ac6403002d991758 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:47.763109 7604 raft_consensus.cc:374] T 4392cd97a4334d77ac6403002d991758 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:47.763147 7604 raft_consensus.cc:729] T 4392cd97a4334d77ac6403002d991758 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 01:59:47.763265 7604 consensus_queue.cc:260] T 4392cd97a4334d77ac6403002d991758 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:47.763510 7604 ts_tablet_manager.cc:1398] T 4392cd97a4334d77ac6403002d991758 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:47.889922 7661 raft_consensus.cc:482] T 4392cd97a4334d77ac6403002d991758 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:47.890076 7661 raft_consensus.cc:504] T 4392cd97a4334d77ac6403002d991758 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:47.890571 7661 leader_election.cc:288] T 4392cd97a4334d77ac6403002d991758 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 01:59:47.891085 6457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4392cd97a4334d77ac6403002d991758" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 01:59:47.891083 6719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4392cd97a4334d77ac6403002d991758" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 01:59:47.891233 6457 raft_consensus.cc:2454] T 4392cd97a4334d77ac6403002d991758 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 01:59:47.891245 6719 raft_consensus.cc:2454] T 4392cd97a4334d77ac6403002d991758 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 01:59:47.891559 6784 leader_election.cc:302] T 4392cd97a4334d77ac6403002d991758 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:47.891726 7661 raft_consensus.cc:2781] T 4392cd97a4334d77ac6403002d991758 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:47.891798 7661 raft_consensus.cc:482] T 4392cd97a4334d77ac6403002d991758 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:47.891834 7661 raft_consensus.cc:3037] T 4392cd97a4334d77ac6403002d991758 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:47.892686 7661 raft_consensus.cc:504] T 4392cd97a4334d77ac6403002d991758 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:47.893028 7661 leader_election.cc:288] T 4392cd97a4334d77ac6403002d991758 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Requested vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 01:59:47.893425 6457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4392cd97a4334d77ac6403002d991758" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 01:59:47.893425 6719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4392cd97a4334d77ac6403002d991758" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 01:59:47.893536 6719 raft_consensus.cc:3037] T 4392cd97a4334d77ac6403002d991758 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:47.893543 6457 raft_consensus.cc:3037] T 4392cd97a4334d77ac6403002d991758 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:47.894284 6457 raft_consensus.cc:2454] T 4392cd97a4334d77ac6403002d991758 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 01:59:47.894285 6719 raft_consensus.cc:2454] T 4392cd97a4334d77ac6403002d991758 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 01:59:47.894585 6784 leader_election.cc:302] T 4392cd97a4334d77ac6403002d991758 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:47.894749 7661 raft_consensus.cc:2781] T 4392cd97a4334d77ac6403002d991758 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:47.894835 7661 raft_consensus.cc:686] T 4392cd97a4334d77ac6403002d991758 P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Becoming Leader. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Running, Role: LEADER
I20240503 01:59:47.894979 7661 consensus_queue.cc:237] T 4392cd97a4334d77ac6403002d991758 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:47.896765 6313 catalog_manager.cc:5484] T 4392cd97a4334d77ac6403002d991758 P 492448570cc94ed1879feb2ec3b9051d reported cstate change: term changed from 0 to 1, leader changed from <none> to 492448570cc94ed1879feb2ec3b9051d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:47.913936 7652 raft_consensus.cc:482] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:47.914088 7652 raft_consensus.cc:504] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:47.914572 7652 leader_election.cc:288] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 01:59:47.914779 7545 raft_consensus.cc:482] T ac7fa0c3722a461c91fb237fd85794a3 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:47.914885 7545 raft_consensus.cc:504] T ac7fa0c3722a461c91fb237fd85794a3 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:47.915038 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac7fa0c3722a461c91fb237fd85794a3" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 01:59:47.915055 6719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac7fa0c3722a461c91fb237fd85794a3" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 01:59:47.915146 6589 raft_consensus.cc:2454] T ac7fa0c3722a461c91fb237fd85794a3 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 01:59:47.915217 6719 raft_consensus.cc:2454] T ac7fa0c3722a461c91fb237fd85794a3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 01:59:47.915347 7545 leader_election.cc:288] T ac7fa0c3722a461c91fb237fd85794a3 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 01:59:47.915479 6393 leader_election.cc:302] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:47.915791 6719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac7fa0c3722a461c91fb237fd85794a3" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 01:59:47.915840 7652 raft_consensus.cc:2781] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:47.915895 7652 raft_consensus.cc:482] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:47.915937 7652 raft_consensus.cc:3037] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:47.915921 6719 raft_consensus.cc:2454] T ac7fa0c3722a461c91fb237fd85794a3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 01:59:47.916198 6525 leader_election.cc:302] T ac7fa0c3722a461c91fb237fd85794a3 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 01:59:47.916363 7545 raft_consensus.cc:2781] T ac7fa0c3722a461c91fb237fd85794a3 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:47.916437 7545 raft_consensus.cc:482] T ac7fa0c3722a461c91fb237fd85794a3 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:47.916479 7545 raft_consensus.cc:3037] T ac7fa0c3722a461c91fb237fd85794a3 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:47.917071 7652 raft_consensus.cc:504] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:47.917448 7652 leader_election.cc:288] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Requested vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 01:59:47.917431 7545 raft_consensus.cc:504] T ac7fa0c3722a461c91fb237fd85794a3 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:47.917759 7545 leader_election.cc:288] T ac7fa0c3722a461c91fb237fd85794a3 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Requested vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 01:59:47.917796 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac7fa0c3722a461c91fb237fd85794a3" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 01:59:47.917939 6589 raft_consensus.cc:2379] T ac7fa0c3722a461c91fb237fd85794a3 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate dec3e4a04d604f7eaaa3f077d081218a in current term 1: Already voted for candidate 80fd18ef4d604c86b909d5c008a46b4f in this term.
I20240503 01:59:47.917918 6719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac7fa0c3722a461c91fb237fd85794a3" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 01:59:47.918042 6719 raft_consensus.cc:3037] T ac7fa0c3722a461c91fb237fd85794a3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:47.918269 6720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac7fa0c3722a461c91fb237fd85794a3" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 01:59:47.918924 6719 raft_consensus.cc:2454] T ac7fa0c3722a461c91fb237fd85794a3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 01:59:47.919162 6394 leader_election.cc:302] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c2ac2eb7cc264047b9ab32fe3a4fc025, dec3e4a04d604f7eaaa3f077d081218a; no voters: 80fd18ef4d604c86b909d5c008a46b4f
I20240503 01:59:47.919360 7652 raft_consensus.cc:2781] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:47.919435 7652 raft_consensus.cc:686] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Becoming Leader. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Running, Role: LEADER
I20240503 01:59:47.919589 7652 consensus_queue.cc:237] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:47.919627 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac7fa0c3722a461c91fb237fd85794a3" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 01:59:47.919612 6457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac7fa0c3722a461c91fb237fd85794a3" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 01:59:47.919972 6522 leader_election.cc:302] T ac7fa0c3722a461c91fb237fd85794a3 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f; no voters: c2ac2eb7cc264047b9ab32fe3a4fc025, dec3e4a04d604f7eaaa3f077d081218a
I20240503 01:59:47.920137 7545 raft_consensus.cc:2726] T ac7fa0c3722a461c91fb237fd85794a3 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 01:59:47.921221 6313 catalog_manager.cc:5484] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: term changed from 0 to 1, leader changed from <none> to dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129). New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:48.060921 7545 raft_consensus.cc:482] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:48.061077 7545 raft_consensus.cc:504] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:48.061652 7545 leader_election.cc:288] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 01:59:48.062144 6719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "915028b3478341d588ed6031e657c259" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 01:59:48.062229 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "915028b3478341d588ed6031e657c259" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 01:59:48.062297 6719 raft_consensus.cc:2454] T 915028b3478341d588ed6031e657c259 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 01:59:48.062377 6851 raft_consensus.cc:2454] T 915028b3478341d588ed6031e657c259 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 01:59:48.062667 6525 leader_election.cc:302] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 01:59:48.062836 7545 raft_consensus.cc:2781] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:48.062899 7545 raft_consensus.cc:482] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:48.062937 7545 raft_consensus.cc:3037] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:48.063786 7545 raft_consensus.cc:504] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:48.064143 7545 leader_election.cc:288] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Requested vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 01:59:48.064639 6719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "915028b3478341d588ed6031e657c259" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 01:59:48.064639 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "915028b3478341d588ed6031e657c259" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 01:59:48.064771 6719 raft_consensus.cc:3037] T 915028b3478341d588ed6031e657c259 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:48.064771 6851 raft_consensus.cc:3037] T 915028b3478341d588ed6031e657c259 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:48.065723 6851 raft_consensus.cc:2454] T 915028b3478341d588ed6031e657c259 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 01:59:48.065723 6719 raft_consensus.cc:2454] T 915028b3478341d588ed6031e657c259 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 01:59:48.066089 6525 leader_election.cc:302] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 80fd18ef4d604c86b909d5c008a46b4f; no voters:
I20240503 01:59:48.066300 7545 raft_consensus.cc:2781] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:48.066408 7545 raft_consensus.cc:686] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Becoming Leader. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Running, Role: LEADER
I20240503 01:59:48.066562 7545 consensus_queue.cc:237] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:48.068423 6313 catalog_manager.cc:5484] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } }
W20240503 01:59:48.229686 7679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:48.229947 7679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:48.233340 7679 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240503 01:59:48.282452 7661 consensus_queue.cc:1035] T 4392cd97a4334d77ac6403002d991758 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:48.290405 7587 consensus_queue.cc:1035] T 4392cd97a4334d77ac6403002d991758 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:48.337802 6719 raft_consensus.cc:1261] T ac7fa0c3722a461c91fb237fd85794a3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:48.338183 6589 raft_consensus.cc:1261] T ac7fa0c3722a461c91fb237fd85794a3 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:48.338667 7652 consensus_queue.cc:1035] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:48.339396 7652 consensus_queue.cc:1035] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:48.342553 6851 raft_consensus.cc:1261] T 915028b3478341d588ed6031e657c259 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:48.344302 6720 raft_consensus.cc:1261] T 915028b3478341d588ed6031e657c259 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:48.344812 7545 consensus_queue.cc:1035] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:48.345149 7545 consensus_queue.cc:1035] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 139.74 ms
time per row: 0.0698702 ms
I20240503 01:59:48.418707 6310 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:48.424018 6699 tablet_service.cc:1467] Processing CreateTablet for tablet fa51b5a29d10491a8437bd5cb84f1479 (DEFAULT_TABLE table=rebalance_test_table_4 [id=d930c636ed124c0eb64e1d6ab7c8d106]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:48.424057 6568 tablet_service.cc:1467] Processing CreateTablet for tablet fa51b5a29d10491a8437bd5cb84f1479 (DEFAULT_TABLE table=rebalance_test_table_4 [id=d930c636ed124c0eb64e1d6ab7c8d106]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:48.424337 6699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa51b5a29d10491a8437bd5cb84f1479. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:48.424355 6568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa51b5a29d10491a8437bd5cb84f1479. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:48.424881 6569 tablet_service.cc:1467] Processing CreateTablet for tablet 5f176537159143e8a770d65ba63ad62e (DEFAULT_TABLE table=rebalance_test_table_4 [id=d930c636ed124c0eb64e1d6ab7c8d106]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:48.425104 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f176537159143e8a770d65ba63ad62e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:48.425742 6830 tablet_service.cc:1467] Processing CreateTablet for tablet 5f176537159143e8a770d65ba63ad62e (DEFAULT_TABLE table=rebalance_test_table_4 [id=d930c636ed124c0eb64e1d6ab7c8d106]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:48.426007 6830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f176537159143e8a770d65ba63ad62e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:48.426582 7741 tablet_bootstrap.cc:492] T fa51b5a29d10491a8437bd5cb84f1479 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 01:59:48.426759 6436 tablet_service.cc:1467] Processing CreateTablet for tablet fa51b5a29d10491a8437bd5cb84f1479 (DEFAULT_TABLE table=rebalance_test_table_4 [id=d930c636ed124c0eb64e1d6ab7c8d106]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:48.427018 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa51b5a29d10491a8437bd5cb84f1479. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:48.427367 7741 tablet_bootstrap.cc:654] T fa51b5a29d10491a8437bd5cb84f1479 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:48.428042 7744 tablet_bootstrap.cc:492] T 5f176537159143e8a770d65ba63ad62e P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 01:59:48.428216 6831 tablet_service.cc:1467] Processing CreateTablet for tablet 396a2a8083d240529a2db48519f6139e (DEFAULT_TABLE table=rebalance_test_table_4 [id=d930c636ed124c0eb64e1d6ab7c8d106]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:48.428509 7741 tablet_bootstrap.cc:492] T fa51b5a29d10491a8437bd5cb84f1479 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 01:59:48.428642 7741 ts_tablet_manager.cc:1367] T fa51b5a29d10491a8437bd5cb84f1479 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:48.428830 6699 tablet_service.cc:1467] Processing CreateTablet for tablet 396a2a8083d240529a2db48519f6139e (DEFAULT_TABLE table=rebalance_test_table_4 [id=d930c636ed124c0eb64e1d6ab7c8d106]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:48.428922 7744 tablet_bootstrap.cc:654] T 5f176537159143e8a770d65ba63ad62e P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:48.429054 6699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 396a2a8083d240529a2db48519f6139e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:48.429276 6438 tablet_service.cc:1467] Processing CreateTablet for tablet 5f176537159143e8a770d65ba63ad62e (DEFAULT_TABLE table=rebalance_test_table_4 [id=d930c636ed124c0eb64e1d6ab7c8d106]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:48.429292 7741 raft_consensus.cc:348] T fa51b5a29d10491a8437bd5cb84f1479 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:48.429489 6438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f176537159143e8a770d65ba63ad62e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:48.429560 7741 raft_consensus.cc:374] T fa51b5a29d10491a8437bd5cb84f1479 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:48.429641 7741 raft_consensus.cc:729] T fa51b5a29d10491a8437bd5cb84f1479 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 01:59:48.429819 7741 consensus_queue.cc:260] T fa51b5a29d10491a8437bd5cb84f1479 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:48.429915 7744 tablet_bootstrap.cc:492] T 5f176537159143e8a770d65ba63ad62e P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 01:59:48.429988 7744 ts_tablet_manager.cc:1367] T 5f176537159143e8a770d65ba63ad62e P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:48.430223 7741 ts_tablet_manager.cc:1398] T fa51b5a29d10491a8437bd5cb84f1479 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:48.430399 7744 raft_consensus.cc:348] T 5f176537159143e8a770d65ba63ad62e P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:48.430481 7744 raft_consensus.cc:374] T 5f176537159143e8a770d65ba63ad62e P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:48.430508 7744 raft_consensus.cc:729] T 5f176537159143e8a770d65ba63ad62e P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 01:59:48.430614 7744 consensus_queue.cc:260] T 5f176537159143e8a770d65ba63ad62e P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:48.430774 7741 tablet_bootstrap.cc:492] T 396a2a8083d240529a2db48519f6139e P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 01:59:48.430873 7744 ts_tablet_manager.cc:1398] T 5f176537159143e8a770d65ba63ad62e P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:48.431427 6567 tablet_service.cc:1467] Processing CreateTablet for tablet 396a2a8083d240529a2db48519f6139e (DEFAULT_TABLE table=rebalance_test_table_4 [id=d930c636ed124c0eb64e1d6ab7c8d106]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:48.431509 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 396a2a8083d240529a2db48519f6139e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:48.431691 6567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 396a2a8083d240529a2db48519f6139e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:48.431694 7741 tablet_bootstrap.cc:654] T 396a2a8083d240529a2db48519f6139e P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:48.433410 7742 tablet_bootstrap.cc:492] T fa51b5a29d10491a8437bd5cb84f1479 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 01:59:48.433538 7744 tablet_bootstrap.cc:492] T 396a2a8083d240529a2db48519f6139e P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 01:59:48.433858 7745 tablet_bootstrap.cc:492] T fa51b5a29d10491a8437bd5cb84f1479 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 01:59:48.434391 7744 tablet_bootstrap.cc:654] T 396a2a8083d240529a2db48519f6139e P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:48.434415 7742 tablet_bootstrap.cc:654] T fa51b5a29d10491a8437bd5cb84f1479 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:48.435297 7745 tablet_bootstrap.cc:654] T fa51b5a29d10491a8437bd5cb84f1479 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:48.435487 7744 tablet_bootstrap.cc:492] T 396a2a8083d240529a2db48519f6139e P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 01:59:48.435575 7744 ts_tablet_manager.cc:1367] T 396a2a8083d240529a2db48519f6139e P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:48.435593 7741 tablet_bootstrap.cc:492] T 396a2a8083d240529a2db48519f6139e P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 01:59:48.435688 7741 ts_tablet_manager.cc:1367] T 396a2a8083d240529a2db48519f6139e P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 01:59:48.436007 7744 raft_consensus.cc:348] T 396a2a8083d240529a2db48519f6139e P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:48.436116 7744 raft_consensus.cc:374] T 396a2a8083d240529a2db48519f6139e P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:48.436156 7744 raft_consensus.cc:729] T 396a2a8083d240529a2db48519f6139e P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 01:59:48.436112 7741 raft_consensus.cc:348] T 396a2a8083d240529a2db48519f6139e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:48.436213 7741 raft_consensus.cc:374] T 396a2a8083d240529a2db48519f6139e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:48.436246 7741 raft_consensus.cc:729] T 396a2a8083d240529a2db48519f6139e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 01:59:48.436297 7744 consensus_queue.cc:260] T 396a2a8083d240529a2db48519f6139e P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:48.436436 7745 tablet_bootstrap.cc:492] T fa51b5a29d10491a8437bd5cb84f1479 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 01:59:48.436518 7745 ts_tablet_manager.cc:1367] T fa51b5a29d10491a8437bd5cb84f1479 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:48.436475 7741 consensus_queue.cc:260] T 396a2a8083d240529a2db48519f6139e P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:48.436637 7744 ts_tablet_manager.cc:1398] T 396a2a8083d240529a2db48519f6139e P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:48.436767 7741 ts_tablet_manager.cc:1398] T 396a2a8083d240529a2db48519f6139e P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:48.437045 7745 raft_consensus.cc:348] T fa51b5a29d10491a8437bd5cb84f1479 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:48.437170 7745 raft_consensus.cc:374] T fa51b5a29d10491a8437bd5cb84f1479 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:48.437211 7745 raft_consensus.cc:729] T fa51b5a29d10491a8437bd5cb84f1479 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 01:59:48.437330 7745 consensus_queue.cc:260] T fa51b5a29d10491a8437bd5cb84f1479 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:48.437626 7745 ts_tablet_manager.cc:1398] T fa51b5a29d10491a8437bd5cb84f1479 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:48.437819 7745 tablet_bootstrap.cc:492] T 5f176537159143e8a770d65ba63ad62e P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 01:59:48.438593 7742 tablet_bootstrap.cc:492] T fa51b5a29d10491a8437bd5cb84f1479 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 01:59:48.438697 7742 ts_tablet_manager.cc:1367] T fa51b5a29d10491a8437bd5cb84f1479 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 01:59:48.439010 7745 tablet_bootstrap.cc:654] T 5f176537159143e8a770d65ba63ad62e P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:48.439107 7742 raft_consensus.cc:348] T fa51b5a29d10491a8437bd5cb84f1479 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:48.439203 7742 raft_consensus.cc:374] T fa51b5a29d10491a8437bd5cb84f1479 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:48.439232 7742 raft_consensus.cc:729] T fa51b5a29d10491a8437bd5cb84f1479 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 01:59:48.439579 7742 consensus_queue.cc:260] T fa51b5a29d10491a8437bd5cb84f1479 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:48.440457 7745 tablet_bootstrap.cc:492] T 5f176537159143e8a770d65ba63ad62e P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 01:59:48.440587 7742 ts_tablet_manager.cc:1398] T fa51b5a29d10491a8437bd5cb84f1479 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:48.440586 7745 ts_tablet_manager.cc:1367] T 5f176537159143e8a770d65ba63ad62e P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:48.440753 7742 tablet_bootstrap.cc:492] T 5f176537159143e8a770d65ba63ad62e P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 01:59:48.441051 7745 raft_consensus.cc:348] T 5f176537159143e8a770d65ba63ad62e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:48.441148 7745 raft_consensus.cc:374] T 5f176537159143e8a770d65ba63ad62e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:48.441179 7745 raft_consensus.cc:729] T 5f176537159143e8a770d65ba63ad62e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 01:59:48.441293 7745 consensus_queue.cc:260] T 5f176537159143e8a770d65ba63ad62e P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:48.441552 7745 ts_tablet_manager.cc:1398] T 5f176537159143e8a770d65ba63ad62e P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:48.441852 7742 tablet_bootstrap.cc:654] T 5f176537159143e8a770d65ba63ad62e P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:48.442844 7742 tablet_bootstrap.cc:492] T 5f176537159143e8a770d65ba63ad62e P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 01:59:48.442916 7742 ts_tablet_manager.cc:1367] T 5f176537159143e8a770d65ba63ad62e P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:48.443320 7742 raft_consensus.cc:348] T 5f176537159143e8a770d65ba63ad62e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:48.443416 7742 raft_consensus.cc:374] T 5f176537159143e8a770d65ba63ad62e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:48.443449 7742 raft_consensus.cc:729] T 5f176537159143e8a770d65ba63ad62e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 01:59:48.443554 7742 consensus_queue.cc:260] T 5f176537159143e8a770d65ba63ad62e P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:48.443773 7742 ts_tablet_manager.cc:1398] T 5f176537159143e8a770d65ba63ad62e P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:48.444011 7742 tablet_bootstrap.cc:492] T 396a2a8083d240529a2db48519f6139e P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 01:59:48.444913 7742 tablet_bootstrap.cc:654] T 396a2a8083d240529a2db48519f6139e P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:48.446072 7742 tablet_bootstrap.cc:492] T 396a2a8083d240529a2db48519f6139e P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 01:59:48.446133 7742 ts_tablet_manager.cc:1367] T 396a2a8083d240529a2db48519f6139e P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:48.446520 7742 raft_consensus.cc:348] T 396a2a8083d240529a2db48519f6139e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:48.446645 7742 raft_consensus.cc:374] T 396a2a8083d240529a2db48519f6139e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:48.446671 7742 raft_consensus.cc:729] T 396a2a8083d240529a2db48519f6139e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 01:59:48.446784 7742 consensus_queue.cc:260] T 396a2a8083d240529a2db48519f6139e P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:48.447022 7742 ts_tablet_manager.cc:1398] T 396a2a8083d240529a2db48519f6139e P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:48.456857 7456 raft_consensus.cc:482] T 5f176537159143e8a770d65ba63ad62e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:48.456985 7456 raft_consensus.cc:504] T 5f176537159143e8a770d65ba63ad62e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:48.457392 7456 leader_election.cc:288] T 5f176537159143e8a770d65ba63ad62e P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 01:59:48.457770 6457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f176537159143e8a770d65ba63ad62e" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 01:59:48.457899 6457 raft_consensus.cc:2454] T 5f176537159143e8a770d65ba63ad62e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 01:59:48.457831 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f176537159143e8a770d65ba63ad62e" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 01:59:48.458007 6851 raft_consensus.cc:2454] T 5f176537159143e8a770d65ba63ad62e P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 01:59:48.458225 6522 leader_election.cc:302] T 5f176537159143e8a770d65ba63ad62e P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:48.458400 7456 raft_consensus.cc:2781] T 5f176537159143e8a770d65ba63ad62e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:48.458458 7456 raft_consensus.cc:482] T 5f176537159143e8a770d65ba63ad62e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:48.458484 7456 raft_consensus.cc:3037] T 5f176537159143e8a770d65ba63ad62e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:48.459359 7456 raft_consensus.cc:504] T 5f176537159143e8a770d65ba63ad62e P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:48.459707 7456 leader_election.cc:288] T 5f176537159143e8a770d65ba63ad62e P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Requested vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 01:59:48.460058 6457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f176537159143e8a770d65ba63ad62e" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 01:59:48.460059 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f176537159143e8a770d65ba63ad62e" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 01:59:48.460170 6457 raft_consensus.cc:3037] T 5f176537159143e8a770d65ba63ad62e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:48.460175 6851 raft_consensus.cc:3037] T 5f176537159143e8a770d65ba63ad62e P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:48.460939 6851 raft_consensus.cc:2454] T 5f176537159143e8a770d65ba63ad62e P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 01:59:48.460956 6457 raft_consensus.cc:2454] T 5f176537159143e8a770d65ba63ad62e P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 01:59:48.461232 6522 leader_election.cc:302] T 5f176537159143e8a770d65ba63ad62e P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:48.461392 7456 raft_consensus.cc:2781] T 5f176537159143e8a770d65ba63ad62e P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:48.461470 7456 raft_consensus.cc:686] T 5f176537159143e8a770d65ba63ad62e P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Becoming Leader. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Running, Role: LEADER
I20240503 01:59:48.461648 7456 consensus_queue.cc:237] T 5f176537159143e8a770d65ba63ad62e P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:48.463346 6310 catalog_manager.cc:5484] T 5f176537159143e8a770d65ba63ad62e P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:48.470973 7456 raft_consensus.cc:482] T fa51b5a29d10491a8437bd5cb84f1479 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:48.471125 7456 raft_consensus.cc:504] T fa51b5a29d10491a8437bd5cb84f1479 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:48.471524 7456 leader_election.cc:288] T fa51b5a29d10491a8437bd5cb84f1479 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 01:59:48.471973 6457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa51b5a29d10491a8437bd5cb84f1479" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 01:59:48.472000 6718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa51b5a29d10491a8437bd5cb84f1479" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 01:59:48.472115 6457 raft_consensus.cc:2454] T fa51b5a29d10491a8437bd5cb84f1479 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 01:59:48.472179 6718 raft_consensus.cc:2454] T fa51b5a29d10491a8437bd5cb84f1479 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 01:59:48.472411 6522 leader_election.cc:302] T fa51b5a29d10491a8437bd5cb84f1479 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:48.472572 7456 raft_consensus.cc:2781] T fa51b5a29d10491a8437bd5cb84f1479 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:48.472621 7456 raft_consensus.cc:482] T fa51b5a29d10491a8437bd5cb84f1479 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:48.472656 7456 raft_consensus.cc:3037] T fa51b5a29d10491a8437bd5cb84f1479 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:48.473460 7456 raft_consensus.cc:504] T fa51b5a29d10491a8437bd5cb84f1479 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:48.473798 7456 leader_election.cc:288] T fa51b5a29d10491a8437bd5cb84f1479 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Requested vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 01:59:48.474187 6718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa51b5a29d10491a8437bd5cb84f1479" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 01:59:48.474303 6718 raft_consensus.cc:3037] T fa51b5a29d10491a8437bd5cb84f1479 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:48.474277 6457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa51b5a29d10491a8437bd5cb84f1479" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 01:59:48.474388 6457 raft_consensus.cc:3037] T fa51b5a29d10491a8437bd5cb84f1479 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:48.475116 6457 raft_consensus.cc:2454] T fa51b5a29d10491a8437bd5cb84f1479 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 01:59:48.475127 6718 raft_consensus.cc:2454] T fa51b5a29d10491a8437bd5cb84f1479 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 01:59:48.475414 6522 leader_election.cc:302] T fa51b5a29d10491a8437bd5cb84f1479 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:48.475574 7456 raft_consensus.cc:2781] T fa51b5a29d10491a8437bd5cb84f1479 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:48.475644 7456 raft_consensus.cc:686] T fa51b5a29d10491a8437bd5cb84f1479 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Becoming Leader. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Running, Role: LEADER
I20240503 01:59:48.475777 7456 consensus_queue.cc:237] T fa51b5a29d10491a8437bd5cb84f1479 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:48.477726 6315 catalog_manager.cc:5484] T fa51b5a29d10491a8437bd5cb84f1479 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:48.526396 7456 raft_consensus.cc:482] T 396a2a8083d240529a2db48519f6139e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:48.526520 7456 raft_consensus.cc:504] T 396a2a8083d240529a2db48519f6139e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:48.526916 7456 leader_election.cc:288] T 396a2a8083d240529a2db48519f6139e P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 01:59:48.527371 6718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "396a2a8083d240529a2db48519f6139e" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 01:59:48.527418 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "396a2a8083d240529a2db48519f6139e" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 01:59:48.527498 6718 raft_consensus.cc:2454] T 396a2a8083d240529a2db48519f6139e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 01:59:48.527542 6851 raft_consensus.cc:2454] T 396a2a8083d240529a2db48519f6139e P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 01:59:48.527825 6525 leader_election.cc:302] T 396a2a8083d240529a2db48519f6139e P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 01:59:48.527990 7456 raft_consensus.cc:2781] T 396a2a8083d240529a2db48519f6139e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:48.528059 7456 raft_consensus.cc:482] T 396a2a8083d240529a2db48519f6139e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:48.528100 7456 raft_consensus.cc:3037] T 396a2a8083d240529a2db48519f6139e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:48.528946 7456 raft_consensus.cc:504] T 396a2a8083d240529a2db48519f6139e P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:48.529253 7456 leader_election.cc:288] T 396a2a8083d240529a2db48519f6139e P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Requested vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 01:59:48.529592 6718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "396a2a8083d240529a2db48519f6139e" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 01:59:48.529641 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "396a2a8083d240529a2db48519f6139e" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 01:59:48.529695 6718 raft_consensus.cc:3037] T 396a2a8083d240529a2db48519f6139e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:48.529754 6851 raft_consensus.cc:3037] T 396a2a8083d240529a2db48519f6139e P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:48.530393 6718 raft_consensus.cc:2454] T 396a2a8083d240529a2db48519f6139e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 01:59:48.530454 6851 raft_consensus.cc:2454] T 396a2a8083d240529a2db48519f6139e P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 01:59:48.530673 6525 leader_election.cc:302] T 396a2a8083d240529a2db48519f6139e P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 01:59:48.530826 7456 raft_consensus.cc:2781] T 396a2a8083d240529a2db48519f6139e P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:48.530923 7456 raft_consensus.cc:686] T 396a2a8083d240529a2db48519f6139e P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Becoming Leader. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Running, Role: LEADER
I20240503 01:59:48.531224 7456 consensus_queue.cc:237] T 396a2a8083d240529a2db48519f6139e P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:48.532955 6315 catalog_manager.cc:5484] T 396a2a8083d240529a2db48519f6139e P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } }
W20240503 01:59:48.642180 7757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:48.642428 7757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:48.645649 7757 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240503 01:59:48.737092 6457 raft_consensus.cc:1261] T fa51b5a29d10491a8437bd5cb84f1479 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:48.737310 6851 raft_consensus.cc:1261] T 396a2a8083d240529a2db48519f6139e P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:48.737483 6718 raft_consensus.cc:1261] T fa51b5a29d10491a8437bd5cb84f1479 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:48.738133 7545 consensus_queue.cc:1035] T fa51b5a29d10491a8437bd5cb84f1479 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:48.738408 7545 consensus_queue.cc:1035] T fa51b5a29d10491a8437bd5cb84f1479 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:48.738898 6851 raft_consensus.cc:1261] T 5f176537159143e8a770d65ba63ad62e P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:48.739163 7456 consensus_queue.cc:1035] T 396a2a8083d240529a2db48519f6139e P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:48.739519 6457 raft_consensus.cc:1261] T 5f176537159143e8a770d65ba63ad62e P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:48.744555 7456 consensus_queue.cc:1035] T 5f176537159143e8a770d65ba63ad62e P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:48.744947 7784 consensus_queue.cc:1035] T 5f176537159143e8a770d65ba63ad62e P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:48.749099 6720 raft_consensus.cc:1261] T 396a2a8083d240529a2db48519f6139e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 01:59:48.749574 7786 consensus_queue.cc:1035] T 396a2a8083d240529a2db48519f6139e P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 115.865 ms
time per row: 0.0579327 ms
I20240503 01:59:48.805197 6310 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:48.810642 6438 tablet_service.cc:1467] Processing CreateTablet for tablet 90ef3584830240acb0d8aad14ae851a6 (DEFAULT_TABLE table=rebalance_test_table_5 [id=9b77b848294749e09f0854d24da5d9f5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:48.810745 6831 tablet_service.cc:1467] Processing CreateTablet for tablet 90ef3584830240acb0d8aad14ae851a6 (DEFAULT_TABLE table=rebalance_test_table_5 [id=9b77b848294749e09f0854d24da5d9f5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:48.811004 6438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90ef3584830240acb0d8aad14ae851a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:48.811031 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90ef3584830240acb0d8aad14ae851a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:48.811239 6830 tablet_service.cc:1467] Processing CreateTablet for tablet d897c21e630d4770832bb10f43778e82 (DEFAULT_TABLE table=rebalance_test_table_5 [id=9b77b848294749e09f0854d24da5d9f5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:48.811455 6830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d897c21e630d4770832bb10f43778e82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:48.811492 6567 tablet_service.cc:1467] Processing CreateTablet for tablet b91a933fb66b4075acb40c5b92ef4824 (DEFAULT_TABLE table=rebalance_test_table_5 [id=9b77b848294749e09f0854d24da5d9f5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:48.811746 6567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b91a933fb66b4075acb40c5b92ef4824. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:48.813123 7744 tablet_bootstrap.cc:492] T 90ef3584830240acb0d8aad14ae851a6 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 01:59:48.813252 7745 tablet_bootstrap.cc:492] T 90ef3584830240acb0d8aad14ae851a6 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 01:59:48.813745 7742 tablet_bootstrap.cc:492] T b91a933fb66b4075acb40c5b92ef4824 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 01:59:48.813993 7745 tablet_bootstrap.cc:654] T 90ef3584830240acb0d8aad14ae851a6 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:48.814018 7744 tablet_bootstrap.cc:654] T 90ef3584830240acb0d8aad14ae851a6 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:48.814548 7742 tablet_bootstrap.cc:654] T b91a933fb66b4075acb40c5b92ef4824 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:48.814908 6436 tablet_service.cc:1467] Processing CreateTablet for tablet b91a933fb66b4075acb40c5b92ef4824 (DEFAULT_TABLE table=rebalance_test_table_5 [id=9b77b848294749e09f0854d24da5d9f5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:48.815156 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b91a933fb66b4075acb40c5b92ef4824. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:48.815224 6438 tablet_service.cc:1467] Processing CreateTablet for tablet d897c21e630d4770832bb10f43778e82 (DEFAULT_TABLE table=rebalance_test_table_5 [id=9b77b848294749e09f0854d24da5d9f5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:48.815429 6438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d897c21e630d4770832bb10f43778e82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:48.815409 6699 tablet_service.cc:1467] Processing CreateTablet for tablet 90ef3584830240acb0d8aad14ae851a6 (DEFAULT_TABLE table=rebalance_test_table_5 [id=9b77b848294749e09f0854d24da5d9f5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:48.815553 7742 tablet_bootstrap.cc:492] T b91a933fb66b4075acb40c5b92ef4824 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 01:59:48.815625 7742 ts_tablet_manager.cc:1367] T b91a933fb66b4075acb40c5b92ef4824 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:48.815656 6699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90ef3584830240acb0d8aad14ae851a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:48.816040 7742 raft_consensus.cc:348] T b91a933fb66b4075acb40c5b92ef4824 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:48.816145 7742 raft_consensus.cc:374] T b91a933fb66b4075acb40c5b92ef4824 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:48.816174 7742 raft_consensus.cc:729] T b91a933fb66b4075acb40c5b92ef4824 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 01:59:48.816301 7742 consensus_queue.cc:260] T b91a933fb66b4075acb40c5b92ef4824 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:48.816732 7742 ts_tablet_manager.cc:1398] T b91a933fb66b4075acb40c5b92ef4824 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:48.817200 6698 tablet_service.cc:1467] Processing CreateTablet for tablet b91a933fb66b4075acb40c5b92ef4824 (DEFAULT_TABLE table=rebalance_test_table_5 [id=9b77b848294749e09f0854d24da5d9f5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:48.817348 6700 tablet_service.cc:1467] Processing CreateTablet for tablet d897c21e630d4770832bb10f43778e82 (DEFAULT_TABLE table=rebalance_test_table_5 [id=9b77b848294749e09f0854d24da5d9f5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:48.817445 6698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b91a933fb66b4075acb40c5b92ef4824. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:48.817567 7741 tablet_bootstrap.cc:492] T 90ef3584830240acb0d8aad14ae851a6 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 01:59:48.817566 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d897c21e630d4770832bb10f43778e82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:48.818260 7744 tablet_bootstrap.cc:492] T 90ef3584830240acb0d8aad14ae851a6 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 01:59:48.818342 7744 ts_tablet_manager.cc:1367] T 90ef3584830240acb0d8aad14ae851a6 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240503 01:59:48.818753 7741 tablet_bootstrap.cc:654] T 90ef3584830240acb0d8aad14ae851a6 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:48.818770 7744 raft_consensus.cc:348] T 90ef3584830240acb0d8aad14ae851a6 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:48.818907 7744 raft_consensus.cc:374] T 90ef3584830240acb0d8aad14ae851a6 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:48.818943 7744 raft_consensus.cc:729] T 90ef3584830240acb0d8aad14ae851a6 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 01:59:48.819075 7744 consensus_queue.cc:260] T 90ef3584830240acb0d8aad14ae851a6 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:48.819339 7744 ts_tablet_manager.cc:1398] T 90ef3584830240acb0d8aad14ae851a6 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:48.819501 7744 tablet_bootstrap.cc:492] T d897c21e630d4770832bb10f43778e82 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 01:59:48.820125 7745 tablet_bootstrap.cc:492] T 90ef3584830240acb0d8aad14ae851a6 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 01:59:48.820202 7745 ts_tablet_manager.cc:1367] T 90ef3584830240acb0d8aad14ae851a6 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240503 01:59:48.820546 7744 tablet_bootstrap.cc:654] T d897c21e630d4770832bb10f43778e82 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:48.820670 7745 raft_consensus.cc:348] T 90ef3584830240acb0d8aad14ae851a6 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:48.820793 7745 raft_consensus.cc:374] T 90ef3584830240acb0d8aad14ae851a6 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:48.820822 7745 raft_consensus.cc:729] T 90ef3584830240acb0d8aad14ae851a6 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 01:59:48.820927 7745 consensus_queue.cc:260] T 90ef3584830240acb0d8aad14ae851a6 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:48.821188 7745 ts_tablet_manager.cc:1398] T 90ef3584830240acb0d8aad14ae851a6 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:48.821345 7745 tablet_bootstrap.cc:492] T b91a933fb66b4075acb40c5b92ef4824 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 01:59:48.822206 7745 tablet_bootstrap.cc:654] T b91a933fb66b4075acb40c5b92ef4824 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:48.822481 7741 tablet_bootstrap.cc:492] T 90ef3584830240acb0d8aad14ae851a6 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 01:59:48.822563 7741 ts_tablet_manager.cc:1367] T 90ef3584830240acb0d8aad14ae851a6 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 01:59:48.822944 7741 raft_consensus.cc:348] T 90ef3584830240acb0d8aad14ae851a6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:48.823047 7741 raft_consensus.cc:374] T 90ef3584830240acb0d8aad14ae851a6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:48.823092 7741 raft_consensus.cc:729] T 90ef3584830240acb0d8aad14ae851a6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 01:59:48.823196 7741 consensus_queue.cc:260] T 90ef3584830240acb0d8aad14ae851a6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:48.823449 7741 ts_tablet_manager.cc:1398] T 90ef3584830240acb0d8aad14ae851a6 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:48.823601 7741 tablet_bootstrap.cc:492] T d897c21e630d4770832bb10f43778e82 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 01:59:48.824333 7745 tablet_bootstrap.cc:492] T b91a933fb66b4075acb40c5b92ef4824 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 01:59:48.824412 7741 tablet_bootstrap.cc:654] T d897c21e630d4770832bb10f43778e82 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:48.824440 7745 ts_tablet_manager.cc:1367] T b91a933fb66b4075acb40c5b92ef4824 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:48.824851 7745 raft_consensus.cc:348] T b91a933fb66b4075acb40c5b92ef4824 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:48.824949 7745 raft_consensus.cc:374] T b91a933fb66b4075acb40c5b92ef4824 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:48.824987 7745 raft_consensus.cc:729] T b91a933fb66b4075acb40c5b92ef4824 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 01:59:48.825105 7745 consensus_queue.cc:260] T b91a933fb66b4075acb40c5b92ef4824 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:48.825362 7745 ts_tablet_manager.cc:1398] T b91a933fb66b4075acb40c5b92ef4824 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:48.825474 7744 tablet_bootstrap.cc:492] T d897c21e630d4770832bb10f43778e82 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 01:59:48.825524 7745 tablet_bootstrap.cc:492] T d897c21e630d4770832bb10f43778e82 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 01:59:48.825560 7744 ts_tablet_manager.cc:1367] T d897c21e630d4770832bb10f43778e82 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 01:59:48.825958 7744 raft_consensus.cc:348] T d897c21e630d4770832bb10f43778e82 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:48.826066 7744 raft_consensus.cc:374] T d897c21e630d4770832bb10f43778e82 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:48.826100 7744 raft_consensus.cc:729] T d897c21e630d4770832bb10f43778e82 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 01:59:48.826341 7745 tablet_bootstrap.cc:654] T d897c21e630d4770832bb10f43778e82 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:48.826531 7741 tablet_bootstrap.cc:492] T d897c21e630d4770832bb10f43778e82 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 01:59:48.826596 7741 ts_tablet_manager.cc:1367] T d897c21e630d4770832bb10f43778e82 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:48.826972 7741 raft_consensus.cc:348] T d897c21e630d4770832bb10f43778e82 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:48.827064 7741 raft_consensus.cc:374] T d897c21e630d4770832bb10f43778e82 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:48.827091 7741 raft_consensus.cc:729] T d897c21e630d4770832bb10f43778e82 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 01:59:48.827214 7741 consensus_queue.cc:260] T d897c21e630d4770832bb10f43778e82 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:48.827363 7744 consensus_queue.cc:260] T d897c21e630d4770832bb10f43778e82 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:48.827682 7741 ts_tablet_manager.cc:1398] T d897c21e630d4770832bb10f43778e82 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:48.827831 7741 tablet_bootstrap.cc:492] T b91a933fb66b4075acb40c5b92ef4824 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 01:59:48.828435 7745 tablet_bootstrap.cc:492] T d897c21e630d4770832bb10f43778e82 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 01:59:48.828567 7745 ts_tablet_manager.cc:1367] T d897c21e630d4770832bb10f43778e82 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 01:59:48.828686 7744 ts_tablet_manager.cc:1398] T d897c21e630d4770832bb10f43778e82 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 01:59:48.829098 7741 tablet_bootstrap.cc:654] T b91a933fb66b4075acb40c5b92ef4824 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:48.829052 7745 raft_consensus.cc:348] T d897c21e630d4770832bb10f43778e82 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:48.829303 7745 raft_consensus.cc:374] T d897c21e630d4770832bb10f43778e82 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:48.829382 7745 raft_consensus.cc:729] T d897c21e630d4770832bb10f43778e82 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 01:59:48.829538 7745 consensus_queue.cc:260] T d897c21e630d4770832bb10f43778e82 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:48.829790 7745 ts_tablet_manager.cc:1398] T d897c21e630d4770832bb10f43778e82 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:48.830474 7741 tablet_bootstrap.cc:492] T b91a933fb66b4075acb40c5b92ef4824 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 01:59:48.830556 7741 ts_tablet_manager.cc:1367] T b91a933fb66b4075acb40c5b92ef4824 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:48.830929 7741 raft_consensus.cc:348] T b91a933fb66b4075acb40c5b92ef4824 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:48.831039 7741 raft_consensus.cc:374] T b91a933fb66b4075acb40c5b92ef4824 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:48.831072 7741 raft_consensus.cc:729] T b91a933fb66b4075acb40c5b92ef4824 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 01:59:48.831179 7741 consensus_queue.cc:260] T b91a933fb66b4075acb40c5b92ef4824 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:48.831790 7741 ts_tablet_manager.cc:1398] T b91a933fb66b4075acb40c5b92ef4824 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:48.882385 7469 raft_consensus.cc:482] T b91a933fb66b4075acb40c5b92ef4824 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:48.882544 7469 raft_consensus.cc:504] T b91a933fb66b4075acb40c5b92ef4824 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:48.883034 7469 leader_election.cc:288] T b91a933fb66b4075acb40c5b92ef4824 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 01:59:48.883450 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91a933fb66b4075acb40c5b92ef4824" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 01:59:48.883466 6457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91a933fb66b4075acb40c5b92ef4824" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 01:59:48.883612 6589 raft_consensus.cc:2454] T b91a933fb66b4075acb40c5b92ef4824 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 01:59:48.883615 6457 raft_consensus.cc:2454] T b91a933fb66b4075acb40c5b92ef4824 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 01:59:48.883951 6655 leader_election.cc:302] T b91a933fb66b4075acb40c5b92ef4824 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 01:59:48.884153 7469 raft_consensus.cc:2781] T b91a933fb66b4075acb40c5b92ef4824 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:48.884218 7469 raft_consensus.cc:482] T b91a933fb66b4075acb40c5b92ef4824 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:48.884259 7469 raft_consensus.cc:3037] T b91a933fb66b4075acb40c5b92ef4824 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:48.885212 7469 raft_consensus.cc:504] T b91a933fb66b4075acb40c5b92ef4824 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:48.885574 7469 leader_election.cc:288] T b91a933fb66b4075acb40c5b92ef4824 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Requested vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 01:59:48.885962 6457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91a933fb66b4075acb40c5b92ef4824" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 01:59:48.886018 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91a933fb66b4075acb40c5b92ef4824" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 01:59:48.886082 6457 raft_consensus.cc:3037] T b91a933fb66b4075acb40c5b92ef4824 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:48.886171 6589 raft_consensus.cc:3037] T b91a933fb66b4075acb40c5b92ef4824 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:48.886906 6457 raft_consensus.cc:2454] T b91a933fb66b4075acb40c5b92ef4824 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 01:59:48.886914 6589 raft_consensus.cc:2454] T b91a933fb66b4075acb40c5b92ef4824 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 01:59:48.887223 6653 leader_election.cc:302] T b91a933fb66b4075acb40c5b92ef4824 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2ac2eb7cc264047b9ab32fe3a4fc025, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:48.887387 7469 raft_consensus.cc:2781] T b91a933fb66b4075acb40c5b92ef4824 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:48.887475 7469 raft_consensus.cc:686] T b91a933fb66b4075acb40c5b92ef4824 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Becoming Leader. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Running, Role: LEADER
I20240503 01:59:48.887603 7469 consensus_queue.cc:237] T b91a933fb66b4075acb40c5b92ef4824 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:48.889417 6315 catalog_manager.cc:5484] T b91a933fb66b4075acb40c5b92ef4824 P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:48.897392 7469 raft_consensus.cc:482] T d897c21e630d4770832bb10f43778e82 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:48.897539 7469 raft_consensus.cc:504] T d897c21e630d4770832bb10f43778e82 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:48.898487 6457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d897c21e630d4770832bb10f43778e82" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 01:59:48.898561 7469 leader_election.cc:288] T d897c21e630d4770832bb10f43778e82 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 01:59:48.898679 6457 raft_consensus.cc:2454] T d897c21e630d4770832bb10f43778e82 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 01:59:48.898979 6653 leader_election.cc:302] T d897c21e630d4770832bb10f43778e82 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2ac2eb7cc264047b9ab32fe3a4fc025, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:48.899147 7469 raft_consensus.cc:2781] T d897c21e630d4770832bb10f43778e82 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:48.899197 7469 raft_consensus.cc:482] T d897c21e630d4770832bb10f43778e82 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:48.899232 7469 raft_consensus.cc:3037] T d897c21e630d4770832bb10f43778e82 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:48.902477 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d897c21e630d4770832bb10f43778e82" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 01:59:48.902642 6851 raft_consensus.cc:2454] T d897c21e630d4770832bb10f43778e82 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 01:59:48.913633 7469 raft_consensus.cc:504] T d897c21e630d4770832bb10f43778e82 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:48.914111 7469 leader_election.cc:288] T d897c21e630d4770832bb10f43778e82 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Requested vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 01:59:48.914471 6457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d897c21e630d4770832bb10f43778e82" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 01:59:48.914629 6457 raft_consensus.cc:3037] T d897c21e630d4770832bb10f43778e82 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:48.914595 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d897c21e630d4770832bb10f43778e82" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 01:59:48.914712 6851 raft_consensus.cc:3037] T d897c21e630d4770832bb10f43778e82 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:48.915601 6457 raft_consensus.cc:2454] T d897c21e630d4770832bb10f43778e82 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 01:59:48.915620 6851 raft_consensus.cc:2454] T d897c21e630d4770832bb10f43778e82 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 01:59:48.915958 6653 leader_election.cc:302] T d897c21e630d4770832bb10f43778e82 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2ac2eb7cc264047b9ab32fe3a4fc025, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:48.916198 7469 raft_consensus.cc:2781] T d897c21e630d4770832bb10f43778e82 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:48.916299 7469 raft_consensus.cc:686] T d897c21e630d4770832bb10f43778e82 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Becoming Leader. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Running, Role: LEADER
I20240503 01:59:48.916471 7469 consensus_queue.cc:237] T d897c21e630d4770832bb10f43778e82 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:48.918318 6315 catalog_manager.cc:5484] T d897c21e630d4770832bb10f43778e82 P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:48.931391 7725 raft_consensus.cc:482] T 90ef3584830240acb0d8aad14ae851a6 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:48.931542 7725 raft_consensus.cc:504] T 90ef3584830240acb0d8aad14ae851a6 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:48.931946 7725 leader_election.cc:288] T 90ef3584830240acb0d8aad14ae851a6 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 01:59:48.932428 6457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90ef3584830240acb0d8aad14ae851a6" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 01:59:48.932444 6718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90ef3584830240acb0d8aad14ae851a6" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 01:59:48.932581 6718 raft_consensus.cc:2454] T 90ef3584830240acb0d8aad14ae851a6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 01:59:48.932583 6457 raft_consensus.cc:2454] T 90ef3584830240acb0d8aad14ae851a6 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 01:59:48.932881 6784 leader_election.cc:302] T 90ef3584830240acb0d8aad14ae851a6 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:48.933046 7725 raft_consensus.cc:2781] T 90ef3584830240acb0d8aad14ae851a6 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:48.933100 7725 raft_consensus.cc:482] T 90ef3584830240acb0d8aad14ae851a6 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:48.933126 7725 raft_consensus.cc:3037] T 90ef3584830240acb0d8aad14ae851a6 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:48.933948 7725 raft_consensus.cc:504] T 90ef3584830240acb0d8aad14ae851a6 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:48.934294 7725 leader_election.cc:288] T 90ef3584830240acb0d8aad14ae851a6 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Requested vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 01:59:48.934616 6457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90ef3584830240acb0d8aad14ae851a6" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 01:59:48.934636 6718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90ef3584830240acb0d8aad14ae851a6" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 01:59:48.934715 6457 raft_consensus.cc:3037] T 90ef3584830240acb0d8aad14ae851a6 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:48.934743 6718 raft_consensus.cc:3037] T 90ef3584830240acb0d8aad14ae851a6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:48.935457 6718 raft_consensus.cc:2454] T 90ef3584830240acb0d8aad14ae851a6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 01:59:48.935459 6457 raft_consensus.cc:2454] T 90ef3584830240acb0d8aad14ae851a6 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 01:59:48.935735 6787 leader_election.cc:302] T 90ef3584830240acb0d8aad14ae851a6 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 01:59:48.935876 7725 raft_consensus.cc:2781] T 90ef3584830240acb0d8aad14ae851a6 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:48.935984 7725 raft_consensus.cc:686] T 90ef3584830240acb0d8aad14ae851a6 P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Becoming Leader. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Running, Role: LEADER
I20240503 01:59:48.936121 7725 consensus_queue.cc:237] T 90ef3584830240acb0d8aad14ae851a6 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:48.937758 6315 catalog_manager.cc:5484] T 90ef3584830240acb0d8aad14ae851a6 P 492448570cc94ed1879feb2ec3b9051d reported cstate change: term changed from 0 to 1, leader changed from <none> to 492448570cc94ed1879feb2ec3b9051d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } }
W20240503 01:59:49.055563 7805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:49.055848 7805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:49.059067 7805 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240503 01:59:49.138890 6589 raft_consensus.cc:1261] T b91a933fb66b4075acb40c5b92ef4824 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:49.139578 6457 raft_consensus.cc:1261] T d897c21e630d4770832bb10f43778e82 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:49.139683 7469 consensus_queue.cc:1035] T b91a933fb66b4075acb40c5b92ef4824 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:49.139883 6458 raft_consensus.cc:1261] T b91a933fb66b4075acb40c5b92ef4824 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:49.140656 6850 raft_consensus.cc:1261] T d897c21e630d4770832bb10f43778e82 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:49.141299 7469 consensus_queue.cc:1035] T d897c21e630d4770832bb10f43778e82 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:49.141487 7705 consensus_queue.cc:1035] T d897c21e630d4770832bb10f43778e82 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:49.141587 7469 consensus_queue.cc:1035] T b91a933fb66b4075acb40c5b92ef4824 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:49.144232 6718 raft_consensus.cc:1261] T 90ef3584830240acb0d8aad14ae851a6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:49.146373 7661 consensus_queue.cc:1035] T 90ef3584830240acb0d8aad14ae851a6 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:49.146782 6458 raft_consensus.cc:1261] T 90ef3584830240acb0d8aad14ae851a6 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:49.150487 7661 consensus_queue.cc:1035] T 90ef3584830240acb0d8aad14ae851a6 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 89.8766 ms
time per row: 0.0449383 ms
I20240503 01:59:49.195757 7707 mvcc.cc:205] Tried to move back new op lower bound from 7023417709103710208 to 7023417708203454464. Current Snapshot: MvccSnapshot[applied={T|T < 7023417709103710208}]
I20240503 01:59:49.196995 6309 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:49.206589 6567 tablet_service.cc:1467] Processing CreateTablet for tablet f14c8008ab914d2091e6cc3f712cb1f0 (DEFAULT_TABLE table=rebalance_test_table_6 [id=65ed012903ed4deabf9e7b0dfe803452]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:49.206831 6438 tablet_service.cc:1467] Processing CreateTablet for tablet f14c8008ab914d2091e6cc3f712cb1f0 (DEFAULT_TABLE table=rebalance_test_table_6 [id=65ed012903ed4deabf9e7b0dfe803452]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:49.206930 6567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f14c8008ab914d2091e6cc3f712cb1f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:49.207073 6438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f14c8008ab914d2091e6cc3f712cb1f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:49.208693 6568 tablet_service.cc:1467] Processing CreateTablet for tablet a9795c5271c548b9867bcdaeed3414e5 (DEFAULT_TABLE table=rebalance_test_table_6 [id=65ed012903ed4deabf9e7b0dfe803452]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:49.208930 6568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9795c5271c548b9867bcdaeed3414e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:49.209300 6436 tablet_service.cc:1467] Processing CreateTablet for tablet a9795c5271c548b9867bcdaeed3414e5 (DEFAULT_TABLE table=rebalance_test_table_6 [id=65ed012903ed4deabf9e7b0dfe803452]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:49.209488 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9795c5271c548b9867bcdaeed3414e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:49.206589 6569 tablet_service.cc:1467] Processing CreateTablet for tablet 6fc09cd6e45f449fa15e486d018751ae (DEFAULT_TABLE table=rebalance_test_table_6 [id=65ed012903ed4deabf9e7b0dfe803452]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:49.210914 7742 tablet_bootstrap.cc:492] T f14c8008ab914d2091e6cc3f712cb1f0 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 01:59:49.210938 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fc09cd6e45f449fa15e486d018751ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:49.211076 7745 tablet_bootstrap.cc:492] T f14c8008ab914d2091e6cc3f712cb1f0 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 01:59:49.211903 7745 tablet_bootstrap.cc:654] T f14c8008ab914d2091e6cc3f712cb1f0 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:49.211903 7742 tablet_bootstrap.cc:654] T f14c8008ab914d2091e6cc3f712cb1f0 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:49.213940 7742 tablet_bootstrap.cc:492] T f14c8008ab914d2091e6cc3f712cb1f0 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 01:59:49.214038 7742 ts_tablet_manager.cc:1367] T f14c8008ab914d2091e6cc3f712cb1f0 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 01:59:49.213985 6831 tablet_service.cc:1467] Processing CreateTablet for tablet 6fc09cd6e45f449fa15e486d018751ae (DEFAULT_TABLE table=rebalance_test_table_6 [id=65ed012903ed4deabf9e7b0dfe803452]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:49.213985 6698 tablet_service.cc:1467] Processing CreateTablet for tablet 6fc09cd6e45f449fa15e486d018751ae (DEFAULT_TABLE table=rebalance_test_table_6 [id=65ed012903ed4deabf9e7b0dfe803452]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:49.214021 6829 tablet_service.cc:1467] Processing CreateTablet for tablet a9795c5271c548b9867bcdaeed3414e5 (DEFAULT_TABLE table=rebalance_test_table_6 [id=65ed012903ed4deabf9e7b0dfe803452]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:49.214254 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fc09cd6e45f449fa15e486d018751ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:49.214255 6698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fc09cd6e45f449fa15e486d018751ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:49.214422 6829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9795c5271c548b9867bcdaeed3414e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:49.214509 7742 raft_consensus.cc:348] T f14c8008ab914d2091e6cc3f712cb1f0 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:49.214607 7742 raft_consensus.cc:374] T f14c8008ab914d2091e6cc3f712cb1f0 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:49.214635 7742 raft_consensus.cc:729] T f14c8008ab914d2091e6cc3f712cb1f0 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 01:59:49.214668 6830 tablet_service.cc:1467] Processing CreateTablet for tablet f14c8008ab914d2091e6cc3f712cb1f0 (DEFAULT_TABLE table=rebalance_test_table_6 [id=65ed012903ed4deabf9e7b0dfe803452]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:49.214756 7742 consensus_queue.cc:260] T f14c8008ab914d2091e6cc3f712cb1f0 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:49.214869 6830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f14c8008ab914d2091e6cc3f712cb1f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:49.215020 7742 ts_tablet_manager.cc:1398] T f14c8008ab914d2091e6cc3f712cb1f0 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:49.215183 7742 tablet_bootstrap.cc:492] T a9795c5271c548b9867bcdaeed3414e5 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 01:59:49.216063 7742 tablet_bootstrap.cc:654] T a9795c5271c548b9867bcdaeed3414e5 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:49.216548 7744 tablet_bootstrap.cc:492] T a9795c5271c548b9867bcdaeed3414e5 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 01:59:49.216822 7741 tablet_bootstrap.cc:492] T 6fc09cd6e45f449fa15e486d018751ae P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 01:59:49.217557 7744 tablet_bootstrap.cc:654] T a9795c5271c548b9867bcdaeed3414e5 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:49.217670 7741 tablet_bootstrap.cc:654] T 6fc09cd6e45f449fa15e486d018751ae P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:49.219238 7745 tablet_bootstrap.cc:492] T f14c8008ab914d2091e6cc3f712cb1f0 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 01:59:49.219336 7745 ts_tablet_manager.cc:1367] T f14c8008ab914d2091e6cc3f712cb1f0 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.001s
I20240503 01:59:49.219766 7745 raft_consensus.cc:348] T f14c8008ab914d2091e6cc3f712cb1f0 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:49.219882 7745 raft_consensus.cc:374] T f14c8008ab914d2091e6cc3f712cb1f0 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:49.219911 7745 raft_consensus.cc:729] T f14c8008ab914d2091e6cc3f712cb1f0 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 01:59:49.220021 7745 consensus_queue.cc:260] T f14c8008ab914d2091e6cc3f712cb1f0 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:49.220278 7745 ts_tablet_manager.cc:1398] T f14c8008ab914d2091e6cc3f712cb1f0 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:49.220722 7744 tablet_bootstrap.cc:492] T a9795c5271c548b9867bcdaeed3414e5 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 01:59:49.220810 7744 ts_tablet_manager.cc:1367] T a9795c5271c548b9867bcdaeed3414e5 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:49.221020 7745 tablet_bootstrap.cc:492] T a9795c5271c548b9867bcdaeed3414e5 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 01:59:49.221271 7744 raft_consensus.cc:348] T a9795c5271c548b9867bcdaeed3414e5 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:49.221381 7744 raft_consensus.cc:374] T a9795c5271c548b9867bcdaeed3414e5 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:49.221411 7744 raft_consensus.cc:729] T a9795c5271c548b9867bcdaeed3414e5 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 01:59:49.221546 7744 consensus_queue.cc:260] T a9795c5271c548b9867bcdaeed3414e5 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:49.221819 7744 ts_tablet_manager.cc:1398] T a9795c5271c548b9867bcdaeed3414e5 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:49.221968 7745 tablet_bootstrap.cc:654] T a9795c5271c548b9867bcdaeed3414e5 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:49.221992 7744 tablet_bootstrap.cc:492] T f14c8008ab914d2091e6cc3f712cb1f0 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 01:59:49.222838 7744 tablet_bootstrap.cc:654] T f14c8008ab914d2091e6cc3f712cb1f0 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:49.223709 7741 tablet_bootstrap.cc:492] T 6fc09cd6e45f449fa15e486d018751ae P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 01:59:49.223816 7741 ts_tablet_manager.cc:1367] T 6fc09cd6e45f449fa15e486d018751ae P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 01:59:49.224402 7745 tablet_bootstrap.cc:492] T a9795c5271c548b9867bcdaeed3414e5 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 01:59:49.224308 7741 raft_consensus.cc:348] T 6fc09cd6e45f449fa15e486d018751ae P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:49.224447 7741 raft_consensus.cc:374] T 6fc09cd6e45f449fa15e486d018751ae P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:49.224470 7745 ts_tablet_manager.cc:1367] T a9795c5271c548b9867bcdaeed3414e5 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 01:59:49.224481 7741 raft_consensus.cc:729] T 6fc09cd6e45f449fa15e486d018751ae P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 01:59:49.224597 7741 consensus_queue.cc:260] T 6fc09cd6e45f449fa15e486d018751ae P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:49.224640 7742 tablet_bootstrap.cc:492] T a9795c5271c548b9867bcdaeed3414e5 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 01:59:49.224735 7742 ts_tablet_manager.cc:1367] T a9795c5271c548b9867bcdaeed3414e5 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.002s
I20240503 01:59:49.224861 7741 ts_tablet_manager.cc:1398] T 6fc09cd6e45f449fa15e486d018751ae P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:49.224833 7745 raft_consensus.cc:348] T a9795c5271c548b9867bcdaeed3414e5 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:49.224918 7745 raft_consensus.cc:374] T a9795c5271c548b9867bcdaeed3414e5 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:49.224946 7745 raft_consensus.cc:729] T a9795c5271c548b9867bcdaeed3414e5 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 01:59:49.225042 7745 consensus_queue.cc:260] T a9795c5271c548b9867bcdaeed3414e5 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:49.225279 7745 ts_tablet_manager.cc:1398] T a9795c5271c548b9867bcdaeed3414e5 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:49.225312 7742 raft_consensus.cc:348] T a9795c5271c548b9867bcdaeed3414e5 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:49.225410 7742 raft_consensus.cc:374] T a9795c5271c548b9867bcdaeed3414e5 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:49.225440 7742 raft_consensus.cc:729] T a9795c5271c548b9867bcdaeed3414e5 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 01:59:49.225548 7742 consensus_queue.cc:260] T a9795c5271c548b9867bcdaeed3414e5 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:49.225811 7742 ts_tablet_manager.cc:1398] T a9795c5271c548b9867bcdaeed3414e5 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:49.225965 7742 tablet_bootstrap.cc:492] T 6fc09cd6e45f449fa15e486d018751ae P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 01:59:49.226871 7742 tablet_bootstrap.cc:654] T 6fc09cd6e45f449fa15e486d018751ae P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:49.228994 7744 tablet_bootstrap.cc:492] T f14c8008ab914d2091e6cc3f712cb1f0 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 01:59:49.229146 7744 ts_tablet_manager.cc:1367] T f14c8008ab914d2091e6cc3f712cb1f0 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240503 01:59:49.229537 7742 tablet_bootstrap.cc:492] T 6fc09cd6e45f449fa15e486d018751ae P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 01:59:49.229622 7742 ts_tablet_manager.cc:1367] T 6fc09cd6e45f449fa15e486d018751ae P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 01:59:49.229884 7744 raft_consensus.cc:348] T f14c8008ab914d2091e6cc3f712cb1f0 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:49.229995 7744 raft_consensus.cc:374] T f14c8008ab914d2091e6cc3f712cb1f0 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:49.230027 7744 raft_consensus.cc:729] T f14c8008ab914d2091e6cc3f712cb1f0 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 01:59:49.230159 7744 consensus_queue.cc:260] T f14c8008ab914d2091e6cc3f712cb1f0 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:49.230150 7742 raft_consensus.cc:348] T 6fc09cd6e45f449fa15e486d018751ae P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:49.230255 7742 raft_consensus.cc:374] T 6fc09cd6e45f449fa15e486d018751ae P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:49.230291 7742 raft_consensus.cc:729] T 6fc09cd6e45f449fa15e486d018751ae P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 01:59:49.230444 7742 consensus_queue.cc:260] T 6fc09cd6e45f449fa15e486d018751ae P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:49.230513 7744 ts_tablet_manager.cc:1398] T f14c8008ab914d2091e6cc3f712cb1f0 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:49.230666 7744 tablet_bootstrap.cc:492] T 6fc09cd6e45f449fa15e486d018751ae P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 01:59:49.230911 7742 ts_tablet_manager.cc:1398] T 6fc09cd6e45f449fa15e486d018751ae P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:49.231788 7744 tablet_bootstrap.cc:654] T 6fc09cd6e45f449fa15e486d018751ae P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:49.232914 7744 tablet_bootstrap.cc:492] T 6fc09cd6e45f449fa15e486d018751ae P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 01:59:49.232996 7744 ts_tablet_manager.cc:1367] T 6fc09cd6e45f449fa15e486d018751ae P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 01:59:49.233426 7744 raft_consensus.cc:348] T 6fc09cd6e45f449fa15e486d018751ae P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:49.233525 7744 raft_consensus.cc:374] T 6fc09cd6e45f449fa15e486d018751ae P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:49.233551 7744 raft_consensus.cc:729] T 6fc09cd6e45f449fa15e486d018751ae P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 01:59:49.233669 7744 consensus_queue.cc:260] T 6fc09cd6e45f449fa15e486d018751ae P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:49.233899 7744 ts_tablet_manager.cc:1398] T 6fc09cd6e45f449fa15e486d018751ae P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:49.240665 7784 raft_consensus.cc:482] T 6fc09cd6e45f449fa15e486d018751ae P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:49.240785 7784 raft_consensus.cc:504] T 6fc09cd6e45f449fa15e486d018751ae P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:49.241153 7784 leader_election.cc:288] T 6fc09cd6e45f449fa15e486d018751ae P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 01:59:49.241567 6718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fc09cd6e45f449fa15e486d018751ae" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 01:59:49.241600 6850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fc09cd6e45f449fa15e486d018751ae" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 01:59:49.241729 6850 raft_consensus.cc:2454] T 6fc09cd6e45f449fa15e486d018751ae P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 01:59:49.241724 6718 raft_consensus.cc:2454] T 6fc09cd6e45f449fa15e486d018751ae P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 01:59:49.242060 6525 leader_election.cc:302] T 6fc09cd6e45f449fa15e486d018751ae P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 80fd18ef4d604c86b909d5c008a46b4f; no voters:
I20240503 01:59:49.242233 7784 raft_consensus.cc:2781] T 6fc09cd6e45f449fa15e486d018751ae P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:49.242292 7784 raft_consensus.cc:482] T 6fc09cd6e45f449fa15e486d018751ae P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:49.242411 7784 raft_consensus.cc:3037] T 6fc09cd6e45f449fa15e486d018751ae P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:49.243201 7784 raft_consensus.cc:504] T 6fc09cd6e45f449fa15e486d018751ae P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:49.243523 7784 leader_election.cc:288] T 6fc09cd6e45f449fa15e486d018751ae P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Requested vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 01:59:49.243906 6718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fc09cd6e45f449fa15e486d018751ae" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 01:59:49.243921 6850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fc09cd6e45f449fa15e486d018751ae" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 01:59:49.244014 6718 raft_consensus.cc:3037] T 6fc09cd6e45f449fa15e486d018751ae P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:49.244028 6850 raft_consensus.cc:3037] T 6fc09cd6e45f449fa15e486d018751ae P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:49.244872 6718 raft_consensus.cc:2454] T 6fc09cd6e45f449fa15e486d018751ae P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 01:59:49.245051 6850 raft_consensus.cc:2454] T 6fc09cd6e45f449fa15e486d018751ae P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 01:59:49.245193 6525 leader_election.cc:302] T 6fc09cd6e45f449fa15e486d018751ae P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 01:59:49.245364 7784 raft_consensus.cc:2781] T 6fc09cd6e45f449fa15e486d018751ae P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:49.245443 7784 raft_consensus.cc:686] T 6fc09cd6e45f449fa15e486d018751ae P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Becoming Leader. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Running, Role: LEADER
I20240503 01:59:49.245576 7784 consensus_queue.cc:237] T 6fc09cd6e45f449fa15e486d018751ae P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:49.247247 6316 catalog_manager.cc:5484] T 6fc09cd6e45f449fa15e486d018751ae P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:49.299016 7652 raft_consensus.cc:482] T a9795c5271c548b9867bcdaeed3414e5 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:49.299178 7652 raft_consensus.cc:504] T a9795c5271c548b9867bcdaeed3414e5 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:49.299623 7652 leader_election.cc:288] T a9795c5271c548b9867bcdaeed3414e5 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 01:59:49.300047 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9795c5271c548b9867bcdaeed3414e5" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 01:59:49.300117 6850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9795c5271c548b9867bcdaeed3414e5" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 01:59:49.300208 6589 raft_consensus.cc:2454] T a9795c5271c548b9867bcdaeed3414e5 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 01:59:49.300230 6850 raft_consensus.cc:2454] T a9795c5271c548b9867bcdaeed3414e5 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 01:59:49.300549 6393 leader_election.cc:302] T a9795c5271c548b9867bcdaeed3414e5 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:49.300722 7652 raft_consensus.cc:2781] T a9795c5271c548b9867bcdaeed3414e5 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:49.300792 7652 raft_consensus.cc:482] T a9795c5271c548b9867bcdaeed3414e5 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:49.300818 7652 raft_consensus.cc:3037] T a9795c5271c548b9867bcdaeed3414e5 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:49.301602 7652 raft_consensus.cc:504] T a9795c5271c548b9867bcdaeed3414e5 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:49.301931 7652 leader_election.cc:288] T a9795c5271c548b9867bcdaeed3414e5 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Requested vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 01:59:49.302408 6850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9795c5271c548b9867bcdaeed3414e5" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 01:59:49.302408 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9795c5271c548b9867bcdaeed3414e5" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 01:59:49.302542 6589 raft_consensus.cc:3037] T a9795c5271c548b9867bcdaeed3414e5 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:49.302560 6850 raft_consensus.cc:3037] T a9795c5271c548b9867bcdaeed3414e5 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:49.303471 6850 raft_consensus.cc:2454] T a9795c5271c548b9867bcdaeed3414e5 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 01:59:49.303471 6589 raft_consensus.cc:2454] T a9795c5271c548b9867bcdaeed3414e5 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 01:59:49.303810 6393 leader_election.cc:302] T a9795c5271c548b9867bcdaeed3414e5 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:49.304008 7652 raft_consensus.cc:2781] T a9795c5271c548b9867bcdaeed3414e5 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:49.304104 7652 raft_consensus.cc:686] T a9795c5271c548b9867bcdaeed3414e5 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Becoming Leader. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Running, Role: LEADER
I20240503 01:59:49.304231 7652 consensus_queue.cc:237] T a9795c5271c548b9867bcdaeed3414e5 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:49.306254 6316 catalog_manager.cc:5484] T a9795c5271c548b9867bcdaeed3414e5 P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: term changed from 0 to 1, leader changed from <none> to dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129). New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:49.439654 7661 raft_consensus.cc:482] T f14c8008ab914d2091e6cc3f712cb1f0 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:49.439802 7661 raft_consensus.cc:504] T f14c8008ab914d2091e6cc3f712cb1f0 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:49.440203 7661 leader_election.cc:288] T f14c8008ab914d2091e6cc3f712cb1f0 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 01:59:49.440763 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f14c8008ab914d2091e6cc3f712cb1f0" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 01:59:49.440768 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f14c8008ab914d2091e6cc3f712cb1f0" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 01:59:49.440899 6456 raft_consensus.cc:2454] T f14c8008ab914d2091e6cc3f712cb1f0 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 01:59:49.440904 6589 raft_consensus.cc:2454] T f14c8008ab914d2091e6cc3f712cb1f0 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 01:59:49.441229 6786 leader_election.cc:302] T f14c8008ab914d2091e6cc3f712cb1f0 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 80fd18ef4d604c86b909d5c008a46b4f; no voters:
I20240503 01:59:49.441483 7661 raft_consensus.cc:2781] T f14c8008ab914d2091e6cc3f712cb1f0 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:49.441555 7661 raft_consensus.cc:482] T f14c8008ab914d2091e6cc3f712cb1f0 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:49.441596 7661 raft_consensus.cc:3037] T f14c8008ab914d2091e6cc3f712cb1f0 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:49.442519 7661 raft_consensus.cc:504] T f14c8008ab914d2091e6cc3f712cb1f0 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:49.442890 7661 leader_election.cc:288] T f14c8008ab914d2091e6cc3f712cb1f0 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Requested vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 01:59:49.443336 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f14c8008ab914d2091e6cc3f712cb1f0" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 01:59:49.443390 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f14c8008ab914d2091e6cc3f712cb1f0" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 01:59:49.443459 6589 raft_consensus.cc:3037] T f14c8008ab914d2091e6cc3f712cb1f0 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:49.443526 6456 raft_consensus.cc:3037] T f14c8008ab914d2091e6cc3f712cb1f0 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:49.444270 6589 raft_consensus.cc:2454] T f14c8008ab914d2091e6cc3f712cb1f0 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 01:59:49.444527 6456 raft_consensus.cc:2454] T f14c8008ab914d2091e6cc3f712cb1f0 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 01:59:49.444634 6786 leader_election.cc:302] T f14c8008ab914d2091e6cc3f712cb1f0 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 80fd18ef4d604c86b909d5c008a46b4f; no voters:
I20240503 01:59:49.444839 7661 raft_consensus.cc:2781] T f14c8008ab914d2091e6cc3f712cb1f0 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:49.444948 7661 raft_consensus.cc:686] T f14c8008ab914d2091e6cc3f712cb1f0 P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Becoming Leader. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Running, Role: LEADER
I20240503 01:59:49.445091 7661 consensus_queue.cc:237] T f14c8008ab914d2091e6cc3f712cb1f0 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:49.447127 6316 catalog_manager.cc:5484] T f14c8008ab914d2091e6cc3f712cb1f0 P 492448570cc94ed1879feb2ec3b9051d reported cstate change: term changed from 0 to 1, leader changed from <none> to 492448570cc94ed1879feb2ec3b9051d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } }
W20240503 01:59:49.576581 7857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:49.576884 7857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:49.580404 7857 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240503 01:59:49.634402 7784 consensus_queue.cc:1035] T 6fc09cd6e45f449fa15e486d018751ae P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:49.640960 7786 consensus_queue.cc:1035] T 6fc09cd6e45f449fa15e486d018751ae P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:49.669440 6456 raft_consensus.cc:1261] T f14c8008ab914d2091e6cc3f712cb1f0 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:49.669613 6589 raft_consensus.cc:1261] T f14c8008ab914d2091e6cc3f712cb1f0 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:49.669875 6850 raft_consensus.cc:1261] T a9795c5271c548b9867bcdaeed3414e5 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:49.670059 7661 consensus_queue.cc:1035] T f14c8008ab914d2091e6cc3f712cb1f0 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:49.670470 7652 consensus_queue.cc:1035] T a9795c5271c548b9867bcdaeed3414e5 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:49.672009 6582 raft_consensus.cc:1261] T a9795c5271c548b9867bcdaeed3414e5 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:49.672119 7725 consensus_queue.cc:1035] T f14c8008ab914d2091e6cc3f712cb1f0 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:49.672605 7448 consensus_queue.cc:1035] T a9795c5271c548b9867bcdaeed3414e5 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:49.699110 7718 mvcc.cc:205] Tried to move back new op lower bound from 7023417711278870528 to 7023417710368583680. Current Snapshot: MvccSnapshot[applied={T|T < 7023417711278870528}]
INSERT report
rows total: 2000
time total: 111.758 ms
time per row: 0.055879 ms
I20240503 01:59:49.739934 6316 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:49.746863 6698 tablet_service.cc:1467] Processing CreateTablet for tablet 2e967e377984468d90799041c29882bd (DEFAULT_TABLE table=rebalance_test_table_7 [id=e0e5e479de824fb18516cfe56d5892c6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:49.747103 6569 tablet_service.cc:1467] Processing CreateTablet for tablet 2e967e377984468d90799041c29882bd (DEFAULT_TABLE table=rebalance_test_table_7 [id=e0e5e479de824fb18516cfe56d5892c6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:49.747262 6698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e967e377984468d90799041c29882bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:49.747400 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e967e377984468d90799041c29882bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:49.749342 7907 tablet_bootstrap.cc:492] T 2e967e377984468d90799041c29882bd P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 01:59:49.749415 7908 tablet_bootstrap.cc:492] T 2e967e377984468d90799041c29882bd P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 01:59:49.749557 6568 tablet_service.cc:1467] Processing CreateTablet for tablet 3416b6107e1848f299ac49ccd88ae7c8 (DEFAULT_TABLE table=rebalance_test_table_7 [id=e0e5e479de824fb18516cfe56d5892c6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:49.749806 6568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3416b6107e1848f299ac49ccd88ae7c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:49.750276 7908 tablet_bootstrap.cc:654] T 2e967e377984468d90799041c29882bd P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:49.750289 7907 tablet_bootstrap.cc:654] T 2e967e377984468d90799041c29882bd P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:49.751235 7908 tablet_bootstrap.cc:492] T 2e967e377984468d90799041c29882bd P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 01:59:49.751329 7908 ts_tablet_manager.cc:1367] T 2e967e377984468d90799041c29882bd P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:49.751703 7908 raft_consensus.cc:348] T 2e967e377984468d90799041c29882bd P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:49.751832 7908 raft_consensus.cc:374] T 2e967e377984468d90799041c29882bd P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:49.751864 7908 raft_consensus.cc:729] T 2e967e377984468d90799041c29882bd P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 01:59:49.752007 7908 consensus_queue.cc:260] T 2e967e377984468d90799041c29882bd P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:49.752293 7908 ts_tablet_manager.cc:1398] T 2e967e377984468d90799041c29882bd P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:49.752628 6438 tablet_service.cc:1467] Processing CreateTablet for tablet 2fc834abc6a94c638d6202b03320a2d2 (DEFAULT_TABLE table=rebalance_test_table_7 [id=e0e5e479de824fb18516cfe56d5892c6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:49.752897 6438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fc834abc6a94c638d6202b03320a2d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:49.753336 7907 tablet_bootstrap.cc:492] T 2e967e377984468d90799041c29882bd P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 01:59:49.753437 7907 ts_tablet_manager.cc:1367] T 2e967e377984468d90799041c29882bd P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:49.754063 7907 raft_consensus.cc:348] T 2e967e377984468d90799041c29882bd P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:49.754184 7907 raft_consensus.cc:374] T 2e967e377984468d90799041c29882bd P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:49.754225 7907 raft_consensus.cc:729] T 2e967e377984468d90799041c29882bd P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 01:59:49.754400 7907 consensus_queue.cc:260] T 2e967e377984468d90799041c29882bd P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:49.755221 7907 ts_tablet_manager.cc:1398] T 2e967e377984468d90799041c29882bd P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:49.755420 7907 tablet_bootstrap.cc:492] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 01:59:49.755419 7911 tablet_bootstrap.cc:492] T 2fc834abc6a94c638d6202b03320a2d2 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 01:59:49.755456 6436 tablet_service.cc:1467] Processing CreateTablet for tablet 2e967e377984468d90799041c29882bd (DEFAULT_TABLE table=rebalance_test_table_7 [id=e0e5e479de824fb18516cfe56d5892c6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:49.755722 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e967e377984468d90799041c29882bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:49.756522 7911 tablet_bootstrap.cc:654] T 2fc834abc6a94c638d6202b03320a2d2 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:49.757269 7907 tablet_bootstrap.cc:654] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:49.759230 7907 tablet_bootstrap.cc:492] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 01:59:49.759264 6831 tablet_service.cc:1467] Processing CreateTablet for tablet 3416b6107e1848f299ac49ccd88ae7c8 (DEFAULT_TABLE table=rebalance_test_table_7 [id=e0e5e479de824fb18516cfe56d5892c6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:49.759336 7907 ts_tablet_manager.cc:1367] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:49.759421 6698 tablet_service.cc:1467] Processing CreateTablet for tablet 3416b6107e1848f299ac49ccd88ae7c8 (DEFAULT_TABLE table=rebalance_test_table_7 [id=e0e5e479de824fb18516cfe56d5892c6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:49.759567 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3416b6107e1848f299ac49ccd88ae7c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:49.759689 6698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3416b6107e1848f299ac49ccd88ae7c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:49.759894 7907 raft_consensus.cc:348] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:49.760008 7907 raft_consensus.cc:374] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:49.760037 7907 raft_consensus.cc:729] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 01:59:49.760143 7907 consensus_queue.cc:260] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:49.760555 7907 ts_tablet_manager.cc:1398] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:49.761982 7908 tablet_bootstrap.cc:492] T 3416b6107e1848f299ac49ccd88ae7c8 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 01:59:49.762187 6700 tablet_service.cc:1467] Processing CreateTablet for tablet 2fc834abc6a94c638d6202b03320a2d2 (DEFAULT_TABLE table=rebalance_test_table_7 [id=e0e5e479de824fb18516cfe56d5892c6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:49.762264 6830 tablet_service.cc:1467] Processing CreateTablet for tablet 2fc834abc6a94c638d6202b03320a2d2 (DEFAULT_TABLE table=rebalance_test_table_7 [id=e0e5e479de824fb18516cfe56d5892c6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:49.762424 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fc834abc6a94c638d6202b03320a2d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:49.762519 6830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fc834abc6a94c638d6202b03320a2d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:49.762909 7908 tablet_bootstrap.cc:654] T 3416b6107e1848f299ac49ccd88ae7c8 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:49.765311 7914 tablet_bootstrap.cc:492] T 3416b6107e1848f299ac49ccd88ae7c8 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 01:59:49.765887 7911 tablet_bootstrap.cc:492] T 2fc834abc6a94c638d6202b03320a2d2 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 01:59:49.766005 7911 ts_tablet_manager.cc:1367] T 2fc834abc6a94c638d6202b03320a2d2 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.001s
I20240503 01:59:49.766554 7908 tablet_bootstrap.cc:492] T 3416b6107e1848f299ac49ccd88ae7c8 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 01:59:49.766530 7911 raft_consensus.cc:348] T 2fc834abc6a94c638d6202b03320a2d2 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:49.766589 7914 tablet_bootstrap.cc:654] T 3416b6107e1848f299ac49ccd88ae7c8 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:49.766673 7908 ts_tablet_manager.cc:1367] T 3416b6107e1848f299ac49ccd88ae7c8 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 01:59:49.766690 7911 raft_consensus.cc:374] T 2fc834abc6a94c638d6202b03320a2d2 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:49.766724 7911 raft_consensus.cc:729] T 2fc834abc6a94c638d6202b03320a2d2 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 01:59:49.766845 7911 consensus_queue.cc:260] T 2fc834abc6a94c638d6202b03320a2d2 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:49.767127 7911 ts_tablet_manager.cc:1398] T 2fc834abc6a94c638d6202b03320a2d2 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:49.767172 7908 raft_consensus.cc:348] T 3416b6107e1848f299ac49ccd88ae7c8 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:49.767302 7911 tablet_bootstrap.cc:492] T 2e967e377984468d90799041c29882bd P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 01:59:49.767324 7908 raft_consensus.cc:374] T 3416b6107e1848f299ac49ccd88ae7c8 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:49.767369 7908 raft_consensus.cc:729] T 3416b6107e1848f299ac49ccd88ae7c8 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 01:59:49.767498 7908 consensus_queue.cc:260] T 3416b6107e1848f299ac49ccd88ae7c8 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:49.767791 7908 ts_tablet_manager.cc:1398] T 3416b6107e1848f299ac49ccd88ae7c8 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:49.767995 7908 tablet_bootstrap.cc:492] T 2fc834abc6a94c638d6202b03320a2d2 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 01:59:49.768244 7911 tablet_bootstrap.cc:654] T 2e967e377984468d90799041c29882bd P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:49.768313 7914 tablet_bootstrap.cc:492] T 3416b6107e1848f299ac49ccd88ae7c8 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 01:59:49.768445 7914 ts_tablet_manager.cc:1367] T 3416b6107e1848f299ac49ccd88ae7c8 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:49.768900 7914 raft_consensus.cc:348] T 3416b6107e1848f299ac49ccd88ae7c8 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:49.769013 7914 raft_consensus.cc:374] T 3416b6107e1848f299ac49ccd88ae7c8 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:49.769043 7914 raft_consensus.cc:729] T 3416b6107e1848f299ac49ccd88ae7c8 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 01:59:49.769316 7914 consensus_queue.cc:260] T 3416b6107e1848f299ac49ccd88ae7c8 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:49.770202 7914 ts_tablet_manager.cc:1398] T 3416b6107e1848f299ac49ccd88ae7c8 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:49.770445 7914 tablet_bootstrap.cc:492] T 2fc834abc6a94c638d6202b03320a2d2 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 01:59:49.771581 7914 tablet_bootstrap.cc:654] T 2fc834abc6a94c638d6202b03320a2d2 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:49.772926 7914 tablet_bootstrap.cc:492] T 2fc834abc6a94c638d6202b03320a2d2 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 01:59:49.773025 7914 ts_tablet_manager.cc:1367] T 2fc834abc6a94c638d6202b03320a2d2 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:49.773038 7908 tablet_bootstrap.cc:654] T 2fc834abc6a94c638d6202b03320a2d2 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:49.773725 7914 raft_consensus.cc:348] T 2fc834abc6a94c638d6202b03320a2d2 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:49.773850 7914 raft_consensus.cc:374] T 2fc834abc6a94c638d6202b03320a2d2 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:49.773885 7914 raft_consensus.cc:729] T 2fc834abc6a94c638d6202b03320a2d2 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 01:59:49.774013 7914 consensus_queue.cc:260] T 2fc834abc6a94c638d6202b03320a2d2 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:49.774313 7914 ts_tablet_manager.cc:1398] T 2fc834abc6a94c638d6202b03320a2d2 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:49.774879 7908 tablet_bootstrap.cc:492] T 2fc834abc6a94c638d6202b03320a2d2 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 01:59:49.774991 7908 ts_tablet_manager.cc:1367] T 2fc834abc6a94c638d6202b03320a2d2 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240503 01:59:49.775566 7908 raft_consensus.cc:348] T 2fc834abc6a94c638d6202b03320a2d2 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:49.775689 7908 raft_consensus.cc:374] T 2fc834abc6a94c638d6202b03320a2d2 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:49.775719 7908 raft_consensus.cc:729] T 2fc834abc6a94c638d6202b03320a2d2 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 01:59:49.775846 7908 consensus_queue.cc:260] T 2fc834abc6a94c638d6202b03320a2d2 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:49.776120 7908 ts_tablet_manager.cc:1398] T 2fc834abc6a94c638d6202b03320a2d2 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:49.777484 7911 tablet_bootstrap.cc:492] T 2e967e377984468d90799041c29882bd P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 01:59:49.777590 7911 ts_tablet_manager.cc:1367] T 2e967e377984468d90799041c29882bd P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.001s
I20240503 01:59:49.778088 7911 raft_consensus.cc:348] T 2e967e377984468d90799041c29882bd P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:49.778231 7911 raft_consensus.cc:374] T 2e967e377984468d90799041c29882bd P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:49.778268 7911 raft_consensus.cc:729] T 2e967e377984468d90799041c29882bd P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 01:59:49.778398 7911 consensus_queue.cc:260] T 2e967e377984468d90799041c29882bd P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:49.778694 7911 ts_tablet_manager.cc:1398] T 2e967e377984468d90799041c29882bd P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:49.821394 7835 raft_consensus.cc:482] T 2fc834abc6a94c638d6202b03320a2d2 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:49.821535 7835 raft_consensus.cc:504] T 2fc834abc6a94c638d6202b03320a2d2 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:49.821974 7835 leader_election.cc:288] T 2fc834abc6a94c638d6202b03320a2d2 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 01:59:49.822487 6850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fc834abc6a94c638d6202b03320a2d2" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 01:59:49.822540 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fc834abc6a94c638d6202b03320a2d2" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 01:59:49.822623 6850 raft_consensus.cc:2454] T 2fc834abc6a94c638d6202b03320a2d2 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 01:59:49.822695 6456 raft_consensus.cc:2454] T 2fc834abc6a94c638d6202b03320a2d2 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 01:59:49.823035 6656 leader_election.cc:302] T 2fc834abc6a94c638d6202b03320a2d2 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 01:59:49.823210 7835 raft_consensus.cc:2781] T 2fc834abc6a94c638d6202b03320a2d2 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:49.823268 7835 raft_consensus.cc:482] T 2fc834abc6a94c638d6202b03320a2d2 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:49.823294 7835 raft_consensus.cc:3037] T 2fc834abc6a94c638d6202b03320a2d2 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:49.824142 7835 raft_consensus.cc:504] T 2fc834abc6a94c638d6202b03320a2d2 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:49.824528 7835 leader_election.cc:288] T 2fc834abc6a94c638d6202b03320a2d2 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Requested vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 01:59:49.824894 6850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fc834abc6a94c638d6202b03320a2d2" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 01:59:49.824879 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fc834abc6a94c638d6202b03320a2d2" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 01:59:49.825017 6456 raft_consensus.cc:3037] T 2fc834abc6a94c638d6202b03320a2d2 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:49.825017 6850 raft_consensus.cc:3037] T 2fc834abc6a94c638d6202b03320a2d2 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:49.825805 6456 raft_consensus.cc:2454] T 2fc834abc6a94c638d6202b03320a2d2 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 01:59:49.825817 6850 raft_consensus.cc:2454] T 2fc834abc6a94c638d6202b03320a2d2 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 01:59:49.826102 6653 leader_election.cc:302] T 2fc834abc6a94c638d6202b03320a2d2 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2ac2eb7cc264047b9ab32fe3a4fc025, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:49.826265 7835 raft_consensus.cc:2781] T 2fc834abc6a94c638d6202b03320a2d2 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:49.826340 7835 raft_consensus.cc:686] T 2fc834abc6a94c638d6202b03320a2d2 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Becoming Leader. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Running, Role: LEADER
I20240503 01:59:49.826481 7835 consensus_queue.cc:237] T 2fc834abc6a94c638d6202b03320a2d2 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:49.828285 6315 catalog_manager.cc:5484] T 2fc834abc6a94c638d6202b03320a2d2 P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:49.836347 7904 raft_consensus.cc:482] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:49.836519 7904 raft_consensus.cc:504] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:49.836900 7904 leader_election.cc:288] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 01:59:49.837358 6718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3416b6107e1848f299ac49ccd88ae7c8" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 01:59:49.837412 6850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3416b6107e1848f299ac49ccd88ae7c8" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 01:59:49.837517 6718 raft_consensus.cc:2454] T 3416b6107e1848f299ac49ccd88ae7c8 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 01:59:49.837559 6850 raft_consensus.cc:2454] T 3416b6107e1848f299ac49ccd88ae7c8 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 01:59:49.837852 6525 leader_election.cc:302] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 01:59:49.838009 7904 raft_consensus.cc:2781] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:49.838071 7904 raft_consensus.cc:482] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:49.838107 7904 raft_consensus.cc:3037] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:49.839013 7904 raft_consensus.cc:504] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:49.839358 7904 leader_election.cc:288] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Requested vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 01:59:49.839743 6718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3416b6107e1848f299ac49ccd88ae7c8" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 01:59:49.839856 6718 raft_consensus.cc:3037] T 3416b6107e1848f299ac49ccd88ae7c8 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:49.839816 6850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3416b6107e1848f299ac49ccd88ae7c8" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 01:59:49.839962 6850 raft_consensus.cc:3037] T 3416b6107e1848f299ac49ccd88ae7c8 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:49.840683 6718 raft_consensus.cc:2454] T 3416b6107e1848f299ac49ccd88ae7c8 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 01:59:49.840843 6850 raft_consensus.cc:2454] T 3416b6107e1848f299ac49ccd88ae7c8 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 01:59:49.841039 6525 leader_election.cc:302] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 01:59:49.841207 7904 raft_consensus.cc:2781] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:49.841307 7904 raft_consensus.cc:686] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Becoming Leader. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Running, Role: LEADER
I20240503 01:59:49.841450 7904 consensus_queue.cc:237] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:49.843282 6315 catalog_manager.cc:5484] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:49.979889 7835 raft_consensus.cc:482] T 2e967e377984468d90799041c29882bd P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:49.980067 7835 raft_consensus.cc:504] T 2e967e377984468d90799041c29882bd P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:49.980542 7835 leader_election.cc:288] T 2e967e377984468d90799041c29882bd P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 01:59:49.980995 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e967e377984468d90799041c29882bd" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 01:59:49.980995 6582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e967e377984468d90799041c29882bd" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 01:59:49.981149 6456 raft_consensus.cc:2454] T 2e967e377984468d90799041c29882bd P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 01:59:49.981158 6582 raft_consensus.cc:2454] T 2e967e377984468d90799041c29882bd P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 01:59:49.981479 6655 leader_election.cc:302] T 2e967e377984468d90799041c29882bd P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 01:59:49.981671 7835 raft_consensus.cc:2781] T 2e967e377984468d90799041c29882bd P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:49.981741 7835 raft_consensus.cc:482] T 2e967e377984468d90799041c29882bd P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:49.981768 7835 raft_consensus.cc:3037] T 2e967e377984468d90799041c29882bd P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:49.982621 7835 raft_consensus.cc:504] T 2e967e377984468d90799041c29882bd P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:49.982992 7835 leader_election.cc:288] T 2e967e377984468d90799041c29882bd P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Requested vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 01:59:49.983387 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e967e377984468d90799041c29882bd" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 01:59:49.983441 6582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e967e377984468d90799041c29882bd" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 01:59:49.983503 6456 raft_consensus.cc:3037] T 2e967e377984468d90799041c29882bd P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:49.983578 6582 raft_consensus.cc:3037] T 2e967e377984468d90799041c29882bd P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:49.984481 6456 raft_consensus.cc:2454] T 2e967e377984468d90799041c29882bd P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 01:59:49.984547 6582 raft_consensus.cc:2454] T 2e967e377984468d90799041c29882bd P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 01:59:49.984807 6653 leader_election.cc:302] T 2e967e377984468d90799041c29882bd P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2ac2eb7cc264047b9ab32fe3a4fc025, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:49.985005 7835 raft_consensus.cc:2781] T 2e967e377984468d90799041c29882bd P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:49.985095 7835 raft_consensus.cc:686] T 2e967e377984468d90799041c29882bd P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Becoming Leader. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Running, Role: LEADER
I20240503 01:59:49.985234 7835 consensus_queue.cc:237] T 2e967e377984468d90799041c29882bd P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:49.987195 6315 catalog_manager.cc:5484] T 2e967e377984468d90799041c29882bd P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } }
W20240503 01:59:50.119644 7923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:50.119932 7923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:50.123518 7923 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240503 01:59:50.226959 6850 raft_consensus.cc:1261] T 3416b6107e1848f299ac49ccd88ae7c8 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:50.226944 6718 raft_consensus.cc:1261] T 3416b6107e1848f299ac49ccd88ae7c8 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:50.227502 7904 consensus_queue.cc:1035] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:50.227855 7786 consensus_queue.cc:1035] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:50.229467 6457 raft_consensus.cc:1261] T 2e967e377984468d90799041c29882bd P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:50.230064 7469 consensus_queue.cc:1035] T 2e967e377984468d90799041c29882bd P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:50.230239 6457 raft_consensus.cc:1261] T 2fc834abc6a94c638d6202b03320a2d2 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:50.230597 7469 consensus_queue.cc:1035] T 2fc834abc6a94c638d6202b03320a2d2 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:50.232583 6851 raft_consensus.cc:1261] T 2fc834abc6a94c638d6202b03320a2d2 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:50.233489 6582 raft_consensus.cc:1261] T 2e967e377984468d90799041c29882bd P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:50.236522 7469 consensus_queue.cc:1035] T 2fc834abc6a94c638d6202b03320a2d2 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:50.239447 7469 consensus_queue.cc:1035] T 2e967e377984468d90799041c29882bd P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:50.244544 7954 mvcc.cc:205] Tried to move back new op lower bound from 7023417713609560064 to 7023417711930744832. Current Snapshot: MvccSnapshot[applied={T|T < 7023417713572958208}]
I20240503 01:59:50.273975 7960 mvcc.cc:205] Tried to move back new op lower bound from 7023417713572958208 to 7023417711930744832. Current Snapshot: MvccSnapshot[applied={T|T < 7023417713572958208}]
INSERT report
rows total: 2000
time total: 132.312 ms
time per row: 0.0661559 ms
I20240503 01:59:50.300141 6309 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:50.306566 6700 tablet_service.cc:1467] Processing CreateTablet for tablet 7f637097ad06465cb93c6ff6d97c1d17 (DEFAULT_TABLE table=rebalance_test_table_8 [id=d93ca01e08fd40d886555f16d21192fc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:50.306665 6568 tablet_service.cc:1467] Processing CreateTablet for tablet 7f637097ad06465cb93c6ff6d97c1d17 (DEFAULT_TABLE table=rebalance_test_table_8 [id=d93ca01e08fd40d886555f16d21192fc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:50.306903 6568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f637097ad06465cb93c6ff6d97c1d17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:50.306918 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f637097ad06465cb93c6ff6d97c1d17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:50.307204 6436 tablet_service.cc:1467] Processing CreateTablet for tablet 7f637097ad06465cb93c6ff6d97c1d17 (DEFAULT_TABLE table=rebalance_test_table_8 [id=d93ca01e08fd40d886555f16d21192fc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:50.307204 6438 tablet_service.cc:1467] Processing CreateTablet for tablet 548f43b381f1445c8d3d55705ae5a48a (DEFAULT_TABLE table=rebalance_test_table_8 [id=d93ca01e08fd40d886555f16d21192fc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:50.307505 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f637097ad06465cb93c6ff6d97c1d17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:50.307998 6567 tablet_service.cc:1467] Processing CreateTablet for tablet ce8bfc835792444e8703e73252b1f638 (DEFAULT_TABLE table=rebalance_test_table_8 [id=d93ca01e08fd40d886555f16d21192fc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:50.308220 6567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce8bfc835792444e8703e73252b1f638. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:50.309456 6569 tablet_service.cc:1467] Processing CreateTablet for tablet 548f43b381f1445c8d3d55705ae5a48a (DEFAULT_TABLE table=rebalance_test_table_8 [id=d93ca01e08fd40d886555f16d21192fc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:50.309675 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 548f43b381f1445c8d3d55705ae5a48a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:50.309883 7986 tablet_bootstrap.cc:492] T 7f637097ad06465cb93c6ff6d97c1d17 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 01:59:50.309947 7985 tablet_bootstrap.cc:492] T 7f637097ad06465cb93c6ff6d97c1d17 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 01:59:50.310765 6438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 548f43b381f1445c8d3d55705ae5a48a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:50.310801 7986 tablet_bootstrap.cc:654] T 7f637097ad06465cb93c6ff6d97c1d17 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:50.311097 7985 tablet_bootstrap.cc:654] T 7f637097ad06465cb93c6ff6d97c1d17 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:50.312204 7985 tablet_bootstrap.cc:492] T 7f637097ad06465cb93c6ff6d97c1d17 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 01:59:50.312295 7985 ts_tablet_manager.cc:1367] T 7f637097ad06465cb93c6ff6d97c1d17 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:50.312625 7987 tablet_bootstrap.cc:492] T 7f637097ad06465cb93c6ff6d97c1d17 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 01:59:50.312662 6831 tablet_service.cc:1467] Processing CreateTablet for tablet ce8bfc835792444e8703e73252b1f638 (DEFAULT_TABLE table=rebalance_test_table_8 [id=d93ca01e08fd40d886555f16d21192fc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:50.312757 7985 raft_consensus.cc:348] T 7f637097ad06465cb93c6ff6d97c1d17 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:50.312871 7985 raft_consensus.cc:374] T 7f637097ad06465cb93c6ff6d97c1d17 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:50.312902 7985 raft_consensus.cc:729] T 7f637097ad06465cb93c6ff6d97c1d17 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 01:59:50.312933 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce8bfc835792444e8703e73252b1f638. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:50.313027 7985 consensus_queue.cc:260] T 7f637097ad06465cb93c6ff6d97c1d17 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:50.313213 6830 tablet_service.cc:1467] Processing CreateTablet for tablet 548f43b381f1445c8d3d55705ae5a48a (DEFAULT_TABLE table=rebalance_test_table_8 [id=d93ca01e08fd40d886555f16d21192fc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:50.313330 7985 ts_tablet_manager.cc:1398] T 7f637097ad06465cb93c6ff6d97c1d17 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:50.313418 6830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 548f43b381f1445c8d3d55705ae5a48a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:50.313544 7987 tablet_bootstrap.cc:654] T 7f637097ad06465cb93c6ff6d97c1d17 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:50.314464 6700 tablet_service.cc:1467] Processing CreateTablet for tablet ce8bfc835792444e8703e73252b1f638 (DEFAULT_TABLE table=rebalance_test_table_8 [id=d93ca01e08fd40d886555f16d21192fc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:50.314688 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce8bfc835792444e8703e73252b1f638. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:50.315109 7991 tablet_bootstrap.cc:492] T ce8bfc835792444e8703e73252b1f638 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 01:59:50.315850 7986 tablet_bootstrap.cc:492] T 7f637097ad06465cb93c6ff6d97c1d17 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 01:59:50.315964 7986 ts_tablet_manager.cc:1367] T 7f637097ad06465cb93c6ff6d97c1d17 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 01:59:50.316236 7991 tablet_bootstrap.cc:654] T ce8bfc835792444e8703e73252b1f638 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:50.316560 7986 raft_consensus.cc:348] T 7f637097ad06465cb93c6ff6d97c1d17 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:50.316684 7986 raft_consensus.cc:374] T 7f637097ad06465cb93c6ff6d97c1d17 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:50.316691 7985 tablet_bootstrap.cc:492] T ce8bfc835792444e8703e73252b1f638 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 01:59:50.316715 7986 raft_consensus.cc:729] T 7f637097ad06465cb93c6ff6d97c1d17 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 01:59:50.317046 7986 consensus_queue.cc:260] T 7f637097ad06465cb93c6ff6d97c1d17 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:50.317454 7986 ts_tablet_manager.cc:1398] T 7f637097ad06465cb93c6ff6d97c1d17 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 01:59:50.317569 7985 tablet_bootstrap.cc:654] T ce8bfc835792444e8703e73252b1f638 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:50.317692 7986 tablet_bootstrap.cc:492] T ce8bfc835792444e8703e73252b1f638 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 01:59:50.318658 7986 tablet_bootstrap.cc:654] T ce8bfc835792444e8703e73252b1f638 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:50.318917 7991 tablet_bootstrap.cc:492] T ce8bfc835792444e8703e73252b1f638 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 01:59:50.319041 7991 ts_tablet_manager.cc:1367] T ce8bfc835792444e8703e73252b1f638 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:50.319628 7991 raft_consensus.cc:348] T ce8bfc835792444e8703e73252b1f638 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:50.319710 7986 tablet_bootstrap.cc:492] T ce8bfc835792444e8703e73252b1f638 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 01:59:50.319723 7991 raft_consensus.cc:374] T ce8bfc835792444e8703e73252b1f638 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:50.319746 7991 raft_consensus.cc:729] T ce8bfc835792444e8703e73252b1f638 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 01:59:50.319783 7986 ts_tablet_manager.cc:1367] T ce8bfc835792444e8703e73252b1f638 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:50.319850 7991 consensus_queue.cc:260] T ce8bfc835792444e8703e73252b1f638 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:50.320111 7991 ts_tablet_manager.cc:1398] T ce8bfc835792444e8703e73252b1f638 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:50.320188 7986 raft_consensus.cc:348] T ce8bfc835792444e8703e73252b1f638 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:50.320281 7991 tablet_bootstrap.cc:492] T 548f43b381f1445c8d3d55705ae5a48a P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 01:59:50.320286 7986 raft_consensus.cc:374] T ce8bfc835792444e8703e73252b1f638 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:50.320314 7986 raft_consensus.cc:729] T ce8bfc835792444e8703e73252b1f638 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 01:59:50.320494 7986 consensus_queue.cc:260] T ce8bfc835792444e8703e73252b1f638 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:50.320767 7986 ts_tablet_manager.cc:1398] T ce8bfc835792444e8703e73252b1f638 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:50.320890 7986 tablet_bootstrap.cc:492] T 548f43b381f1445c8d3d55705ae5a48a P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 01:59:50.321261 7991 tablet_bootstrap.cc:654] T 548f43b381f1445c8d3d55705ae5a48a P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:50.321676 7986 tablet_bootstrap.cc:654] T 548f43b381f1445c8d3d55705ae5a48a P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:50.322671 7987 tablet_bootstrap.cc:492] T 7f637097ad06465cb93c6ff6d97c1d17 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 01:59:50.322784 7987 ts_tablet_manager.cc:1367] T 7f637097ad06465cb93c6ff6d97c1d17 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 01:59:50.323282 7987 raft_consensus.cc:348] T 7f637097ad06465cb93c6ff6d97c1d17 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:50.323421 7987 raft_consensus.cc:374] T 7f637097ad06465cb93c6ff6d97c1d17 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:50.323462 7987 raft_consensus.cc:729] T 7f637097ad06465cb93c6ff6d97c1d17 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 01:59:50.323598 7987 consensus_queue.cc:260] T 7f637097ad06465cb93c6ff6d97c1d17 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:50.323819 7991 tablet_bootstrap.cc:492] T 548f43b381f1445c8d3d55705ae5a48a P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 01:59:50.323891 7991 ts_tablet_manager.cc:1367] T 548f43b381f1445c8d3d55705ae5a48a P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:50.323895 7987 ts_tablet_manager.cc:1398] T 7f637097ad06465cb93c6ff6d97c1d17 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:50.324090 7987 tablet_bootstrap.cc:492] T 548f43b381f1445c8d3d55705ae5a48a P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 01:59:50.324271 7991 raft_consensus.cc:348] T 548f43b381f1445c8d3d55705ae5a48a P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:50.324353 7991 raft_consensus.cc:374] T 548f43b381f1445c8d3d55705ae5a48a P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:50.324410 7991 raft_consensus.cc:729] T 548f43b381f1445c8d3d55705ae5a48a P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 01:59:50.324546 7991 consensus_queue.cc:260] T 548f43b381f1445c8d3d55705ae5a48a P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:50.324833 7991 ts_tablet_manager.cc:1398] T 548f43b381f1445c8d3d55705ae5a48a P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:50.325073 7987 tablet_bootstrap.cc:654] T 548f43b381f1445c8d3d55705ae5a48a P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:50.325464 7986 tablet_bootstrap.cc:492] T 548f43b381f1445c8d3d55705ae5a48a P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 01:59:50.325590 7986 ts_tablet_manager.cc:1367] T 548f43b381f1445c8d3d55705ae5a48a P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 01:59:50.325664 7985 tablet_bootstrap.cc:492] T ce8bfc835792444e8703e73252b1f638 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 01:59:50.325762 7985 ts_tablet_manager.cc:1367] T ce8bfc835792444e8703e73252b1f638 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 01:59:50.326232 7985 raft_consensus.cc:348] T ce8bfc835792444e8703e73252b1f638 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:50.326229 7986 raft_consensus.cc:348] T 548f43b381f1445c8d3d55705ae5a48a P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:50.326339 7985 raft_consensus.cc:374] T ce8bfc835792444e8703e73252b1f638 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:50.326342 7986 raft_consensus.cc:374] T 548f43b381f1445c8d3d55705ae5a48a P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:50.326367 7985 raft_consensus.cc:729] T ce8bfc835792444e8703e73252b1f638 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 01:59:50.326375 7986 raft_consensus.cc:729] T 548f43b381f1445c8d3d55705ae5a48a P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 01:59:50.326493 7985 consensus_queue.cc:260] T ce8bfc835792444e8703e73252b1f638 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:50.326659 7986 consensus_queue.cc:260] T 548f43b381f1445c8d3d55705ae5a48a P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:50.326777 7985 ts_tablet_manager.cc:1398] T ce8bfc835792444e8703e73252b1f638 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:50.326963 7986 ts_tablet_manager.cc:1398] T 548f43b381f1445c8d3d55705ae5a48a P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:50.329007 7987 tablet_bootstrap.cc:492] T 548f43b381f1445c8d3d55705ae5a48a P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 01:59:50.329099 7987 ts_tablet_manager.cc:1367] T 548f43b381f1445c8d3d55705ae5a48a P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 01:59:50.329560 7987 raft_consensus.cc:348] T 548f43b381f1445c8d3d55705ae5a48a P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:50.329778 7987 raft_consensus.cc:374] T 548f43b381f1445c8d3d55705ae5a48a P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:50.329842 7987 raft_consensus.cc:729] T 548f43b381f1445c8d3d55705ae5a48a P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 01:59:50.329994 7987 consensus_queue.cc:260] T 548f43b381f1445c8d3d55705ae5a48a P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:50.330768 7987 ts_tablet_manager.cc:1398] T 548f43b381f1445c8d3d55705ae5a48a P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:50.362008 7725 raft_consensus.cc:482] T 548f43b381f1445c8d3d55705ae5a48a P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:50.362156 7725 raft_consensus.cc:504] T 548f43b381f1445c8d3d55705ae5a48a P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:50.362617 7725 leader_election.cc:288] T 548f43b381f1445c8d3d55705ae5a48a P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 01:59:50.363020 6457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "548f43b381f1445c8d3d55705ae5a48a" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 01:59:50.363029 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "548f43b381f1445c8d3d55705ae5a48a" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 01:59:50.363157 6457 raft_consensus.cc:2454] T 548f43b381f1445c8d3d55705ae5a48a P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 01:59:50.363173 6589 raft_consensus.cc:2454] T 548f43b381f1445c8d3d55705ae5a48a P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 01:59:50.363495 6786 leader_election.cc:302] T 548f43b381f1445c8d3d55705ae5a48a P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 80fd18ef4d604c86b909d5c008a46b4f; no voters:
I20240503 01:59:50.363718 7725 raft_consensus.cc:2781] T 548f43b381f1445c8d3d55705ae5a48a P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:50.363777 7725 raft_consensus.cc:482] T 548f43b381f1445c8d3d55705ae5a48a P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:50.363803 7725 raft_consensus.cc:3037] T 548f43b381f1445c8d3d55705ae5a48a P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:50.364624 7725 raft_consensus.cc:504] T 548f43b381f1445c8d3d55705ae5a48a P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:50.364988 7725 leader_election.cc:288] T 548f43b381f1445c8d3d55705ae5a48a P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Requested vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 01:59:50.365401 6457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "548f43b381f1445c8d3d55705ae5a48a" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 01:59:50.365435 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "548f43b381f1445c8d3d55705ae5a48a" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 01:59:50.365540 6589 raft_consensus.cc:3037] T 548f43b381f1445c8d3d55705ae5a48a P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:50.365536 6457 raft_consensus.cc:3037] T 548f43b381f1445c8d3d55705ae5a48a P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:50.366343 6457 raft_consensus.cc:2454] T 548f43b381f1445c8d3d55705ae5a48a P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 01:59:50.366334 6589 raft_consensus.cc:2454] T 548f43b381f1445c8d3d55705ae5a48a P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 01:59:50.366688 6784 leader_election.cc:302] T 548f43b381f1445c8d3d55705ae5a48a P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:50.366883 7725 raft_consensus.cc:2781] T 548f43b381f1445c8d3d55705ae5a48a P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:50.366981 7725 raft_consensus.cc:686] T 548f43b381f1445c8d3d55705ae5a48a P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Becoming Leader. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Running, Role: LEADER
I20240503 01:59:50.367143 7725 consensus_queue.cc:237] T 548f43b381f1445c8d3d55705ae5a48a P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:50.369170 6316 catalog_manager.cc:5484] T 548f43b381f1445c8d3d55705ae5a48a P 492448570cc94ed1879feb2ec3b9051d reported cstate change: term changed from 0 to 1, leader changed from <none> to 492448570cc94ed1879feb2ec3b9051d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:50.548425 7958 raft_consensus.cc:482] T ce8bfc835792444e8703e73252b1f638 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:50.548609 7958 raft_consensus.cc:504] T ce8bfc835792444e8703e73252b1f638 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:50.548629 7957 raft_consensus.cc:482] T 7f637097ad06465cb93c6ff6d97c1d17 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:50.548743 7957 raft_consensus.cc:504] T 7f637097ad06465cb93c6ff6d97c1d17 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:50.549041 7958 leader_election.cc:288] T ce8bfc835792444e8703e73252b1f638 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 01:59:50.549132 7957 leader_election.cc:288] T 7f637097ad06465cb93c6ff6d97c1d17 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 01:59:50.549554 6457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f637097ad06465cb93c6ff6d97c1d17" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 01:59:50.549554 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce8bfc835792444e8703e73252b1f638" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 01:59:50.549706 6457 raft_consensus.cc:2454] T 7f637097ad06465cb93c6ff6d97c1d17 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 01:59:50.549706 6589 raft_consensus.cc:2454] T ce8bfc835792444e8703e73252b1f638 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 01:59:50.549692 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce8bfc835792444e8703e73252b1f638" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 01:59:50.549821 6851 raft_consensus.cc:2454] T ce8bfc835792444e8703e73252b1f638 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 01:59:50.550045 6653 leader_election.cc:302] T 7f637097ad06465cb93c6ff6d97c1d17 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2ac2eb7cc264047b9ab32fe3a4fc025, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:50.550053 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f637097ad06465cb93c6ff6d97c1d17" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 01:59:50.550153 6589 raft_consensus.cc:2454] T 7f637097ad06465cb93c6ff6d97c1d17 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 01:59:50.550233 7957 raft_consensus.cc:2781] T 7f637097ad06465cb93c6ff6d97c1d17 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:50.550280 7957 raft_consensus.cc:482] T 7f637097ad06465cb93c6ff6d97c1d17 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:50.550314 7957 raft_consensus.cc:3037] T 7f637097ad06465cb93c6ff6d97c1d17 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:50.550402 6656 leader_election.cc:302] T ce8bfc835792444e8703e73252b1f638 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 01:59:50.550702 7958 raft_consensus.cc:2781] T ce8bfc835792444e8703e73252b1f638 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:50.550765 7958 raft_consensus.cc:482] T ce8bfc835792444e8703e73252b1f638 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:50.550797 7958 raft_consensus.cc:3037] T ce8bfc835792444e8703e73252b1f638 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:50.551287 7957 raft_consensus.cc:504] T 7f637097ad06465cb93c6ff6d97c1d17 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:50.551674 7957 leader_election.cc:288] T 7f637097ad06465cb93c6ff6d97c1d17 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Requested vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 01:59:50.551671 7958 raft_consensus.cc:504] T ce8bfc835792444e8703e73252b1f638 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:50.552085 6457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f637097ad06465cb93c6ff6d97c1d17" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 01:59:50.552136 7958 leader_election.cc:288] T ce8bfc835792444e8703e73252b1f638 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Requested vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 01:59:50.552102 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f637097ad06465cb93c6ff6d97c1d17" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 01:59:50.552211 6457 raft_consensus.cc:3037] T 7f637097ad06465cb93c6ff6d97c1d17 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:50.552225 6589 raft_consensus.cc:3037] T 7f637097ad06465cb93c6ff6d97c1d17 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:50.552798 6582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce8bfc835792444e8703e73252b1f638" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 01:59:50.552892 6582 raft_consensus.cc:3037] T ce8bfc835792444e8703e73252b1f638 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:50.553179 6457 raft_consensus.cc:2454] T 7f637097ad06465cb93c6ff6d97c1d17 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 01:59:50.553210 6589 raft_consensus.cc:2454] T 7f637097ad06465cb93c6ff6d97c1d17 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 01:59:50.553440 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce8bfc835792444e8703e73252b1f638" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 01:59:50.553536 6653 leader_election.cc:302] T 7f637097ad06465cb93c6ff6d97c1d17 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2ac2eb7cc264047b9ab32fe3a4fc025, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:50.553570 6851 raft_consensus.cc:3037] T ce8bfc835792444e8703e73252b1f638 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:50.553694 7958 raft_consensus.cc:2781] T 7f637097ad06465cb93c6ff6d97c1d17 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:50.553772 7958 raft_consensus.cc:686] T 7f637097ad06465cb93c6ff6d97c1d17 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Becoming Leader. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Running, Role: LEADER
I20240503 01:59:50.553804 6582 raft_consensus.cc:2454] T ce8bfc835792444e8703e73252b1f638 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 01:59:50.553946 7958 consensus_queue.cc:237] T 7f637097ad06465cb93c6ff6d97c1d17 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:50.554466 6851 raft_consensus.cc:2454] T ce8bfc835792444e8703e73252b1f638 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 01:59:50.555316 6656 leader_election.cc:302] T ce8bfc835792444e8703e73252b1f638 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 01:59:50.555532 7957 raft_consensus.cc:2781] T ce8bfc835792444e8703e73252b1f638 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:50.555608 7957 raft_consensus.cc:686] T ce8bfc835792444e8703e73252b1f638 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Becoming Leader. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Running, Role: LEADER
I20240503 01:59:50.555720 7957 consensus_queue.cc:237] T ce8bfc835792444e8703e73252b1f638 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:50.556077 6316 catalog_manager.cc:5484] T 7f637097ad06465cb93c6ff6d97c1d17 P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:50.559751 6316 catalog_manager.cc:5484] T ce8bfc835792444e8703e73252b1f638 P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } }
W20240503 01:59:50.733217 8001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:50.733508 8001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:50.736964 8001 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240503 01:59:50.845571 6851 raft_consensus.cc:1261] T ce8bfc835792444e8703e73252b1f638 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:50.845769 6589 raft_consensus.cc:1261] T 7f637097ad06465cb93c6ff6d97c1d17 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:50.845901 6582 raft_consensus.cc:1261] T ce8bfc835792444e8703e73252b1f638 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:50.846257 6457 raft_consensus.cc:1261] T 7f637097ad06465cb93c6ff6d97c1d17 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:50.846288 7957 consensus_queue.cc:1035] T 7f637097ad06465cb93c6ff6d97c1d17 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:50.846581 7957 consensus_queue.cc:1035] T ce8bfc835792444e8703e73252b1f638 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:50.846835 7957 consensus_queue.cc:1035] T ce8bfc835792444e8703e73252b1f638 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:50.847095 7957 consensus_queue.cc:1035] T 7f637097ad06465cb93c6ff6d97c1d17 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:50.859037 6588 raft_consensus.cc:1261] T 548f43b381f1445c8d3d55705ae5a48a P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:50.859182 6457 raft_consensus.cc:1261] T 548f43b381f1445c8d3d55705ae5a48a P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:50.859527 7725 consensus_queue.cc:1035] T 548f43b381f1445c8d3d55705ae5a48a P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:50.859846 7725 consensus_queue.cc:1035] T 548f43b381f1445c8d3d55705ae5a48a P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 127.184 ms
time per row: 0.0635921 ms
I20240503 01:59:50.915664 6313 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:50.922063 6438 tablet_service.cc:1467] Processing CreateTablet for tablet 46ad540deb624a25b97d3b967fe1f6f4 (DEFAULT_TABLE table=rebalance_test_table_9 [id=c65135643094446abc8eba15be23cd99]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:50.922063 6700 tablet_service.cc:1467] Processing CreateTablet for tablet 46ad540deb624a25b97d3b967fe1f6f4 (DEFAULT_TABLE table=rebalance_test_table_9 [id=c65135643094446abc8eba15be23cd99]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:50.922405 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46ad540deb624a25b97d3b967fe1f6f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:50.922475 6438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46ad540deb624a25b97d3b967fe1f6f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:50.922719 6569 tablet_service.cc:1467] Processing CreateTablet for tablet ac1632cc5eb94554b1faa7eb715706ef (DEFAULT_TABLE table=rebalance_test_table_9 [id=c65135643094446abc8eba15be23cd99]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:50.922993 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac1632cc5eb94554b1faa7eb715706ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:50.924353 8061 tablet_bootstrap.cc:492] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 01:59:50.924990 6438 tablet_service.cc:1467] Processing CreateTablet for tablet 8a76d3e2d4264687aa2972c179341b5e (DEFAULT_TABLE table=rebalance_test_table_9 [id=c65135643094446abc8eba15be23cd99]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:50.925237 6438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a76d3e2d4264687aa2972c179341b5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:50.925191 6436 tablet_service.cc:1467] Processing CreateTablet for tablet ac1632cc5eb94554b1faa7eb715706ef (DEFAULT_TABLE table=rebalance_test_table_9 [id=c65135643094446abc8eba15be23cd99]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:50.925397 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac1632cc5eb94554b1faa7eb715706ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:50.925534 8061 tablet_bootstrap.cc:654] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:50.926716 8061 tablet_bootstrap.cc:492] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 01:59:50.926875 8061 ts_tablet_manager.cc:1367] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 01:59:50.926968 6831 tablet_service.cc:1467] Processing CreateTablet for tablet 46ad540deb624a25b97d3b967fe1f6f4 (DEFAULT_TABLE table=rebalance_test_table_9 [id=c65135643094446abc8eba15be23cd99]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:50.927273 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46ad540deb624a25b97d3b967fe1f6f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:50.927675 6698 tablet_service.cc:1467] Processing CreateTablet for tablet ac1632cc5eb94554b1faa7eb715706ef (DEFAULT_TABLE table=rebalance_test_table_9 [id=c65135643094446abc8eba15be23cd99]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:50.927776 6830 tablet_service.cc:1467] Processing CreateTablet for tablet 8a76d3e2d4264687aa2972c179341b5e (DEFAULT_TABLE table=rebalance_test_table_9 [id=c65135643094446abc8eba15be23cd99]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:50.927901 6698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac1632cc5eb94554b1faa7eb715706ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:50.928011 6830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a76d3e2d4264687aa2972c179341b5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:50.928803 8061 raft_consensus.cc:348] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:50.928917 8061 raft_consensus.cc:374] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:50.928948 8061 raft_consensus.cc:729] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 01:59:50.929107 8061 consensus_queue.cc:260] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:50.930372 6700 tablet_service.cc:1467] Processing CreateTablet for tablet 8a76d3e2d4264687aa2972c179341b5e (DEFAULT_TABLE table=rebalance_test_table_9 [id=c65135643094446abc8eba15be23cd99]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:50.930650 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a76d3e2d4264687aa2972c179341b5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:50.931007 8065 tablet_bootstrap.cc:492] T 46ad540deb624a25b97d3b967fe1f6f4 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 01:59:50.931200 8061 ts_tablet_manager.cc:1398] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240503 01:59:50.931569 8061 tablet_bootstrap.cc:492] T ac1632cc5eb94554b1faa7eb715706ef P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 01:59:50.932147 8065 tablet_bootstrap.cc:654] T 46ad540deb624a25b97d3b967fe1f6f4 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:50.932740 8061 tablet_bootstrap.cc:654] T ac1632cc5eb94554b1faa7eb715706ef P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:50.932920 8064 tablet_bootstrap.cc:492] T 46ad540deb624a25b97d3b967fe1f6f4 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 01:59:50.933002 8062 tablet_bootstrap.cc:492] T ac1632cc5eb94554b1faa7eb715706ef P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 01:59:50.933944 8064 tablet_bootstrap.cc:654] T 46ad540deb624a25b97d3b967fe1f6f4 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:50.933986 8062 tablet_bootstrap.cc:654] T ac1632cc5eb94554b1faa7eb715706ef P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:50.934345 8061 tablet_bootstrap.cc:492] T ac1632cc5eb94554b1faa7eb715706ef P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 01:59:50.934441 8061 ts_tablet_manager.cc:1367] T ac1632cc5eb94554b1faa7eb715706ef P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 01:59:50.934854 8061 raft_consensus.cc:348] T ac1632cc5eb94554b1faa7eb715706ef P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:50.934976 8061 raft_consensus.cc:374] T ac1632cc5eb94554b1faa7eb715706ef P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:50.935010 8061 raft_consensus.cc:729] T ac1632cc5eb94554b1faa7eb715706ef P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 01:59:50.935133 8062 tablet_bootstrap.cc:492] T ac1632cc5eb94554b1faa7eb715706ef P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 01:59:50.935133 8065 tablet_bootstrap.cc:492] T 46ad540deb624a25b97d3b967fe1f6f4 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 01:59:50.935133 8061 consensus_queue.cc:260] T ac1632cc5eb94554b1faa7eb715706ef P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:50.935211 8062 ts_tablet_manager.cc:1367] T ac1632cc5eb94554b1faa7eb715706ef P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:50.935221 8065 ts_tablet_manager.cc:1367] T 46ad540deb624a25b97d3b967fe1f6f4 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 01:59:50.935238 8064 tablet_bootstrap.cc:492] T 46ad540deb624a25b97d3b967fe1f6f4 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 01:59:50.935307 8064 ts_tablet_manager.cc:1367] T 46ad540deb624a25b97d3b967fe1f6f4 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:50.935426 8061 ts_tablet_manager.cc:1398] T ac1632cc5eb94554b1faa7eb715706ef P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:50.935600 8061 tablet_bootstrap.cc:492] T 8a76d3e2d4264687aa2972c179341b5e P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 01:59:50.935650 8062 raft_consensus.cc:348] T ac1632cc5eb94554b1faa7eb715706ef P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:50.935669 8065 raft_consensus.cc:348] T 46ad540deb624a25b97d3b967fe1f6f4 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:50.935742 8062 raft_consensus.cc:374] T ac1632cc5eb94554b1faa7eb715706ef P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:50.935703 8064 raft_consensus.cc:348] T 46ad540deb624a25b97d3b967fe1f6f4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:50.935775 8062 raft_consensus.cc:729] T ac1632cc5eb94554b1faa7eb715706ef P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 01:59:50.935782 8065 raft_consensus.cc:374] T 46ad540deb624a25b97d3b967fe1f6f4 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:50.935788 8064 raft_consensus.cc:374] T 46ad540deb624a25b97d3b967fe1f6f4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:50.935817 8064 raft_consensus.cc:729] T 46ad540deb624a25b97d3b967fe1f6f4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 01:59:50.935814 8065 raft_consensus.cc:729] T 46ad540deb624a25b97d3b967fe1f6f4 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 01:59:50.935889 8062 consensus_queue.cc:260] T ac1632cc5eb94554b1faa7eb715706ef P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:50.935964 8065 consensus_queue.cc:260] T 46ad540deb624a25b97d3b967fe1f6f4 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:50.936178 8062 ts_tablet_manager.cc:1398] T ac1632cc5eb94554b1faa7eb715706ef P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:50.936300 8065 ts_tablet_manager.cc:1398] T 46ad540deb624a25b97d3b967fe1f6f4 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:50.936476 8065 tablet_bootstrap.cc:492] T 8a76d3e2d4264687aa2972c179341b5e P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 01:59:50.935964 8064 consensus_queue.cc:260] T 46ad540deb624a25b97d3b967fe1f6f4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:50.937014 8064 ts_tablet_manager.cc:1398] T 46ad540deb624a25b97d3b967fe1f6f4 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:50.937220 8064 tablet_bootstrap.cc:492] T ac1632cc5eb94554b1faa7eb715706ef P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 01:59:50.937860 8065 tablet_bootstrap.cc:654] T 8a76d3e2d4264687aa2972c179341b5e P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:50.938860 8064 tablet_bootstrap.cc:654] T ac1632cc5eb94554b1faa7eb715706ef P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:50.939172 8065 tablet_bootstrap.cc:492] T 8a76d3e2d4264687aa2972c179341b5e P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 01:59:50.939260 8065 ts_tablet_manager.cc:1367] T 8a76d3e2d4264687aa2972c179341b5e P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:50.939726 8065 raft_consensus.cc:348] T 8a76d3e2d4264687aa2972c179341b5e P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:50.939832 8065 raft_consensus.cc:374] T 8a76d3e2d4264687aa2972c179341b5e P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:50.939874 8065 raft_consensus.cc:729] T 8a76d3e2d4264687aa2972c179341b5e P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 01:59:50.939970 8064 tablet_bootstrap.cc:492] T ac1632cc5eb94554b1faa7eb715706ef P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 01:59:50.940042 8064 ts_tablet_manager.cc:1367] T ac1632cc5eb94554b1faa7eb715706ef P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:50.940021 8065 consensus_queue.cc:260] T 8a76d3e2d4264687aa2972c179341b5e P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:50.940304 8065 ts_tablet_manager.cc:1398] T 8a76d3e2d4264687aa2972c179341b5e P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:50.940500 8064 raft_consensus.cc:348] T ac1632cc5eb94554b1faa7eb715706ef P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:50.940601 8064 raft_consensus.cc:374] T ac1632cc5eb94554b1faa7eb715706ef P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:50.940634 8064 raft_consensus.cc:729] T ac1632cc5eb94554b1faa7eb715706ef P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 01:59:50.940737 8064 consensus_queue.cc:260] T ac1632cc5eb94554b1faa7eb715706ef P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:50.940990 8064 ts_tablet_manager.cc:1398] T ac1632cc5eb94554b1faa7eb715706ef P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:50.941176 8064 tablet_bootstrap.cc:492] T 8a76d3e2d4264687aa2972c179341b5e P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 01:59:50.942075 8061 tablet_bootstrap.cc:654] T 8a76d3e2d4264687aa2972c179341b5e P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:50.942093 8064 tablet_bootstrap.cc:654] T 8a76d3e2d4264687aa2972c179341b5e P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:50.943233 8064 tablet_bootstrap.cc:492] T 8a76d3e2d4264687aa2972c179341b5e P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 01:59:50.943316 8064 ts_tablet_manager.cc:1367] T 8a76d3e2d4264687aa2972c179341b5e P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:50.943675 8064 raft_consensus.cc:348] T 8a76d3e2d4264687aa2972c179341b5e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:50.943806 8064 raft_consensus.cc:374] T 8a76d3e2d4264687aa2972c179341b5e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:50.943847 8064 raft_consensus.cc:729] T 8a76d3e2d4264687aa2972c179341b5e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 01:59:50.943990 8064 consensus_queue.cc:260] T 8a76d3e2d4264687aa2972c179341b5e P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:50.944272 8064 ts_tablet_manager.cc:1398] T 8a76d3e2d4264687aa2972c179341b5e P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:50.945128 8061 tablet_bootstrap.cc:492] T 8a76d3e2d4264687aa2972c179341b5e P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 01:59:50.945235 8061 ts_tablet_manager.cc:1367] T 8a76d3e2d4264687aa2972c179341b5e P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 01:59:50.945696 8061 raft_consensus.cc:348] T 8a76d3e2d4264687aa2972c179341b5e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:50.945849 8061 raft_consensus.cc:374] T 8a76d3e2d4264687aa2972c179341b5e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:50.945890 8061 raft_consensus.cc:729] T 8a76d3e2d4264687aa2972c179341b5e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 01:59:50.946038 8061 consensus_queue.cc:260] T 8a76d3e2d4264687aa2972c179341b5e P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:50.946313 8061 ts_tablet_manager.cc:1398] T 8a76d3e2d4264687aa2972c179341b5e P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:50.965394 7959 raft_consensus.cc:482] T ac1632cc5eb94554b1faa7eb715706ef P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:50.965538 7959 raft_consensus.cc:504] T ac1632cc5eb94554b1faa7eb715706ef P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:50.965961 7959 leader_election.cc:288] T ac1632cc5eb94554b1faa7eb715706ef P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 01:59:50.966444 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac1632cc5eb94554b1faa7eb715706ef" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 01:59:50.966444 6718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac1632cc5eb94554b1faa7eb715706ef" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 01:59:50.966583 6589 raft_consensus.cc:2454] T ac1632cc5eb94554b1faa7eb715706ef P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 01:59:50.966605 6718 raft_consensus.cc:2454] T ac1632cc5eb94554b1faa7eb715706ef P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 01:59:50.966882 6393 leader_election.cc:302] T ac1632cc5eb94554b1faa7eb715706ef P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:50.967056 7959 raft_consensus.cc:2781] T ac1632cc5eb94554b1faa7eb715706ef P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:50.967116 7959 raft_consensus.cc:482] T ac1632cc5eb94554b1faa7eb715706ef P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:50.967175 7959 raft_consensus.cc:3037] T ac1632cc5eb94554b1faa7eb715706ef P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:50.968348 7959 raft_consensus.cc:504] T ac1632cc5eb94554b1faa7eb715706ef P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:50.968752 7959 leader_election.cc:288] T ac1632cc5eb94554b1faa7eb715706ef P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Requested vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 01:59:50.969190 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac1632cc5eb94554b1faa7eb715706ef" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 01:59:50.969305 6589 raft_consensus.cc:3037] T ac1632cc5eb94554b1faa7eb715706ef P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:50.970254 6589 raft_consensus.cc:2454] T ac1632cc5eb94554b1faa7eb715706ef P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 01:59:50.970590 6393 leader_election.cc:302] T ac1632cc5eb94554b1faa7eb715706ef P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:50.970568 6718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac1632cc5eb94554b1faa7eb715706ef" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 01:59:50.970706 6718 raft_consensus.cc:3037] T ac1632cc5eb94554b1faa7eb715706ef P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:50.970799 7959 raft_consensus.cc:2781] T ac1632cc5eb94554b1faa7eb715706ef P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:50.971012 7959 raft_consensus.cc:686] T ac1632cc5eb94554b1faa7eb715706ef P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Becoming Leader. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Running, Role: LEADER
I20240503 01:59:50.971172 7959 consensus_queue.cc:237] T ac1632cc5eb94554b1faa7eb715706ef P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:50.971665 6718 raft_consensus.cc:2454] T ac1632cc5eb94554b1faa7eb715706ef P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 01:59:50.973134 6307 catalog_manager.cc:5484] T ac1632cc5eb94554b1faa7eb715706ef P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: term changed from 0 to 1, leader changed from <none> to dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129). New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:51.020471 7959 raft_consensus.cc:482] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:51.020630 7959 raft_consensus.cc:504] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:51.021050 7959 leader_election.cc:288] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 01:59:51.021538 6718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46ad540deb624a25b97d3b967fe1f6f4" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 01:59:51.021538 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46ad540deb624a25b97d3b967fe1f6f4" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 01:59:51.021685 6718 raft_consensus.cc:2454] T 46ad540deb624a25b97d3b967fe1f6f4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 01:59:51.021699 6851 raft_consensus.cc:2454] T 46ad540deb624a25b97d3b967fe1f6f4 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 01:59:51.021991 6394 leader_election.cc:302] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:51.022150 7959 raft_consensus.cc:2781] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:51.022209 7959 raft_consensus.cc:482] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:51.022244 7959 raft_consensus.cc:3037] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:51.023078 7959 raft_consensus.cc:504] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:51.023429 7959 leader_election.cc:288] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Requested vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 01:59:51.023792 6718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46ad540deb624a25b97d3b967fe1f6f4" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 01:59:51.023861 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46ad540deb624a25b97d3b967fe1f6f4" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 01:59:51.023943 6718 raft_consensus.cc:3037] T 46ad540deb624a25b97d3b967fe1f6f4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:51.023991 6851 raft_consensus.cc:3037] T 46ad540deb624a25b97d3b967fe1f6f4 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:51.024940 6851 raft_consensus.cc:2454] T 46ad540deb624a25b97d3b967fe1f6f4 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 01:59:51.024940 6718 raft_consensus.cc:2454] T 46ad540deb624a25b97d3b967fe1f6f4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 01:59:51.025266 6394 leader_election.cc:302] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2ac2eb7cc264047b9ab32fe3a4fc025, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:51.025444 7959 raft_consensus.cc:2781] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:51.025545 7959 raft_consensus.cc:686] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Becoming Leader. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Running, Role: LEADER
I20240503 01:59:51.025688 7959 consensus_queue.cc:237] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:51.027662 6307 catalog_manager.cc:5484] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: term changed from 0 to 1, leader changed from <none> to dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129). New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:51.049118 7959 raft_consensus.cc:482] T 8a76d3e2d4264687aa2972c179341b5e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:51.049268 7959 raft_consensus.cc:504] T 8a76d3e2d4264687aa2972c179341b5e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:51.049644 7959 leader_election.cc:288] T 8a76d3e2d4264687aa2972c179341b5e P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 01:59:51.050215 6718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a76d3e2d4264687aa2972c179341b5e" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 01:59:51.050215 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a76d3e2d4264687aa2972c179341b5e" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 01:59:51.050396 6851 raft_consensus.cc:2454] T 8a76d3e2d4264687aa2972c179341b5e P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 01:59:51.050396 6718 raft_consensus.cc:2454] T 8a76d3e2d4264687aa2972c179341b5e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 01:59:51.050702 6394 leader_election.cc:302] T 8a76d3e2d4264687aa2972c179341b5e P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:51.050940 7959 raft_consensus.cc:2781] T 8a76d3e2d4264687aa2972c179341b5e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:51.051005 7959 raft_consensus.cc:482] T 8a76d3e2d4264687aa2972c179341b5e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:51.051061 7959 raft_consensus.cc:3037] T 8a76d3e2d4264687aa2972c179341b5e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:51.051905 7959 raft_consensus.cc:504] T 8a76d3e2d4264687aa2972c179341b5e P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:51.052307 7959 leader_election.cc:288] T 8a76d3e2d4264687aa2972c179341b5e P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Requested vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 01:59:51.052805 6718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a76d3e2d4264687aa2972c179341b5e" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 01:59:51.052930 6718 raft_consensus.cc:3037] T 8a76d3e2d4264687aa2972c179341b5e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:51.052887 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a76d3e2d4264687aa2972c179341b5e" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 01:59:51.053047 6851 raft_consensus.cc:3037] T 8a76d3e2d4264687aa2972c179341b5e P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:51.053786 6718 raft_consensus.cc:2454] T 8a76d3e2d4264687aa2972c179341b5e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 01:59:51.053793 6851 raft_consensus.cc:2454] T 8a76d3e2d4264687aa2972c179341b5e P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 01:59:51.054143 6394 leader_election.cc:302] T 8a76d3e2d4264687aa2972c179341b5e P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:51.054386 7959 raft_consensus.cc:2781] T 8a76d3e2d4264687aa2972c179341b5e P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:51.054477 7959 raft_consensus.cc:686] T 8a76d3e2d4264687aa2972c179341b5e P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Becoming Leader. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Running, Role: LEADER
I20240503 01:59:51.054627 7959 consensus_queue.cc:237] T 8a76d3e2d4264687aa2972c179341b5e P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:51.057451 6307 catalog_manager.cc:5484] T 8a76d3e2d4264687aa2972c179341b5e P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: term changed from 0 to 1, leader changed from <none> to dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129). New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } }
W20240503 01:59:51.204238 8078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:51.204645 8078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:51.209337 8078 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240503 01:59:51.306880 6589 raft_consensus.cc:1261] T ac1632cc5eb94554b1faa7eb715706ef P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:51.307415 8074 consensus_queue.cc:1035] T ac1632cc5eb94554b1faa7eb715706ef P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:51.312361 6718 raft_consensus.cc:1261] T ac1632cc5eb94554b1faa7eb715706ef P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:51.315378 6851 raft_consensus.cc:1261] T 46ad540deb624a25b97d3b967fe1f6f4 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:51.316296 8109 consensus_queue.cc:1035] T ac1632cc5eb94554b1faa7eb715706ef P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:51.316690 7959 consensus_queue.cc:1035] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:51.316797 6718 raft_consensus.cc:1261] T 46ad540deb624a25b97d3b967fe1f6f4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:51.318661 6851 raft_consensus.cc:1261] T 8a76d3e2d4264687aa2972c179341b5e P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:51.320109 6718 raft_consensus.cc:1261] T 8a76d3e2d4264687aa2972c179341b5e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:51.322047 8107 consensus_queue.cc:1035] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:51.325860 8074 consensus_queue.cc:1035] T 8a76d3e2d4264687aa2972c179341b5e P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:51.326258 8074 consensus_queue.cc:1035] T 8a76d3e2d4264687aa2972c179341b5e P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 117.961 ms
time per row: 0.0589805 ms
I20240503 01:59:51.383414 6313 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalance_test_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:51.390094 6438 tablet_service.cc:1467] Processing CreateTablet for tablet 7f61f9a345d94a00a0929d8ebd197c23 (DEFAULT_TABLE table=rebalance_test_table_10 [id=14361c7197694d579466ed5d4f2922f7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:51.390411 6436 tablet_service.cc:1467] Processing CreateTablet for tablet 20b9d73abd6a424193fc102180f1dd4b (DEFAULT_TABLE table=rebalance_test_table_10 [id=14361c7197694d579466ed5d4f2922f7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:51.390422 6438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f61f9a345d94a00a0929d8ebd197c23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:51.391086 6700 tablet_service.cc:1467] Processing CreateTablet for tablet 1832eb75b09d4823ad07c56868f3aede (DEFAULT_TABLE table=rebalance_test_table_10 [id=14361c7197694d579466ed5d4f2922f7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:51.391345 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1832eb75b09d4823ad07c56868f3aede. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:51.392685 6830 tablet_service.cc:1467] Processing CreateTablet for tablet 7f61f9a345d94a00a0929d8ebd197c23 (DEFAULT_TABLE table=rebalance_test_table_10 [id=14361c7197694d579466ed5d4f2922f7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:51.392848 8061 tablet_bootstrap.cc:492] T 7f61f9a345d94a00a0929d8ebd197c23 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 01:59:51.393033 6830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f61f9a345d94a00a0929d8ebd197c23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:51.393069 6831 tablet_service.cc:1467] Processing CreateTablet for tablet 1832eb75b09d4823ad07c56868f3aede (DEFAULT_TABLE table=rebalance_test_table_10 [id=14361c7197694d579466ed5d4f2922f7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:51.393265 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1832eb75b09d4823ad07c56868f3aede. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:51.393780 8061 tablet_bootstrap.cc:654] T 7f61f9a345d94a00a0929d8ebd197c23 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:51.394019 6569 tablet_service.cc:1467] Processing CreateTablet for tablet 7f61f9a345d94a00a0929d8ebd197c23 (DEFAULT_TABLE table=rebalance_test_table_10 [id=14361c7197694d579466ed5d4f2922f7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:51.394282 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f61f9a345d94a00a0929d8ebd197c23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:51.394565 6568 tablet_service.cc:1467] Processing CreateTablet for tablet 20b9d73abd6a424193fc102180f1dd4b (DEFAULT_TABLE table=rebalance_test_table_10 [id=14361c7197694d579466ed5d4f2922f7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:51.394811 6568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20b9d73abd6a424193fc102180f1dd4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:51.395293 6567 tablet_service.cc:1467] Processing CreateTablet for tablet 1832eb75b09d4823ad07c56868f3aede (DEFAULT_TABLE table=rebalance_test_table_10 [id=14361c7197694d579466ed5d4f2922f7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:51.395494 6829 tablet_service.cc:1467] Processing CreateTablet for tablet 20b9d73abd6a424193fc102180f1dd4b (DEFAULT_TABLE table=rebalance_test_table_10 [id=14361c7197694d579466ed5d4f2922f7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:51.395589 8065 tablet_bootstrap.cc:492] T 1832eb75b09d4823ad07c56868f3aede P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 01:59:51.396247 6829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20b9d73abd6a424193fc102180f1dd4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:51.396486 8065 tablet_bootstrap.cc:654] T 1832eb75b09d4823ad07c56868f3aede P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:51.396538 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20b9d73abd6a424193fc102180f1dd4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:51.397205 6567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1832eb75b09d4823ad07c56868f3aede. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:51.399418 8062 tablet_bootstrap.cc:492] T 1832eb75b09d4823ad07c56868f3aede P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 01:59:51.400321 8062 tablet_bootstrap.cc:654] T 1832eb75b09d4823ad07c56868f3aede P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:51.401468 8062 tablet_bootstrap.cc:492] T 1832eb75b09d4823ad07c56868f3aede P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 01:59:51.401561 8062 ts_tablet_manager.cc:1367] T 1832eb75b09d4823ad07c56868f3aede P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:51.402027 8062 raft_consensus.cc:348] T 1832eb75b09d4823ad07c56868f3aede P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:51.402144 8062 raft_consensus.cc:374] T 1832eb75b09d4823ad07c56868f3aede P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:51.402174 8062 raft_consensus.cc:729] T 1832eb75b09d4823ad07c56868f3aede P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 01:59:51.402295 8062 consensus_queue.cc:260] T 1832eb75b09d4823ad07c56868f3aede P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:51.402587 8062 ts_tablet_manager.cc:1398] T 1832eb75b09d4823ad07c56868f3aede P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:51.402763 8062 tablet_bootstrap.cc:492] T 20b9d73abd6a424193fc102180f1dd4b P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 01:59:51.402760 8061 tablet_bootstrap.cc:492] T 7f61f9a345d94a00a0929d8ebd197c23 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 01:59:51.402859 8061 ts_tablet_manager.cc:1367] T 7f61f9a345d94a00a0929d8ebd197c23 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 01:59:51.403499 8061 raft_consensus.cc:348] T 7f61f9a345d94a00a0929d8ebd197c23 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:51.403616 8061 raft_consensus.cc:374] T 7f61f9a345d94a00a0929d8ebd197c23 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:51.403648 8061 raft_consensus.cc:729] T 7f61f9a345d94a00a0929d8ebd197c23 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 01:59:51.403766 8061 consensus_queue.cc:260] T 7f61f9a345d94a00a0929d8ebd197c23 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:51.403853 8062 tablet_bootstrap.cc:654] T 20b9d73abd6a424193fc102180f1dd4b P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:51.404043 8061 ts_tablet_manager.cc:1398] T 7f61f9a345d94a00a0929d8ebd197c23 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:51.404232 8061 tablet_bootstrap.cc:492] T 20b9d73abd6a424193fc102180f1dd4b P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 01:59:51.405369 8061 tablet_bootstrap.cc:654] T 20b9d73abd6a424193fc102180f1dd4b P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:51.407065 8061 tablet_bootstrap.cc:492] T 20b9d73abd6a424193fc102180f1dd4b P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 01:59:51.407261 8061 ts_tablet_manager.cc:1367] T 20b9d73abd6a424193fc102180f1dd4b P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240503 01:59:51.408099 8061 raft_consensus.cc:348] T 20b9d73abd6a424193fc102180f1dd4b P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:51.408309 8061 raft_consensus.cc:374] T 20b9d73abd6a424193fc102180f1dd4b P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:51.408366 8061 raft_consensus.cc:729] T 20b9d73abd6a424193fc102180f1dd4b P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 01:59:51.408576 8061 consensus_queue.cc:260] T 20b9d73abd6a424193fc102180f1dd4b P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:51.409037 8061 ts_tablet_manager.cc:1398] T 20b9d73abd6a424193fc102180f1dd4b P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 01:59:51.409497 8035 raft_consensus.cc:482] T 1832eb75b09d4823ad07c56868f3aede P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:51.409606 8035 raft_consensus.cc:504] T 1832eb75b09d4823ad07c56868f3aede P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:51.410080 8064 tablet_bootstrap.cc:492] T 1832eb75b09d4823ad07c56868f3aede P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 01:59:51.410827 8035 leader_election.cc:288] T 1832eb75b09d4823ad07c56868f3aede P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 01:59:51.411686 8064 tablet_bootstrap.cc:654] T 1832eb75b09d4823ad07c56868f3aede P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:51.411957 8062 tablet_bootstrap.cc:492] T 20b9d73abd6a424193fc102180f1dd4b P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 01:59:51.412042 8062 ts_tablet_manager.cc:1367] T 20b9d73abd6a424193fc102180f1dd4b P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20240503 01:59:51.412060 8065 tablet_bootstrap.cc:492] T 1832eb75b09d4823ad07c56868f3aede P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 01:59:51.412158 8065 ts_tablet_manager.cc:1367] T 1832eb75b09d4823ad07c56868f3aede P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.001s
I20240503 01:59:51.412569 8062 raft_consensus.cc:348] T 20b9d73abd6a424193fc102180f1dd4b P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:51.412683 8062 raft_consensus.cc:374] T 20b9d73abd6a424193fc102180f1dd4b P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:51.412695 8064 tablet_bootstrap.cc:492] T 1832eb75b09d4823ad07c56868f3aede P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 01:59:51.412712 8062 raft_consensus.cc:729] T 20b9d73abd6a424193fc102180f1dd4b P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 01:59:51.412667 8065 raft_consensus.cc:348] T 1832eb75b09d4823ad07c56868f3aede P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:51.412772 8064 ts_tablet_manager.cc:1367] T 1832eb75b09d4823ad07c56868f3aede P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:51.412782 8065 raft_consensus.cc:374] T 1832eb75b09d4823ad07c56868f3aede P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:51.412812 8065 raft_consensus.cc:729] T 1832eb75b09d4823ad07c56868f3aede P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 01:59:51.412822 8062 consensus_queue.cc:260] T 20b9d73abd6a424193fc102180f1dd4b P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:51.412928 8065 consensus_queue.cc:260] T 1832eb75b09d4823ad07c56868f3aede P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:51.413141 8062 ts_tablet_manager.cc:1398] T 20b9d73abd6a424193fc102180f1dd4b P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:51.413167 8064 raft_consensus.cc:348] T 1832eb75b09d4823ad07c56868f3aede P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:51.413244 8065 ts_tablet_manager.cc:1398] T 1832eb75b09d4823ad07c56868f3aede P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:51.413257 8064 raft_consensus.cc:374] T 1832eb75b09d4823ad07c56868f3aede P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:51.413285 8064 raft_consensus.cc:729] T 1832eb75b09d4823ad07c56868f3aede P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 01:59:51.413450 8065 tablet_bootstrap.cc:492] T 20b9d73abd6a424193fc102180f1dd4b P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 01:59:51.413405 8064 consensus_queue.cc:260] T 1832eb75b09d4823ad07c56868f3aede P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:51.413677 8064 ts_tablet_manager.cc:1398] T 1832eb75b09d4823ad07c56868f3aede P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:51.413323 8062 tablet_bootstrap.cc:492] T 7f61f9a345d94a00a0929d8ebd197c23 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 01:59:51.414031 8108 raft_consensus.cc:482] T 7f61f9a345d94a00a0929d8ebd197c23 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:51.414139 8108 raft_consensus.cc:504] T 7f61f9a345d94a00a0929d8ebd197c23 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:51.414328 8065 tablet_bootstrap.cc:654] T 20b9d73abd6a424193fc102180f1dd4b P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:51.414577 8108 leader_election.cc:288] T 7f61f9a345d94a00a0929d8ebd197c23 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 01:59:51.415071 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f61f9a345d94a00a0929d8ebd197c23" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 01:59:51.415156 8062 tablet_bootstrap.cc:654] T 7f61f9a345d94a00a0929d8ebd197c23 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
W20240503 01:59:51.415596 6394 leader_election.cc:341] T 7f61f9a345d94a00a0929d8ebd197c23 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719): Illegal state: must be running to vote when last-logged opid is not known
I20240503 01:59:51.416643 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f61f9a345d94a00a0929d8ebd197c23" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
W20240503 01:59:51.417208 6393 leader_election.cc:341] T 7f61f9a345d94a00a0929d8ebd197c23 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): Illegal state: must be running to vote when last-logged opid is not known
I20240503 01:59:51.417282 6393 leader_election.cc:302] T 7f61f9a345d94a00a0929d8ebd197c23 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dec3e4a04d604f7eaaa3f077d081218a; no voters: 492448570cc94ed1879feb2ec3b9051d, 80fd18ef4d604c86b909d5c008a46b4f
I20240503 01:59:51.417430 8108 raft_consensus.cc:2726] T 7f61f9a345d94a00a0929d8ebd197c23 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 01:59:51.417709 8062 tablet_bootstrap.cc:492] T 7f61f9a345d94a00a0929d8ebd197c23 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 01:59:51.417801 8062 ts_tablet_manager.cc:1367] T 7f61f9a345d94a00a0929d8ebd197c23 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240503 01:59:51.418331 8062 raft_consensus.cc:348] T 7f61f9a345d94a00a0929d8ebd197c23 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:51.418430 8062 raft_consensus.cc:374] T 7f61f9a345d94a00a0929d8ebd197c23 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:51.418460 8062 raft_consensus.cc:729] T 7f61f9a345d94a00a0929d8ebd197c23 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 01:59:51.419127 6720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1832eb75b09d4823ad07c56868f3aede" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 01:59:51.419272 6720 raft_consensus.cc:2454] T 1832eb75b09d4823ad07c56868f3aede P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 01:59:51.419373 8062 consensus_queue.cc:260] T 7f61f9a345d94a00a0929d8ebd197c23 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:51.419652 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1832eb75b09d4823ad07c56868f3aede" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 01:59:51.419780 6849 raft_consensus.cc:2454] T 1832eb75b09d4823ad07c56868f3aede P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 01:59:51.420322 6525 leader_election.cc:302] T 1832eb75b09d4823ad07c56868f3aede P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 01:59:51.420655 8035 raft_consensus.cc:2781] T 1832eb75b09d4823ad07c56868f3aede P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:51.420706 8035 raft_consensus.cc:482] T 1832eb75b09d4823ad07c56868f3aede P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:51.420732 8035 raft_consensus.cc:3037] T 1832eb75b09d4823ad07c56868f3aede P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:51.420900 8062 ts_tablet_manager.cc:1398] T 7f61f9a345d94a00a0929d8ebd197c23 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 01:59:51.421053 8065 tablet_bootstrap.cc:492] T 20b9d73abd6a424193fc102180f1dd4b P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 01:59:51.421151 8065 ts_tablet_manager.cc:1367] T 20b9d73abd6a424193fc102180f1dd4b P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.001s
I20240503 01:59:51.421722 8035 raft_consensus.cc:504] T 1832eb75b09d4823ad07c56868f3aede P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:51.421788 8065 raft_consensus.cc:348] T 20b9d73abd6a424193fc102180f1dd4b P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:51.421933 8065 raft_consensus.cc:374] T 20b9d73abd6a424193fc102180f1dd4b P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:51.421967 8065 raft_consensus.cc:729] T 20b9d73abd6a424193fc102180f1dd4b P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 01:59:51.422111 8035 leader_election.cc:288] T 1832eb75b09d4823ad07c56868f3aede P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Requested vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 01:59:51.422083 8065 consensus_queue.cc:260] T 20b9d73abd6a424193fc102180f1dd4b P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:51.422868 8065 ts_tablet_manager.cc:1398] T 20b9d73abd6a424193fc102180f1dd4b P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 01:59:51.422863 6720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1832eb75b09d4823ad07c56868f3aede" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 01:59:51.422977 6720 raft_consensus.cc:3037] T 1832eb75b09d4823ad07c56868f3aede P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:51.423163 8065 tablet_bootstrap.cc:492] T 7f61f9a345d94a00a0929d8ebd197c23 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 01:59:51.423398 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1832eb75b09d4823ad07c56868f3aede" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 01:59:51.423493 6849 raft_consensus.cc:3037] T 1832eb75b09d4823ad07c56868f3aede P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:51.423986 6720 raft_consensus.cc:2454] T 1832eb75b09d4823ad07c56868f3aede P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 01:59:51.424160 8065 tablet_bootstrap.cc:654] T 7f61f9a345d94a00a0929d8ebd197c23 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:51.424508 6849 raft_consensus.cc:2454] T 1832eb75b09d4823ad07c56868f3aede P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 01:59:51.424686 6525 leader_election.cc:302] T 1832eb75b09d4823ad07c56868f3aede P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 01:59:51.424860 8035 raft_consensus.cc:2781] T 1832eb75b09d4823ad07c56868f3aede P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:51.425026 8035 raft_consensus.cc:686] T 1832eb75b09d4823ad07c56868f3aede P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Becoming Leader. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Running, Role: LEADER
I20240503 01:59:51.425172 8035 consensus_queue.cc:237] T 1832eb75b09d4823ad07c56868f3aede P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:51.427245 6313 catalog_manager.cc:5484] T 1832eb75b09d4823ad07c56868f3aede P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:51.427531 8065 tablet_bootstrap.cc:492] T 7f61f9a345d94a00a0929d8ebd197c23 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 01:59:51.427618 8065 ts_tablet_manager.cc:1367] T 7f61f9a345d94a00a0929d8ebd197c23 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:51.428371 8065 raft_consensus.cc:348] T 7f61f9a345d94a00a0929d8ebd197c23 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:51.428519 8065 raft_consensus.cc:374] T 7f61f9a345d94a00a0929d8ebd197c23 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:51.428551 8065 raft_consensus.cc:729] T 7f61f9a345d94a00a0929d8ebd197c23 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 01:59:51.428659 8065 consensus_queue.cc:260] T 7f61f9a345d94a00a0929d8ebd197c23 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:51.430055 8065 ts_tablet_manager.cc:1398] T 7f61f9a345d94a00a0929d8ebd197c23 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:51.432365 7905 raft_consensus.cc:482] T 20b9d73abd6a424193fc102180f1dd4b P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:51.432523 7905 raft_consensus.cc:504] T 20b9d73abd6a424193fc102180f1dd4b P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:51.432934 7905 leader_election.cc:288] T 20b9d73abd6a424193fc102180f1dd4b P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 01:59:51.433359 6457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20b9d73abd6a424193fc102180f1dd4b" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 01:59:51.433496 6457 raft_consensus.cc:2454] T 20b9d73abd6a424193fc102180f1dd4b P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 01:59:51.433775 6784 leader_election.cc:302] T 20b9d73abd6a424193fc102180f1dd4b P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:51.433959 7905 raft_consensus.cc:2781] T 20b9d73abd6a424193fc102180f1dd4b P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:51.434051 7905 raft_consensus.cc:482] T 20b9d73abd6a424193fc102180f1dd4b P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:51.434093 7905 raft_consensus.cc:3037] T 20b9d73abd6a424193fc102180f1dd4b P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:51.435214 7905 raft_consensus.cc:504] T 20b9d73abd6a424193fc102180f1dd4b P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:51.435577 7905 leader_election.cc:288] T 20b9d73abd6a424193fc102180f1dd4b P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Requested vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 01:59:51.435969 6457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20b9d73abd6a424193fc102180f1dd4b" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 01:59:51.436107 6457 raft_consensus.cc:3037] T 20b9d73abd6a424193fc102180f1dd4b P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:51.436125 6588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20b9d73abd6a424193fc102180f1dd4b" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 01:59:51.436259 6588 raft_consensus.cc:3037] T 20b9d73abd6a424193fc102180f1dd4b P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:51.437074 6457 raft_consensus.cc:2454] T 20b9d73abd6a424193fc102180f1dd4b P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 01:59:51.437150 6588 raft_consensus.cc:2454] T 20b9d73abd6a424193fc102180f1dd4b P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 01:59:51.437368 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20b9d73abd6a424193fc102180f1dd4b" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 01:59:51.437528 6786 leader_election.cc:302] T 20b9d73abd6a424193fc102180f1dd4b P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 80fd18ef4d604c86b909d5c008a46b4f; no voters:
I20240503 01:59:51.437577 6589 raft_consensus.cc:2362] T 20b9d73abd6a424193fc102180f1dd4b P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1. Re-sending same reply.
I20240503 01:59:51.437968 7905 raft_consensus.cc:2781] T 20b9d73abd6a424193fc102180f1dd4b P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:51.438119 7905 raft_consensus.cc:686] T 20b9d73abd6a424193fc102180f1dd4b P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Becoming Leader. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Running, Role: LEADER
I20240503 01:59:51.438261 7905 consensus_queue.cc:237] T 20b9d73abd6a424193fc102180f1dd4b P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:51.440405 6308 catalog_manager.cc:5484] T 20b9d73abd6a424193fc102180f1dd4b P 492448570cc94ed1879feb2ec3b9051d reported cstate change: term changed from 0 to 1, leader changed from <none> to 492448570cc94ed1879feb2ec3b9051d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:51.635865 7905 raft_consensus.cc:482] T 7f61f9a345d94a00a0929d8ebd197c23 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:51.636003 7905 raft_consensus.cc:504] T 7f61f9a345d94a00a0929d8ebd197c23 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:51.636493 7905 leader_election.cc:288] T 7f61f9a345d94a00a0929d8ebd197c23 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 01:59:51.636883 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f61f9a345d94a00a0929d8ebd197c23" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 01:59:51.636880 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f61f9a345d94a00a0929d8ebd197c23" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 01:59:51.637032 6589 raft_consensus.cc:2454] T 7f61f9a345d94a00a0929d8ebd197c23 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 01:59:51.637032 6458 raft_consensus.cc:2454] T 7f61f9a345d94a00a0929d8ebd197c23 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 01:59:51.637334 6786 leader_election.cc:302] T 7f61f9a345d94a00a0929d8ebd197c23 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 80fd18ef4d604c86b909d5c008a46b4f; no voters:
I20240503 01:59:51.637513 7905 raft_consensus.cc:2781] T 7f61f9a345d94a00a0929d8ebd197c23 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:51.637566 7905 raft_consensus.cc:482] T 7f61f9a345d94a00a0929d8ebd197c23 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:51.637594 7905 raft_consensus.cc:3037] T 7f61f9a345d94a00a0929d8ebd197c23 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:51.638447 7905 raft_consensus.cc:504] T 7f61f9a345d94a00a0929d8ebd197c23 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:51.638821 7905 leader_election.cc:288] T 7f61f9a345d94a00a0929d8ebd197c23 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Requested vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 01:59:51.639236 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f61f9a345d94a00a0929d8ebd197c23" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 01:59:51.639364 6458 raft_consensus.cc:3037] T 7f61f9a345d94a00a0929d8ebd197c23 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:51.639348 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f61f9a345d94a00a0929d8ebd197c23" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 01:59:51.639468 6589 raft_consensus.cc:3037] T 7f61f9a345d94a00a0929d8ebd197c23 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:51.640228 6458 raft_consensus.cc:2454] T 7f61f9a345d94a00a0929d8ebd197c23 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 01:59:51.640321 6589 raft_consensus.cc:2454] T 7f61f9a345d94a00a0929d8ebd197c23 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 01:59:51.640620 6784 leader_election.cc:302] T 7f61f9a345d94a00a0929d8ebd197c23 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:51.640789 7905 raft_consensus.cc:2781] T 7f61f9a345d94a00a0929d8ebd197c23 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:51.640880 7905 raft_consensus.cc:686] T 7f61f9a345d94a00a0929d8ebd197c23 P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Becoming Leader. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Running, Role: LEADER
I20240503 01:59:51.641054 7905 consensus_queue.cc:237] T 7f61f9a345d94a00a0929d8ebd197c23 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:51.643011 6308 catalog_manager.cc:5484] T 7f61f9a345d94a00a0929d8ebd197c23 P 492448570cc94ed1879feb2ec3b9051d reported cstate change: term changed from 0 to 1, leader changed from <none> to 492448570cc94ed1879feb2ec3b9051d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } }
W20240503 01:59:51.782377 8131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:51.782653 8131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:51.786345 8131 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 01:59:51.803594 8035 consensus_queue.cc:1035] T 1832eb75b09d4823ad07c56868f3aede P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:51.808523 8125 consensus_queue.cc:1035] T 1832eb75b09d4823ad07c56868f3aede P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_10'
I20240503 01:59:51.837846 7905 consensus_queue.cc:1035] T 20b9d73abd6a424193fc102180f1dd4b P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:51.843521 7905 consensus_queue.cc:1035] T 20b9d73abd6a424193fc102180f1dd4b P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:51.884886 6589 raft_consensus.cc:1261] T 7f61f9a345d94a00a0929d8ebd197c23 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:51.885893 6458 raft_consensus.cc:1261] T 7f61f9a345d94a00a0929d8ebd197c23 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:51.887053 7905 consensus_queue.cc:1035] T 7f61f9a345d94a00a0929d8ebd197c23 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:51.890367 8128 consensus_queue.cc:1035] T 7f61f9a345d94a00a0929d8ebd197c23 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:51.911878 8104 mvcc.cc:205] Tried to move back new op lower bound from 7023417720343703552 to 7023417719363387392. Current Snapshot: MvccSnapshot[applied={T|T < 7023417720343703552}]
INSERT report
rows total: 2000
time total: 114.51 ms
time per row: 0.0572551 ms
I20240503 01:59:51.949746 6308 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalance_test_table_11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:51.956308 6436 tablet_service.cc:1467] Processing CreateTablet for tablet d73ea6e550a04c04a2695b058e38c566 (DEFAULT_TABLE table=rebalance_test_table_11 [id=805c97ce87c74e019deaa2acee6a5329]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:51.956701 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d73ea6e550a04c04a2695b058e38c566. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:51.956822 6699 tablet_service.cc:1467] Processing CreateTablet for tablet 3cf186a0901f420390c8fda54316c5af (DEFAULT_TABLE table=rebalance_test_table_11 [id=805c97ce87c74e019deaa2acee6a5329]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:51.956991 6829 tablet_service.cc:1467] Processing CreateTablet for tablet 3cf186a0901f420390c8fda54316c5af (DEFAULT_TABLE table=rebalance_test_table_11 [id=805c97ce87c74e019deaa2acee6a5329]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:51.957017 6698 tablet_service.cc:1467] Processing CreateTablet for tablet d0d5dd42db3e4883af7fd16d8813b550 (DEFAULT_TABLE table=rebalance_test_table_11 [id=805c97ce87c74e019deaa2acee6a5329]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:51.957098 6699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cf186a0901f420390c8fda54316c5af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:51.957195 6698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0d5dd42db3e4883af7fd16d8813b550. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:51.957247 6829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cf186a0901f420390c8fda54316c5af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:51.958865 6569 tablet_service.cc:1467] Processing CreateTablet for tablet d73ea6e550a04c04a2695b058e38c566 (DEFAULT_TABLE table=rebalance_test_table_11 [id=805c97ce87c74e019deaa2acee6a5329]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:51.959148 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d73ea6e550a04c04a2695b058e38c566. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:51.959271 6830 tablet_service.cc:1467] Processing CreateTablet for tablet d0d5dd42db3e4883af7fd16d8813b550 (DEFAULT_TABLE table=rebalance_test_table_11 [id=805c97ce87c74e019deaa2acee6a5329]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:51.959350 6568 tablet_service.cc:1467] Processing CreateTablet for tablet d0d5dd42db3e4883af7fd16d8813b550 (DEFAULT_TABLE table=rebalance_test_table_11 [id=805c97ce87c74e019deaa2acee6a5329]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:51.959502 6830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0d5dd42db3e4883af7fd16d8813b550. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:51.959583 6568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0d5dd42db3e4883af7fd16d8813b550. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:51.961345 8186 tablet_bootstrap.cc:492] T 3cf186a0901f420390c8fda54316c5af P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 01:59:51.961704 8188 tablet_bootstrap.cc:492] T d73ea6e550a04c04a2695b058e38c566 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 01:59:51.961841 6436 tablet_service.cc:1467] Processing CreateTablet for tablet 3cf186a0901f420390c8fda54316c5af (DEFAULT_TABLE table=rebalance_test_table_11 [id=805c97ce87c74e019deaa2acee6a5329]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:51.962137 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cf186a0901f420390c8fda54316c5af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:51.962401 8187 tablet_bootstrap.cc:492] T 3cf186a0901f420390c8fda54316c5af P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 01:59:51.962684 8188 tablet_bootstrap.cc:654] T d73ea6e550a04c04a2695b058e38c566 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:51.963214 8187 tablet_bootstrap.cc:654] T 3cf186a0901f420390c8fda54316c5af P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:51.963390 8186 tablet_bootstrap.cc:654] T 3cf186a0901f420390c8fda54316c5af P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:51.964224 8188 tablet_bootstrap.cc:492] T d73ea6e550a04c04a2695b058e38c566 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 01:59:51.964267 8187 tablet_bootstrap.cc:492] T 3cf186a0901f420390c8fda54316c5af P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 01:59:51.964309 8188 ts_tablet_manager.cc:1367] T d73ea6e550a04c04a2695b058e38c566 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 01:59:51.964354 8187 ts_tablet_manager.cc:1367] T 3cf186a0901f420390c8fda54316c5af P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:51.956822 6700 tablet_service.cc:1467] Processing CreateTablet for tablet d73ea6e550a04c04a2695b058e38c566 (DEFAULT_TABLE table=rebalance_test_table_11 [id=805c97ce87c74e019deaa2acee6a5329]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:51.964792 8188 raft_consensus.cc:348] T d73ea6e550a04c04a2695b058e38c566 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:51.964891 8188 raft_consensus.cc:374] T d73ea6e550a04c04a2695b058e38c566 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:51.964912 8188 raft_consensus.cc:729] T d73ea6e550a04c04a2695b058e38c566 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 01:59:51.964965 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d73ea6e550a04c04a2695b058e38c566. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:51.964991 8187 raft_consensus.cc:348] T 3cf186a0901f420390c8fda54316c5af P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:51.965049 8188 consensus_queue.cc:260] T d73ea6e550a04c04a2695b058e38c566 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:51.965085 8187 raft_consensus.cc:374] T 3cf186a0901f420390c8fda54316c5af P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:51.965113 8187 raft_consensus.cc:729] T 3cf186a0901f420390c8fda54316c5af P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 01:59:51.965226 8187 consensus_queue.cc:260] T 3cf186a0901f420390c8fda54316c5af P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:51.965310 8188 ts_tablet_manager.cc:1398] T d73ea6e550a04c04a2695b058e38c566 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:51.965469 8188 tablet_bootstrap.cc:492] T d0d5dd42db3e4883af7fd16d8813b550 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 01:59:51.965497 8187 ts_tablet_manager.cc:1398] T 3cf186a0901f420390c8fda54316c5af P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:51.965641 8187 tablet_bootstrap.cc:492] T d0d5dd42db3e4883af7fd16d8813b550 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 01:59:51.966349 8188 tablet_bootstrap.cc:654] T d0d5dd42db3e4883af7fd16d8813b550 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:51.966745 8187 tablet_bootstrap.cc:654] T d0d5dd42db3e4883af7fd16d8813b550 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:51.967268 8188 tablet_bootstrap.cc:492] T d0d5dd42db3e4883af7fd16d8813b550 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 01:59:51.967268 8186 tablet_bootstrap.cc:492] T 3cf186a0901f420390c8fda54316c5af P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 01:59:51.967341 8188 ts_tablet_manager.cc:1367] T d0d5dd42db3e4883af7fd16d8813b550 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:51.967344 8186 ts_tablet_manager.cc:1367] T 3cf186a0901f420390c8fda54316c5af P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 01:59:51.967777 8186 raft_consensus.cc:348] T 3cf186a0901f420390c8fda54316c5af P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:51.967880 8186 raft_consensus.cc:374] T 3cf186a0901f420390c8fda54316c5af P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:51.967906 8186 raft_consensus.cc:729] T 3cf186a0901f420390c8fda54316c5af P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 01:59:51.967831 8188 raft_consensus.cc:348] T d0d5dd42db3e4883af7fd16d8813b550 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:51.968034 8186 consensus_queue.cc:260] T 3cf186a0901f420390c8fda54316c5af P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:51.968078 8188 raft_consensus.cc:374] T d0d5dd42db3e4883af7fd16d8813b550 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:51.968109 8188 raft_consensus.cc:729] T d0d5dd42db3e4883af7fd16d8813b550 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 01:59:51.968313 8186 ts_tablet_manager.cc:1398] T 3cf186a0901f420390c8fda54316c5af P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:51.968319 8188 consensus_queue.cc:260] T d0d5dd42db3e4883af7fd16d8813b550 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:51.968492 8186 tablet_bootstrap.cc:492] T d73ea6e550a04c04a2695b058e38c566 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 01:59:51.968688 8188 ts_tablet_manager.cc:1398] T d0d5dd42db3e4883af7fd16d8813b550 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:51.969305 8186 tablet_bootstrap.cc:654] T d73ea6e550a04c04a2695b058e38c566 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:51.970981 8185 tablet_bootstrap.cc:492] T d73ea6e550a04c04a2695b058e38c566 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 01:59:51.971784 8185 tablet_bootstrap.cc:654] T d73ea6e550a04c04a2695b058e38c566 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:51.972844 8186 tablet_bootstrap.cc:492] T d73ea6e550a04c04a2695b058e38c566 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 01:59:51.972878 8185 tablet_bootstrap.cc:492] T d73ea6e550a04c04a2695b058e38c566 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 01:59:51.972936 8186 ts_tablet_manager.cc:1367] T d73ea6e550a04c04a2695b058e38c566 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 01:59:51.973001 8185 ts_tablet_manager.cc:1367] T d73ea6e550a04c04a2695b058e38c566 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:51.973284 8187 tablet_bootstrap.cc:492] T d0d5dd42db3e4883af7fd16d8813b550 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 01:59:51.973387 8187 ts_tablet_manager.cc:1367] T d0d5dd42db3e4883af7fd16d8813b550 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 01:59:51.973656 8186 raft_consensus.cc:348] T d73ea6e550a04c04a2695b058e38c566 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:51.973760 8186 raft_consensus.cc:374] T d73ea6e550a04c04a2695b058e38c566 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:51.973788 8186 raft_consensus.cc:729] T d73ea6e550a04c04a2695b058e38c566 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 01:59:51.973501 8185 raft_consensus.cc:348] T d73ea6e550a04c04a2695b058e38c566 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:51.973872 8185 raft_consensus.cc:374] T d73ea6e550a04c04a2695b058e38c566 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:51.973819 8187 raft_consensus.cc:348] T d0d5dd42db3e4883af7fd16d8813b550 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:51.973904 8185 raft_consensus.cc:729] T d73ea6e550a04c04a2695b058e38c566 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 01:59:51.973923 8187 raft_consensus.cc:374] T d0d5dd42db3e4883af7fd16d8813b550 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:51.973907 8186 consensus_queue.cc:260] T d73ea6e550a04c04a2695b058e38c566 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:51.973953 8187 raft_consensus.cc:729] T d0d5dd42db3e4883af7fd16d8813b550 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 01:59:51.974061 8185 consensus_queue.cc:260] T d73ea6e550a04c04a2695b058e38c566 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:51.974195 8186 ts_tablet_manager.cc:1398] T d73ea6e550a04c04a2695b058e38c566 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:51.974362 8186 tablet_bootstrap.cc:492] T d0d5dd42db3e4883af7fd16d8813b550 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 01:59:51.974357 8187 consensus_queue.cc:260] T d0d5dd42db3e4883af7fd16d8813b550 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:51.974431 8185 ts_tablet_manager.cc:1398] T d73ea6e550a04c04a2695b058e38c566 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:51.974666 8187 ts_tablet_manager.cc:1398] T d0d5dd42db3e4883af7fd16d8813b550 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:51.974702 8185 tablet_bootstrap.cc:492] T 3cf186a0901f420390c8fda54316c5af P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 01:59:51.975332 8186 tablet_bootstrap.cc:654] T d0d5dd42db3e4883af7fd16d8813b550 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:51.976694 8185 tablet_bootstrap.cc:654] T 3cf186a0901f420390c8fda54316c5af P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:51.977804 8186 tablet_bootstrap.cc:492] T d0d5dd42db3e4883af7fd16d8813b550 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 01:59:51.977870 8185 tablet_bootstrap.cc:492] T 3cf186a0901f420390c8fda54316c5af P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 01:59:51.977916 8186 ts_tablet_manager.cc:1367] T d0d5dd42db3e4883af7fd16d8813b550 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:51.977958 8185 ts_tablet_manager.cc:1367] T 3cf186a0901f420390c8fda54316c5af P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 01:59:51.978379 8185 raft_consensus.cc:348] T 3cf186a0901f420390c8fda54316c5af P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:51.978485 8185 raft_consensus.cc:374] T 3cf186a0901f420390c8fda54316c5af P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:51.978416 8186 raft_consensus.cc:348] T d0d5dd42db3e4883af7fd16d8813b550 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:51.978515 8185 raft_consensus.cc:729] T 3cf186a0901f420390c8fda54316c5af P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 01:59:51.978528 8186 raft_consensus.cc:374] T d0d5dd42db3e4883af7fd16d8813b550 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:51.978560 8186 raft_consensus.cc:729] T d0d5dd42db3e4883af7fd16d8813b550 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 01:59:51.978631 8185 consensus_queue.cc:260] T 3cf186a0901f420390c8fda54316c5af P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:51.978674 8186 consensus_queue.cc:260] T d0d5dd42db3e4883af7fd16d8813b550 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:51.978905 8185 ts_tablet_manager.cc:1398] T 3cf186a0901f420390c8fda54316c5af P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:51.978968 8186 ts_tablet_manager.cc:1398] T d0d5dd42db3e4883af7fd16d8813b550 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:52.011476 8035 raft_consensus.cc:482] T d0d5dd42db3e4883af7fd16d8813b550 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:52.011639 8035 raft_consensus.cc:504] T d0d5dd42db3e4883af7fd16d8813b550 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:52.012118 8035 leader_election.cc:288] T d0d5dd42db3e4883af7fd16d8813b550 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 01:59:52.012554 6720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0d5dd42db3e4883af7fd16d8813b550" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 01:59:52.012604 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0d5dd42db3e4883af7fd16d8813b550" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 01:59:52.012712 6720 raft_consensus.cc:2454] T d0d5dd42db3e4883af7fd16d8813b550 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 01:59:52.012769 6849 raft_consensus.cc:2454] T d0d5dd42db3e4883af7fd16d8813b550 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 01:59:52.013056 6525 leader_election.cc:302] T d0d5dd42db3e4883af7fd16d8813b550 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 80fd18ef4d604c86b909d5c008a46b4f; no voters:
I20240503 01:59:52.013263 8035 raft_consensus.cc:2781] T d0d5dd42db3e4883af7fd16d8813b550 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:52.013329 8035 raft_consensus.cc:482] T d0d5dd42db3e4883af7fd16d8813b550 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:52.013360 8035 raft_consensus.cc:3037] T d0d5dd42db3e4883af7fd16d8813b550 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:52.014206 8035 raft_consensus.cc:504] T d0d5dd42db3e4883af7fd16d8813b550 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:52.014582 8035 leader_election.cc:288] T d0d5dd42db3e4883af7fd16d8813b550 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Requested vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 01:59:52.015039 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0d5dd42db3e4883af7fd16d8813b550" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 01:59:52.015187 6849 raft_consensus.cc:3037] T d0d5dd42db3e4883af7fd16d8813b550 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:52.015997 6849 raft_consensus.cc:2454] T d0d5dd42db3e4883af7fd16d8813b550 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 01:59:52.016287 6720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0d5dd42db3e4883af7fd16d8813b550" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 01:59:52.016314 6525 leader_election.cc:302] T d0d5dd42db3e4883af7fd16d8813b550 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 80fd18ef4d604c86b909d5c008a46b4f; no voters:
I20240503 01:59:52.016434 6720 raft_consensus.cc:3037] T d0d5dd42db3e4883af7fd16d8813b550 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:52.016528 8035 raft_consensus.cc:2781] T d0d5dd42db3e4883af7fd16d8813b550 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:52.016603 8035 raft_consensus.cc:686] T d0d5dd42db3e4883af7fd16d8813b550 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Becoming Leader. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Running, Role: LEADER
I20240503 01:59:52.016731 8035 consensus_queue.cc:237] T d0d5dd42db3e4883af7fd16d8813b550 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:52.017422 6720 raft_consensus.cc:2454] T d0d5dd42db3e4883af7fd16d8813b550 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 01:59:52.019009 6311 catalog_manager.cc:5484] T d0d5dd42db3e4883af7fd16d8813b550 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:52.026087 8034 raft_consensus.cc:482] T d73ea6e550a04c04a2695b058e38c566 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:52.026253 8034 raft_consensus.cc:504] T d73ea6e550a04c04a2695b058e38c566 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:52.026762 8034 leader_election.cc:288] T d73ea6e550a04c04a2695b058e38c566 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 01:59:52.027202 6457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d73ea6e550a04c04a2695b058e38c566" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 01:59:52.027230 6588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d73ea6e550a04c04a2695b058e38c566" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 01:59:52.027352 6457 raft_consensus.cc:2454] T d73ea6e550a04c04a2695b058e38c566 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 01:59:52.027367 6588 raft_consensus.cc:2454] T d73ea6e550a04c04a2695b058e38c566 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 01:59:52.027683 6653 leader_election.cc:302] T d73ea6e550a04c04a2695b058e38c566 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2ac2eb7cc264047b9ab32fe3a4fc025, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:52.027994 8034 raft_consensus.cc:2781] T d73ea6e550a04c04a2695b058e38c566 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:52.028053 8034 raft_consensus.cc:482] T d73ea6e550a04c04a2695b058e38c566 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:52.028095 8034 raft_consensus.cc:3037] T d73ea6e550a04c04a2695b058e38c566 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:52.029083 8034 raft_consensus.cc:504] T d73ea6e550a04c04a2695b058e38c566 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:52.029489 8034 leader_election.cc:288] T d73ea6e550a04c04a2695b058e38c566 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Requested vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 01:59:52.029909 6457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d73ea6e550a04c04a2695b058e38c566" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 01:59:52.029928 6588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d73ea6e550a04c04a2695b058e38c566" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 01:59:52.030047 6588 raft_consensus.cc:3037] T d73ea6e550a04c04a2695b058e38c566 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:52.030048 6457 raft_consensus.cc:3037] T d73ea6e550a04c04a2695b058e38c566 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:52.030875 6457 raft_consensus.cc:2454] T d73ea6e550a04c04a2695b058e38c566 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 01:59:52.031008 6588 raft_consensus.cc:2454] T d73ea6e550a04c04a2695b058e38c566 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 01:59:52.031244 6653 leader_election.cc:302] T d73ea6e550a04c04a2695b058e38c566 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2ac2eb7cc264047b9ab32fe3a4fc025, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:52.031534 8034 raft_consensus.cc:2781] T d73ea6e550a04c04a2695b058e38c566 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:52.031633 8034 raft_consensus.cc:686] T d73ea6e550a04c04a2695b058e38c566 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Becoming Leader. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Running, Role: LEADER
I20240503 01:59:52.031790 8034 consensus_queue.cc:237] T d73ea6e550a04c04a2695b058e38c566 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:52.034111 6311 catalog_manager.cc:5484] T d73ea6e550a04c04a2695b058e38c566 P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:52.151297 7957 raft_consensus.cc:482] T 3cf186a0901f420390c8fda54316c5af P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:52.151465 7957 raft_consensus.cc:504] T 3cf186a0901f420390c8fda54316c5af P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:52.151901 7957 leader_election.cc:288] T 3cf186a0901f420390c8fda54316c5af P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 01:59:52.152422 6457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cf186a0901f420390c8fda54316c5af" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 01:59:52.152422 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cf186a0901f420390c8fda54316c5af" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 01:59:52.152570 6849 raft_consensus.cc:2454] T 3cf186a0901f420390c8fda54316c5af P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 01:59:52.152576 6457 raft_consensus.cc:2454] T 3cf186a0901f420390c8fda54316c5af P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 01:59:52.152895 6656 leader_election.cc:302] T 3cf186a0901f420390c8fda54316c5af P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, c2ac2eb7cc264047b9ab32fe3a4fc025, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:52.153110 7957 raft_consensus.cc:2781] T 3cf186a0901f420390c8fda54316c5af P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:52.153167 7957 raft_consensus.cc:482] T 3cf186a0901f420390c8fda54316c5af P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:52.153213 7957 raft_consensus.cc:3037] T 3cf186a0901f420390c8fda54316c5af P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:52.154037 7957 raft_consensus.cc:504] T 3cf186a0901f420390c8fda54316c5af P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:52.154407 7957 leader_election.cc:288] T 3cf186a0901f420390c8fda54316c5af P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Requested vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 01:59:52.154788 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cf186a0901f420390c8fda54316c5af" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 01:59:52.154903 6849 raft_consensus.cc:3037] T 3cf186a0901f420390c8fda54316c5af P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:52.155023 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cf186a0901f420390c8fda54316c5af" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 01:59:52.155166 6458 raft_consensus.cc:3037] T 3cf186a0901f420390c8fda54316c5af P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:52.155752 6849 raft_consensus.cc:2454] T 3cf186a0901f420390c8fda54316c5af P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 01:59:52.156083 6656 leader_election.cc:302] T 3cf186a0901f420390c8fda54316c5af P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 01:59:52.156247 7957 raft_consensus.cc:2781] T 3cf186a0901f420390c8fda54316c5af P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:52.156157 6458 raft_consensus.cc:2454] T 3cf186a0901f420390c8fda54316c5af P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 01:59:52.156337 7957 raft_consensus.cc:686] T 3cf186a0901f420390c8fda54316c5af P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Becoming Leader. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Running, Role: LEADER
I20240503 01:59:52.156504 7957 consensus_queue.cc:237] T 3cf186a0901f420390c8fda54316c5af P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:52.158473 6311 catalog_manager.cc:5484] T 3cf186a0901f420390c8fda54316c5af P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } }
W20240503 01:59:52.281184 8201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:52.281466 8201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:52.284866 8201 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_11'
I20240503 01:59:52.386993 6849 raft_consensus.cc:1261] T 3cf186a0901f420390c8fda54316c5af P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:52.387411 6458 raft_consensus.cc:1261] T 3cf186a0901f420390c8fda54316c5af P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:52.388051 7957 consensus_queue.cc:1035] T 3cf186a0901f420390c8fda54316c5af P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:52.388413 7957 consensus_queue.cc:1035] T 3cf186a0901f420390c8fda54316c5af P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:52.389622 6849 raft_consensus.cc:1261] T d0d5dd42db3e4883af7fd16d8813b550 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:52.390105 6718 raft_consensus.cc:1261] T d0d5dd42db3e4883af7fd16d8813b550 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:52.390588 8169 consensus_queue.cc:1035] T d0d5dd42db3e4883af7fd16d8813b550 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:52.392416 6589 raft_consensus.cc:1261] T d73ea6e550a04c04a2695b058e38c566 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:52.392877 7957 consensus_queue.cc:1035] T d73ea6e550a04c04a2695b058e38c566 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:52.406255 8035 consensus_queue.cc:1035] T d0d5dd42db3e4883af7fd16d8813b550 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:52.408285 6457 raft_consensus.cc:1261] T d73ea6e550a04c04a2695b058e38c566 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:52.409448 8034 consensus_queue.cc:1035] T d73ea6e550a04c04a2695b058e38c566 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:52.411672 8036 mvcc.cc:205] Tried to move back new op lower bound from 7023417722431651840 to 7023417720963764224. Current Snapshot: MvccSnapshot[applied={T|T < 7023417722431651840}]
I20240503 01:59:52.447474 8233 mvcc.cc:205] Tried to move back new op lower bound from 7023417722422345728 to 7023417720902291456. Current Snapshot: MvccSnapshot[applied={T|T < 7023417722422345728}]
INSERT report
rows total: 2000
time total: 134.548 ms
time per row: 0.0672742 ms
I20240503 01:59:52.470803 6313 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalance_test_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:52.476774 6436 tablet_service.cc:1467] Processing CreateTablet for tablet e08ab73460e746fc86a9b77c405da86e (DEFAULT_TABLE table=rebalance_test_table_12 [id=043b44e4ca1745038882551026afda53]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:52.476774 6698 tablet_service.cc:1467] Processing CreateTablet for tablet e08ab73460e746fc86a9b77c405da86e (DEFAULT_TABLE table=rebalance_test_table_12 [id=043b44e4ca1745038882551026afda53]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:52.477128 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e08ab73460e746fc86a9b77c405da86e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:52.477138 6698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e08ab73460e746fc86a9b77c405da86e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:52.477182 6569 tablet_service.cc:1467] Processing CreateTablet for tablet 34120f4948504cbabcd28e0c4f08f9f8 (DEFAULT_TABLE table=rebalance_test_table_12 [id=043b44e4ca1745038882551026afda53]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:52.477303 6567 tablet_service.cc:1467] Processing CreateTablet for tablet 7b392142a4d948e7a36b49314ae15129 (DEFAULT_TABLE table=rebalance_test_table_12 [id=043b44e4ca1745038882551026afda53]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:52.477423 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34120f4948504cbabcd28e0c4f08f9f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:52.477519 6567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b392142a4d948e7a36b49314ae15129. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:52.479596 8186 tablet_bootstrap.cc:492] T e08ab73460e746fc86a9b77c405da86e P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 01:59:52.479665 6568 tablet_service.cc:1467] Processing CreateTablet for tablet e08ab73460e746fc86a9b77c405da86e (DEFAULT_TABLE table=rebalance_test_table_12 [id=043b44e4ca1745038882551026afda53]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:52.479933 8185 tablet_bootstrap.cc:492] T e08ab73460e746fc86a9b77c405da86e P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 01:59:52.480139 6568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e08ab73460e746fc86a9b77c405da86e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:52.480804 8185 tablet_bootstrap.cc:654] T e08ab73460e746fc86a9b77c405da86e P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:52.480973 8186 tablet_bootstrap.cc:654] T e08ab73460e746fc86a9b77c405da86e P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:52.481832 8185 tablet_bootstrap.cc:492] T e08ab73460e746fc86a9b77c405da86e P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 01:59:52.481928 8185 ts_tablet_manager.cc:1367] T e08ab73460e746fc86a9b77c405da86e P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 01:59:52.482403 8185 raft_consensus.cc:348] T e08ab73460e746fc86a9b77c405da86e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:52.482538 8185 raft_consensus.cc:374] T e08ab73460e746fc86a9b77c405da86e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:52.482575 8185 raft_consensus.cc:729] T e08ab73460e746fc86a9b77c405da86e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 01:59:52.482478 6436 tablet_service.cc:1467] Processing CreateTablet for tablet 34120f4948504cbabcd28e0c4f08f9f8 (DEFAULT_TABLE table=rebalance_test_table_12 [id=043b44e4ca1745038882551026afda53]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:52.482689 8185 consensus_queue.cc:260] T e08ab73460e746fc86a9b77c405da86e P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:52.482882 6830 tablet_service.cc:1467] Processing CreateTablet for tablet 34120f4948504cbabcd28e0c4f08f9f8 (DEFAULT_TABLE table=rebalance_test_table_12 [id=043b44e4ca1745038882551026afda53]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:52.482990 8185 ts_tablet_manager.cc:1398] T e08ab73460e746fc86a9b77c405da86e P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 01:59:52.482998 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34120f4948504cbabcd28e0c4f08f9f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:52.482990 6829 tablet_service.cc:1467] Processing CreateTablet for tablet 7b392142a4d948e7a36b49314ae15129 (DEFAULT_TABLE table=rebalance_test_table_12 [id=043b44e4ca1745038882551026afda53]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:52.483152 6830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34120f4948504cbabcd28e0c4f08f9f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:52.483796 8186 tablet_bootstrap.cc:492] T e08ab73460e746fc86a9b77c405da86e P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 01:59:52.483888 8186 ts_tablet_manager.cc:1367] T e08ab73460e746fc86a9b77c405da86e P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:52.484002 6829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b392142a4d948e7a36b49314ae15129. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:52.484365 8186 raft_consensus.cc:348] T e08ab73460e746fc86a9b77c405da86e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:52.484550 8186 raft_consensus.cc:374] T e08ab73460e746fc86a9b77c405da86e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:52.484583 8186 raft_consensus.cc:729] T e08ab73460e746fc86a9b77c405da86e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 01:59:52.484723 8186 consensus_queue.cc:260] T e08ab73460e746fc86a9b77c405da86e P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:52.485008 8186 ts_tablet_manager.cc:1398] T e08ab73460e746fc86a9b77c405da86e P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:52.486487 8249 tablet_bootstrap.cc:492] T 34120f4948504cbabcd28e0c4f08f9f8 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 01:59:52.486640 6698 tablet_service.cc:1467] Processing CreateTablet for tablet 7b392142a4d948e7a36b49314ae15129 (DEFAULT_TABLE table=rebalance_test_table_12 [id=043b44e4ca1745038882551026afda53]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:52.486913 6698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b392142a4d948e7a36b49314ae15129. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:52.487579 8249 tablet_bootstrap.cc:654] T 34120f4948504cbabcd28e0c4f08f9f8 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:52.488052 8185 tablet_bootstrap.cc:492] T 34120f4948504cbabcd28e0c4f08f9f8 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 01:59:52.488708 8249 tablet_bootstrap.cc:492] T 34120f4948504cbabcd28e0c4f08f9f8 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 01:59:52.488811 8249 ts_tablet_manager.cc:1367] T 34120f4948504cbabcd28e0c4f08f9f8 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:52.489161 8185 tablet_bootstrap.cc:654] T 34120f4948504cbabcd28e0c4f08f9f8 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:52.489291 8249 raft_consensus.cc:348] T 34120f4948504cbabcd28e0c4f08f9f8 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:52.489408 8249 raft_consensus.cc:374] T 34120f4948504cbabcd28e0c4f08f9f8 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:52.489441 8249 raft_consensus.cc:729] T 34120f4948504cbabcd28e0c4f08f9f8 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 01:59:52.489569 8249 consensus_queue.cc:260] T 34120f4948504cbabcd28e0c4f08f9f8 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:52.489858 8249 ts_tablet_manager.cc:1398] T 34120f4948504cbabcd28e0c4f08f9f8 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:52.489990 8186 tablet_bootstrap.cc:492] T 7b392142a4d948e7a36b49314ae15129 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 01:59:52.490725 8185 tablet_bootstrap.cc:492] T 34120f4948504cbabcd28e0c4f08f9f8 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 01:59:52.490823 8185 ts_tablet_manager.cc:1367] T 34120f4948504cbabcd28e0c4f08f9f8 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:52.490907 8249 tablet_bootstrap.cc:492] T 7b392142a4d948e7a36b49314ae15129 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 01:59:52.490957 8186 tablet_bootstrap.cc:654] T 7b392142a4d948e7a36b49314ae15129 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:52.491176 8185 raft_consensus.cc:348] T 34120f4948504cbabcd28e0c4f08f9f8 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:52.491256 8185 raft_consensus.cc:374] T 34120f4948504cbabcd28e0c4f08f9f8 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:52.491276 8185 raft_consensus.cc:729] T 34120f4948504cbabcd28e0c4f08f9f8 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 01:59:52.491381 8185 consensus_queue.cc:260] T 34120f4948504cbabcd28e0c4f08f9f8 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:52.491664 8185 ts_tablet_manager.cc:1398] T 34120f4948504cbabcd28e0c4f08f9f8 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:52.491837 8249 tablet_bootstrap.cc:654] T 7b392142a4d948e7a36b49314ae15129 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:52.492143 8186 tablet_bootstrap.cc:492] T 7b392142a4d948e7a36b49314ae15129 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 01:59:52.492226 8186 ts_tablet_manager.cc:1367] T 7b392142a4d948e7a36b49314ae15129 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 01:59:52.492674 8186 raft_consensus.cc:348] T 7b392142a4d948e7a36b49314ae15129 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:52.492792 8186 raft_consensus.cc:374] T 7b392142a4d948e7a36b49314ae15129 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:52.492836 8186 raft_consensus.cc:729] T 7b392142a4d948e7a36b49314ae15129 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 01:59:52.493153 8186 consensus_queue.cc:260] T 7b392142a4d948e7a36b49314ae15129 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:52.493602 8186 ts_tablet_manager.cc:1398] T 7b392142a4d948e7a36b49314ae15129 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:52.494403 8249 tablet_bootstrap.cc:492] T 7b392142a4d948e7a36b49314ae15129 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 01:59:52.494518 8249 ts_tablet_manager.cc:1367] T 7b392142a4d948e7a36b49314ae15129 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:52.495903 8246 tablet_bootstrap.cc:492] T 7b392142a4d948e7a36b49314ae15129 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 01:59:52.495100 8249 raft_consensus.cc:348] T 7b392142a4d948e7a36b49314ae15129 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:52.496538 8249 raft_consensus.cc:374] T 7b392142a4d948e7a36b49314ae15129 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:52.496611 8249 raft_consensus.cc:729] T 7b392142a4d948e7a36b49314ae15129 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 01:59:52.496809 8249 consensus_queue.cc:260] T 7b392142a4d948e7a36b49314ae15129 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:52.497709 8249 ts_tablet_manager.cc:1398] T 7b392142a4d948e7a36b49314ae15129 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:52.498529 8246 tablet_bootstrap.cc:654] T 7b392142a4d948e7a36b49314ae15129 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:52.499707 8246 tablet_bootstrap.cc:492] T 7b392142a4d948e7a36b49314ae15129 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 01:59:52.499809 8246 ts_tablet_manager.cc:1367] T 7b392142a4d948e7a36b49314ae15129 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:52.500468 8246 raft_consensus.cc:348] T 7b392142a4d948e7a36b49314ae15129 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:52.500586 8246 raft_consensus.cc:374] T 7b392142a4d948e7a36b49314ae15129 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:52.500617 8246 raft_consensus.cc:729] T 7b392142a4d948e7a36b49314ae15129 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 01:59:52.500763 8246 consensus_queue.cc:260] T 7b392142a4d948e7a36b49314ae15129 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:52.501025 8246 ts_tablet_manager.cc:1398] T 7b392142a4d948e7a36b49314ae15129 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:52.501194 8246 tablet_bootstrap.cc:492] T 34120f4948504cbabcd28e0c4f08f9f8 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 01:59:52.502247 8246 tablet_bootstrap.cc:654] T 34120f4948504cbabcd28e0c4f08f9f8 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:52.503535 8246 tablet_bootstrap.cc:492] T 34120f4948504cbabcd28e0c4f08f9f8 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 01:59:52.503629 8246 ts_tablet_manager.cc:1367] T 34120f4948504cbabcd28e0c4f08f9f8 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:52.504046 8246 raft_consensus.cc:348] T 34120f4948504cbabcd28e0c4f08f9f8 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:52.504148 8246 raft_consensus.cc:374] T 34120f4948504cbabcd28e0c4f08f9f8 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:52.504174 8246 raft_consensus.cc:729] T 34120f4948504cbabcd28e0c4f08f9f8 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 01:59:52.504307 8246 consensus_queue.cc:260] T 34120f4948504cbabcd28e0c4f08f9f8 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:52.504642 8246 ts_tablet_manager.cc:1398] T 34120f4948504cbabcd28e0c4f08f9f8 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:52.504827 8246 tablet_bootstrap.cc:492] T e08ab73460e746fc86a9b77c405da86e P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 01:59:52.506047 8246 tablet_bootstrap.cc:654] T e08ab73460e746fc86a9b77c405da86e P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:52.507277 8246 tablet_bootstrap.cc:492] T e08ab73460e746fc86a9b77c405da86e P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 01:59:52.507364 8246 ts_tablet_manager.cc:1367] T e08ab73460e746fc86a9b77c405da86e P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:52.507750 8246 raft_consensus.cc:348] T e08ab73460e746fc86a9b77c405da86e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:52.507874 8246 raft_consensus.cc:374] T e08ab73460e746fc86a9b77c405da86e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:52.507911 8246 raft_consensus.cc:729] T e08ab73460e746fc86a9b77c405da86e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 01:59:52.508037 8246 consensus_queue.cc:260] T e08ab73460e746fc86a9b77c405da86e P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:52.508307 8246 ts_tablet_manager.cc:1398] T e08ab73460e746fc86a9b77c405da86e P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:52.509047 8128 raft_consensus.cc:482] T 7b392142a4d948e7a36b49314ae15129 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:52.509164 8128 raft_consensus.cc:504] T 7b392142a4d948e7a36b49314ae15129 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:52.509632 8128 leader_election.cc:288] T 7b392142a4d948e7a36b49314ae15129 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 01:59:52.509984 6718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b392142a4d948e7a36b49314ae15129" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 01:59:52.509982 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b392142a4d948e7a36b49314ae15129" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 01:59:52.510123 6589 raft_consensus.cc:2454] T 7b392142a4d948e7a36b49314ae15129 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 01:59:52.510123 6718 raft_consensus.cc:2454] T 7b392142a4d948e7a36b49314ae15129 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 01:59:52.510476 6787 leader_election.cc:302] T 7b392142a4d948e7a36b49314ae15129 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 01:59:52.510757 8128 raft_consensus.cc:2781] T 7b392142a4d948e7a36b49314ae15129 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:52.510813 8128 raft_consensus.cc:482] T 7b392142a4d948e7a36b49314ae15129 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:52.510850 8128 raft_consensus.cc:3037] T 7b392142a4d948e7a36b49314ae15129 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:52.511776 8128 raft_consensus.cc:504] T 7b392142a4d948e7a36b49314ae15129 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:52.512166 8128 leader_election.cc:288] T 7b392142a4d948e7a36b49314ae15129 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Requested vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 01:59:52.512539 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b392142a4d948e7a36b49314ae15129" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 01:59:52.512539 6718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b392142a4d948e7a36b49314ae15129" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 01:59:52.512668 6718 raft_consensus.cc:3037] T 7b392142a4d948e7a36b49314ae15129 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:52.512668 6589 raft_consensus.cc:3037] T 7b392142a4d948e7a36b49314ae15129 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:52.513648 6718 raft_consensus.cc:2454] T 7b392142a4d948e7a36b49314ae15129 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 01:59:52.513648 6589 raft_consensus.cc:2454] T 7b392142a4d948e7a36b49314ae15129 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 01:59:52.513975 6787 leader_election.cc:302] T 7b392142a4d948e7a36b49314ae15129 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 01:59:52.514163 8128 raft_consensus.cc:2781] T 7b392142a4d948e7a36b49314ae15129 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:52.514257 8128 raft_consensus.cc:686] T 7b392142a4d948e7a36b49314ae15129 P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Becoming Leader. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Running, Role: LEADER
I20240503 01:59:52.514427 8128 consensus_queue.cc:237] T 7b392142a4d948e7a36b49314ae15129 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:52.516304 6315 catalog_manager.cc:5484] T 7b392142a4d948e7a36b49314ae15129 P 492448570cc94ed1879feb2ec3b9051d reported cstate change: term changed from 0 to 1, leader changed from <none> to 492448570cc94ed1879feb2ec3b9051d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:52.530076 8128 raft_consensus.cc:482] T 34120f4948504cbabcd28e0c4f08f9f8 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:52.530251 8128 raft_consensus.cc:504] T 34120f4948504cbabcd28e0c4f08f9f8 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:52.530751 8128 leader_election.cc:288] T 34120f4948504cbabcd28e0c4f08f9f8 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 01:59:52.531229 6457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34120f4948504cbabcd28e0c4f08f9f8" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 01:59:52.531235 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34120f4948504cbabcd28e0c4f08f9f8" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 01:59:52.531392 6589 raft_consensus.cc:2454] T 34120f4948504cbabcd28e0c4f08f9f8 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 01:59:52.531388 6457 raft_consensus.cc:2454] T 34120f4948504cbabcd28e0c4f08f9f8 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 01:59:52.531733 6786 leader_election.cc:302] T 34120f4948504cbabcd28e0c4f08f9f8 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 80fd18ef4d604c86b909d5c008a46b4f, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:52.531931 8128 raft_consensus.cc:2781] T 34120f4948504cbabcd28e0c4f08f9f8 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:52.531993 8128 raft_consensus.cc:482] T 34120f4948504cbabcd28e0c4f08f9f8 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:52.532019 8128 raft_consensus.cc:3037] T 34120f4948504cbabcd28e0c4f08f9f8 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:52.532996 8128 raft_consensus.cc:504] T 34120f4948504cbabcd28e0c4f08f9f8 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:52.533391 8128 leader_election.cc:288] T 34120f4948504cbabcd28e0c4f08f9f8 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Requested vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 01:59:52.533762 6457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34120f4948504cbabcd28e0c4f08f9f8" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 01:59:52.533847 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34120f4948504cbabcd28e0c4f08f9f8" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 01:59:52.533896 6457 raft_consensus.cc:3037] T 34120f4948504cbabcd28e0c4f08f9f8 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:52.533959 6589 raft_consensus.cc:3037] T 34120f4948504cbabcd28e0c4f08f9f8 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:52.534904 6457 raft_consensus.cc:2454] T 34120f4948504cbabcd28e0c4f08f9f8 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 01:59:52.534904 6589 raft_consensus.cc:2454] T 34120f4948504cbabcd28e0c4f08f9f8 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 01:59:52.535248 6786 leader_election.cc:302] T 34120f4948504cbabcd28e0c4f08f9f8 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 80fd18ef4d604c86b909d5c008a46b4f; no voters:
I20240503 01:59:52.535447 8128 raft_consensus.cc:2781] T 34120f4948504cbabcd28e0c4f08f9f8 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:52.535544 8128 raft_consensus.cc:686] T 34120f4948504cbabcd28e0c4f08f9f8 P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Becoming Leader. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Running, Role: LEADER
I20240503 01:59:52.535705 8128 consensus_queue.cc:237] T 34120f4948504cbabcd28e0c4f08f9f8 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:52.537775 6315 catalog_manager.cc:5484] T 34120f4948504cbabcd28e0c4f08f9f8 P 492448570cc94ed1879feb2ec3b9051d reported cstate change: term changed from 0 to 1, leader changed from <none> to 492448570cc94ed1879feb2ec3b9051d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:52.700683 8238 raft_consensus.cc:482] T e08ab73460e746fc86a9b77c405da86e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:52.700841 8238 raft_consensus.cc:504] T e08ab73460e746fc86a9b77c405da86e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:52.701316 8238 leader_election.cc:288] T e08ab73460e746fc86a9b77c405da86e P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 01:59:52.701898 6457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e08ab73460e746fc86a9b77c405da86e" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 01:59:52.701900 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e08ab73460e746fc86a9b77c405da86e" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 01:59:52.702066 6589 raft_consensus.cc:2454] T e08ab73460e746fc86a9b77c405da86e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 01:59:52.702076 6457 raft_consensus.cc:2454] T e08ab73460e746fc86a9b77c405da86e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 01:59:52.702423 6655 leader_election.cc:302] T e08ab73460e746fc86a9b77c405da86e P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 01:59:52.702594 8238 raft_consensus.cc:2781] T e08ab73460e746fc86a9b77c405da86e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:52.702651 8238 raft_consensus.cc:482] T e08ab73460e746fc86a9b77c405da86e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:52.702689 8238 raft_consensus.cc:3037] T e08ab73460e746fc86a9b77c405da86e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:52.703559 8238 raft_consensus.cc:504] T e08ab73460e746fc86a9b77c405da86e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:52.703923 8238 leader_election.cc:288] T e08ab73460e746fc86a9b77c405da86e P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Requested vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 01:59:52.704329 6457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e08ab73460e746fc86a9b77c405da86e" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 01:59:52.704406 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e08ab73460e746fc86a9b77c405da86e" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 01:59:52.704496 6457 raft_consensus.cc:3037] T e08ab73460e746fc86a9b77c405da86e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:52.704530 6589 raft_consensus.cc:3037] T e08ab73460e746fc86a9b77c405da86e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:52.705317 6589 raft_consensus.cc:2454] T e08ab73460e746fc86a9b77c405da86e P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 01:59:52.705319 6457 raft_consensus.cc:2454] T e08ab73460e746fc86a9b77c405da86e P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 01:59:52.705670 6653 leader_election.cc:302] T e08ab73460e746fc86a9b77c405da86e P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2ac2eb7cc264047b9ab32fe3a4fc025, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:52.705868 8238 raft_consensus.cc:2781] T e08ab73460e746fc86a9b77c405da86e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:52.705961 8238 raft_consensus.cc:686] T e08ab73460e746fc86a9b77c405da86e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Becoming Leader. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Running, Role: LEADER
I20240503 01:59:52.706112 8238 consensus_queue.cc:237] T e08ab73460e746fc86a9b77c405da86e P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:52.708086 6315 catalog_manager.cc:5484] T e08ab73460e746fc86a9b77c405da86e P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } }
W20240503 01:59:52.846225 8260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:52.846529 8260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:52.849934 8260 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_12'
I20240503 01:59:52.913900 8128 consensus_queue.cc:1035] T 7b392142a4d948e7a36b49314ae15129 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:52.919474 8127 consensus_queue.cc:1035] T 7b392142a4d948e7a36b49314ae15129 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:52.952560 6588 raft_consensus.cc:1261] T e08ab73460e746fc86a9b77c405da86e P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:52.952560 6457 raft_consensus.cc:1261] T e08ab73460e746fc86a9b77c405da86e P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:52.953510 8238 consensus_queue.cc:1035] T e08ab73460e746fc86a9b77c405da86e P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:52.953922 8238 consensus_queue.cc:1035] T e08ab73460e746fc86a9b77c405da86e P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:52.956838 8128 consensus_queue.cc:1035] T 34120f4948504cbabcd28e0c4f08f9f8 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:52.968478 6582 raft_consensus.cc:1261] T 34120f4948504cbabcd28e0c4f08f9f8 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:52.970777 8128 consensus_queue.cc:1035] T 34120f4948504cbabcd28e0c4f08f9f8 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 124.795 ms
time per row: 0.0623976 ms
I20240503 01:59:53.024583 6307 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalance_test_table_13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:53.031345 6700 tablet_service.cc:1467] Processing CreateTablet for tablet 2b1d700d09534fa3adcf51bb3c97c8f5 (DEFAULT_TABLE table=rebalance_test_table_13 [id=029e2476b8c44a8d8cdb98005eb84a4b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:53.031345 6436 tablet_service.cc:1467] Processing CreateTablet for tablet 1feaf29d8c6b4842829ce66e0f4ec23b (DEFAULT_TABLE table=rebalance_test_table_13 [id=029e2476b8c44a8d8cdb98005eb84a4b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:53.031553 6438 tablet_service.cc:1467] Processing CreateTablet for tablet 2b1d700d09534fa3adcf51bb3c97c8f5 (DEFAULT_TABLE table=rebalance_test_table_13 [id=029e2476b8c44a8d8cdb98005eb84a4b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:53.031695 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1feaf29d8c6b4842829ce66e0f4ec23b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:53.031706 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b1d700d09534fa3adcf51bb3c97c8f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:53.031399 6699 tablet_service.cc:1467] Processing CreateTablet for tablet d82268925a934c559dfcf51a84f03d2e (DEFAULT_TABLE table=rebalance_test_table_13 [id=029e2476b8c44a8d8cdb98005eb84a4b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:53.031884 6699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d82268925a934c559dfcf51a84f03d2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:53.032014 6829 tablet_service.cc:1467] Processing CreateTablet for tablet 1feaf29d8c6b4842829ce66e0f4ec23b (DEFAULT_TABLE table=rebalance_test_table_13 [id=029e2476b8c44a8d8cdb98005eb84a4b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:53.032279 6829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1feaf29d8c6b4842829ce66e0f4ec23b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:53.031345 6698 tablet_service.cc:1467] Processing CreateTablet for tablet 1feaf29d8c6b4842829ce66e0f4ec23b (DEFAULT_TABLE table=rebalance_test_table_13 [id=029e2476b8c44a8d8cdb98005eb84a4b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:53.032996 6698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1feaf29d8c6b4842829ce66e0f4ec23b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:53.034462 6830 tablet_service.cc:1467] Processing CreateTablet for tablet 2b1d700d09534fa3adcf51bb3c97c8f5 (DEFAULT_TABLE table=rebalance_test_table_13 [id=029e2476b8c44a8d8cdb98005eb84a4b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:53.034741 6830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b1d700d09534fa3adcf51bb3c97c8f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:53.035169 6438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b1d700d09534fa3adcf51bb3c97c8f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:53.037600 6568 tablet_service.cc:1467] Processing CreateTablet for tablet d82268925a934c559dfcf51a84f03d2e (DEFAULT_TABLE table=rebalance_test_table_13 [id=029e2476b8c44a8d8cdb98005eb84a4b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:53.037781 6436 tablet_service.cc:1467] Processing CreateTablet for tablet d82268925a934c559dfcf51a84f03d2e (DEFAULT_TABLE table=rebalance_test_table_13 [id=029e2476b8c44a8d8cdb98005eb84a4b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:53.037925 6568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d82268925a934c559dfcf51a84f03d2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:53.037993 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d82268925a934c559dfcf51a84f03d2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:53.040035 8319 tablet_bootstrap.cc:492] T 1feaf29d8c6b4842829ce66e0f4ec23b P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 01:59:53.040913 8318 tablet_bootstrap.cc:492] T d82268925a934c559dfcf51a84f03d2e P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 01:59:53.040971 8319 tablet_bootstrap.cc:654] T 1feaf29d8c6b4842829ce66e0f4ec23b P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:53.041858 8318 tablet_bootstrap.cc:654] T d82268925a934c559dfcf51a84f03d2e P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:53.042150 8319 tablet_bootstrap.cc:492] T 1feaf29d8c6b4842829ce66e0f4ec23b P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 01:59:53.042258 8319 ts_tablet_manager.cc:1367] T 1feaf29d8c6b4842829ce66e0f4ec23b P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:53.042695 8319 raft_consensus.cc:348] T 1feaf29d8c6b4842829ce66e0f4ec23b P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:53.042809 8319 raft_consensus.cc:374] T 1feaf29d8c6b4842829ce66e0f4ec23b P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:53.042838 8319 raft_consensus.cc:729] T 1feaf29d8c6b4842829ce66e0f4ec23b P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 01:59:53.042954 8319 consensus_queue.cc:260] T 1feaf29d8c6b4842829ce66e0f4ec23b P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:53.043219 8321 tablet_bootstrap.cc:492] T d82268925a934c559dfcf51a84f03d2e P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 01:59:53.043234 8319 ts_tablet_manager.cc:1398] T 1feaf29d8c6b4842829ce66e0f4ec23b P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:53.043397 8319 tablet_bootstrap.cc:492] T 2b1d700d09534fa3adcf51bb3c97c8f5 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 01:59:53.044176 8321 tablet_bootstrap.cc:654] T d82268925a934c559dfcf51a84f03d2e P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:53.044180 8319 tablet_bootstrap.cc:654] T 2b1d700d09534fa3adcf51bb3c97c8f5 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:53.044890 8320 tablet_bootstrap.cc:492] T 2b1d700d09534fa3adcf51bb3c97c8f5 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 01:59:53.045313 8321 tablet_bootstrap.cc:492] T d82268925a934c559dfcf51a84f03d2e P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 01:59:53.045439 8321 ts_tablet_manager.cc:1367] T d82268925a934c559dfcf51a84f03d2e P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:53.045786 8320 tablet_bootstrap.cc:654] T 2b1d700d09534fa3adcf51bb3c97c8f5 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:53.045934 8321 raft_consensus.cc:348] T d82268925a934c559dfcf51a84f03d2e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:53.046137 8321 raft_consensus.cc:374] T d82268925a934c559dfcf51a84f03d2e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:53.046219 8321 raft_consensus.cc:729] T d82268925a934c559dfcf51a84f03d2e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 01:59:53.046427 8321 consensus_queue.cc:260] T d82268925a934c559dfcf51a84f03d2e P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:53.046669 8319 tablet_bootstrap.cc:492] T 2b1d700d09534fa3adcf51bb3c97c8f5 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 01:59:53.046767 8319 ts_tablet_manager.cc:1367] T 2b1d700d09534fa3adcf51bb3c97c8f5 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 01:59:53.046826 8320 tablet_bootstrap.cc:492] T 2b1d700d09534fa3adcf51bb3c97c8f5 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 01:59:53.046905 8320 ts_tablet_manager.cc:1367] T 2b1d700d09534fa3adcf51bb3c97c8f5 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:53.047228 8319 raft_consensus.cc:348] T 2b1d700d09534fa3adcf51bb3c97c8f5 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:53.047329 8319 raft_consensus.cc:374] T 2b1d700d09534fa3adcf51bb3c97c8f5 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:53.047302 8320 raft_consensus.cc:348] T 2b1d700d09534fa3adcf51bb3c97c8f5 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:53.047384 8319 raft_consensus.cc:729] T 2b1d700d09534fa3adcf51bb3c97c8f5 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 01:59:53.047394 8320 raft_consensus.cc:374] T 2b1d700d09534fa3adcf51bb3c97c8f5 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:53.047423 8320 raft_consensus.cc:729] T 2b1d700d09534fa3adcf51bb3c97c8f5 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 01:59:53.047509 8319 consensus_queue.cc:260] T 2b1d700d09534fa3adcf51bb3c97c8f5 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:53.047536 8320 consensus_queue.cc:260] T 2b1d700d09534fa3adcf51bb3c97c8f5 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:53.047799 8320 ts_tablet_manager.cc:1398] T 2b1d700d09534fa3adcf51bb3c97c8f5 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:53.047796 8319 ts_tablet_manager.cc:1398] T 2b1d700d09534fa3adcf51bb3c97c8f5 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:53.047972 8320 tablet_bootstrap.cc:492] T 1feaf29d8c6b4842829ce66e0f4ec23b P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 01:59:53.047973 8319 tablet_bootstrap.cc:492] T d82268925a934c559dfcf51a84f03d2e P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 01:59:53.048712 8321 ts_tablet_manager.cc:1398] T d82268925a934c559dfcf51a84f03d2e P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 01:59:53.048959 8319 tablet_bootstrap.cc:654] T d82268925a934c559dfcf51a84f03d2e P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:53.048959 8320 tablet_bootstrap.cc:654] T 1feaf29d8c6b4842829ce66e0f4ec23b P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:53.050639 8318 tablet_bootstrap.cc:492] T d82268925a934c559dfcf51a84f03d2e P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 01:59:53.050767 8318 ts_tablet_manager.cc:1367] T d82268925a934c559dfcf51a84f03d2e P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 01:59:53.051306 8318 raft_consensus.cc:348] T d82268925a934c559dfcf51a84f03d2e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:53.051435 8318 raft_consensus.cc:374] T d82268925a934c559dfcf51a84f03d2e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:53.051467 8318 raft_consensus.cc:729] T d82268925a934c559dfcf51a84f03d2e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 01:59:53.051599 8318 consensus_queue.cc:260] T d82268925a934c559dfcf51a84f03d2e P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:53.051898 8318 ts_tablet_manager.cc:1398] T d82268925a934c559dfcf51a84f03d2e P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:53.052062 8318 tablet_bootstrap.cc:492] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 01:59:53.052240 8319 tablet_bootstrap.cc:492] T d82268925a934c559dfcf51a84f03d2e P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 01:59:53.052320 8319 ts_tablet_manager.cc:1367] T d82268925a934c559dfcf51a84f03d2e P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 01:59:53.052811 8319 raft_consensus.cc:348] T d82268925a934c559dfcf51a84f03d2e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:53.052960 8319 raft_consensus.cc:374] T d82268925a934c559dfcf51a84f03d2e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:53.052994 8319 raft_consensus.cc:729] T d82268925a934c559dfcf51a84f03d2e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 01:59:53.053082 8318 tablet_bootstrap.cc:654] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:53.053112 8319 consensus_queue.cc:260] T d82268925a934c559dfcf51a84f03d2e P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:53.053392 8319 ts_tablet_manager.cc:1398] T d82268925a934c559dfcf51a84f03d2e P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:53.054193 8320 tablet_bootstrap.cc:492] T 1feaf29d8c6b4842829ce66e0f4ec23b P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 01:59:53.054301 8320 ts_tablet_manager.cc:1367] T 1feaf29d8c6b4842829ce66e0f4ec23b P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 01:59:53.054775 8320 raft_consensus.cc:348] T 1feaf29d8c6b4842829ce66e0f4ec23b P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:53.054888 8320 raft_consensus.cc:374] T 1feaf29d8c6b4842829ce66e0f4ec23b P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:53.054917 8320 raft_consensus.cc:729] T 1feaf29d8c6b4842829ce66e0f4ec23b P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 01:59:53.055037 8320 consensus_queue.cc:260] T 1feaf29d8c6b4842829ce66e0f4ec23b P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:53.055322 8320 ts_tablet_manager.cc:1398] T 1feaf29d8c6b4842829ce66e0f4ec23b P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:53.056098 8318 tablet_bootstrap.cc:492] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 01:59:53.056206 8318 ts_tablet_manager.cc:1367] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:53.056749 8318 raft_consensus.cc:348] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:53.056874 8318 raft_consensus.cc:374] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:53.056907 8318 raft_consensus.cc:729] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 01:59:53.057060 8318 consensus_queue.cc:260] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:53.057420 8318 ts_tablet_manager.cc:1398] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:53.057622 8318 tablet_bootstrap.cc:492] T 2b1d700d09534fa3adcf51bb3c97c8f5 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 01:59:53.058694 8318 tablet_bootstrap.cc:654] T 2b1d700d09534fa3adcf51bb3c97c8f5 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:53.060261 8318 tablet_bootstrap.cc:492] T 2b1d700d09534fa3adcf51bb3c97c8f5 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 01:59:53.060369 8318 ts_tablet_manager.cc:1367] T 2b1d700d09534fa3adcf51bb3c97c8f5 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:53.060879 8318 raft_consensus.cc:348] T 2b1d700d09534fa3adcf51bb3c97c8f5 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:53.060994 8318 raft_consensus.cc:374] T 2b1d700d09534fa3adcf51bb3c97c8f5 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:53.061020 8318 raft_consensus.cc:729] T 2b1d700d09534fa3adcf51bb3c97c8f5 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 01:59:53.061164 8318 consensus_queue.cc:260] T 2b1d700d09534fa3adcf51bb3c97c8f5 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:53.061503 8318 ts_tablet_manager.cc:1398] T 2b1d700d09534fa3adcf51bb3c97c8f5 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:53.200716 8108 raft_consensus.cc:482] T 2b1d700d09534fa3adcf51bb3c97c8f5 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:53.200896 8108 raft_consensus.cc:504] T 2b1d700d09534fa3adcf51bb3c97c8f5 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:53.201339 8108 leader_election.cc:288] T 2b1d700d09534fa3adcf51bb3c97c8f5 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 01:59:53.201853 6718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b1d700d09534fa3adcf51bb3c97c8f5" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 01:59:53.201854 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b1d700d09534fa3adcf51bb3c97c8f5" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 01:59:53.202009 6718 raft_consensus.cc:2454] T 2b1d700d09534fa3adcf51bb3c97c8f5 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 01:59:53.202036 6849 raft_consensus.cc:2454] T 2b1d700d09534fa3adcf51bb3c97c8f5 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 01:59:53.202379 6394 leader_election.cc:302] T 2b1d700d09534fa3adcf51bb3c97c8f5 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:53.202566 8108 raft_consensus.cc:2781] T 2b1d700d09534fa3adcf51bb3c97c8f5 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:53.202643 8108 raft_consensus.cc:482] T 2b1d700d09534fa3adcf51bb3c97c8f5 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:53.202697 8108 raft_consensus.cc:3037] T 2b1d700d09534fa3adcf51bb3c97c8f5 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:53.203697 8108 raft_consensus.cc:504] T 2b1d700d09534fa3adcf51bb3c97c8f5 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:53.204072 8108 leader_election.cc:288] T 2b1d700d09534fa3adcf51bb3c97c8f5 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Requested vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 01:59:53.204617 6718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b1d700d09534fa3adcf51bb3c97c8f5" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 01:59:53.204622 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b1d700d09534fa3adcf51bb3c97c8f5" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 01:59:53.204751 6849 raft_consensus.cc:3037] T 2b1d700d09534fa3adcf51bb3c97c8f5 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:53.204764 6718 raft_consensus.cc:3037] T 2b1d700d09534fa3adcf51bb3c97c8f5 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:53.205585 6718 raft_consensus.cc:2454] T 2b1d700d09534fa3adcf51bb3c97c8f5 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 01:59:53.205734 6849 raft_consensus.cc:2454] T 2b1d700d09534fa3adcf51bb3c97c8f5 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 01:59:53.206231 6394 leader_election.cc:302] T 2b1d700d09534fa3adcf51bb3c97c8f5 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:53.206434 8108 raft_consensus.cc:2781] T 2b1d700d09534fa3adcf51bb3c97c8f5 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:53.206672 8108 raft_consensus.cc:686] T 2b1d700d09534fa3adcf51bb3c97c8f5 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Becoming Leader. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Running, Role: LEADER
I20240503 01:59:53.206831 8108 consensus_queue.cc:237] T 2b1d700d09534fa3adcf51bb3c97c8f5 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:53.209681 6309 catalog_manager.cc:5484] T 2b1d700d09534fa3adcf51bb3c97c8f5 P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: term changed from 0 to 1, leader changed from <none> to dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129). New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:53.230597 8294 raft_consensus.cc:482] T d82268925a934c559dfcf51a84f03d2e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:53.230731 8294 raft_consensus.cc:504] T d82268925a934c559dfcf51a84f03d2e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:53.231173 8294 leader_election.cc:288] T d82268925a934c559dfcf51a84f03d2e P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 01:59:53.231524 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d82268925a934c559dfcf51a84f03d2e" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 01:59:53.231532 6587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d82268925a934c559dfcf51a84f03d2e" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 01:59:53.231664 6587 raft_consensus.cc:2454] T d82268925a934c559dfcf51a84f03d2e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 01:59:53.231680 6458 raft_consensus.cc:2454] T d82268925a934c559dfcf51a84f03d2e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 01:59:53.231966 6655 leader_election.cc:302] T d82268925a934c559dfcf51a84f03d2e P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 01:59:53.232270 8294 raft_consensus.cc:2781] T d82268925a934c559dfcf51a84f03d2e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:53.232335 8294 raft_consensus.cc:482] T d82268925a934c559dfcf51a84f03d2e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:53.232370 8294 raft_consensus.cc:3037] T d82268925a934c559dfcf51a84f03d2e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:53.233511 8294 raft_consensus.cc:504] T d82268925a934c559dfcf51a84f03d2e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:53.233867 8294 leader_election.cc:288] T d82268925a934c559dfcf51a84f03d2e P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Requested vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 01:59:53.234236 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d82268925a934c559dfcf51a84f03d2e" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 01:59:53.234262 6587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d82268925a934c559dfcf51a84f03d2e" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 01:59:53.234349 6458 raft_consensus.cc:3037] T d82268925a934c559dfcf51a84f03d2e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:53.234387 6587 raft_consensus.cc:3037] T d82268925a934c559dfcf51a84f03d2e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:53.235179 6587 raft_consensus.cc:2454] T d82268925a934c559dfcf51a84f03d2e P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 01:59:53.235190 6458 raft_consensus.cc:2454] T d82268925a934c559dfcf51a84f03d2e P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 01:59:53.235510 6653 leader_election.cc:302] T d82268925a934c559dfcf51a84f03d2e P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2ac2eb7cc264047b9ab32fe3a4fc025, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:53.235687 8294 raft_consensus.cc:2781] T d82268925a934c559dfcf51a84f03d2e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:53.235778 8294 raft_consensus.cc:686] T d82268925a934c559dfcf51a84f03d2e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Becoming Leader. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Running, Role: LEADER
I20240503 01:59:53.235921 8294 consensus_queue.cc:237] T d82268925a934c559dfcf51a84f03d2e P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:53.238242 6309 catalog_manager.cc:5484] T d82268925a934c559dfcf51a84f03d2e P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:53.255668 8294 raft_consensus.cc:482] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:53.255800 8294 raft_consensus.cc:504] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:53.256332 8294 leader_election.cc:288] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 01:59:53.256738 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1feaf29d8c6b4842829ce66e0f4ec23b" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 01:59:53.256724 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1feaf29d8c6b4842829ce66e0f4ec23b" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 01:59:53.256870 6458 raft_consensus.cc:2454] T 1feaf29d8c6b4842829ce66e0f4ec23b P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 01:59:53.256882 6849 raft_consensus.cc:2454] T 1feaf29d8c6b4842829ce66e0f4ec23b P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 01:59:53.257210 6656 leader_election.cc:302] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 01:59:53.257373 8294 raft_consensus.cc:2781] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:53.257429 8294 raft_consensus.cc:482] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:53.257454 8294 raft_consensus.cc:3037] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:53.258206 8294 raft_consensus.cc:504] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:53.258674 8294 leader_election.cc:288] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Requested vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 01:59:53.259027 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1feaf29d8c6b4842829ce66e0f4ec23b" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 01:59:53.259195 6458 raft_consensus.cc:3037] T 1feaf29d8c6b4842829ce66e0f4ec23b P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:53.259162 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1feaf29d8c6b4842829ce66e0f4ec23b" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 01:59:53.259282 6849 raft_consensus.cc:3037] T 1feaf29d8c6b4842829ce66e0f4ec23b P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:53.260187 6849 raft_consensus.cc:2454] T 1feaf29d8c6b4842829ce66e0f4ec23b P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 01:59:53.260188 6458 raft_consensus.cc:2454] T 1feaf29d8c6b4842829ce66e0f4ec23b P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 01:59:53.260543 6656 leader_election.cc:302] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 01:59:53.260727 8294 raft_consensus.cc:2781] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:53.260823 8294 raft_consensus.cc:686] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Becoming Leader. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Running, Role: LEADER
I20240503 01:59:53.260980 8294 consensus_queue.cc:237] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:53.262935 6309 catalog_manager.cc:5484] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } }
W20240503 01:59:53.399317 8334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:53.399585 8334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:53.402896 8334 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_13'
I20240503 01:59:53.487394 6849 raft_consensus.cc:1261] T 2b1d700d09534fa3adcf51bb3c97c8f5 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:53.487394 6718 raft_consensus.cc:1261] T 2b1d700d09534fa3adcf51bb3c97c8f5 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:53.487834 8110 consensus_queue.cc:1035] T 2b1d700d09534fa3adcf51bb3c97c8f5 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:53.488420 8110 consensus_queue.cc:1035] T 2b1d700d09534fa3adcf51bb3c97c8f5 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:53.489295 6851 raft_consensus.cc:1261] T 1feaf29d8c6b4842829ce66e0f4ec23b P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:53.489840 8294 consensus_queue.cc:1035] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:53.493556 6458 raft_consensus.cc:1261] T 1feaf29d8c6b4842829ce66e0f4ec23b P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:53.495157 8294 consensus_queue.cc:1035] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:53.497933 6457 raft_consensus.cc:1261] T d82268925a934c559dfcf51a84f03d2e P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:53.498345 8034 consensus_queue.cc:1035] T d82268925a934c559dfcf51a84f03d2e P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:53.500326 6587 raft_consensus.cc:1261] T d82268925a934c559dfcf51a84f03d2e P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:53.501158 8294 consensus_queue.cc:1035] T d82268925a934c559dfcf51a84f03d2e P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 118.156 ms
time per row: 0.0590782 ms
I20240503 01:59:53.569471 6310 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalance_test_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:53.575718 6829 tablet_service.cc:1467] Processing CreateTablet for tablet 7a4a910a8b9d4134adebb52a73dfae1d (DEFAULT_TABLE table=rebalance_test_table_14 [id=7455f5480e88494b83e1ac1d180ca4e6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:53.576071 6829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a4a910a8b9d4134adebb52a73dfae1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:53.576643 6568 tablet_service.cc:1467] Processing CreateTablet for tablet 7a4a910a8b9d4134adebb52a73dfae1d (DEFAULT_TABLE table=rebalance_test_table_14 [id=7455f5480e88494b83e1ac1d180ca4e6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:53.576752 6567 tablet_service.cc:1467] Processing CreateTablet for tablet 70d0c699b33e44dcb7136f3d0456ce9c (DEFAULT_TABLE table=rebalance_test_table_14 [id=7455f5480e88494b83e1ac1d180ca4e6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:53.576913 6568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a4a910a8b9d4134adebb52a73dfae1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:53.577100 6436 tablet_service.cc:1467] Processing CreateTablet for tablet 7a4a910a8b9d4134adebb52a73dfae1d (DEFAULT_TABLE table=rebalance_test_table_14 [id=7455f5480e88494b83e1ac1d180ca4e6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:53.577179 6567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70d0c699b33e44dcb7136f3d0456ce9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:53.577350 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a4a910a8b9d4134adebb52a73dfae1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:53.578572 8385 tablet_bootstrap.cc:492] T 7a4a910a8b9d4134adebb52a73dfae1d P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 01:59:53.579245 6569 tablet_service.cc:1467] Processing CreateTablet for tablet ceca04391cd446ae932b3485d958237a (DEFAULT_TABLE table=rebalance_test_table_14 [id=7455f5480e88494b83e1ac1d180ca4e6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:53.579484 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ceca04391cd446ae932b3485d958237a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:53.579505 8385 tablet_bootstrap.cc:654] T 7a4a910a8b9d4134adebb52a73dfae1d P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:53.580192 8386 tablet_bootstrap.cc:492] T 7a4a910a8b9d4134adebb52a73dfae1d P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 01:59:53.580225 8387 tablet_bootstrap.cc:492] T 7a4a910a8b9d4134adebb52a73dfae1d P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 01:59:53.580845 6829 tablet_service.cc:1467] Processing CreateTablet for tablet ceca04391cd446ae932b3485d958237a (DEFAULT_TABLE table=rebalance_test_table_14 [id=7455f5480e88494b83e1ac1d180ca4e6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:53.581097 6829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ceca04391cd446ae932b3485d958237a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:53.581184 8387 tablet_bootstrap.cc:654] T 7a4a910a8b9d4134adebb52a73dfae1d P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:53.581353 8386 tablet_bootstrap.cc:654] T 7a4a910a8b9d4134adebb52a73dfae1d P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:53.581737 6438 tablet_service.cc:1467] Processing CreateTablet for tablet 70d0c699b33e44dcb7136f3d0456ce9c (DEFAULT_TABLE table=rebalance_test_table_14 [id=7455f5480e88494b83e1ac1d180ca4e6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:53.581984 6438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70d0c699b33e44dcb7136f3d0456ce9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:53.582386 8387 tablet_bootstrap.cc:492] T 7a4a910a8b9d4134adebb52a73dfae1d P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 01:59:53.582482 8387 ts_tablet_manager.cc:1367] T 7a4a910a8b9d4134adebb52a73dfae1d P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:53.582971 8387 raft_consensus.cc:348] T 7a4a910a8b9d4134adebb52a73dfae1d P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:53.583094 8387 raft_consensus.cc:374] T 7a4a910a8b9d4134adebb52a73dfae1d P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:53.583132 8387 raft_consensus.cc:729] T 7a4a910a8b9d4134adebb52a73dfae1d P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 01:59:53.583248 8387 consensus_queue.cc:260] T 7a4a910a8b9d4134adebb52a73dfae1d P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:53.583432 8386 tablet_bootstrap.cc:492] T 7a4a910a8b9d4134adebb52a73dfae1d P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 01:59:53.583381 6830 tablet_service.cc:1467] Processing CreateTablet for tablet 70d0c699b33e44dcb7136f3d0456ce9c (DEFAULT_TABLE table=rebalance_test_table_14 [id=7455f5480e88494b83e1ac1d180ca4e6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:53.583535 8387 ts_tablet_manager.cc:1398] T 7a4a910a8b9d4134adebb52a73dfae1d P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:53.583564 8386 ts_tablet_manager.cc:1367] T 7a4a910a8b9d4134adebb52a73dfae1d P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:53.583626 6830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70d0c699b33e44dcb7136f3d0456ce9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:53.584066 8385 tablet_bootstrap.cc:492] T 7a4a910a8b9d4134adebb52a73dfae1d P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 01:59:53.584151 8385 ts_tablet_manager.cc:1367] T 7a4a910a8b9d4134adebb52a73dfae1d P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 01:59:53.584106 8386 raft_consensus.cc:348] T 7a4a910a8b9d4134adebb52a73dfae1d P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:53.584203 8386 raft_consensus.cc:374] T 7a4a910a8b9d4134adebb52a73dfae1d P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:53.584244 8386 raft_consensus.cc:729] T 7a4a910a8b9d4134adebb52a73dfae1d P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 01:59:53.584360 8386 consensus_queue.cc:260] T 7a4a910a8b9d4134adebb52a73dfae1d P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:53.584609 8385 raft_consensus.cc:348] T 7a4a910a8b9d4134adebb52a73dfae1d P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:53.584705 8385 raft_consensus.cc:374] T 7a4a910a8b9d4134adebb52a73dfae1d P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:53.584745 8385 raft_consensus.cc:729] T 7a4a910a8b9d4134adebb52a73dfae1d P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 01:59:53.584872 8385 consensus_queue.cc:260] T 7a4a910a8b9d4134adebb52a73dfae1d P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:53.585137 8385 ts_tablet_manager.cc:1398] T 7a4a910a8b9d4134adebb52a73dfae1d P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:53.585301 8385 tablet_bootstrap.cc:492] T ceca04391cd446ae932b3485d958237a P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 01:59:53.586197 8385 tablet_bootstrap.cc:654] T ceca04391cd446ae932b3485d958237a P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:53.586481 6700 tablet_service.cc:1467] Processing CreateTablet for tablet ceca04391cd446ae932b3485d958237a (DEFAULT_TABLE table=rebalance_test_table_14 [id=7455f5480e88494b83e1ac1d180ca4e6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:53.586737 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ceca04391cd446ae932b3485d958237a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:53.587309 8385 tablet_bootstrap.cc:492] T ceca04391cd446ae932b3485d958237a P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 01:59:53.587395 8385 ts_tablet_manager.cc:1367] T ceca04391cd446ae932b3485d958237a P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:53.587873 8385 raft_consensus.cc:348] T ceca04391cd446ae932b3485d958237a P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:53.587992 8385 raft_consensus.cc:374] T ceca04391cd446ae932b3485d958237a P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:53.588023 8385 raft_consensus.cc:729] T ceca04391cd446ae932b3485d958237a P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 01:59:53.588155 8385 consensus_queue.cc:260] T ceca04391cd446ae932b3485d958237a P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:53.588603 8385 ts_tablet_manager.cc:1398] T ceca04391cd446ae932b3485d958237a P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:53.588783 8385 tablet_bootstrap.cc:492] T 70d0c699b33e44dcb7136f3d0456ce9c P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 01:59:53.588987 8387 tablet_bootstrap.cc:492] T 70d0c699b33e44dcb7136f3d0456ce9c P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 01:59:53.590013 8387 tablet_bootstrap.cc:654] T 70d0c699b33e44dcb7136f3d0456ce9c P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:53.590281 8386 ts_tablet_manager.cc:1398] T 7a4a910a8b9d4134adebb52a73dfae1d P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240503 01:59:53.590413 8385 tablet_bootstrap.cc:654] T 70d0c699b33e44dcb7136f3d0456ce9c P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:53.591077 8392 tablet_bootstrap.cc:492] T ceca04391cd446ae932b3485d958237a P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 01:59:53.591662 8385 tablet_bootstrap.cc:492] T 70d0c699b33e44dcb7136f3d0456ce9c P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 01:59:53.591768 8385 ts_tablet_manager.cc:1367] T 70d0c699b33e44dcb7136f3d0456ce9c P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:53.591885 8387 tablet_bootstrap.cc:492] T 70d0c699b33e44dcb7136f3d0456ce9c P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 01:59:53.591972 8387 ts_tablet_manager.cc:1367] T 70d0c699b33e44dcb7136f3d0456ce9c P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:53.592207 8386 tablet_bootstrap.cc:492] T 70d0c699b33e44dcb7136f3d0456ce9c P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 01:59:53.592227 8385 raft_consensus.cc:348] T 70d0c699b33e44dcb7136f3d0456ce9c P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:53.592341 8385 raft_consensus.cc:374] T 70d0c699b33e44dcb7136f3d0456ce9c P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:53.592393 8385 raft_consensus.cc:729] T 70d0c699b33e44dcb7136f3d0456ce9c P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 01:59:53.592402 8387 raft_consensus.cc:348] T 70d0c699b33e44dcb7136f3d0456ce9c P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:53.592511 8387 raft_consensus.cc:374] T 70d0c699b33e44dcb7136f3d0456ce9c P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:53.592545 8387 raft_consensus.cc:729] T 70d0c699b33e44dcb7136f3d0456ce9c P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 01:59:53.592515 8385 consensus_queue.cc:260] T 70d0c699b33e44dcb7136f3d0456ce9c P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:53.592669 8387 consensus_queue.cc:260] T 70d0c699b33e44dcb7136f3d0456ce9c P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:53.593039 8387 ts_tablet_manager.cc:1398] T 70d0c699b33e44dcb7136f3d0456ce9c P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:53.593307 8386 tablet_bootstrap.cc:654] T 70d0c699b33e44dcb7136f3d0456ce9c P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:53.593425 8385 ts_tablet_manager.cc:1398] T 70d0c699b33e44dcb7136f3d0456ce9c P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 01:59:53.594727 8386 tablet_bootstrap.cc:492] T 70d0c699b33e44dcb7136f3d0456ce9c P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 01:59:53.594832 8386 ts_tablet_manager.cc:1367] T 70d0c699b33e44dcb7136f3d0456ce9c P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:53.595085 8392 tablet_bootstrap.cc:654] T ceca04391cd446ae932b3485d958237a P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:53.595278 8386 raft_consensus.cc:348] T 70d0c699b33e44dcb7136f3d0456ce9c P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:53.595386 8386 raft_consensus.cc:374] T 70d0c699b33e44dcb7136f3d0456ce9c P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:53.595424 8386 raft_consensus.cc:729] T 70d0c699b33e44dcb7136f3d0456ce9c P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 01:59:53.595574 8386 consensus_queue.cc:260] T 70d0c699b33e44dcb7136f3d0456ce9c P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:53.595939 8386 ts_tablet_manager.cc:1398] T 70d0c699b33e44dcb7136f3d0456ce9c P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:53.596092 8386 tablet_bootstrap.cc:492] T ceca04391cd446ae932b3485d958237a P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 01:59:53.596117 8392 tablet_bootstrap.cc:492] T ceca04391cd446ae932b3485d958237a P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 01:59:53.596206 8392 ts_tablet_manager.cc:1367] T ceca04391cd446ae932b3485d958237a P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 01:59:53.596616 8392 raft_consensus.cc:348] T ceca04391cd446ae932b3485d958237a P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:53.596719 8392 raft_consensus.cc:374] T ceca04391cd446ae932b3485d958237a P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:53.596750 8392 raft_consensus.cc:729] T ceca04391cd446ae932b3485d958237a P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 01:59:53.596880 8392 consensus_queue.cc:260] T ceca04391cd446ae932b3485d958237a P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:53.597124 8386 tablet_bootstrap.cc:654] T ceca04391cd446ae932b3485d958237a P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:53.597178 8392 ts_tablet_manager.cc:1398] T ceca04391cd446ae932b3485d958237a P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:53.599242 8386 tablet_bootstrap.cc:492] T ceca04391cd446ae932b3485d958237a P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 01:59:53.599350 8386 ts_tablet_manager.cc:1367] T ceca04391cd446ae932b3485d958237a P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:53.599823 8386 raft_consensus.cc:348] T ceca04391cd446ae932b3485d958237a P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:53.599925 8386 raft_consensus.cc:374] T ceca04391cd446ae932b3485d958237a P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:53.599953 8386 raft_consensus.cc:729] T ceca04391cd446ae932b3485d958237a P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 01:59:53.600081 8386 consensus_queue.cc:260] T ceca04391cd446ae932b3485d958237a P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:53.600417 8386 ts_tablet_manager.cc:1398] T ceca04391cd446ae932b3485d958237a P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 01:59:53.637316 8034 raft_consensus.cc:482] T ceca04391cd446ae932b3485d958237a P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:53.637459 8034 raft_consensus.cc:504] T ceca04391cd446ae932b3485d958237a P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:53.637878 8034 leader_election.cc:288] T ceca04391cd446ae932b3485d958237a P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 01:59:53.638371 6587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceca04391cd446ae932b3485d958237a" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 01:59:53.638381 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceca04391cd446ae932b3485d958237a" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 01:59:53.638523 6587 raft_consensus.cc:2454] T ceca04391cd446ae932b3485d958237a P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 01:59:53.638545 6849 raft_consensus.cc:2454] T ceca04391cd446ae932b3485d958237a P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 01:59:53.638846 6655 leader_election.cc:302] T ceca04391cd446ae932b3485d958237a P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 01:59:53.639060 8034 raft_consensus.cc:2781] T ceca04391cd446ae932b3485d958237a P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:53.639123 8034 raft_consensus.cc:482] T ceca04391cd446ae932b3485d958237a P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:53.639161 8034 raft_consensus.cc:3037] T ceca04391cd446ae932b3485d958237a P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:53.640053 8034 raft_consensus.cc:504] T ceca04391cd446ae932b3485d958237a P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:53.640446 8034 leader_election.cc:288] T ceca04391cd446ae932b3485d958237a P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Requested vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 01:59:53.640858 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceca04391cd446ae932b3485d958237a" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 01:59:53.640887 6587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceca04391cd446ae932b3485d958237a" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 01:59:53.641016 6849 raft_consensus.cc:3037] T ceca04391cd446ae932b3485d958237a P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:53.641014 6587 raft_consensus.cc:3037] T ceca04391cd446ae932b3485d958237a P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:53.642010 6587 raft_consensus.cc:2454] T ceca04391cd446ae932b3485d958237a P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 01:59:53.642009 6849 raft_consensus.cc:2454] T ceca04391cd446ae932b3485d958237a P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 01:59:53.642365 6656 leader_election.cc:302] T ceca04391cd446ae932b3485d958237a P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 01:59:53.642554 8034 raft_consensus.cc:2781] T ceca04391cd446ae932b3485d958237a P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:53.642653 8034 raft_consensus.cc:686] T ceca04391cd446ae932b3485d958237a P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Becoming Leader. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Running, Role: LEADER
I20240503 01:59:53.642799 8034 consensus_queue.cc:237] T ceca04391cd446ae932b3485d958237a P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:53.644773 6315 catalog_manager.cc:5484] T ceca04391cd446ae932b3485d958237a P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:53.780499 8108 raft_consensus.cc:482] T 70d0c699b33e44dcb7136f3d0456ce9c P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:53.780638 8108 raft_consensus.cc:504] T 70d0c699b33e44dcb7136f3d0456ce9c P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:53.781073 8108 leader_election.cc:288] T 70d0c699b33e44dcb7136f3d0456ce9c P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 01:59:53.781569 6587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70d0c699b33e44dcb7136f3d0456ce9c" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 01:59:53.781558 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70d0c699b33e44dcb7136f3d0456ce9c" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 01:59:53.781697 6849 raft_consensus.cc:2454] T 70d0c699b33e44dcb7136f3d0456ce9c P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 01:59:53.781699 6587 raft_consensus.cc:2454] T 70d0c699b33e44dcb7136f3d0456ce9c P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 01:59:53.781989 6393 leader_election.cc:302] T 70d0c699b33e44dcb7136f3d0456ce9c P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:53.782161 8108 raft_consensus.cc:2781] T 70d0c699b33e44dcb7136f3d0456ce9c P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:53.782214 8108 raft_consensus.cc:482] T 70d0c699b33e44dcb7136f3d0456ce9c P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:53.782251 8108 raft_consensus.cc:3037] T 70d0c699b33e44dcb7136f3d0456ce9c P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:53.783103 8108 raft_consensus.cc:504] T 70d0c699b33e44dcb7136f3d0456ce9c P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:53.783486 8108 leader_election.cc:288] T 70d0c699b33e44dcb7136f3d0456ce9c P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Requested vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 01:59:53.783862 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70d0c699b33e44dcb7136f3d0456ce9c" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 01:59:53.783986 6849 raft_consensus.cc:3037] T 70d0c699b33e44dcb7136f3d0456ce9c P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:53.783999 6587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70d0c699b33e44dcb7136f3d0456ce9c" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 01:59:53.784123 6587 raft_consensus.cc:3037] T 70d0c699b33e44dcb7136f3d0456ce9c P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:53.784945 6849 raft_consensus.cc:2454] T 70d0c699b33e44dcb7136f3d0456ce9c P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 01:59:53.785066 6587 raft_consensus.cc:2454] T 70d0c699b33e44dcb7136f3d0456ce9c P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 01:59:53.785265 6394 leader_election.cc:302] T 70d0c699b33e44dcb7136f3d0456ce9c P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:53.785431 8108 raft_consensus.cc:2781] T 70d0c699b33e44dcb7136f3d0456ce9c P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:53.785521 8108 raft_consensus.cc:686] T 70d0c699b33e44dcb7136f3d0456ce9c P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Becoming Leader. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Running, Role: LEADER
I20240503 01:59:53.785765 8108 consensus_queue.cc:237] T 70d0c699b33e44dcb7136f3d0456ce9c P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:53.787974 6315 catalog_manager.cc:5484] T 70d0c699b33e44dcb7136f3d0456ce9c P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: term changed from 0 to 1, leader changed from <none> to dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129). New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:53.885071 8399 raft_consensus.cc:482] T 7a4a910a8b9d4134adebb52a73dfae1d P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:53.885238 8399 raft_consensus.cc:504] T 7a4a910a8b9d4134adebb52a73dfae1d P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:53.885674 8399 leader_election.cc:288] T 7a4a910a8b9d4134adebb52a73dfae1d P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 01:59:53.886060 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a4a910a8b9d4134adebb52a73dfae1d" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 01:59:53.886060 6457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a4a910a8b9d4134adebb52a73dfae1d" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 01:59:53.886195 6457 raft_consensus.cc:2454] T 7a4a910a8b9d4134adebb52a73dfae1d P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 01:59:53.886265 6849 raft_consensus.cc:2454] T 7a4a910a8b9d4134adebb52a73dfae1d P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 01:59:53.886493 6522 leader_election.cc:302] T 7a4a910a8b9d4134adebb52a73dfae1d P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:53.886672 8399 raft_consensus.cc:2781] T 7a4a910a8b9d4134adebb52a73dfae1d P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:53.886725 8399 raft_consensus.cc:482] T 7a4a910a8b9d4134adebb52a73dfae1d P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:53.886760 8399 raft_consensus.cc:3037] T 7a4a910a8b9d4134adebb52a73dfae1d P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:53.887604 8399 raft_consensus.cc:504] T 7a4a910a8b9d4134adebb52a73dfae1d P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:53.887935 8399 leader_election.cc:288] T 7a4a910a8b9d4134adebb52a73dfae1d P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Requested vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 01:59:53.888298 6457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a4a910a8b9d4134adebb52a73dfae1d" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 01:59:53.888447 6457 raft_consensus.cc:3037] T 7a4a910a8b9d4134adebb52a73dfae1d P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:53.889235 6457 raft_consensus.cc:2454] T 7a4a910a8b9d4134adebb52a73dfae1d P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 01:59:53.889310 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a4a910a8b9d4134adebb52a73dfae1d" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 01:59:53.889456 6849 raft_consensus.cc:3037] T 7a4a910a8b9d4134adebb52a73dfae1d P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:53.890198 6849 raft_consensus.cc:2454] T 7a4a910a8b9d4134adebb52a73dfae1d P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 01:59:53.890460 6522 leader_election.cc:302] T 7a4a910a8b9d4134adebb52a73dfae1d P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:53.890627 8399 raft_consensus.cc:2781] T 7a4a910a8b9d4134adebb52a73dfae1d P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:53.890712 8399 raft_consensus.cc:686] T 7a4a910a8b9d4134adebb52a73dfae1d P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Becoming Leader. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Running, Role: LEADER
I20240503 01:59:53.890844 8399 consensus_queue.cc:237] T 7a4a910a8b9d4134adebb52a73dfae1d P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:53.892930 6315 catalog_manager.cc:5484] T 7a4a910a8b9d4134adebb52a73dfae1d P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } }
W20240503 01:59:54.073626 8402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:54.073931 8402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:54.077505 8402 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_14'
I20240503 01:59:54.137634 8294 consensus_queue.cc:1035] T ceca04391cd446ae932b3485d958237a P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:54.154649 8294 consensus_queue.cc:1035] T ceca04391cd446ae932b3485d958237a P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:54.196818 6851 raft_consensus.cc:1261] T 7a4a910a8b9d4134adebb52a73dfae1d P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:54.196820 6849 raft_consensus.cc:1261] T 70d0c699b33e44dcb7136f3d0456ce9c P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:54.197412 8108 consensus_queue.cc:1035] T 70d0c699b33e44dcb7136f3d0456ce9c P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:54.197536 6587 raft_consensus.cc:1261] T 70d0c699b33e44dcb7136f3d0456ce9c P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:54.197682 8169 consensus_queue.cc:1035] T 7a4a910a8b9d4134adebb52a73dfae1d P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:54.197886 6457 raft_consensus.cc:1261] T 7a4a910a8b9d4134adebb52a73dfae1d P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:54.198020 8108 consensus_queue.cc:1035] T 70d0c699b33e44dcb7136f3d0456ce9c P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:54.198546 8399 consensus_queue.cc:1035] T 7a4a910a8b9d4134adebb52a73dfae1d P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:54.230682 8427 mvcc.cc:205] Tried to move back new op lower bound from 7023417729795219456 to 7023417728578588672. Current Snapshot: MvccSnapshot[applied={T|T < 7023417729795219456}]
INSERT report
rows total: 2000
time total: 154.826 ms
time per row: 0.0774131 ms
I20240503 01:59:54.279157 6312 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalance_test_table_15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:54.285392 6438 tablet_service.cc:1467] Processing CreateTablet for tablet 7df4c083550046e883e5e61d3d4e1572 (DEFAULT_TABLE table=rebalance_test_table_15 [id=4e740b0167ed439291b74d77653ec386]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:54.285459 6569 tablet_service.cc:1467] Processing CreateTablet for tablet 7df4c083550046e883e5e61d3d4e1572 (DEFAULT_TABLE table=rebalance_test_table_15 [id=4e740b0167ed439291b74d77653ec386]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:54.285776 6438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7df4c083550046e883e5e61d3d4e1572. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:54.285777 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7df4c083550046e883e5e61d3d4e1572. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:54.287585 8463 tablet_bootstrap.cc:492] T 7df4c083550046e883e5e61d3d4e1572 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 01:59:54.287705 8462 tablet_bootstrap.cc:492] T 7df4c083550046e883e5e61d3d4e1572 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 01:59:54.288213 6569 tablet_service.cc:1467] Processing CreateTablet for tablet c3485701bf3b48f5ae2e1568afeed133 (DEFAULT_TABLE table=rebalance_test_table_15 [id=4e740b0167ed439291b74d77653ec386]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:54.288477 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3485701bf3b48f5ae2e1568afeed133. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:54.288543 8463 tablet_bootstrap.cc:654] T 7df4c083550046e883e5e61d3d4e1572 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:54.288983 8462 tablet_bootstrap.cc:654] T 7df4c083550046e883e5e61d3d4e1572 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:54.289152 6700 tablet_service.cc:1467] Processing CreateTablet for tablet 7df4c083550046e883e5e61d3d4e1572 (DEFAULT_TABLE table=rebalance_test_table_15 [id=4e740b0167ed439291b74d77653ec386]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:54.289245 6698 tablet_service.cc:1467] Processing CreateTablet for tablet c3485701bf3b48f5ae2e1568afeed133 (DEFAULT_TABLE table=rebalance_test_table_15 [id=4e740b0167ed439291b74d77653ec386]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:54.289410 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7df4c083550046e883e5e61d3d4e1572. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:54.289775 6698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3485701bf3b48f5ae2e1568afeed133. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:54.291121 6830 tablet_service.cc:1467] Processing CreateTablet for tablet c3485701bf3b48f5ae2e1568afeed133 (DEFAULT_TABLE table=rebalance_test_table_15 [id=4e740b0167ed439291b74d77653ec386]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:54.291258 6829 tablet_service.cc:1467] Processing CreateTablet for tablet 485d1d89a8b94505969d663be92124cd (DEFAULT_TABLE table=rebalance_test_table_15 [id=4e740b0167ed439291b74d77653ec386]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:54.291391 6830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3485701bf3b48f5ae2e1568afeed133. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:54.291679 6699 tablet_service.cc:1467] Processing CreateTablet for tablet 485d1d89a8b94505969d663be92124cd (DEFAULT_TABLE table=rebalance_test_table_15 [id=4e740b0167ed439291b74d77653ec386]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:54.291901 6699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 485d1d89a8b94505969d663be92124cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:54.291990 6829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 485d1d89a8b94505969d663be92124cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:54.292129 6438 tablet_service.cc:1467] Processing CreateTablet for tablet 485d1d89a8b94505969d663be92124cd (DEFAULT_TABLE table=rebalance_test_table_15 [id=4e740b0167ed439291b74d77653ec386]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:54.292413 6438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 485d1d89a8b94505969d663be92124cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:54.294893 8466 tablet_bootstrap.cc:492] T c3485701bf3b48f5ae2e1568afeed133 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 01:59:54.294879 8465 tablet_bootstrap.cc:492] T 7df4c083550046e883e5e61d3d4e1572 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 01:59:54.295977 8463 tablet_bootstrap.cc:492] T 7df4c083550046e883e5e61d3d4e1572 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 01:59:54.296110 8463 ts_tablet_manager.cc:1367] T 7df4c083550046e883e5e61d3d4e1572 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20240503 01:59:54.296247 8465 tablet_bootstrap.cc:654] T 7df4c083550046e883e5e61d3d4e1572 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:54.296304 8466 tablet_bootstrap.cc:654] T c3485701bf3b48f5ae2e1568afeed133 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:54.296676 8463 raft_consensus.cc:348] T 7df4c083550046e883e5e61d3d4e1572 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:54.296816 8463 raft_consensus.cc:374] T 7df4c083550046e883e5e61d3d4e1572 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:54.296847 8463 raft_consensus.cc:729] T 7df4c083550046e883e5e61d3d4e1572 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 01:59:54.296979 8463 consensus_queue.cc:260] T 7df4c083550046e883e5e61d3d4e1572 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:54.297260 8463 ts_tablet_manager.cc:1398] T 7df4c083550046e883e5e61d3d4e1572 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 01:59:54.297425 8463 tablet_bootstrap.cc:492] T c3485701bf3b48f5ae2e1568afeed133 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 01:59:54.297667 8462 tablet_bootstrap.cc:492] T 7df4c083550046e883e5e61d3d4e1572 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 01:59:54.297755 8462 ts_tablet_manager.cc:1367] T 7df4c083550046e883e5e61d3d4e1572 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 01:59:54.298262 8463 tablet_bootstrap.cc:654] T c3485701bf3b48f5ae2e1568afeed133 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:54.298486 8466 tablet_bootstrap.cc:492] T c3485701bf3b48f5ae2e1568afeed133 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 01:59:54.298559 8466 ts_tablet_manager.cc:1367] T c3485701bf3b48f5ae2e1568afeed133 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:54.298858 8465 tablet_bootstrap.cc:492] T 7df4c083550046e883e5e61d3d4e1572 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 01:59:54.298938 8465 ts_tablet_manager.cc:1367] T 7df4c083550046e883e5e61d3d4e1572 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:54.299166 8466 raft_consensus.cc:348] T c3485701bf3b48f5ae2e1568afeed133 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:54.299263 8466 raft_consensus.cc:374] T c3485701bf3b48f5ae2e1568afeed133 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:54.299283 8466 raft_consensus.cc:729] T c3485701bf3b48f5ae2e1568afeed133 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 01:59:54.299396 8466 consensus_queue.cc:260] T c3485701bf3b48f5ae2e1568afeed133 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:54.299446 8465 raft_consensus.cc:348] T 7df4c083550046e883e5e61d3d4e1572 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:54.299548 8465 raft_consensus.cc:374] T 7df4c083550046e883e5e61d3d4e1572 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:54.299576 8465 raft_consensus.cc:729] T 7df4c083550046e883e5e61d3d4e1572 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 01:59:54.300062 8465 consensus_queue.cc:260] T 7df4c083550046e883e5e61d3d4e1572 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:54.300432 8463 tablet_bootstrap.cc:492] T c3485701bf3b48f5ae2e1568afeed133 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 01:59:54.300518 8463 ts_tablet_manager.cc:1367] T c3485701bf3b48f5ae2e1568afeed133 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 01:59:54.300530 8466 ts_tablet_manager.cc:1398] T c3485701bf3b48f5ae2e1568afeed133 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:54.300552 8465 ts_tablet_manager.cc:1398] T 7df4c083550046e883e5e61d3d4e1572 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:54.300715 8465 tablet_bootstrap.cc:492] T c3485701bf3b48f5ae2e1568afeed133 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 01:59:54.300716 8466 tablet_bootstrap.cc:492] T 485d1d89a8b94505969d663be92124cd P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 01:59:54.300968 8463 raft_consensus.cc:348] T c3485701bf3b48f5ae2e1568afeed133 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:54.301069 8463 raft_consensus.cc:374] T c3485701bf3b48f5ae2e1568afeed133 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:54.301095 8463 raft_consensus.cc:729] T c3485701bf3b48f5ae2e1568afeed133 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 01:59:54.301353 8463 consensus_queue.cc:260] T c3485701bf3b48f5ae2e1568afeed133 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:54.301798 8466 tablet_bootstrap.cc:654] T 485d1d89a8b94505969d663be92124cd P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:54.302403 8463 ts_tablet_manager.cc:1398] T c3485701bf3b48f5ae2e1568afeed133 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:54.302664 8462 raft_consensus.cc:348] T 7df4c083550046e883e5e61d3d4e1572 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:54.303256 8466 tablet_bootstrap.cc:492] T 485d1d89a8b94505969d663be92124cd P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 01:59:54.303354 8462 raft_consensus.cc:374] T 7df4c083550046e883e5e61d3d4e1572 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:54.303480 8465 tablet_bootstrap.cc:654] T c3485701bf3b48f5ae2e1568afeed133 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:54.303442 8462 raft_consensus.cc:729] T 7df4c083550046e883e5e61d3d4e1572 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 01:59:54.304169 8462 consensus_queue.cc:260] T 7df4c083550046e883e5e61d3d4e1572 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:54.304534 8466 ts_tablet_manager.cc:1367] T 485d1d89a8b94505969d663be92124cd P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:54.305685 8462 ts_tablet_manager.cc:1398] T 7df4c083550046e883e5e61d3d4e1572 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240503 01:59:54.306177 8462 tablet_bootstrap.cc:492] T 485d1d89a8b94505969d663be92124cd P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 01:59:54.305109 8466 raft_consensus.cc:348] T 485d1d89a8b94505969d663be92124cd P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:54.306403 8466 raft_consensus.cc:374] T 485d1d89a8b94505969d663be92124cd P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:54.306459 8466 raft_consensus.cc:729] T 485d1d89a8b94505969d663be92124cd P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 01:59:54.306567 8465 tablet_bootstrap.cc:492] T c3485701bf3b48f5ae2e1568afeed133 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 01:59:54.306596 8466 consensus_queue.cc:260] T 485d1d89a8b94505969d663be92124cd P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:54.306671 8465 ts_tablet_manager.cc:1367] T c3485701bf3b48f5ae2e1568afeed133 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 01:59:54.306875 8466 ts_tablet_manager.cc:1398] T 485d1d89a8b94505969d663be92124cd P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:54.307137 8465 raft_consensus.cc:348] T c3485701bf3b48f5ae2e1568afeed133 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:54.307250 8465 raft_consensus.cc:374] T c3485701bf3b48f5ae2e1568afeed133 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:54.307279 8465 raft_consensus.cc:729] T c3485701bf3b48f5ae2e1568afeed133 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 01:59:54.307397 8465 consensus_queue.cc:260] T c3485701bf3b48f5ae2e1568afeed133 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:54.307718 8465 ts_tablet_manager.cc:1398] T c3485701bf3b48f5ae2e1568afeed133 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:54.307873 8465 tablet_bootstrap.cc:492] T 485d1d89a8b94505969d663be92124cd P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 01:59:54.307951 8462 tablet_bootstrap.cc:654] T 485d1d89a8b94505969d663be92124cd P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:54.308810 8465 tablet_bootstrap.cc:654] T 485d1d89a8b94505969d663be92124cd P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:54.309774 8462 tablet_bootstrap.cc:492] T 485d1d89a8b94505969d663be92124cd P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 01:59:54.309865 8462 ts_tablet_manager.cc:1367] T 485d1d89a8b94505969d663be92124cd P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:54.310132 8465 tablet_bootstrap.cc:492] T 485d1d89a8b94505969d663be92124cd P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 01:59:54.310209 8465 ts_tablet_manager.cc:1367] T 485d1d89a8b94505969d663be92124cd P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 01:59:54.310340 8462 raft_consensus.cc:348] T 485d1d89a8b94505969d663be92124cd P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:54.310465 8462 raft_consensus.cc:374] T 485d1d89a8b94505969d663be92124cd P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:54.310501 8462 raft_consensus.cc:729] T 485d1d89a8b94505969d663be92124cd P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 01:59:54.310604 8465 raft_consensus.cc:348] T 485d1d89a8b94505969d663be92124cd P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:54.310631 8462 consensus_queue.cc:260] T 485d1d89a8b94505969d663be92124cd P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:54.310715 8465 raft_consensus.cc:374] T 485d1d89a8b94505969d663be92124cd P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:54.310750 8465 raft_consensus.cc:729] T 485d1d89a8b94505969d663be92124cd P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 01:59:54.310870 8465 consensus_queue.cc:260] T 485d1d89a8b94505969d663be92124cd P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:54.310914 8462 ts_tablet_manager.cc:1398] T 485d1d89a8b94505969d663be92124cd P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:54.311132 8465 ts_tablet_manager.cc:1398] T 485d1d89a8b94505969d663be92124cd P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:54.321295 8438 raft_consensus.cc:482] T 485d1d89a8b94505969d663be92124cd P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:54.321417 8438 raft_consensus.cc:504] T 485d1d89a8b94505969d663be92124cd P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:54.322033 8438 leader_election.cc:288] T 485d1d89a8b94505969d663be92124cd P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 01:59:54.322312 6457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "485d1d89a8b94505969d663be92124cd" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 01:59:54.322468 6457 raft_consensus.cc:2454] T 485d1d89a8b94505969d663be92124cd P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 01:59:54.322419 6850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "485d1d89a8b94505969d663be92124cd" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 01:59:54.322563 6850 raft_consensus.cc:2454] T 485d1d89a8b94505969d663be92124cd P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 01:59:54.322744 6653 leader_election.cc:302] T 485d1d89a8b94505969d663be92124cd P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2ac2eb7cc264047b9ab32fe3a4fc025, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:54.322927 8438 raft_consensus.cc:2781] T 485d1d89a8b94505969d663be92124cd P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:54.322997 8438 raft_consensus.cc:482] T 485d1d89a8b94505969d663be92124cd P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:54.323036 8438 raft_consensus.cc:3037] T 485d1d89a8b94505969d663be92124cd P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:54.323849 8438 raft_consensus.cc:504] T 485d1d89a8b94505969d663be92124cd P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:54.324205 8438 leader_election.cc:288] T 485d1d89a8b94505969d663be92124cd P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Requested vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 01:59:54.324615 6850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "485d1d89a8b94505969d663be92124cd" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 01:59:54.324749 6850 raft_consensus.cc:3037] T 485d1d89a8b94505969d663be92124cd P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:54.324797 6457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "485d1d89a8b94505969d663be92124cd" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 01:59:54.324920 6457 raft_consensus.cc:3037] T 485d1d89a8b94505969d663be92124cd P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:54.325604 6850 raft_consensus.cc:2454] T 485d1d89a8b94505969d663be92124cd P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 01:59:54.325680 6457 raft_consensus.cc:2454] T 485d1d89a8b94505969d663be92124cd P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 01:59:54.325946 6656 leader_election.cc:302] T 485d1d89a8b94505969d663be92124cd P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 01:59:54.326212 8438 raft_consensus.cc:2781] T 485d1d89a8b94505969d663be92124cd P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:54.326319 8438 raft_consensus.cc:686] T 485d1d89a8b94505969d663be92124cd P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Becoming Leader. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Running, Role: LEADER
I20240503 01:59:54.326493 8438 consensus_queue.cc:237] T 485d1d89a8b94505969d663be92124cd P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:54.328713 6312 catalog_manager.cc:5484] T 485d1d89a8b94505969d663be92124cd P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:54.396911 8459 raft_consensus.cc:482] T 7df4c083550046e883e5e61d3d4e1572 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:54.397065 8459 raft_consensus.cc:504] T 7df4c083550046e883e5e61d3d4e1572 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:54.397569 8459 leader_election.cc:288] T 7df4c083550046e883e5e61d3d4e1572 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 01:59:54.397879 6457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7df4c083550046e883e5e61d3d4e1572" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 01:59:54.398011 6457 raft_consensus.cc:2454] T 7df4c083550046e883e5e61d3d4e1572 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 01:59:54.398311 6522 leader_election.cc:302] T 7df4c083550046e883e5e61d3d4e1572 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:54.398504 8459 raft_consensus.cc:2781] T 7df4c083550046e883e5e61d3d4e1572 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:54.398564 8459 raft_consensus.cc:482] T 7df4c083550046e883e5e61d3d4e1572 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:54.398594 8459 raft_consensus.cc:3037] T 7df4c083550046e883e5e61d3d4e1572 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:54.399564 6718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7df4c083550046e883e5e61d3d4e1572" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 01:59:54.399721 6718 raft_consensus.cc:2454] T 7df4c083550046e883e5e61d3d4e1572 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 01:59:54.399758 8459 raft_consensus.cc:504] T 7df4c083550046e883e5e61d3d4e1572 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:54.400192 8459 leader_election.cc:288] T 7df4c083550046e883e5e61d3d4e1572 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Requested vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 01:59:54.400584 6718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7df4c083550046e883e5e61d3d4e1572" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 01:59:54.400704 6718 raft_consensus.cc:3037] T 7df4c083550046e883e5e61d3d4e1572 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:54.401679 6718 raft_consensus.cc:2454] T 7df4c083550046e883e5e61d3d4e1572 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 01:59:54.402158 6525 leader_election.cc:302] T 7df4c083550046e883e5e61d3d4e1572 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 01:59:54.402421 8459 raft_consensus.cc:2781] T 7df4c083550046e883e5e61d3d4e1572 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:54.402542 8459 raft_consensus.cc:686] T 7df4c083550046e883e5e61d3d4e1572 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Becoming Leader. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Running, Role: LEADER
I20240503 01:59:54.402686 8459 consensus_queue.cc:237] T 7df4c083550046e883e5e61d3d4e1572 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:54.404157 6457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7df4c083550046e883e5e61d3d4e1572" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 01:59:54.404299 6457 raft_consensus.cc:3037] T 7df4c083550046e883e5e61d3d4e1572 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:54.405752 6457 raft_consensus.cc:2454] T 7df4c083550046e883e5e61d3d4e1572 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 01:59:54.405865 6312 catalog_manager.cc:5484] T 7df4c083550046e883e5e61d3d4e1572 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:54.506165 8459 raft_consensus.cc:482] T c3485701bf3b48f5ae2e1568afeed133 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:54.506312 8459 raft_consensus.cc:504] T c3485701bf3b48f5ae2e1568afeed133 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:54.506754 8459 leader_election.cc:288] T c3485701bf3b48f5ae2e1568afeed133 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 01:59:54.507288 6850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3485701bf3b48f5ae2e1568afeed133" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 01:59:54.507321 6718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3485701bf3b48f5ae2e1568afeed133" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 01:59:54.507454 6718 raft_consensus.cc:2454] T c3485701bf3b48f5ae2e1568afeed133 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 01:59:54.507445 6850 raft_consensus.cc:2454] T c3485701bf3b48f5ae2e1568afeed133 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 01:59:54.507797 6525 leader_election.cc:302] T c3485701bf3b48f5ae2e1568afeed133 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 01:59:54.507956 8459 raft_consensus.cc:2781] T c3485701bf3b48f5ae2e1568afeed133 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:54.508010 8459 raft_consensus.cc:482] T c3485701bf3b48f5ae2e1568afeed133 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:54.508049 8459 raft_consensus.cc:3037] T c3485701bf3b48f5ae2e1568afeed133 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:54.509091 8459 raft_consensus.cc:504] T c3485701bf3b48f5ae2e1568afeed133 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:54.509420 8459 leader_election.cc:288] T c3485701bf3b48f5ae2e1568afeed133 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Requested vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 01:59:54.510020 6850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3485701bf3b48f5ae2e1568afeed133" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 01:59:54.510083 6718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3485701bf3b48f5ae2e1568afeed133" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 01:59:54.510140 6850 raft_consensus.cc:3037] T c3485701bf3b48f5ae2e1568afeed133 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:54.510210 6718 raft_consensus.cc:3037] T c3485701bf3b48f5ae2e1568afeed133 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:54.510948 6850 raft_consensus.cc:2454] T c3485701bf3b48f5ae2e1568afeed133 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 01:59:54.510948 6718 raft_consensus.cc:2454] T c3485701bf3b48f5ae2e1568afeed133 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 01:59:54.511297 6525 leader_election.cc:302] T c3485701bf3b48f5ae2e1568afeed133 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 80fd18ef4d604c86b909d5c008a46b4f; no voters:
I20240503 01:59:54.511498 8459 raft_consensus.cc:2781] T c3485701bf3b48f5ae2e1568afeed133 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:54.511698 8459 raft_consensus.cc:686] T c3485701bf3b48f5ae2e1568afeed133 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Becoming Leader. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Running, Role: LEADER
I20240503 01:59:54.511865 8459 consensus_queue.cc:237] T c3485701bf3b48f5ae2e1568afeed133 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:54.514473 6312 catalog_manager.cc:5484] T c3485701bf3b48f5ae2e1568afeed133 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } }
W20240503 01:59:54.654260 8478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:54.654531 8478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:54.657788 8478 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_15'
I20240503 01:59:54.718290 8294 consensus_queue.cc:1035] T 485d1d89a8b94505969d663be92124cd P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:54.725427 8034 consensus_queue.cc:1035] T 485d1d89a8b94505969d663be92124cd P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:54.756758 6458 raft_consensus.cc:1261] T 7df4c083550046e883e5e61d3d4e1572 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:54.757284 8459 consensus_queue.cc:1035] T 7df4c083550046e883e5e61d3d4e1572 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:54.760993 6718 raft_consensus.cc:1261] T 7df4c083550046e883e5e61d3d4e1572 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:54.761322 6720 raft_consensus.cc:1261] T c3485701bf3b48f5ae2e1568afeed133 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:54.761951 8459 consensus_queue.cc:1035] T 7df4c083550046e883e5e61d3d4e1572 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:54.763758 6850 raft_consensus.cc:1261] T c3485701bf3b48f5ae2e1568afeed133 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:54.764536 8399 consensus_queue.cc:1035] T c3485701bf3b48f5ae2e1568afeed133 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:54.766121 8399 consensus_queue.cc:1035] T c3485701bf3b48f5ae2e1568afeed133 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 126.261 ms
time per row: 0.0631303 ms
I20240503 01:59:54.831633 6312 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalance_test_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:54.838078 6569 tablet_service.cc:1467] Processing CreateTablet for tablet 9b5c2152b64449de9a4a0990a15ee3f8 (DEFAULT_TABLE table=rebalance_test_table_16 [id=eecab7e70bfa4621832bb99bd9ef066c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:54.838464 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b5c2152b64449de9a4a0990a15ee3f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:54.838606 6699 tablet_service.cc:1467] Processing CreateTablet for tablet 0b659e9d57db4ebf93c9143938970090 (DEFAULT_TABLE table=rebalance_test_table_16 [id=eecab7e70bfa4621832bb99bd9ef066c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:54.838744 6829 tablet_service.cc:1467] Processing CreateTablet for tablet 9b5c2152b64449de9a4a0990a15ee3f8 (DEFAULT_TABLE table=rebalance_test_table_16 [id=eecab7e70bfa4621832bb99bd9ef066c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:54.839013 6699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b659e9d57db4ebf93c9143938970090. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:54.839082 6829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b5c2152b64449de9a4a0990a15ee3f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:54.840654 8531 tablet_bootstrap.cc:492] T 9b5c2152b64449de9a4a0990a15ee3f8 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 01:59:54.840835 6438 tablet_service.cc:1467] Processing CreateTablet for tablet 9b5c2152b64449de9a4a0990a15ee3f8 (DEFAULT_TABLE table=rebalance_test_table_16 [id=eecab7e70bfa4621832bb99bd9ef066c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:54.840997 6436 tablet_service.cc:1467] Processing CreateTablet for tablet 0b659e9d57db4ebf93c9143938970090 (DEFAULT_TABLE table=rebalance_test_table_16 [id=eecab7e70bfa4621832bb99bd9ef066c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:54.841135 6438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b5c2152b64449de9a4a0990a15ee3f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:54.841974 8531 tablet_bootstrap.cc:654] T 9b5c2152b64449de9a4a0990a15ee3f8 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:54.842093 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b659e9d57db4ebf93c9143938970090. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:54.843366 8535 tablet_bootstrap.cc:492] T 9b5c2152b64449de9a4a0990a15ee3f8 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 01:59:54.843333 6699 tablet_service.cc:1467] Processing CreateTablet for tablet c85a74a168964629a6c2b527f0e02632 (DEFAULT_TABLE table=rebalance_test_table_16 [id=eecab7e70bfa4621832bb99bd9ef066c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:54.843581 6699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c85a74a168964629a6c2b527f0e02632. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:54.843569 6830 tablet_service.cc:1467] Processing CreateTablet for tablet c85a74a168964629a6c2b527f0e02632 (DEFAULT_TABLE table=rebalance_test_table_16 [id=eecab7e70bfa4621832bb99bd9ef066c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:54.843822 6830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c85a74a168964629a6c2b527f0e02632. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:54.844141 8532 tablet_bootstrap.cc:492] T 0b659e9d57db4ebf93c9143938970090 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 01:59:54.844265 8535 tablet_bootstrap.cc:654] T 9b5c2152b64449de9a4a0990a15ee3f8 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:54.845089 8532 tablet_bootstrap.cc:654] T 0b659e9d57db4ebf93c9143938970090 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:54.845592 8535 tablet_bootstrap.cc:492] T 9b5c2152b64449de9a4a0990a15ee3f8 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 01:59:54.845679 8535 ts_tablet_manager.cc:1367] T 9b5c2152b64449de9a4a0990a15ee3f8 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:54.846009 8532 tablet_bootstrap.cc:492] T 0b659e9d57db4ebf93c9143938970090 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 01:59:54.846096 8532 ts_tablet_manager.cc:1367] T 0b659e9d57db4ebf93c9143938970090 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:54.846143 8535 raft_consensus.cc:348] T 9b5c2152b64449de9a4a0990a15ee3f8 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:54.846253 8535 raft_consensus.cc:374] T 9b5c2152b64449de9a4a0990a15ee3f8 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:54.846285 8535 raft_consensus.cc:729] T 9b5c2152b64449de9a4a0990a15ee3f8 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 01:59:54.846410 8535 consensus_queue.cc:260] T 9b5c2152b64449de9a4a0990a15ee3f8 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:54.846477 8532 raft_consensus.cc:348] T 0b659e9d57db4ebf93c9143938970090 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:54.846573 8532 raft_consensus.cc:374] T 0b659e9d57db4ebf93c9143938970090 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:54.846603 8532 raft_consensus.cc:729] T 0b659e9d57db4ebf93c9143938970090 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 01:59:54.846686 8535 ts_tablet_manager.cc:1398] T 9b5c2152b64449de9a4a0990a15ee3f8 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:54.846719 8532 consensus_queue.cc:260] T 0b659e9d57db4ebf93c9143938970090 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:54.846853 8535 tablet_bootstrap.cc:492] T 0b659e9d57db4ebf93c9143938970090 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 01:59:54.846975 8532 ts_tablet_manager.cc:1398] T 0b659e9d57db4ebf93c9143938970090 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:54.847127 8532 tablet_bootstrap.cc:492] T c85a74a168964629a6c2b527f0e02632 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 01:59:54.847249 8533 tablet_bootstrap.cc:492] T 9b5c2152b64449de9a4a0990a15ee3f8 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 01:59:54.847855 8532 tablet_bootstrap.cc:654] T c85a74a168964629a6c2b527f0e02632 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:54.847875 8535 tablet_bootstrap.cc:654] T 0b659e9d57db4ebf93c9143938970090 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:54.848448 6569 tablet_service.cc:1467] Processing CreateTablet for tablet 0b659e9d57db4ebf93c9143938970090 (DEFAULT_TABLE table=rebalance_test_table_16 [id=eecab7e70bfa4621832bb99bd9ef066c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:54.848524 8533 tablet_bootstrap.cc:654] T 9b5c2152b64449de9a4a0990a15ee3f8 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:54.848711 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b659e9d57db4ebf93c9143938970090. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:54.848731 6568 tablet_service.cc:1467] Processing CreateTablet for tablet c85a74a168964629a6c2b527f0e02632 (DEFAULT_TABLE table=rebalance_test_table_16 [id=eecab7e70bfa4621832bb99bd9ef066c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:54.848910 6568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c85a74a168964629a6c2b527f0e02632. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:54.849917 8533 tablet_bootstrap.cc:492] T 9b5c2152b64449de9a4a0990a15ee3f8 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 01:59:54.850004 8533 ts_tablet_manager.cc:1367] T 9b5c2152b64449de9a4a0990a15ee3f8 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 01:59:54.850471 8533 raft_consensus.cc:348] T 9b5c2152b64449de9a4a0990a15ee3f8 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:54.850581 8533 raft_consensus.cc:374] T 9b5c2152b64449de9a4a0990a15ee3f8 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:54.850606 8533 raft_consensus.cc:729] T 9b5c2152b64449de9a4a0990a15ee3f8 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 01:59:54.850723 8533 consensus_queue.cc:260] T 9b5c2152b64449de9a4a0990a15ee3f8 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:54.850975 8533 ts_tablet_manager.cc:1398] T 9b5c2152b64449de9a4a0990a15ee3f8 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:54.851145 8533 tablet_bootstrap.cc:492] T c85a74a168964629a6c2b527f0e02632 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 01:59:54.851570 8532 tablet_bootstrap.cc:492] T c85a74a168964629a6c2b527f0e02632 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 01:59:54.851660 8532 ts_tablet_manager.cc:1367] T c85a74a168964629a6c2b527f0e02632 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 01:59:54.851753 8531 tablet_bootstrap.cc:492] T 9b5c2152b64449de9a4a0990a15ee3f8 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 01:59:54.851828 8531 ts_tablet_manager.cc:1367] T 9b5c2152b64449de9a4a0990a15ee3f8 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 01:59:54.852070 8533 tablet_bootstrap.cc:654] T c85a74a168964629a6c2b527f0e02632 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:54.852097 8532 raft_consensus.cc:348] T c85a74a168964629a6c2b527f0e02632 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:54.852200 8532 raft_consensus.cc:374] T c85a74a168964629a6c2b527f0e02632 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:54.852231 8532 raft_consensus.cc:729] T c85a74a168964629a6c2b527f0e02632 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 01:59:54.852232 8531 raft_consensus.cc:348] T 9b5c2152b64449de9a4a0990a15ee3f8 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:54.852320 8531 raft_consensus.cc:374] T 9b5c2152b64449de9a4a0990a15ee3f8 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:54.852347 8531 raft_consensus.cc:729] T 9b5c2152b64449de9a4a0990a15ee3f8 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 01:59:54.852355 8532 consensus_queue.cc:260] T c85a74a168964629a6c2b527f0e02632 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:54.852469 8531 consensus_queue.cc:260] T 9b5c2152b64449de9a4a0990a15ee3f8 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:54.852675 8532 ts_tablet_manager.cc:1398] T c85a74a168964629a6c2b527f0e02632 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:54.852707 8531 ts_tablet_manager.cc:1398] T 9b5c2152b64449de9a4a0990a15ee3f8 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:54.852885 8531 tablet_bootstrap.cc:492] T c85a74a168964629a6c2b527f0e02632 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 01:59:54.853053 8533 tablet_bootstrap.cc:492] T c85a74a168964629a6c2b527f0e02632 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 01:59:54.853125 8533 ts_tablet_manager.cc:1367] T c85a74a168964629a6c2b527f0e02632 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:54.853518 8533 raft_consensus.cc:348] T c85a74a168964629a6c2b527f0e02632 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:54.853622 8533 raft_consensus.cc:374] T c85a74a168964629a6c2b527f0e02632 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:54.853659 8531 tablet_bootstrap.cc:654] T c85a74a168964629a6c2b527f0e02632 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:54.853663 8533 raft_consensus.cc:729] T c85a74a168964629a6c2b527f0e02632 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 01:59:54.853778 8533 consensus_queue.cc:260] T c85a74a168964629a6c2b527f0e02632 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:54.854056 8533 ts_tablet_manager.cc:1398] T c85a74a168964629a6c2b527f0e02632 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:54.855909 8531 tablet_bootstrap.cc:492] T c85a74a168964629a6c2b527f0e02632 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 01:59:54.856004 8531 ts_tablet_manager.cc:1367] T c85a74a168964629a6c2b527f0e02632 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:54.856472 8531 raft_consensus.cc:348] T c85a74a168964629a6c2b527f0e02632 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:54.856580 8531 raft_consensus.cc:374] T c85a74a168964629a6c2b527f0e02632 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:54.856609 8531 raft_consensus.cc:729] T c85a74a168964629a6c2b527f0e02632 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 01:59:54.856703 8531 consensus_queue.cc:260] T c85a74a168964629a6c2b527f0e02632 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:54.856957 8531 ts_tablet_manager.cc:1398] T c85a74a168964629a6c2b527f0e02632 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:54.857089 8531 tablet_bootstrap.cc:492] T 0b659e9d57db4ebf93c9143938970090 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 01:59:54.857868 8531 tablet_bootstrap.cc:654] T 0b659e9d57db4ebf93c9143938970090 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:54.859051 8531 tablet_bootstrap.cc:492] T 0b659e9d57db4ebf93c9143938970090 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 01:59:54.859138 8531 ts_tablet_manager.cc:1367] T 0b659e9d57db4ebf93c9143938970090 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:54.859602 8531 raft_consensus.cc:348] T 0b659e9d57db4ebf93c9143938970090 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:54.859714 8531 raft_consensus.cc:374] T 0b659e9d57db4ebf93c9143938970090 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:54.859746 8531 raft_consensus.cc:729] T 0b659e9d57db4ebf93c9143938970090 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 01:59:54.859846 8531 consensus_queue.cc:260] T 0b659e9d57db4ebf93c9143938970090 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:54.859994 8535 tablet_bootstrap.cc:492] T 0b659e9d57db4ebf93c9143938970090 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 01:59:54.860088 8531 ts_tablet_manager.cc:1398] T 0b659e9d57db4ebf93c9143938970090 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:54.860095 8535 ts_tablet_manager.cc:1367] T 0b659e9d57db4ebf93c9143938970090 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 01:59:54.860553 8535 raft_consensus.cc:348] T 0b659e9d57db4ebf93c9143938970090 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:54.860661 8535 raft_consensus.cc:374] T 0b659e9d57db4ebf93c9143938970090 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:54.860685 8535 raft_consensus.cc:729] T 0b659e9d57db4ebf93c9143938970090 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 01:59:54.860814 8535 consensus_queue.cc:260] T 0b659e9d57db4ebf93c9143938970090 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:54.861124 8535 ts_tablet_manager.cc:1398] T 0b659e9d57db4ebf93c9143938970090 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:54.921976 8294 raft_consensus.cc:482] T c85a74a168964629a6c2b527f0e02632 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:54.922113 8294 raft_consensus.cc:504] T c85a74a168964629a6c2b527f0e02632 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:54.922511 8294 leader_election.cc:288] T c85a74a168964629a6c2b527f0e02632 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 01:59:54.922989 6850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85a74a168964629a6c2b527f0e02632" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 01:59:54.922982 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85a74a168964629a6c2b527f0e02632" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 01:59:54.923125 6850 raft_consensus.cc:2454] T c85a74a168964629a6c2b527f0e02632 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 01:59:54.923130 6589 raft_consensus.cc:2454] T c85a74a168964629a6c2b527f0e02632 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 01:59:54.923434 6655 leader_election.cc:302] T c85a74a168964629a6c2b527f0e02632 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 80fd18ef4d604c86b909d5c008a46b4f, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 01:59:54.923619 8294 raft_consensus.cc:2781] T c85a74a168964629a6c2b527f0e02632 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:54.923673 8294 raft_consensus.cc:482] T c85a74a168964629a6c2b527f0e02632 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:54.923698 8294 raft_consensus.cc:3037] T c85a74a168964629a6c2b527f0e02632 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:54.924542 8294 raft_consensus.cc:504] T c85a74a168964629a6c2b527f0e02632 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:54.924890 8294 leader_election.cc:288] T c85a74a168964629a6c2b527f0e02632 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Requested vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 01:59:54.925274 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85a74a168964629a6c2b527f0e02632" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 01:59:54.925422 6589 raft_consensus.cc:3037] T c85a74a168964629a6c2b527f0e02632 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:54.925383 6850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85a74a168964629a6c2b527f0e02632" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 01:59:54.925499 6850 raft_consensus.cc:3037] T c85a74a168964629a6c2b527f0e02632 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:54.926261 6850 raft_consensus.cc:2454] T c85a74a168964629a6c2b527f0e02632 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 01:59:54.926268 6589 raft_consensus.cc:2454] T c85a74a168964629a6c2b527f0e02632 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 01:59:54.926548 6656 leader_election.cc:302] T c85a74a168964629a6c2b527f0e02632 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 01:59:54.926738 8294 raft_consensus.cc:2781] T c85a74a168964629a6c2b527f0e02632 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:54.926841 8294 raft_consensus.cc:686] T c85a74a168964629a6c2b527f0e02632 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Becoming Leader. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Running, Role: LEADER
I20240503 01:59:54.926977 8294 consensus_queue.cc:237] T c85a74a168964629a6c2b527f0e02632 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:54.928854 6309 catalog_manager.cc:5484] T c85a74a168964629a6c2b527f0e02632 P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:55.004221 8110 raft_consensus.cc:482] T 0b659e9d57db4ebf93c9143938970090 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:55.004361 8110 raft_consensus.cc:504] T 0b659e9d57db4ebf93c9143938970090 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:55.004865 8110 leader_election.cc:288] T 0b659e9d57db4ebf93c9143938970090 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 01:59:55.005211 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b659e9d57db4ebf93c9143938970090" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 01:59:55.005306 6718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b659e9d57db4ebf93c9143938970090" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 01:59:55.005365 6589 raft_consensus.cc:2454] T 0b659e9d57db4ebf93c9143938970090 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 01:59:55.005441 6718 raft_consensus.cc:2454] T 0b659e9d57db4ebf93c9143938970090 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 01:59:55.005698 6393 leader_election.cc:302] T 0b659e9d57db4ebf93c9143938970090 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:55.005894 8110 raft_consensus.cc:2781] T 0b659e9d57db4ebf93c9143938970090 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:55.005960 8110 raft_consensus.cc:482] T 0b659e9d57db4ebf93c9143938970090 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:55.006003 8110 raft_consensus.cc:3037] T 0b659e9d57db4ebf93c9143938970090 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:55.006922 8110 raft_consensus.cc:504] T 0b659e9d57db4ebf93c9143938970090 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:55.007256 8110 leader_election.cc:288] T 0b659e9d57db4ebf93c9143938970090 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Requested vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 01:59:55.007589 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b659e9d57db4ebf93c9143938970090" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 01:59:55.007634 6718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b659e9d57db4ebf93c9143938970090" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 01:59:55.007732 6718 raft_consensus.cc:3037] T 0b659e9d57db4ebf93c9143938970090 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:55.007732 6589 raft_consensus.cc:3037] T 0b659e9d57db4ebf93c9143938970090 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:55.008700 6718 raft_consensus.cc:2454] T 0b659e9d57db4ebf93c9143938970090 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 01:59:55.008700 6589 raft_consensus.cc:2454] T 0b659e9d57db4ebf93c9143938970090 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 01:59:55.009037 6393 leader_election.cc:302] T 0b659e9d57db4ebf93c9143938970090 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:55.009238 8110 raft_consensus.cc:2781] T 0b659e9d57db4ebf93c9143938970090 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:55.009325 8110 raft_consensus.cc:686] T 0b659e9d57db4ebf93c9143938970090 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Becoming Leader. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Running, Role: LEADER
I20240503 01:59:55.009461 8110 consensus_queue.cc:237] T 0b659e9d57db4ebf93c9143938970090 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:55.011497 6309 catalog_manager.cc:5484] T 0b659e9d57db4ebf93c9143938970090 P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: term changed from 0 to 1, leader changed from <none> to dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129). New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:55.152900 8459 raft_consensus.cc:482] T 9b5c2152b64449de9a4a0990a15ee3f8 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:55.153043 8459 raft_consensus.cc:504] T 9b5c2152b64449de9a4a0990a15ee3f8 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:55.153441 8459 leader_election.cc:288] T 9b5c2152b64449de9a4a0990a15ee3f8 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 01:59:55.153877 6850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b5c2152b64449de9a4a0990a15ee3f8" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 01:59:55.153877 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b5c2152b64449de9a4a0990a15ee3f8" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 01:59:55.154023 6458 raft_consensus.cc:2454] T 9b5c2152b64449de9a4a0990a15ee3f8 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 01:59:55.154047 6850 raft_consensus.cc:2454] T 9b5c2152b64449de9a4a0990a15ee3f8 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 01:59:55.154327 6525 leader_election.cc:302] T 9b5c2152b64449de9a4a0990a15ee3f8 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 80fd18ef4d604c86b909d5c008a46b4f; no voters:
I20240503 01:59:55.154597 8459 raft_consensus.cc:2781] T 9b5c2152b64449de9a4a0990a15ee3f8 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:55.154676 8459 raft_consensus.cc:482] T 9b5c2152b64449de9a4a0990a15ee3f8 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:55.154717 8459 raft_consensus.cc:3037] T 9b5c2152b64449de9a4a0990a15ee3f8 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:55.155637 8459 raft_consensus.cc:504] T 9b5c2152b64449de9a4a0990a15ee3f8 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:55.156060 8459 leader_election.cc:288] T 9b5c2152b64449de9a4a0990a15ee3f8 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Requested vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 01:59:55.156459 6850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b5c2152b64449de9a4a0990a15ee3f8" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 01:59:55.156461 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b5c2152b64449de9a4a0990a15ee3f8" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 01:59:55.156594 6850 raft_consensus.cc:3037] T 9b5c2152b64449de9a4a0990a15ee3f8 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:55.156598 6458 raft_consensus.cc:3037] T 9b5c2152b64449de9a4a0990a15ee3f8 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:55.157356 6850 raft_consensus.cc:2454] T 9b5c2152b64449de9a4a0990a15ee3f8 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 01:59:55.157358 6458 raft_consensus.cc:2454] T 9b5c2152b64449de9a4a0990a15ee3f8 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 01:59:55.157627 6525 leader_election.cc:302] T 9b5c2152b64449de9a4a0990a15ee3f8 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 80fd18ef4d604c86b909d5c008a46b4f; no voters:
I20240503 01:59:55.157788 8459 raft_consensus.cc:2781] T 9b5c2152b64449de9a4a0990a15ee3f8 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:55.157872 8459 raft_consensus.cc:686] T 9b5c2152b64449de9a4a0990a15ee3f8 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Becoming Leader. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Running, Role: LEADER
I20240503 01:59:55.158010 8459 consensus_queue.cc:237] T 9b5c2152b64449de9a4a0990a15ee3f8 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:55.160102 6309 catalog_manager.cc:5484] T 9b5c2152b64449de9a4a0990a15ee3f8 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } }
W20240503 01:59:55.339577 8547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:55.339865 8547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:55.343271 8547 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 01:59:55.355217 8034 consensus_queue.cc:1035] T c85a74a168964629a6c2b527f0e02632 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:55.358871 8034 consensus_queue.cc:1035] T c85a74a168964629a6c2b527f0e02632 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_16'
I20240503 01:59:55.436189 6718 raft_consensus.cc:1261] T 0b659e9d57db4ebf93c9143938970090 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:55.436653 8110 consensus_queue.cc:1035] T 0b659e9d57db4ebf93c9143938970090 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:55.437247 6589 raft_consensus.cc:1261] T 0b659e9d57db4ebf93c9143938970090 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:55.437693 8110 consensus_queue.cc:1035] T 0b659e9d57db4ebf93c9143938970090 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:55.449199 6850 raft_consensus.cc:1261] T 9b5c2152b64449de9a4a0990a15ee3f8 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:55.449292 6458 raft_consensus.cc:1261] T 9b5c2152b64449de9a4a0990a15ee3f8 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:55.450590 8459 consensus_queue.cc:1035] T 9b5c2152b64449de9a4a0990a15ee3f8 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:55.450963 8459 consensus_queue.cc:1035] T 9b5c2152b64449de9a4a0990a15ee3f8 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 112.769 ms
time per row: 0.0563846 ms
I20240503 01:59:55.511737 6309 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalance_test_table_17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:55.518529 6436 tablet_service.cc:1467] Processing CreateTablet for tablet 3b189fba0d6c461694dda002a3c5eb2f (DEFAULT_TABLE table=rebalance_test_table_17 [id=5ccf0adaf2de4b79bbf38c053646e60e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:55.518754 6438 tablet_service.cc:1467] Processing CreateTablet for tablet 2d7244b0dc4d419cb5c3f3677acd70c7 (DEFAULT_TABLE table=rebalance_test_table_17 [id=5ccf0adaf2de4b79bbf38c053646e60e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:55.518894 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b189fba0d6c461694dda002a3c5eb2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:55.519012 6438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d7244b0dc4d419cb5c3f3677acd70c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:55.519276 6699 tablet_service.cc:1467] Processing CreateTablet for tablet 3b189fba0d6c461694dda002a3c5eb2f (DEFAULT_TABLE table=rebalance_test_table_17 [id=5ccf0adaf2de4b79bbf38c053646e60e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:55.519443 6569 tablet_service.cc:1467] Processing CreateTablet for tablet 4385d51d11574192883d01376e2d5620 (DEFAULT_TABLE table=rebalance_test_table_17 [id=5ccf0adaf2de4b79bbf38c053646e60e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:55.519520 6699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b189fba0d6c461694dda002a3c5eb2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:55.519722 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4385d51d11574192883d01376e2d5620. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:55.520648 6700 tablet_service.cc:1467] Processing CreateTablet for tablet 2d7244b0dc4d419cb5c3f3677acd70c7 (DEFAULT_TABLE table=rebalance_test_table_17 [id=5ccf0adaf2de4b79bbf38c053646e60e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:55.520879 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d7244b0dc4d419cb5c3f3677acd70c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:55.522053 6438 tablet_service.cc:1467] Processing CreateTablet for tablet 4385d51d11574192883d01376e2d5620 (DEFAULT_TABLE table=rebalance_test_table_17 [id=5ccf0adaf2de4b79bbf38c053646e60e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:55.522301 6438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4385d51d11574192883d01376e2d5620. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:55.522758 8613 tablet_bootstrap.cc:492] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 01:59:55.522967 8615 tablet_bootstrap.cc:492] T 2d7244b0dc4d419cb5c3f3677acd70c7 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 01:59:55.523658 8613 tablet_bootstrap.cc:654] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:55.523794 8615 tablet_bootstrap.cc:654] T 2d7244b0dc4d419cb5c3f3677acd70c7 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:55.524782 6830 tablet_service.cc:1467] Processing CreateTablet for tablet 3b189fba0d6c461694dda002a3c5eb2f (DEFAULT_TABLE table=rebalance_test_table_17 [id=5ccf0adaf2de4b79bbf38c053646e60e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:55.524802 6829 tablet_service.cc:1467] Processing CreateTablet for tablet 2d7244b0dc4d419cb5c3f3677acd70c7 (DEFAULT_TABLE table=rebalance_test_table_17 [id=5ccf0adaf2de4b79bbf38c053646e60e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:55.525141 6830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b189fba0d6c461694dda002a3c5eb2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:55.525496 8613 tablet_bootstrap.cc:492] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 01:59:55.525586 8613 ts_tablet_manager.cc:1367] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:55.525743 6831 tablet_service.cc:1467] Processing CreateTablet for tablet 4385d51d11574192883d01376e2d5620 (DEFAULT_TABLE table=rebalance_test_table_17 [id=5ccf0adaf2de4b79bbf38c053646e60e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:55.526075 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4385d51d11574192883d01376e2d5620. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:55.526212 8613 raft_consensus.cc:348] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:55.526321 8613 raft_consensus.cc:374] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:55.526352 8613 raft_consensus.cc:729] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 01:59:55.526476 8613 consensus_queue.cc:260] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:55.526744 8613 ts_tablet_manager.cc:1398] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:55.526922 8613 tablet_bootstrap.cc:492] T 3b189fba0d6c461694dda002a3c5eb2f P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 01:59:55.527459 6829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d7244b0dc4d419cb5c3f3677acd70c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:55.527902 8613 tablet_bootstrap.cc:654] T 3b189fba0d6c461694dda002a3c5eb2f P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:55.529047 8613 tablet_bootstrap.cc:492] T 3b189fba0d6c461694dda002a3c5eb2f P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 01:59:55.529138 8613 ts_tablet_manager.cc:1367] T 3b189fba0d6c461694dda002a3c5eb2f P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 01:59:55.529458 8615 tablet_bootstrap.cc:492] T 2d7244b0dc4d419cb5c3f3677acd70c7 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 01:59:55.529562 8615 ts_tablet_manager.cc:1367] T 2d7244b0dc4d419cb5c3f3677acd70c7 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240503 01:59:55.529579 8613 raft_consensus.cc:348] T 3b189fba0d6c461694dda002a3c5eb2f P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:55.529681 8613 raft_consensus.cc:374] T 3b189fba0d6c461694dda002a3c5eb2f P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:55.529711 8613 raft_consensus.cc:729] T 3b189fba0d6c461694dda002a3c5eb2f P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 01:59:55.529829 8613 consensus_queue.cc:260] T 3b189fba0d6c461694dda002a3c5eb2f P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:55.529894 8618 tablet_bootstrap.cc:492] T 3b189fba0d6c461694dda002a3c5eb2f P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 01:59:55.530115 8613 ts_tablet_manager.cc:1398] T 3b189fba0d6c461694dda002a3c5eb2f P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:55.530057 8615 raft_consensus.cc:348] T 2d7244b0dc4d419cb5c3f3677acd70c7 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:55.530159 8615 raft_consensus.cc:374] T 2d7244b0dc4d419cb5c3f3677acd70c7 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:55.530186 8615 raft_consensus.cc:729] T 2d7244b0dc4d419cb5c3f3677acd70c7 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 01:59:55.530279 8613 tablet_bootstrap.cc:492] T 4385d51d11574192883d01376e2d5620 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 01:59:55.530305 8615 consensus_queue.cc:260] T 2d7244b0dc4d419cb5c3f3677acd70c7 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:55.530586 8615 ts_tablet_manager.cc:1398] T 2d7244b0dc4d419cb5c3f3677acd70c7 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:55.530767 8615 tablet_bootstrap.cc:492] T 3b189fba0d6c461694dda002a3c5eb2f P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 01:59:55.530776 8618 tablet_bootstrap.cc:654] T 3b189fba0d6c461694dda002a3c5eb2f P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:55.531149 8613 tablet_bootstrap.cc:654] T 4385d51d11574192883d01376e2d5620 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:55.531505 8614 tablet_bootstrap.cc:492] T 4385d51d11574192883d01376e2d5620 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 01:59:55.531621 8615 tablet_bootstrap.cc:654] T 3b189fba0d6c461694dda002a3c5eb2f P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:55.532474 8614 tablet_bootstrap.cc:654] T 4385d51d11574192883d01376e2d5620 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:55.532774 8613 tablet_bootstrap.cc:492] T 4385d51d11574192883d01376e2d5620 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 01:59:55.532864 8613 ts_tablet_manager.cc:1367] T 4385d51d11574192883d01376e2d5620 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 01:59:55.534376 8613 raft_consensus.cc:348] T 4385d51d11574192883d01376e2d5620 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:55.534492 8613 raft_consensus.cc:374] T 4385d51d11574192883d01376e2d5620 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:55.534524 8613 raft_consensus.cc:729] T 4385d51d11574192883d01376e2d5620 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 01:59:55.534127 8618 tablet_bootstrap.cc:492] T 3b189fba0d6c461694dda002a3c5eb2f P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 01:59:55.534695 8618 ts_tablet_manager.cc:1367] T 3b189fba0d6c461694dda002a3c5eb2f P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 01:59:55.534658 8613 consensus_queue.cc:260] T 4385d51d11574192883d01376e2d5620 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:55.534955 8613 ts_tablet_manager.cc:1398] T 4385d51d11574192883d01376e2d5620 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:55.535269 8618 raft_consensus.cc:348] T 3b189fba0d6c461694dda002a3c5eb2f P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:55.535437 8618 raft_consensus.cc:374] T 3b189fba0d6c461694dda002a3c5eb2f P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:55.535527 8618 raft_consensus.cc:729] T 3b189fba0d6c461694dda002a3c5eb2f P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 01:59:55.535712 8618 consensus_queue.cc:260] T 3b189fba0d6c461694dda002a3c5eb2f P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:55.535879 8614 tablet_bootstrap.cc:492] T 4385d51d11574192883d01376e2d5620 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 01:59:55.536032 8614 ts_tablet_manager.cc:1367] T 4385d51d11574192883d01376e2d5620 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240503 01:59:55.536068 8618 ts_tablet_manager.cc:1398] T 3b189fba0d6c461694dda002a3c5eb2f P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:55.536294 8618 tablet_bootstrap.cc:492] T 4385d51d11574192883d01376e2d5620 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 01:59:55.536648 8614 raft_consensus.cc:348] T 4385d51d11574192883d01376e2d5620 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:55.536764 8615 tablet_bootstrap.cc:492] T 3b189fba0d6c461694dda002a3c5eb2f P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 01:59:55.536870 8615 ts_tablet_manager.cc:1367] T 3b189fba0d6c461694dda002a3c5eb2f P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 01:59:55.537245 8614 raft_consensus.cc:374] T 4385d51d11574192883d01376e2d5620 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:55.537353 8614 raft_consensus.cc:729] T 4385d51d11574192883d01376e2d5620 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 01:59:55.537308 8615 raft_consensus.cc:348] T 3b189fba0d6c461694dda002a3c5eb2f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:55.537406 8615 raft_consensus.cc:374] T 3b189fba0d6c461694dda002a3c5eb2f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:55.537446 8615 raft_consensus.cc:729] T 3b189fba0d6c461694dda002a3c5eb2f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 01:59:55.537567 8615 consensus_queue.cc:260] T 3b189fba0d6c461694dda002a3c5eb2f P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:55.537690 8618 tablet_bootstrap.cc:654] T 4385d51d11574192883d01376e2d5620 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:55.537638 8614 consensus_queue.cc:260] T 4385d51d11574192883d01376e2d5620 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:55.537858 8615 ts_tablet_manager.cc:1398] T 3b189fba0d6c461694dda002a3c5eb2f P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:55.538132 8614 ts_tablet_manager.cc:1398] T 4385d51d11574192883d01376e2d5620 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 01:59:55.542161 8618 tablet_bootstrap.cc:492] T 4385d51d11574192883d01376e2d5620 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 01:59:55.542357 8618 ts_tablet_manager.cc:1367] T 4385d51d11574192883d01376e2d5620 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 01:59:55.543045 8598 raft_consensus.cc:482] T 4385d51d11574192883d01376e2d5620 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:55.543143 8598 raft_consensus.cc:504] T 4385d51d11574192883d01376e2d5620 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:55.543556 8598 leader_election.cc:288] T 4385d51d11574192883d01376e2d5620 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 01:59:55.543908 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4385d51d11574192883d01376e2d5620" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 01:59:55.544057 6458 raft_consensus.cc:2454] T 4385d51d11574192883d01376e2d5620 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 01:59:55.544523 6522 leader_election.cc:302] T 4385d51d11574192883d01376e2d5620 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:55.544804 8598 raft_consensus.cc:2781] T 4385d51d11574192883d01376e2d5620 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:55.544862 8598 raft_consensus.cc:482] T 4385d51d11574192883d01376e2d5620 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:55.544896 8598 raft_consensus.cc:3037] T 4385d51d11574192883d01376e2d5620 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:55.543932 6850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4385d51d11574192883d01376e2d5620" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 01:59:55.542985 8618 raft_consensus.cc:348] T 4385d51d11574192883d01376e2d5620 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:55.545261 8618 raft_consensus.cc:374] T 4385d51d11574192883d01376e2d5620 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:55.545289 8618 raft_consensus.cc:729] T 4385d51d11574192883d01376e2d5620 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 01:59:55.545486 8618 consensus_queue.cc:260] T 4385d51d11574192883d01376e2d5620 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:55.545746 6850 raft_consensus.cc:2454] T 4385d51d11574192883d01376e2d5620 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 01:59:55.545879 8598 raft_consensus.cc:504] T 4385d51d11574192883d01376e2d5620 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:55.546218 8598 leader_election.cc:288] T 4385d51d11574192883d01376e2d5620 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Requested vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 01:59:55.546350 8618 ts_tablet_manager.cc:1398] T 4385d51d11574192883d01376e2d5620 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 01:59:55.546607 8618 tablet_bootstrap.cc:492] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 01:59:55.546924 6850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4385d51d11574192883d01376e2d5620" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 01:59:55.547051 6850 raft_consensus.cc:3037] T 4385d51d11574192883d01376e2d5620 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:55.547704 8618 tablet_bootstrap.cc:654] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:55.548769 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4385d51d11574192883d01376e2d5620" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 01:59:55.548907 6458 raft_consensus.cc:3037] T 4385d51d11574192883d01376e2d5620 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:55.549854 6458 raft_consensus.cc:2454] T 4385d51d11574192883d01376e2d5620 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 01:59:55.548007 6850 raft_consensus.cc:2454] T 4385d51d11574192883d01376e2d5620 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 01:59:55.550170 6522 leader_election.cc:302] T 4385d51d11574192883d01376e2d5620 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:55.550350 8618 tablet_bootstrap.cc:492] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 01:59:55.550348 8598 raft_consensus.cc:2781] T 4385d51d11574192883d01376e2d5620 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:55.550423 8598 raft_consensus.cc:686] T 4385d51d11574192883d01376e2d5620 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Becoming Leader. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Running, Role: LEADER
I20240503 01:59:55.550539 8618 ts_tablet_manager.cc:1367] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:55.550561 8598 consensus_queue.cc:237] T 4385d51d11574192883d01376e2d5620 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:55.551085 8618 raft_consensus.cc:348] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:55.551244 8618 raft_consensus.cc:374] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:55.551314 8618 raft_consensus.cc:729] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 01:59:55.551509 8618 consensus_queue.cc:260] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:55.551900 8618 ts_tablet_manager.cc:1398] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:55.553220 6307 catalog_manager.cc:5484] T 4385d51d11574192883d01376e2d5620 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:55.579036 8108 raft_consensus.cc:482] T 3b189fba0d6c461694dda002a3c5eb2f P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:55.579181 8108 raft_consensus.cc:504] T 3b189fba0d6c461694dda002a3c5eb2f P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:55.579618 8108 leader_election.cc:288] T 3b189fba0d6c461694dda002a3c5eb2f P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 01:59:55.580006 6850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b189fba0d6c461694dda002a3c5eb2f" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 01:59:55.580098 6718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b189fba0d6c461694dda002a3c5eb2f" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 01:59:55.580157 6850 raft_consensus.cc:2454] T 3b189fba0d6c461694dda002a3c5eb2f P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 01:59:55.580232 6718 raft_consensus.cc:2454] T 3b189fba0d6c461694dda002a3c5eb2f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 01:59:55.580485 6394 leader_election.cc:302] T 3b189fba0d6c461694dda002a3c5eb2f P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:55.580930 8108 raft_consensus.cc:2781] T 3b189fba0d6c461694dda002a3c5eb2f P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:55.580996 8108 raft_consensus.cc:482] T 3b189fba0d6c461694dda002a3c5eb2f P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:55.581043 8108 raft_consensus.cc:3037] T 3b189fba0d6c461694dda002a3c5eb2f P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:55.581933 8108 raft_consensus.cc:504] T 3b189fba0d6c461694dda002a3c5eb2f P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:55.582281 8108 leader_election.cc:288] T 3b189fba0d6c461694dda002a3c5eb2f P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Requested vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 01:59:55.582621 6850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b189fba0d6c461694dda002a3c5eb2f" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 01:59:55.582664 6718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b189fba0d6c461694dda002a3c5eb2f" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 01:59:55.582741 6850 raft_consensus.cc:3037] T 3b189fba0d6c461694dda002a3c5eb2f P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:55.582765 6718 raft_consensus.cc:3037] T 3b189fba0d6c461694dda002a3c5eb2f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:55.583554 6718 raft_consensus.cc:2454] T 3b189fba0d6c461694dda002a3c5eb2f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 01:59:55.583561 6850 raft_consensus.cc:2454] T 3b189fba0d6c461694dda002a3c5eb2f P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 01:59:55.583884 6394 leader_election.cc:302] T 3b189fba0d6c461694dda002a3c5eb2f P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2ac2eb7cc264047b9ab32fe3a4fc025, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:55.584089 8108 raft_consensus.cc:2781] T 3b189fba0d6c461694dda002a3c5eb2f P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:55.584190 8108 raft_consensus.cc:686] T 3b189fba0d6c461694dda002a3c5eb2f P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Becoming Leader. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Running, Role: LEADER
I20240503 01:59:55.584340 8108 consensus_queue.cc:237] T 3b189fba0d6c461694dda002a3c5eb2f P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:55.586618 6307 catalog_manager.cc:5484] T 3b189fba0d6c461694dda002a3c5eb2f P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: term changed from 0 to 1, leader changed from <none> to dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129). New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:55.903673 8110 raft_consensus.cc:482] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:55.903841 8110 raft_consensus.cc:504] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:55.904335 8110 leader_election.cc:288] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 01:59:55.904654 6718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7244b0dc4d419cb5c3f3677acd70c7" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 01:59:55.904804 6718 raft_consensus.cc:2454] T 2d7244b0dc4d419cb5c3f3677acd70c7 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 01:59:55.904774 6850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7244b0dc4d419cb5c3f3677acd70c7" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 01:59:55.904909 6850 raft_consensus.cc:2454] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 01:59:55.905107 6394 leader_election.cc:302] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2ac2eb7cc264047b9ab32fe3a4fc025, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:55.905263 8110 raft_consensus.cc:2781] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:55.905316 8110 raft_consensus.cc:482] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:55.905352 8110 raft_consensus.cc:3037] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:55.906188 8110 raft_consensus.cc:504] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:55.906522 8110 leader_election.cc:288] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Requested vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 01:59:55.906864 6718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7244b0dc4d419cb5c3f3677acd70c7" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 01:59:55.906926 6850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7244b0dc4d419cb5c3f3677acd70c7" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 01:59:55.906984 6718 raft_consensus.cc:3037] T 2d7244b0dc4d419cb5c3f3677acd70c7 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:55.907047 6850 raft_consensus.cc:3037] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:55.907809 6850 raft_consensus.cc:2454] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 01:59:55.907959 6718 raft_consensus.cc:2454] T 2d7244b0dc4d419cb5c3f3677acd70c7 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 01:59:55.908169 6394 leader_election.cc:302] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:55.908401 8110 raft_consensus.cc:2781] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:55.908528 8110 raft_consensus.cc:686] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Becoming Leader. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Running, Role: LEADER
I20240503 01:59:55.908689 8110 consensus_queue.cc:237] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 01:59:55.910867 6307 catalog_manager.cc:5484] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: term changed from 0 to 1, leader changed from <none> to dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129). New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:55.956787 8598 consensus_queue.cc:1035] T 4385d51d11574192883d01376e2d5620 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:55.960443 8459 consensus_queue.cc:1035] T 4385d51d11574192883d01376e2d5620 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:56.066771 8110 consensus_queue.cc:1035] T 3b189fba0d6c461694dda002a3c5eb2f P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:56.073292 8633 consensus_queue.cc:1035] T 3b189fba0d6c461694dda002a3c5eb2f P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 01:59:56.114097 8631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:56.114385 8631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:56.118311 8631 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_17'
I20240503 01:59:56.220451 6851 raft_consensus.cc:1261] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:56.220983 8110 consensus_queue.cc:1035] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:56.221661 6718 raft_consensus.cc:1261] T 2d7244b0dc4d419cb5c3f3677acd70c7 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:56.225348 8110 consensus_queue.cc:1035] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 128.766 ms
time per row: 0.0643832 ms
I20240503 01:59:56.291961 6316 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalance_test_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:56.298643 6438 tablet_service.cc:1467] Processing CreateTablet for tablet af00a839bb5e402791f87288493efef6 (DEFAULT_TABLE table=rebalance_test_table_18 [id=d732c32bc9004748904a109f59db08fd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:56.298979 6438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af00a839bb5e402791f87288493efef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:56.299394 6436 tablet_service.cc:1467] Processing CreateTablet for tablet 9f7ade1e2af24e86b6e54e2b5769e4c0 (DEFAULT_TABLE table=rebalance_test_table_18 [id=d732c32bc9004748904a109f59db08fd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:56.299643 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f7ade1e2af24e86b6e54e2b5769e4c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:56.301260 8681 tablet_bootstrap.cc:492] T af00a839bb5e402791f87288493efef6 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 01:59:56.301441 6829 tablet_service.cc:1467] Processing CreateTablet for tablet 4c13642db7fc40a5a934aa5e3c2b08fc (DEFAULT_TABLE table=rebalance_test_table_18 [id=d732c32bc9004748904a109f59db08fd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:56.301733 6829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c13642db7fc40a5a934aa5e3c2b08fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:56.302120 8681 tablet_bootstrap.cc:654] T af00a839bb5e402791f87288493efef6 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:56.303954 8683 tablet_bootstrap.cc:492] T 4c13642db7fc40a5a934aa5e3c2b08fc P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 01:59:56.304561 6569 tablet_service.cc:1467] Processing CreateTablet for tablet af00a839bb5e402791f87288493efef6 (DEFAULT_TABLE table=rebalance_test_table_18 [id=d732c32bc9004748904a109f59db08fd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:56.304612 6567 tablet_service.cc:1467] Processing CreateTablet for tablet 9f7ade1e2af24e86b6e54e2b5769e4c0 (DEFAULT_TABLE table=rebalance_test_table_18 [id=d732c32bc9004748904a109f59db08fd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:56.304893 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af00a839bb5e402791f87288493efef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:56.304965 6698 tablet_service.cc:1467] Processing CreateTablet for tablet 9f7ade1e2af24e86b6e54e2b5769e4c0 (DEFAULT_TABLE table=rebalance_test_table_18 [id=d732c32bc9004748904a109f59db08fd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:56.305203 6698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f7ade1e2af24e86b6e54e2b5769e4c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:56.305181 6699 tablet_service.cc:1467] Processing CreateTablet for tablet af00a839bb5e402791f87288493efef6 (DEFAULT_TABLE table=rebalance_test_table_18 [id=d732c32bc9004748904a109f59db08fd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:56.305243 8683 tablet_bootstrap.cc:654] T 4c13642db7fc40a5a934aa5e3c2b08fc P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:56.305362 6699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af00a839bb5e402791f87288493efef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:56.306869 8685 tablet_bootstrap.cc:492] T af00a839bb5e402791f87288493efef6 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 01:59:56.307039 6568 tablet_service.cc:1467] Processing CreateTablet for tablet 4c13642db7fc40a5a934aa5e3c2b08fc (DEFAULT_TABLE table=rebalance_test_table_18 [id=d732c32bc9004748904a109f59db08fd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:56.307252 6568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c13642db7fc40a5a934aa5e3c2b08fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:56.307294 6700 tablet_service.cc:1467] Processing CreateTablet for tablet 4c13642db7fc40a5a934aa5e3c2b08fc (DEFAULT_TABLE table=rebalance_test_table_18 [id=d732c32bc9004748904a109f59db08fd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:56.307510 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c13642db7fc40a5a934aa5e3c2b08fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:56.307715 8685 tablet_bootstrap.cc:654] T af00a839bb5e402791f87288493efef6 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:56.309736 6567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f7ade1e2af24e86b6e54e2b5769e4c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:56.310285 8683 tablet_bootstrap.cc:492] T 4c13642db7fc40a5a934aa5e3c2b08fc P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 01:59:56.310376 8683 ts_tablet_manager.cc:1367] T 4c13642db7fc40a5a934aa5e3c2b08fc P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 01:59:56.310806 8683 raft_consensus.cc:348] T 4c13642db7fc40a5a934aa5e3c2b08fc P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:56.310914 8683 raft_consensus.cc:374] T 4c13642db7fc40a5a934aa5e3c2b08fc P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:56.310938 8683 raft_consensus.cc:729] T 4c13642db7fc40a5a934aa5e3c2b08fc P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 01:59:56.311039 8683 consensus_queue.cc:260] T 4c13642db7fc40a5a934aa5e3c2b08fc P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:56.311270 8683 ts_tablet_manager.cc:1398] T 4c13642db7fc40a5a934aa5e3c2b08fc P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:56.311689 8681 tablet_bootstrap.cc:492] T af00a839bb5e402791f87288493efef6 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 01:59:56.311805 8681 ts_tablet_manager.cc:1367] T af00a839bb5e402791f87288493efef6 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 01:59:56.312273 8681 raft_consensus.cc:348] T af00a839bb5e402791f87288493efef6 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:56.312471 8681 raft_consensus.cc:374] T af00a839bb5e402791f87288493efef6 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:56.312518 8681 raft_consensus.cc:729] T af00a839bb5e402791f87288493efef6 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 01:59:56.312851 8681 consensus_queue.cc:260] T af00a839bb5e402791f87288493efef6 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:56.313720 8681 ts_tablet_manager.cc:1398] T af00a839bb5e402791f87288493efef6 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:56.313948 8681 tablet_bootstrap.cc:492] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 01:59:56.316012 8681 tablet_bootstrap.cc:654] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:56.316711 8686 tablet_bootstrap.cc:492] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 01:59:56.317678 8686 tablet_bootstrap.cc:654] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:56.317673 8681 tablet_bootstrap.cc:492] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 01:59:56.317770 8681 ts_tablet_manager.cc:1367] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:56.318310 8681 raft_consensus.cc:348] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:56.318516 8681 raft_consensus.cc:374] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:56.318568 8681 raft_consensus.cc:729] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 01:59:56.318712 8681 consensus_queue.cc:260] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:56.318810 8686 tablet_bootstrap.cc:492] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 01:59:56.318899 8686 ts_tablet_manager.cc:1367] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:56.319351 8686 raft_consensus.cc:348] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:56.319519 8686 raft_consensus.cc:374] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:56.319599 8686 raft_consensus.cc:729] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 01:59:56.319727 8686 consensus_queue.cc:260] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:56.320015 8681 ts_tablet_manager.cc:1398] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:56.320061 8686 ts_tablet_manager.cc:1398] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:56.320204 8686 tablet_bootstrap.cc:492] T 4c13642db7fc40a5a934aa5e3c2b08fc P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 01:59:56.321023 8686 tablet_bootstrap.cc:654] T 4c13642db7fc40a5a934aa5e3c2b08fc P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:56.322121 8685 tablet_bootstrap.cc:492] T af00a839bb5e402791f87288493efef6 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 01:59:56.322553 8686 tablet_bootstrap.cc:492] T 4c13642db7fc40a5a934aa5e3c2b08fc P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 01:59:56.322695 8686 ts_tablet_manager.cc:1367] T 4c13642db7fc40a5a934aa5e3c2b08fc P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:56.323194 8686 raft_consensus.cc:348] T 4c13642db7fc40a5a934aa5e3c2b08fc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:56.323297 8686 raft_consensus.cc:374] T 4c13642db7fc40a5a934aa5e3c2b08fc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:56.323318 8686 raft_consensus.cc:729] T 4c13642db7fc40a5a934aa5e3c2b08fc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 01:59:56.323415 8686 consensus_queue.cc:260] T 4c13642db7fc40a5a934aa5e3c2b08fc P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:56.323694 8686 ts_tablet_manager.cc:1398] T 4c13642db7fc40a5a934aa5e3c2b08fc P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:56.323840 8685 ts_tablet_manager.cc:1367] T af00a839bb5e402791f87288493efef6 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240503 01:59:56.324000 8686 tablet_bootstrap.cc:492] T af00a839bb5e402791f87288493efef6 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 01:59:56.324338 8685 raft_consensus.cc:348] T af00a839bb5e402791f87288493efef6 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:56.324469 8685 raft_consensus.cc:374] T af00a839bb5e402791f87288493efef6 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:56.324522 8685 raft_consensus.cc:729] T af00a839bb5e402791f87288493efef6 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 01:59:56.324659 8685 consensus_queue.cc:260] T af00a839bb5e402791f87288493efef6 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:56.324913 8685 ts_tablet_manager.cc:1398] T af00a839bb5e402791f87288493efef6 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:56.325187 8685 tablet_bootstrap.cc:492] T 4c13642db7fc40a5a934aa5e3c2b08fc P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 01:59:56.325371 8686 tablet_bootstrap.cc:654] T af00a839bb5e402791f87288493efef6 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:56.326488 8685 tablet_bootstrap.cc:654] T 4c13642db7fc40a5a934aa5e3c2b08fc P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:56.327651 8686 tablet_bootstrap.cc:492] T af00a839bb5e402791f87288493efef6 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 01:59:56.327960 8686 ts_tablet_manager.cc:1367] T af00a839bb5e402791f87288493efef6 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:56.328028 8685 tablet_bootstrap.cc:492] T 4c13642db7fc40a5a934aa5e3c2b08fc P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 01:59:56.328660 8686 raft_consensus.cc:348] T af00a839bb5e402791f87288493efef6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:56.328783 8686 raft_consensus.cc:374] T af00a839bb5e402791f87288493efef6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:56.328809 8686 raft_consensus.cc:729] T af00a839bb5e402791f87288493efef6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 01:59:56.328909 8686 consensus_queue.cc:260] T af00a839bb5e402791f87288493efef6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:56.329218 8686 ts_tablet_manager.cc:1398] T af00a839bb5e402791f87288493efef6 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:56.329407 8685 ts_tablet_manager.cc:1367] T 4c13642db7fc40a5a934aa5e3c2b08fc P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:56.329834 8685 raft_consensus.cc:348] T 4c13642db7fc40a5a934aa5e3c2b08fc P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:56.329947 8685 raft_consensus.cc:374] T 4c13642db7fc40a5a934aa5e3c2b08fc P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:56.329999 8685 raft_consensus.cc:729] T 4c13642db7fc40a5a934aa5e3c2b08fc P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 01:59:56.330125 8685 consensus_queue.cc:260] T 4c13642db7fc40a5a934aa5e3c2b08fc P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:56.330439 8685 ts_tablet_manager.cc:1398] T 4c13642db7fc40a5a934aa5e3c2b08fc P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:56.330665 8685 tablet_bootstrap.cc:492] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 01:59:56.331988 8685 tablet_bootstrap.cc:654] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:56.333103 8685 tablet_bootstrap.cc:492] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 01:59:56.333211 8685 ts_tablet_manager.cc:1367] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 01:59:56.333596 8685 raft_consensus.cc:348] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:56.333693 8685 raft_consensus.cc:374] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:56.333720 8685 raft_consensus.cc:729] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 01:59:56.333910 8685 consensus_queue.cc:260] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:56.334245 8685 ts_tablet_manager.cc:1398] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:56.493654 8459 raft_consensus.cc:482] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:56.493782 8459 raft_consensus.cc:504] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:56.494177 8459 leader_election.cc:288] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 01:59:56.494561 6457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7ade1e2af24e86b6e54e2b5769e4c0" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 01:59:56.494562 6718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7ade1e2af24e86b6e54e2b5769e4c0" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 01:59:56.494710 6457 raft_consensus.cc:2454] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 01:59:56.494710 6718 raft_consensus.cc:2454] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 01:59:56.494982 6522 leader_election.cc:302] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:56.495280 8459 raft_consensus.cc:2781] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:56.495339 8459 raft_consensus.cc:482] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:56.495376 8459 raft_consensus.cc:3037] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:56.496397 8459 raft_consensus.cc:504] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:56.496881 8459 leader_election.cc:288] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Requested vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 01:59:56.497166 6718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7ade1e2af24e86b6e54e2b5769e4c0" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 01:59:56.497314 6718 raft_consensus.cc:3037] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:56.497476 6457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7ade1e2af24e86b6e54e2b5769e4c0" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 01:59:56.497620 6457 raft_consensus.cc:3037] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:56.498169 6718 raft_consensus.cc:2454] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 01:59:56.498456 6525 leader_election.cc:302] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 01:59:56.498474 6457 raft_consensus.cc:2454] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 01:59:56.498631 8459 raft_consensus.cc:2781] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:56.498749 8459 raft_consensus.cc:686] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Becoming Leader. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Running, Role: LEADER
I20240503 01:59:56.499068 8459 consensus_queue.cc:237] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:56.502216 6315 catalog_manager.cc:5484] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:56.590217 8679 raft_consensus.cc:482] T af00a839bb5e402791f87288493efef6 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:56.590342 8679 raft_consensus.cc:504] T af00a839bb5e402791f87288493efef6 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:56.590781 8679 leader_election.cc:288] T af00a839bb5e402791f87288493efef6 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 01:59:56.591167 6718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af00a839bb5e402791f87288493efef6" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 01:59:56.591313 6718 raft_consensus.cc:2454] T af00a839bb5e402791f87288493efef6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 01:59:56.591307 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af00a839bb5e402791f87288493efef6" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 01:59:56.591437 6589 raft_consensus.cc:2454] T af00a839bb5e402791f87288493efef6 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 01:59:56.591615 6394 leader_election.cc:302] T af00a839bb5e402791f87288493efef6 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2ac2eb7cc264047b9ab32fe3a4fc025, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:56.591764 8679 raft_consensus.cc:2781] T af00a839bb5e402791f87288493efef6 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:56.591817 8679 raft_consensus.cc:482] T af00a839bb5e402791f87288493efef6 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:56.591857 8679 raft_consensus.cc:3037] T af00a839bb5e402791f87288493efef6 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:56.592043 8594 raft_consensus.cc:482] T 4c13642db7fc40a5a934aa5e3c2b08fc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:56.592130 8594 raft_consensus.cc:504] T 4c13642db7fc40a5a934aa5e3c2b08fc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:56.592506 8594 leader_election.cc:288] T 4c13642db7fc40a5a934aa5e3c2b08fc P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 01:59:56.592878 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c13642db7fc40a5a934aa5e3c2b08fc" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 01:59:56.592893 8679 raft_consensus.cc:504] T af00a839bb5e402791f87288493efef6 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:56.592996 6849 raft_consensus.cc:2454] T 4c13642db7fc40a5a934aa5e3c2b08fc P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 01:59:56.593274 6656 leader_election.cc:302] T 4c13642db7fc40a5a934aa5e3c2b08fc P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 01:59:56.593438 8594 raft_consensus.cc:2781] T 4c13642db7fc40a5a934aa5e3c2b08fc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:56.593456 8679 leader_election.cc:288] T af00a839bb5e402791f87288493efef6 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Requested vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 01:59:56.593503 8594 raft_consensus.cc:482] T 4c13642db7fc40a5a934aa5e3c2b08fc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:56.593544 8594 raft_consensus.cc:3037] T 4c13642db7fc40a5a934aa5e3c2b08fc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:56.593506 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c13642db7fc40a5a934aa5e3c2b08fc" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 01:59:56.593621 6589 raft_consensus.cc:2454] T 4c13642db7fc40a5a934aa5e3c2b08fc P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 01:59:56.593603 6587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af00a839bb5e402791f87288493efef6" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 01:59:56.593694 6587 raft_consensus.cc:3037] T af00a839bb5e402791f87288493efef6 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:56.594527 6587 raft_consensus.cc:2454] T af00a839bb5e402791f87288493efef6 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 01:59:56.594528 8594 raft_consensus.cc:504] T 4c13642db7fc40a5a934aa5e3c2b08fc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:56.594841 6393 leader_election.cc:302] T af00a839bb5e402791f87288493efef6 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:56.594841 6718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af00a839bb5e402791f87288493efef6" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 01:59:56.594944 6718 raft_consensus.cc:3037] T af00a839bb5e402791f87288493efef6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:56.595194 8679 raft_consensus.cc:2781] T af00a839bb5e402791f87288493efef6 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:56.595199 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c13642db7fc40a5a934aa5e3c2b08fc" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 01:59:56.595220 6587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c13642db7fc40a5a934aa5e3c2b08fc" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 01:59:56.595305 6849 raft_consensus.cc:3037] T 4c13642db7fc40a5a934aa5e3c2b08fc P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:56.595315 6587 raft_consensus.cc:3037] T 4c13642db7fc40a5a934aa5e3c2b08fc P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:56.595536 8679 raft_consensus.cc:686] T af00a839bb5e402791f87288493efef6 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Becoming Leader. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Running, Role: LEADER
I20240503 01:59:56.595678 8679 consensus_queue.cc:237] T af00a839bb5e402791f87288493efef6 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:56.595949 6718 raft_consensus.cc:2454] T af00a839bb5e402791f87288493efef6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 01:59:56.596237 6587 raft_consensus.cc:2454] T 4c13642db7fc40a5a934aa5e3c2b08fc P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 01:59:56.596338 6849 raft_consensus.cc:2454] T 4c13642db7fc40a5a934aa5e3c2b08fc P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 01:59:56.596693 6656 leader_election.cc:302] T 4c13642db7fc40a5a934aa5e3c2b08fc P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 01:59:56.596894 8034 raft_consensus.cc:2781] T 4c13642db7fc40a5a934aa5e3c2b08fc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:56.597028 8034 raft_consensus.cc:686] T 4c13642db7fc40a5a934aa5e3c2b08fc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Becoming Leader. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Running, Role: LEADER
I20240503 01:59:56.597133 8034 consensus_queue.cc:237] T 4c13642db7fc40a5a934aa5e3c2b08fc P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:56.598286 6315 catalog_manager.cc:5484] T af00a839bb5e402791f87288493efef6 P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: term changed from 0 to 1, leader changed from <none> to dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129). New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:56.598896 8594 leader_election.cc:288] T 4c13642db7fc40a5a934aa5e3c2b08fc P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Requested vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 01:59:56.599980 6316 catalog_manager.cc:5484] T 4c13642db7fc40a5a934aa5e3c2b08fc P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } }
W20240503 01:59:56.790833 8698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:56.791096 8698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:56.794495 8698 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_18'
I20240503 01:59:56.897109 6458 raft_consensus.cc:1261] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:56.897122 6718 raft_consensus.cc:1261] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:56.897557 8459 consensus_queue.cc:1035] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:56.897862 8598 consensus_queue.cc:1035] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:56.900830 6587 raft_consensus.cc:1261] T af00a839bb5e402791f87288493efef6 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:56.903486 8679 consensus_queue.cc:1035] T af00a839bb5e402791f87288493efef6 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:56.911067 8737 mvcc.cc:205] Tried to move back new op lower bound from 7023417740886097920 to 7023417739261419520. Current Snapshot: MvccSnapshot[applied={T|T < 7023417740886097920}]
I20240503 01:59:56.911306 6720 raft_consensus.cc:1261] T af00a839bb5e402791f87288493efef6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:56.911731 8679 consensus_queue.cc:1035] T af00a839bb5e402791f87288493efef6 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:56.919951 6589 raft_consensus.cc:1261] T 4c13642db7fc40a5a934aa5e3c2b08fc P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:56.921311 6849 raft_consensus.cc:1261] T 4c13642db7fc40a5a934aa5e3c2b08fc P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:56.922552 8696 consensus_queue.cc:1035] T 4c13642db7fc40a5a934aa5e3c2b08fc P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:56.923408 8594 consensus_queue.cc:1035] T 4c13642db7fc40a5a934aa5e3c2b08fc P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 127.363 ms
time per row: 0.0636814 ms
I20240503 01:59:56.974989 6307 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalance_test_table_19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:56.980978 6829 tablet_service.cc:1467] Processing CreateTablet for tablet 1e445a3cc928429caf9e3c6193a63010 (DEFAULT_TABLE table=rebalance_test_table_19 [id=250f0720cafa4425b45a0360ebe04a03]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:56.981336 6829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e445a3cc928429caf9e3c6193a63010. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:56.981304 6831 tablet_service.cc:1467] Processing CreateTablet for tablet 3dc213593e4048c99ccbe10161ff8e7f (DEFAULT_TABLE table=rebalance_test_table_19 [id=250f0720cafa4425b45a0360ebe04a03]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:56.981456 6567 tablet_service.cc:1467] Processing CreateTablet for tablet 1e445a3cc928429caf9e3c6193a63010 (DEFAULT_TABLE table=rebalance_test_table_19 [id=250f0720cafa4425b45a0360ebe04a03]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:56.981566 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dc213593e4048c99ccbe10161ff8e7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:56.981711 6567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e445a3cc928429caf9e3c6193a63010. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:56.981772 6830 tablet_service.cc:1467] Processing CreateTablet for tablet 8246488dec204bf6b09e868d7951ac5a (DEFAULT_TABLE table=rebalance_test_table_19 [id=250f0720cafa4425b45a0360ebe04a03]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:56.981956 6830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8246488dec204bf6b09e868d7951ac5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:56.983892 8758 tablet_bootstrap.cc:492] T 3dc213593e4048c99ccbe10161ff8e7f P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 01:59:56.984310 6699 tablet_service.cc:1467] Processing CreateTablet for tablet 3dc213593e4048c99ccbe10161ff8e7f (DEFAULT_TABLE table=rebalance_test_table_19 [id=250f0720cafa4425b45a0360ebe04a03]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:56.984609 6699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dc213593e4048c99ccbe10161ff8e7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:56.984653 8759 tablet_bootstrap.cc:492] T 1e445a3cc928429caf9e3c6193a63010 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 01:59:56.985033 8758 tablet_bootstrap.cc:654] T 3dc213593e4048c99ccbe10161ff8e7f P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:56.985518 8759 tablet_bootstrap.cc:654] T 1e445a3cc928429caf9e3c6193a63010 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:56.986740 6700 tablet_service.cc:1467] Processing CreateTablet for tablet 8246488dec204bf6b09e868d7951ac5a (DEFAULT_TABLE table=rebalance_test_table_19 [id=250f0720cafa4425b45a0360ebe04a03]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:56.986982 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8246488dec204bf6b09e868d7951ac5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:56.988061 8758 tablet_bootstrap.cc:492] T 3dc213593e4048c99ccbe10161ff8e7f P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 01:59:56.988060 8759 tablet_bootstrap.cc:492] T 1e445a3cc928429caf9e3c6193a63010 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 01:59:56.988147 8758 ts_tablet_manager.cc:1367] T 3dc213593e4048c99ccbe10161ff8e7f P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:56.988147 8759 ts_tablet_manager.cc:1367] T 1e445a3cc928429caf9e3c6193a63010 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:56.988639 8758 raft_consensus.cc:348] T 3dc213593e4048c99ccbe10161ff8e7f P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:56.988754 8758 raft_consensus.cc:374] T 3dc213593e4048c99ccbe10161ff8e7f P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:56.988824 8758 raft_consensus.cc:729] T 3dc213593e4048c99ccbe10161ff8e7f P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 01:59:56.988906 8759 raft_consensus.cc:348] T 1e445a3cc928429caf9e3c6193a63010 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:56.989022 8759 raft_consensus.cc:374] T 1e445a3cc928429caf9e3c6193a63010 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:56.989061 8759 raft_consensus.cc:729] T 1e445a3cc928429caf9e3c6193a63010 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 01:59:56.988992 8758 consensus_queue.cc:260] T 3dc213593e4048c99ccbe10161ff8e7f P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:56.989169 8759 consensus_queue.cc:260] T 1e445a3cc928429caf9e3c6193a63010 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:56.989432 8759 ts_tablet_manager.cc:1398] T 1e445a3cc928429caf9e3c6193a63010 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:56.989459 8758 ts_tablet_manager.cc:1398] T 3dc213593e4048c99ccbe10161ff8e7f P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:56.989660 8758 tablet_bootstrap.cc:492] T 8246488dec204bf6b09e868d7951ac5a P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 01:59:56.989774 6436 tablet_service.cc:1467] Processing CreateTablet for tablet 1e445a3cc928429caf9e3c6193a63010 (DEFAULT_TABLE table=rebalance_test_table_19 [id=250f0720cafa4425b45a0360ebe04a03]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:56.990067 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e445a3cc928429caf9e3c6193a63010. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:56.990777 8758 tablet_bootstrap.cc:654] T 8246488dec204bf6b09e868d7951ac5a P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:56.991925 8758 tablet_bootstrap.cc:492] T 8246488dec204bf6b09e868d7951ac5a P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 01:59:56.992004 8758 ts_tablet_manager.cc:1367] T 8246488dec204bf6b09e868d7951ac5a P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:56.992272 8764 tablet_bootstrap.cc:492] T 1e445a3cc928429caf9e3c6193a63010 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 01:59:56.992471 6438 tablet_service.cc:1467] Processing CreateTablet for tablet 8246488dec204bf6b09e868d7951ac5a (DEFAULT_TABLE table=rebalance_test_table_19 [id=250f0720cafa4425b45a0360ebe04a03]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:56.992523 8758 raft_consensus.cc:348] T 8246488dec204bf6b09e868d7951ac5a P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:56.992645 8758 raft_consensus.cc:374] T 8246488dec204bf6b09e868d7951ac5a P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:56.992676 8758 raft_consensus.cc:729] T 8246488dec204bf6b09e868d7951ac5a P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 01:59:56.992764 6438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8246488dec204bf6b09e868d7951ac5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:56.992792 8758 consensus_queue.cc:260] T 8246488dec204bf6b09e868d7951ac5a P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:56.993083 8758 ts_tablet_manager.cc:1398] T 8246488dec204bf6b09e868d7951ac5a P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:56.993240 8758 tablet_bootstrap.cc:492] T 1e445a3cc928429caf9e3c6193a63010 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 01:59:56.993366 8764 tablet_bootstrap.cc:654] T 1e445a3cc928429caf9e3c6193a63010 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:56.994115 8758 tablet_bootstrap.cc:654] T 1e445a3cc928429caf9e3c6193a63010 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:56.994341 8764 tablet_bootstrap.cc:492] T 1e445a3cc928429caf9e3c6193a63010 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 01:59:56.994403 8764 ts_tablet_manager.cc:1367] T 1e445a3cc928429caf9e3c6193a63010 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:56.994791 8764 raft_consensus.cc:348] T 1e445a3cc928429caf9e3c6193a63010 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:56.994890 8764 raft_consensus.cc:374] T 1e445a3cc928429caf9e3c6193a63010 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:56.994921 8764 raft_consensus.cc:729] T 1e445a3cc928429caf9e3c6193a63010 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 01:59:56.995071 8764 consensus_queue.cc:260] T 1e445a3cc928429caf9e3c6193a63010 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:56.995232 8758 tablet_bootstrap.cc:492] T 1e445a3cc928429caf9e3c6193a63010 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 01:59:56.995301 8758 ts_tablet_manager.cc:1367] T 1e445a3cc928429caf9e3c6193a63010 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:56.995366 8764 ts_tablet_manager.cc:1398] T 1e445a3cc928429caf9e3c6193a63010 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:56.996004 8764 tablet_bootstrap.cc:492] T 8246488dec204bf6b09e868d7951ac5a P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 01:59:56.996814 6567 tablet_service.cc:1467] Processing CreateTablet for tablet 3dc213593e4048c99ccbe10161ff8e7f (DEFAULT_TABLE table=rebalance_test_table_19 [id=250f0720cafa4425b45a0360ebe04a03]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:56.997126 6567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dc213593e4048c99ccbe10161ff8e7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:56.997141 8764 tablet_bootstrap.cc:654] T 8246488dec204bf6b09e868d7951ac5a P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:56.998528 8764 tablet_bootstrap.cc:492] T 8246488dec204bf6b09e868d7951ac5a P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 01:59:56.998623 8764 ts_tablet_manager.cc:1367] T 8246488dec204bf6b09e868d7951ac5a P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:56.999091 8759 tablet_bootstrap.cc:492] T 3dc213593e4048c99ccbe10161ff8e7f P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 01:59:56.999156 8764 raft_consensus.cc:348] T 8246488dec204bf6b09e868d7951ac5a P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:56.999260 8764 raft_consensus.cc:374] T 8246488dec204bf6b09e868d7951ac5a P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:56.999290 8764 raft_consensus.cc:729] T 8246488dec204bf6b09e868d7951ac5a P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 01:59:56.999450 8764 consensus_queue.cc:260] T 8246488dec204bf6b09e868d7951ac5a P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:56.999917 8764 ts_tablet_manager.cc:1398] T 8246488dec204bf6b09e868d7951ac5a P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:57.000072 8759 tablet_bootstrap.cc:654] T 3dc213593e4048c99ccbe10161ff8e7f P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:57.000116 8758 raft_consensus.cc:348] T 1e445a3cc928429caf9e3c6193a63010 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:57.000221 8758 raft_consensus.cc:374] T 1e445a3cc928429caf9e3c6193a63010 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:57.000252 8758 raft_consensus.cc:729] T 1e445a3cc928429caf9e3c6193a63010 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 01:59:57.000398 8758 consensus_queue.cc:260] T 1e445a3cc928429caf9e3c6193a63010 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:57.000742 8758 ts_tablet_manager.cc:1398] T 1e445a3cc928429caf9e3c6193a63010 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240503 01:59:57.001061 8761 tablet_bootstrap.cc:492] T 3dc213593e4048c99ccbe10161ff8e7f P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 01:59:57.001267 8759 tablet_bootstrap.cc:492] T 3dc213593e4048c99ccbe10161ff8e7f P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 01:59:57.001354 8759 ts_tablet_manager.cc:1367] T 3dc213593e4048c99ccbe10161ff8e7f P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:57.001797 8759 raft_consensus.cc:348] T 3dc213593e4048c99ccbe10161ff8e7f P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:57.001910 8759 raft_consensus.cc:374] T 3dc213593e4048c99ccbe10161ff8e7f P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:57.001940 8759 raft_consensus.cc:729] T 3dc213593e4048c99ccbe10161ff8e7f P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 01:59:57.001943 8761 tablet_bootstrap.cc:654] T 3dc213593e4048c99ccbe10161ff8e7f P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:57.002051 8759 consensus_queue.cc:260] T 3dc213593e4048c99ccbe10161ff8e7f P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:57.002321 8759 ts_tablet_manager.cc:1398] T 3dc213593e4048c99ccbe10161ff8e7f P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:57.003029 8761 tablet_bootstrap.cc:492] T 3dc213593e4048c99ccbe10161ff8e7f P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 01:59:57.003122 8761 ts_tablet_manager.cc:1367] T 3dc213593e4048c99ccbe10161ff8e7f P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:57.003579 8761 raft_consensus.cc:348] T 3dc213593e4048c99ccbe10161ff8e7f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:57.003690 8761 raft_consensus.cc:374] T 3dc213593e4048c99ccbe10161ff8e7f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:57.003722 8761 raft_consensus.cc:729] T 3dc213593e4048c99ccbe10161ff8e7f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 01:59:57.003831 8761 consensus_queue.cc:260] T 3dc213593e4048c99ccbe10161ff8e7f P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:57.004117 8761 ts_tablet_manager.cc:1398] T 3dc213593e4048c99ccbe10161ff8e7f P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:57.004354 8761 tablet_bootstrap.cc:492] T 8246488dec204bf6b09e868d7951ac5a P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 01:59:57.005275 8761 tablet_bootstrap.cc:654] T 8246488dec204bf6b09e868d7951ac5a P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:57.006668 8761 tablet_bootstrap.cc:492] T 8246488dec204bf6b09e868d7951ac5a P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 01:59:57.006762 8761 ts_tablet_manager.cc:1367] T 8246488dec204bf6b09e868d7951ac5a P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 01:59:57.007140 8761 raft_consensus.cc:348] T 8246488dec204bf6b09e868d7951ac5a P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:57.007226 8761 raft_consensus.cc:374] T 8246488dec204bf6b09e868d7951ac5a P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:57.007246 8761 raft_consensus.cc:729] T 8246488dec204bf6b09e868d7951ac5a P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 01:59:57.007344 8761 consensus_queue.cc:260] T 8246488dec204bf6b09e868d7951ac5a P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:57.007612 8761 ts_tablet_manager.cc:1398] T 8246488dec204bf6b09e868d7951ac5a P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:57.018839 8696 raft_consensus.cc:482] T 3dc213593e4048c99ccbe10161ff8e7f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:57.018958 8696 raft_consensus.cc:504] T 3dc213593e4048c99ccbe10161ff8e7f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:57.019383 8696 leader_election.cc:288] T 3dc213593e4048c99ccbe10161ff8e7f P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 01:59:57.019764 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc213593e4048c99ccbe10161ff8e7f" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 01:59:57.019909 6589 raft_consensus.cc:2454] T 3dc213593e4048c99ccbe10161ff8e7f P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 01:59:57.019886 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc213593e4048c99ccbe10161ff8e7f" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 01:59:57.020018 6849 raft_consensus.cc:2454] T 3dc213593e4048c99ccbe10161ff8e7f P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 01:59:57.020217 6655 leader_election.cc:302] T 3dc213593e4048c99ccbe10161ff8e7f P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 01:59:57.020392 8696 raft_consensus.cc:2781] T 3dc213593e4048c99ccbe10161ff8e7f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:57.020459 8696 raft_consensus.cc:482] T 3dc213593e4048c99ccbe10161ff8e7f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:57.020501 8696 raft_consensus.cc:3037] T 3dc213593e4048c99ccbe10161ff8e7f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:57.021554 8696 raft_consensus.cc:504] T 3dc213593e4048c99ccbe10161ff8e7f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:57.021871 8696 leader_election.cc:288] T 3dc213593e4048c99ccbe10161ff8e7f P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Requested vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 01:59:57.022192 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc213593e4048c99ccbe10161ff8e7f" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 01:59:57.022219 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dc213593e4048c99ccbe10161ff8e7f" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 01:59:57.022295 6849 raft_consensus.cc:3037] T 3dc213593e4048c99ccbe10161ff8e7f P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:57.022332 6589 raft_consensus.cc:3037] T 3dc213593e4048c99ccbe10161ff8e7f P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:57.023108 6849 raft_consensus.cc:2454] T 3dc213593e4048c99ccbe10161ff8e7f P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 01:59:57.023279 6589 raft_consensus.cc:2454] T 3dc213593e4048c99ccbe10161ff8e7f P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 01:59:57.023391 6656 leader_election.cc:302] T 3dc213593e4048c99ccbe10161ff8e7f P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 01:59:57.023694 8696 raft_consensus.cc:2781] T 3dc213593e4048c99ccbe10161ff8e7f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:57.023780 8696 raft_consensus.cc:686] T 3dc213593e4048c99ccbe10161ff8e7f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Becoming Leader. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Running, Role: LEADER
I20240503 01:59:57.023918 8696 consensus_queue.cc:237] T 3dc213593e4048c99ccbe10161ff8e7f P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 01:59:57.026149 6307 catalog_manager.cc:5484] T 3dc213593e4048c99ccbe10161ff8e7f P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:57.109114 8772 raft_consensus.cc:482] T 8246488dec204bf6b09e868d7951ac5a P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:57.109268 8772 raft_consensus.cc:504] T 8246488dec204bf6b09e868d7951ac5a P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:57.109805 8772 leader_election.cc:288] T 8246488dec204bf6b09e868d7951ac5a P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 01:59:57.110234 6720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8246488dec204bf6b09e868d7951ac5a" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 01:59:57.110256 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8246488dec204bf6b09e868d7951ac5a" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 01:59:57.110388 6720 raft_consensus.cc:2454] T 8246488dec204bf6b09e868d7951ac5a P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 01:59:57.110401 6458 raft_consensus.cc:2454] T 8246488dec204bf6b09e868d7951ac5a P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 01:59:57.110693 6787 leader_election.cc:302] T 8246488dec204bf6b09e868d7951ac5a P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 01:59:57.110880 8772 raft_consensus.cc:2781] T 8246488dec204bf6b09e868d7951ac5a P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:57.110946 8772 raft_consensus.cc:482] T 8246488dec204bf6b09e868d7951ac5a P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:57.110983 8772 raft_consensus.cc:3037] T 8246488dec204bf6b09e868d7951ac5a P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:57.111927 8772 raft_consensus.cc:504] T 8246488dec204bf6b09e868d7951ac5a P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:57.112327 8772 leader_election.cc:288] T 8246488dec204bf6b09e868d7951ac5a P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Requested vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 01:59:57.112901 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8246488dec204bf6b09e868d7951ac5a" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 01:59:57.113046 6458 raft_consensus.cc:3037] T 8246488dec204bf6b09e868d7951ac5a P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:57.113212 6720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8246488dec204bf6b09e868d7951ac5a" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 01:59:57.113320 6720 raft_consensus.cc:3037] T 8246488dec204bf6b09e868d7951ac5a P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:57.114109 6458 raft_consensus.cc:2454] T 8246488dec204bf6b09e868d7951ac5a P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 01:59:57.114203 6720 raft_consensus.cc:2454] T 8246488dec204bf6b09e868d7951ac5a P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 01:59:57.114511 6787 leader_election.cc:302] T 8246488dec204bf6b09e868d7951ac5a P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, c2ac2eb7cc264047b9ab32fe3a4fc025, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:57.114768 8772 raft_consensus.cc:2781] T 8246488dec204bf6b09e868d7951ac5a P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:57.114856 8772 raft_consensus.cc:686] T 8246488dec204bf6b09e868d7951ac5a P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Becoming Leader. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Running, Role: LEADER
I20240503 01:59:57.114996 8772 consensus_queue.cc:237] T 8246488dec204bf6b09e868d7951ac5a P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:57.117677 6307 catalog_manager.cc:5484] T 8246488dec204bf6b09e868d7951ac5a P 492448570cc94ed1879feb2ec3b9051d reported cstate change: term changed from 0 to 1, leader changed from <none> to 492448570cc94ed1879feb2ec3b9051d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:57.261472 8459 raft_consensus.cc:482] T 1e445a3cc928429caf9e3c6193a63010 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:57.261619 8459 raft_consensus.cc:504] T 1e445a3cc928429caf9e3c6193a63010 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:57.262106 8459 leader_election.cc:288] T 1e445a3cc928429caf9e3c6193a63010 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 01:59:57.262493 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e445a3cc928429caf9e3c6193a63010" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 01:59:57.262535 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e445a3cc928429caf9e3c6193a63010" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 01:59:57.262655 6849 raft_consensus.cc:2454] T 1e445a3cc928429caf9e3c6193a63010 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 01:59:57.262684 6458 raft_consensus.cc:2454] T 1e445a3cc928429caf9e3c6193a63010 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 01:59:57.262948 6525 leader_election.cc:302] T 1e445a3cc928429caf9e3c6193a63010 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 80fd18ef4d604c86b909d5c008a46b4f; no voters:
I20240503 01:59:57.263134 8459 raft_consensus.cc:2781] T 1e445a3cc928429caf9e3c6193a63010 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:57.263195 8459 raft_consensus.cc:482] T 1e445a3cc928429caf9e3c6193a63010 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:57.263243 8459 raft_consensus.cc:3037] T 1e445a3cc928429caf9e3c6193a63010 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:57.264185 8459 raft_consensus.cc:504] T 1e445a3cc928429caf9e3c6193a63010 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:57.264604 8459 leader_election.cc:288] T 1e445a3cc928429caf9e3c6193a63010 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Requested vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 01:59:57.264911 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e445a3cc928429caf9e3c6193a63010" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 01:59:57.264991 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e445a3cc928429caf9e3c6193a63010" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 01:59:57.265043 6849 raft_consensus.cc:3037] T 1e445a3cc928429caf9e3c6193a63010 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:57.265111 6458 raft_consensus.cc:3037] T 1e445a3cc928429caf9e3c6193a63010 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:57.266024 6849 raft_consensus.cc:2454] T 1e445a3cc928429caf9e3c6193a63010 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 01:59:57.266024 6458 raft_consensus.cc:2454] T 1e445a3cc928429caf9e3c6193a63010 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 01:59:57.266323 6522 leader_election.cc:302] T 1e445a3cc928429caf9e3c6193a63010 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:57.266546 8459 raft_consensus.cc:2781] T 1e445a3cc928429caf9e3c6193a63010 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:57.266618 8459 raft_consensus.cc:686] T 1e445a3cc928429caf9e3c6193a63010 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Becoming Leader. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Running, Role: LEADER
I20240503 01:59:57.266753 8459 consensus_queue.cc:237] T 1e445a3cc928429caf9e3c6193a63010 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:57.268924 6307 catalog_manager.cc:5484] T 1e445a3cc928429caf9e3c6193a63010 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } }
W20240503 01:59:57.403429 8775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:57.403693 8775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:57.407451 8775 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_19'
I20240503 01:59:57.487380 8594 consensus_queue.cc:1035] T 3dc213593e4048c99ccbe10161ff8e7f P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:57.494730 8696 consensus_queue.cc:1035] T 3dc213593e4048c99ccbe10161ff8e7f P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:57.500764 6457 raft_consensus.cc:1261] T 8246488dec204bf6b09e868d7951ac5a P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:57.500800 6720 raft_consensus.cc:1261] T 8246488dec204bf6b09e868d7951ac5a P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:57.501250 8636 consensus_queue.cc:1035] T 8246488dec204bf6b09e868d7951ac5a P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:57.501590 8772 consensus_queue.cc:1035] T 8246488dec204bf6b09e868d7951ac5a P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:57.509457 6457 raft_consensus.cc:1261] T 1e445a3cc928429caf9e3c6193a63010 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:57.509917 8798 consensus_queue.cc:1035] T 1e445a3cc928429caf9e3c6193a63010 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:57.511998 6849 raft_consensus.cc:1261] T 1e445a3cc928429caf9e3c6193a63010 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:57.512430 8598 consensus_queue.cc:1035] T 1e445a3cc928429caf9e3c6193a63010 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:57.523190 8811 mvcc.cc:205] Tried to move back new op lower bound from 7023417743383048192 to 7023417742406193152. Current Snapshot: MvccSnapshot[applied={T|T < 7023417743383048192}]
I20240503 01:59:57.541224 8815 mvcc.cc:205] Tried to move back new op lower bound from 7023417743359262720 to 7023417741784653824. Current Snapshot: MvccSnapshot[applied={T|T < 7023417743359262720}]
INSERT report
rows total: 2000
time total: 131.615 ms
time per row: 0.0658076 ms
I20240503 01:59:57.586563 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
/tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:33211
--local_ip_for_outbound_sockets=127.0.99.133
--tserver_master_addrs=127.0.99.190:35423
--webserver_port=41957
--webserver_interface=127.0.99.133
--builtin_ntp_servers=127.0.99.148:40175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 01:59:57.706756 8835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:57.707072 8835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:57.707134 8835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:57.710510 8835 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:57.710595 8835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:57.710667 8835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 01:59:57.714699 8835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:40175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:33211
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=41957
--tserver_master_addrs=127.0.99.190:35423
--never_fsync=true
--heap_profile_path=/tmp/kudu.8835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:57.715648 8835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:57.716856 8835 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:57.725510 8840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:57.725482 8843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:57.725651 8835 server_base.cc:1034] running on GCE node
W20240503 01:59:57.725780 8841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:57.726203 8835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:57.726948 8835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:57.728147 8835 hybrid_clock.cc:648] HybridClock initialized: now 1714701597728121 us; error 44 us; skew 500 ppm
I20240503 01:59:57.730690 8835 webserver.cc:460] Webserver started at http://127.0.99.133:41957/ using document root <none> and password file <none>
I20240503 01:59:57.731345 8835 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:57.731432 8835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:57.735215 8835 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:57.737589 8849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:57.738916 8835 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 01:59:57.739048 8835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "9942306855404cfd891adf227dd65eb2"
format_stamp: "Formatted at 2024-05-03 01:59:41 on dist-test-slave-pz5w"
I20240503 01:59:57.739558 8835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:57.757100 8835 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:57.757727 8835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:57.757884 8835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:57.758414 8835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:57.759407 8835 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:57.759466 8835 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:57.759510 8835 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:57.759532 8835 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:57.768733 8835 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:33211
I20240503 01:59:57.769874 8835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 01:59:57.773488 8962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:33211 every 8 connection(s)
I20240503 01:59:57.773825 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu as pid 8835
I20240503 01:59:57.774187 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
/tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:46157
--local_ip_for_outbound_sockets=127.0.99.134
--tserver_master_addrs=127.0.99.190:35423
--webserver_port=42917
--webserver_interface=127.0.99.134
--builtin_ntp_servers=127.0.99.148:40175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 01:59:57.785035 8963 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35423
I20240503 01:59:57.785281 8963 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:57.785912 8963 heartbeater.cc:511] Master 127.0.99.190:35423 requested a full tablet report, sending...
I20240503 01:59:57.787181 6307 ts_manager.cc:194] Re-registered known tserver with Master: 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 01:59:57.787963 6307 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:56451
W20240503 01:59:57.894673 8966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:57.894928 8966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:57.894976 8966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:57.898186 8966 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:57.898272 8966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:57.898341 8966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 01:59:57.902577 8966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:40175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:46157
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=42917
--tserver_master_addrs=127.0.99.190:35423
--never_fsync=true
--heap_profile_path=/tmp/kudu.8966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:57.903717 8966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:57.904961 8966 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:57.913651 8974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:57.913654 8972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:57.914084 8966 server_base.cc:1034] running on GCE node
W20240503 01:59:57.914090 8971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:57.914671 8966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:57.915241 8966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:57.916416 8966 hybrid_clock.cc:648] HybridClock initialized: now 1714701597916394 us; error 36 us; skew 500 ppm
I20240503 01:59:57.918546 8966 webserver.cc:460] Webserver started at http://127.0.99.134:42917/ using document root <none> and password file <none>
I20240503 01:59:57.919157 8966 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:57.919242 8966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:57.923115 8966 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20240503 01:59:57.925261 8980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:57.926478 8966 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:57.926599 8966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "99ead02915954b4fab23b033ad6d546b"
format_stamp: "Formatted at 2024-05-03 01:59:41 on dist-test-slave-pz5w"
I20240503 01:59:57.927073 8966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:57.943280 8966 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:57.943926 8966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:57.944161 8966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:57.944844 8966 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:57.946213 8966 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:57.946272 8966 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:57.946327 8966 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:57.946347 8966 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:57.955773 8966 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:46157
I20240503 01:59:57.955813 9093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:46157 every 8 connection(s)
I20240503 01:59:57.957047 8966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 01:59:57.962551 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu as pid 8966
I20240503 01:59:57.963137 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
/tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:34079
--local_ip_for_outbound_sockets=127.0.99.135
--tserver_master_addrs=127.0.99.190:35423
--webserver_port=36139
--webserver_interface=127.0.99.135
--builtin_ntp_servers=127.0.99.148:40175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 01:59:57.968343 9094 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35423
I20240503 01:59:57.968650 9094 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:57.969278 9094 heartbeater.cc:511] Master 127.0.99.190:35423 requested a full tablet report, sending...
I20240503 01:59:57.970463 6307 ts_manager.cc:194] Re-registered known tserver with Master: 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 01:59:57.971073 6307 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:55435
W20240503 01:59:58.090695 9097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:58.090971 9097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:58.091033 9097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:58.094818 9097 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:58.094929 9097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:58.095031 9097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 01:59:58.099280 9097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:40175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:34079
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=36139
--tserver_master_addrs=127.0.99.190:35423
--never_fsync=true
--heap_profile_path=/tmp/kudu.9097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:58.100276 9097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:58.101497 9097 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:58.110729 9102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:58.110729 9103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:58.111284 9105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:58.111550 9097 server_base.cc:1034] running on GCE node
I20240503 01:59:58.111965 9097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:58.112607 9097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:58.113785 9097 hybrid_clock.cc:648] HybridClock initialized: now 1714701598113767 us; error 54 us; skew 500 ppm
I20240503 01:59:58.115902 9097 webserver.cc:460] Webserver started at http://127.0.99.135:36139/ using document root <none> and password file <none>
I20240503 01:59:58.116597 9097 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:58.116686 9097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:58.120555 9097 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20240503 01:59:58.122803 9111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:58.123951 9097 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:58.124079 9097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "2a01ace80baf4057b6bf1995cf88468b"
format_stamp: "Formatted at 2024-05-03 01:59:42 on dist-test-slave-pz5w"
I20240503 01:59:58.124567 9097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:58.141582 9097 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:58.142222 9097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:58.142382 9097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:58.142906 9097 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:58.143882 9097 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:58.143954 9097 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:58.144019 9097 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:58.144047 9097 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:58.153682 9097 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:34079
I20240503 01:59:58.153703 9224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:34079 every 8 connection(s)
I20240503 01:59:58.154817 9097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 01:59:58.156814 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu as pid 9097
I20240503 01:59:58.157253 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
/tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:45311
--local_ip_for_outbound_sockets=127.0.99.136
--tserver_master_addrs=127.0.99.190:35423
--webserver_port=45791
--webserver_interface=127.0.99.136
--builtin_ntp_servers=127.0.99.148:40175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 01:59:58.165623 9225 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35423
I20240503 01:59:58.165882 9225 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:58.166523 9225 heartbeater.cc:511] Master 127.0.99.190:35423 requested a full tablet report, sending...
I20240503 01:59:58.167619 6307 ts_manager.cc:194] Re-registered known tserver with Master: 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 01:59:58.168212 6307 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:46411
W20240503 01:59:58.283119 9228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:58.283465 9228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:58.283602 9228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:58.287600 9228 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:58.287714 9228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:58.287807 9228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240503 01:59:58.292742 9228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:40175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:45311
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=45791
--tserver_master_addrs=127.0.99.190:35423
--never_fsync=true
--heap_profile_path=/tmp/kudu.9228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:58.293782 9228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:58.295225 9228 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:58.305289 9233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:58.305369 9236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:58.305564 9234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:58.305491 9228 server_base.cc:1034] running on GCE node
I20240503 01:59:58.306124 9228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:58.306784 9228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:58.310236 9228 hybrid_clock.cc:648] HybridClock initialized: now 1714701598310167 us; error 74 us; skew 500 ppm
I20240503 01:59:58.312814 9228 webserver.cc:460] Webserver started at http://127.0.99.136:45791/ using document root <none> and password file <none>
I20240503 01:59:58.313469 9228 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:58.313553 9228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:58.317467 9228 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 01:59:58.320312 9242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:58.321892 9228 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 01:59:58.322042 9228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023"
format_stamp: "Formatted at 2024-05-03 01:59:42 on dist-test-slave-pz5w"
I20240503 01:59:58.322516 9228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:58.339634 9228 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:58.340292 9228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:58.340493 9228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:58.341123 9228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:58.342124 9228 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:58.342175 9228 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:58.342217 9228 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:58.342232 9228 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:58.353370 9228 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:45311
I20240503 01:59:58.353438 9355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:45311 every 8 connection(s)
I20240503 01:59:58.354622 9228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 01:59:58.364179 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu as pid 9228
I20240503 01:59:58.365581 9356 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35423
I20240503 01:59:58.366055 9356 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:58.366948 9356 heartbeater.cc:511] Master 127.0.99.190:35423 requested a full tablet report, sending...
I20240503 01:59:58.368153 6307 ts_manager.cc:194] Re-registered known tserver with Master: e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311)
I20240503 01:59:58.368950 6307 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:57335
I20240503 01:59:58.369285 6312 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 01:59:58.375599 6567 tablet_service.cc:1467] Processing CreateTablet for tablet e0921a7de46f4385969a72d9088c3bcb (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c7c1d119289f48aab21886443f61f742]), partition=
I20240503 01:59:58.375932 6567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0921a7de46f4385969a72d9088c3bcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:58.378679 9372 tablet_bootstrap.cc:492] T e0921a7de46f4385969a72d9088c3bcb P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 01:59:58.380002 9372 tablet_bootstrap.cc:654] T e0921a7de46f4385969a72d9088c3bcb P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:58.381228 9372 tablet_bootstrap.cc:492] T e0921a7de46f4385969a72d9088c3bcb P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 01:59:58.381340 9372 ts_tablet_manager.cc:1367] T e0921a7de46f4385969a72d9088c3bcb P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:58.381800 9372 raft_consensus.cc:348] T e0921a7de46f4385969a72d9088c3bcb P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 01:59:58.382020 9372 raft_consensus.cc:374] T e0921a7de46f4385969a72d9088c3bcb P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:58.382073 9372 raft_consensus.cc:729] T e0921a7de46f4385969a72d9088c3bcb P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 01:59:58.382192 9372 consensus_queue.cc:260] T e0921a7de46f4385969a72d9088c3bcb P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 01:59:58.382483 9372 ts_tablet_manager.cc:1398] T e0921a7de46f4385969a72d9088c3bcb P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:58.386507 9027 tablet_service.cc:1467] Processing CreateTablet for tablet e0921a7de46f4385969a72d9088c3bcb (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c7c1d119289f48aab21886443f61f742]), partition=
I20240503 01:59:58.387324 9027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0921a7de46f4385969a72d9088c3bcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:58.387710 9285 tablet_service.cc:1467] Processing CreateTablet for tablet e0921a7de46f4385969a72d9088c3bcb (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c7c1d119289f48aab21886443f61f742]), partition=
I20240503 01:59:58.388310 9285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0921a7de46f4385969a72d9088c3bcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:58.395076 9374 tablet_bootstrap.cc:492] T e0921a7de46f4385969a72d9088c3bcb P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap starting.
I20240503 01:59:58.399121 9375 tablet_bootstrap.cc:492] T e0921a7de46f4385969a72d9088c3bcb P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 01:59:58.402124 9375 tablet_bootstrap.cc:654] T e0921a7de46f4385969a72d9088c3bcb P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:58.403899 9374 tablet_bootstrap.cc:654] T e0921a7de46f4385969a72d9088c3bcb P e1a9251ce1cf4361b8b0ede9fbfc4023: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:58.406044 9375 log.cc:824] T e0921a7de46f4385969a72d9088c3bcb P 99ead02915954b4fab23b033ad6d546b: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:58.406191 9374 log.cc:824] T e0921a7de46f4385969a72d9088c3bcb P e1a9251ce1cf4361b8b0ede9fbfc4023: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:58.408437 9375 tablet_bootstrap.cc:492] T e0921a7de46f4385969a72d9088c3bcb P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 01:59:58.408674 9375 ts_tablet_manager.cc:1367] T e0921a7de46f4385969a72d9088c3bcb P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.004s
I20240503 01:59:58.409725 9374 tablet_bootstrap.cc:492] T e0921a7de46f4385969a72d9088c3bcb P e1a9251ce1cf4361b8b0ede9fbfc4023: No bootstrap required, opened a new log
I20240503 01:59:58.409938 9374 ts_tablet_manager.cc:1367] T e0921a7de46f4385969a72d9088c3bcb P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20240503 01:59:58.412770 9375 raft_consensus.cc:348] T e0921a7de46f4385969a72d9088c3bcb P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 01:59:58.413020 9374 raft_consensus.cc:348] T e0921a7de46f4385969a72d9088c3bcb P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 01:59:58.413166 9375 raft_consensus.cc:374] T e0921a7de46f4385969a72d9088c3bcb P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:58.413246 9374 raft_consensus.cc:374] T e0921a7de46f4385969a72d9088c3bcb P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:58.413316 9374 raft_consensus.cc:729] T e0921a7de46f4385969a72d9088c3bcb P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1a9251ce1cf4361b8b0ede9fbfc4023, State: Initialized, Role: FOLLOWER
I20240503 01:59:58.413334 9375 raft_consensus.cc:729] T e0921a7de46f4385969a72d9088c3bcb P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 01:59:58.413825 9374 consensus_queue.cc:260] T e0921a7de46f4385969a72d9088c3bcb P e1a9251ce1cf4361b8b0ede9fbfc4023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 01:59:58.414759 9356 heartbeater.cc:503] Master 127.0.99.190:35423 was elected leader, sending a full tablet report...
I20240503 01:59:58.414911 9374 ts_tablet_manager.cc:1398] T e0921a7de46f4385969a72d9088c3bcb P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240503 01:59:58.416848 9375 consensus_queue.cc:260] T e0921a7de46f4385969a72d9088c3bcb P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 01:59:58.418722 9375 ts_tablet_manager.cc:1398] T e0921a7de46f4385969a72d9088c3bcb P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20240503 01:59:58.419663 9094 heartbeater.cc:503] Master 127.0.99.190:35423 was elected leader, sending a full tablet report...
W20240503 01:59:58.458917 9095 tablet.cc:2369] T e0921a7de46f4385969a72d9088c3bcb P 99ead02915954b4fab23b033ad6d546b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 01:59:58.605571 9357 tablet.cc:2369] T e0921a7de46f4385969a72d9088c3bcb P e1a9251ce1cf4361b8b0ede9fbfc4023: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 01:59:58.702603 8833 raft_consensus.cc:482] T e0921a7de46f4385969a72d9088c3bcb P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:58.702735 8833 raft_consensus.cc:504] T e0921a7de46f4385969a72d9088c3bcb P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 01:59:58.703359 8833 leader_election.cc:288] T e0921a7de46f4385969a72d9088c3bcb P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 01:59:58.707324 9310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0921a7de46f4385969a72d9088c3bcb" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" is_pre_election: true
I20240503 01:59:58.707687 9310 raft_consensus.cc:2454] T e0921a7de46f4385969a72d9088c3bcb P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 01:59:58.708030 9039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0921a7de46f4385969a72d9088c3bcb" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 01:59:58.708094 6523 leader_election.cc:302] T e0921a7de46f4385969a72d9088c3bcb P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, e1a9251ce1cf4361b8b0ede9fbfc4023; no voters:
I20240503 01:59:58.708281 8833 raft_consensus.cc:2781] T e0921a7de46f4385969a72d9088c3bcb P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:58.708281 9039 raft_consensus.cc:2454] T e0921a7de46f4385969a72d9088c3bcb P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 01:59:58.708334 8833 raft_consensus.cc:482] T e0921a7de46f4385969a72d9088c3bcb P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:58.708360 8833 raft_consensus.cc:3037] T e0921a7de46f4385969a72d9088c3bcb P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:58.709326 8833 raft_consensus.cc:504] T e0921a7de46f4385969a72d9088c3bcb P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 01:59:58.709686 8833 leader_election.cc:288] T e0921a7de46f4385969a72d9088c3bcb P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Requested vote from peers e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 01:59:58.710104 9310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0921a7de46f4385969a72d9088c3bcb" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023"
I20240503 01:59:58.710104 9039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0921a7de46f4385969a72d9088c3bcb" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 01:59:58.710229 9039 raft_consensus.cc:3037] T e0921a7de46f4385969a72d9088c3bcb P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:58.710229 9310 raft_consensus.cc:3037] T e0921a7de46f4385969a72d9088c3bcb P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:58.711300 9310 raft_consensus.cc:2454] T e0921a7de46f4385969a72d9088c3bcb P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 01:59:58.711301 9039 raft_consensus.cc:2454] T e0921a7de46f4385969a72d9088c3bcb P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 01:59:58.711635 6523 leader_election.cc:302] T e0921a7de46f4385969a72d9088c3bcb P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 01:59:58.711805 8833 raft_consensus.cc:2781] T e0921a7de46f4385969a72d9088c3bcb P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:58.711891 8833 raft_consensus.cc:686] T e0921a7de46f4385969a72d9088c3bcb P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Becoming Leader. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Running, Role: LEADER
I20240503 01:59:58.712036 8833 consensus_queue.cc:237] T e0921a7de46f4385969a72d9088c3bcb P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 01:59:58.714274 6307 catalog_manager.cc:5484] T e0921a7de46f4385969a72d9088c3bcb P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:58.731738 9027 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:58.734601 6829 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:58.738664 9159 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:58.740543 6567 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 01:59:58.747457 8880 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:58.752596 6438 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 01:59:58.786659 9285 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:58.787632 6700 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 01:59:58.790251 8963 heartbeater.cc:503] Master 127.0.99.190:35423 was elected leader, sending a full tablet report...
I20240503 01:59:58.862037 6307 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 01:59:58.866451 6307 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_extra_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:58.869236 6315 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240503 01:59:58.869383 9310 raft_consensus.cc:1261] T e0921a7de46f4385969a72d9088c3bcb P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:58.869403 6315 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240503 01:59:58.869776 9039 raft_consensus.cc:1261] T e0921a7de46f4385969a72d9088c3bcb P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:58.869940 8598 consensus_queue.cc:1035] T e0921a7de46f4385969a72d9088c3bcb P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:58.870313 8598 consensus_queue.cc:1035] T e0921a7de46f4385969a72d9088c3bcb P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:58.874217 6315 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet 771f069dd030449489008509f9794d7e
I20240503 01:59:58.874826 6315 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet 37b626b17ee242f6ac73689212b7474b
I20240503 01:59:58.875082 6315 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet 481e1968fa634172a4312352e1a1b6ec
I20240503 01:59:58.876736 6307 catalog_manager.cc:5576] TS 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157) does not have the latest schema for tablet e0921a7de46f4385969a72d9088c3bcb (table rebalancer_dynamic_table [id=c7c1d119289f48aab21886443f61f742]). Expected version 1 got 0
I20240503 01:59:58.876978 6568 tablet_service.cc:1514] Processing DeleteTablet for tablet 481e1968fa634172a4312352e1a1b6ec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:58 UTC) from {username='slave'} at 127.0.0.1:59846
I20240503 01:59:58.876946 6567 tablet_service.cc:1514] Processing DeleteTablet for tablet 37b626b17ee242f6ac73689212b7474b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:58 UTC) from {username='slave'} at 127.0.0.1:59846
I20240503 01:59:58.877038 6438 tablet_service.cc:1514] Processing DeleteTablet for tablet 771f069dd030449489008509f9794d7e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:58 UTC) from {username='slave'} at 127.0.0.1:51338
I20240503 01:59:58.877393 6436 tablet_service.cc:1514] Processing DeleteTablet for tablet 37b626b17ee242f6ac73689212b7474b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:58 UTC) from {username='slave'} at 127.0.0.1:51338
I20240503 01:59:58.877430 6437 tablet_service.cc:1514] Processing DeleteTablet for tablet 481e1968fa634172a4312352e1a1b6ec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:58 UTC) from {username='slave'} at 127.0.0.1:51338
I20240503 01:59:58.878208 6312 catalog_manager.cc:5576] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863) does not have the latest schema for tablet e0921a7de46f4385969a72d9088c3bcb (table rebalancer_dynamic_table [id=c7c1d119289f48aab21886443f61f742]). Expected version 1 got 0
I20240503 01:59:58.878221 6700 tablet_service.cc:1514] Processing DeleteTablet for tablet 771f069dd030449489008509f9794d7e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:58 UTC) from {username='slave'} at 127.0.0.1:40336
I20240503 01:59:58.878319 6699 tablet_service.cc:1514] Processing DeleteTablet for tablet 37b626b17ee242f6ac73689212b7474b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:58 UTC) from {username='slave'} at 127.0.0.1:40336
I20240503 01:59:58.878619 6829 tablet_service.cc:1514] Processing DeleteTablet for tablet 771f069dd030449489008509f9794d7e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:58 UTC) from {username='slave'} at 127.0.0.1:52812
I20240503 01:59:58.878669 6830 tablet_service.cc:1514] Processing DeleteTablet for tablet 481e1968fa634172a4312352e1a1b6ec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:58 UTC) from {username='slave'} at 127.0.0.1:52812
I20240503 01:59:58.879465 9476 tablet_replica.cc:330] T 771f069dd030449489008509f9794d7e P 492448570cc94ed1879feb2ec3b9051d: stopping tablet replica
I20240503 01:59:58.879688 9468 mvcc.cc:205] Tried to move back new op lower bound from 7023417748963176448 to 7023417748326043648. Current Snapshot: MvccSnapshot[applied={T|T < 7023417748963176448}]
I20240503 01:59:58.879933 9476 raft_consensus.cc:2229] T 771f069dd030449489008509f9794d7e P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:58.880561 9476 raft_consensus.cc:2258] T 771f069dd030449489008509f9794d7e P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:58.880096 9473 tablet_replica.cc:330] T 481e1968fa634172a4312352e1a1b6ec P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 01:59:58.880934 9473 raft_consensus.cc:2229] T 481e1968fa634172a4312352e1a1b6ec P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:58.880522 9477 tablet.cc:1722] T e0921a7de46f4385969a72d9088c3bcb P e1a9251ce1cf4361b8b0ede9fbfc4023: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:58.881827 9473 raft_consensus.cc:2258] T 481e1968fa634172a4312352e1a1b6ec P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:58.881860 9476 ts_tablet_manager.cc:1875] T 771f069dd030449489008509f9794d7e P 492448570cc94ed1879feb2ec3b9051d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:58.882193 9474 tablet_replica.cc:330] T 771f069dd030449489008509f9794d7e P dec3e4a04d604f7eaaa3f077d081218a: stopping tablet replica
I20240503 01:59:58.882524 9474 raft_consensus.cc:2229] T 771f069dd030449489008509f9794d7e P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:58.882874 9473 ts_tablet_manager.cc:1875] T 481e1968fa634172a4312352e1a1b6ec P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:58.884286 9027 tablet_service.cc:1467] Processing CreateTablet for tablet 51c0137930464635be1e6a97f5e9d952 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=296df7c9486144058745436d9b814a78]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:58.884685 9477 tablet.cc:1620] T e0921a7de46f4385969a72d9088c3bcb P e1a9251ce1cf4361b8b0ede9fbfc4023: MemRowSet was empty: no flush needed.
I20240503 01:59:58.884708 9027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51c0137930464635be1e6a97f5e9d952. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:58.884945 6830 tablet_service.cc:1467] Processing CreateTablet for tablet f44b59277c3b47adbd4dd6aff9350af1 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=296df7c9486144058745436d9b814a78]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:58.885200 6830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f44b59277c3b47adbd4dd6aff9350af1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:58.885800 9285 tablet_service.cc:1467] Processing CreateTablet for tablet f44b59277c3b47adbd4dd6aff9350af1 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=296df7c9486144058745436d9b814a78]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:58.886147 9285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f44b59277c3b47adbd4dd6aff9350af1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:58.887599 9375 tablet_bootstrap.cc:492] T 51c0137930464635be1e6a97f5e9d952 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 01:59:58.884287 9025 tablet_service.cc:1467] Processing CreateTablet for tablet f5ed01740019467d8c1d55cc2fa18ca3 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=296df7c9486144058745436d9b814a78]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:58.887941 9025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5ed01740019467d8c1d55cc2fa18ca3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:58.888528 9375 tablet_bootstrap.cc:654] T 51c0137930464635be1e6a97f5e9d952 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:58.888932 9473 ts_tablet_manager.cc:1888] T 481e1968fa634172a4312352e1a1b6ec P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:58.889024 9473 log.cc:1196] T 481e1968fa634172a4312352e1a1b6ec P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/481e1968fa634172a4312352e1a1b6ec
I20240503 01:59:58.890244 6315 catalog_manager.cc:5576] TS e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311) does not have the latest schema for tablet e0921a7de46f4385969a72d9088c3bcb (table rebalancer_dynamic_table [id=c7c1d119289f48aab21886443f61f742]). Expected version 1 got 0
I20240503 01:59:58.891250 9473 ts_tablet_manager.cc:1909] T 481e1968fa634172a4312352e1a1b6ec P 80fd18ef4d604c86b909d5c008a46b4f: Deleting consensus metadata
I20240503 01:59:58.890244 6437 tablet_service.cc:1467] Processing CreateTablet for tablet 51c0137930464635be1e6a97f5e9d952 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=296df7c9486144058745436d9b814a78]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:58.891842 6437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51c0137930464635be1e6a97f5e9d952. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:58.892279 9374 tablet_bootstrap.cc:492] T f44b59277c3b47adbd4dd6aff9350af1 P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap starting.
I20240503 01:59:58.893086 9473 tablet_replica.cc:330] T 37b626b17ee242f6ac73689212b7474b P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 01:59:58.893280 9374 tablet_bootstrap.cc:654] T f44b59277c3b47adbd4dd6aff9350af1 P e1a9251ce1cf4361b8b0ede9fbfc4023: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:58.893280 9473 raft_consensus.cc:2229] T 37b626b17ee242f6ac73689212b7474b P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:58.893601 9473 raft_consensus.cc:2258] T 37b626b17ee242f6ac73689212b7474b P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:58.894116 9476 ts_tablet_manager.cc:1888] T 771f069dd030449489008509f9794d7e P 492448570cc94ed1879feb2ec3b9051d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:58.894196 9473 ts_tablet_manager.cc:1875] T 37b626b17ee242f6ac73689212b7474b P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:58.894691 9474 raft_consensus.cc:2258] T 771f069dd030449489008509f9794d7e P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:58.895210 9476 log.cc:1196] T 771f069dd030449489008509f9794d7e P 492448570cc94ed1879feb2ec3b9051d: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/771f069dd030449489008509f9794d7e
I20240503 01:59:58.895603 9476 ts_tablet_manager.cc:1909] T 771f069dd030449489008509f9794d7e P 492448570cc94ed1879feb2ec3b9051d: Deleting consensus metadata
I20240503 01:59:58.896631 9474 ts_tablet_manager.cc:1875] T 771f069dd030449489008509f9794d7e P dec3e4a04d604f7eaaa3f077d081218a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:58.895356 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet 481e1968fa634172a4312352e1a1b6ec (table rebalance_test_table_0 [id=cce4a4befe484b7f83338cb366b6106d]) successfully deleted
I20240503 01:59:58.897172 6303 catalog_manager.cc:4830] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719): tablet 771f069dd030449489008509f9794d7e (table rebalance_test_table_0 [id=cce4a4befe484b7f83338cb366b6106d]) successfully deleted
I20240503 01:59:58.898519 9472 tablet.cc:1722] T e0921a7de46f4385969a72d9088c3bcb P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:58.898763 9375 tablet_bootstrap.cc:492] T 51c0137930464635be1e6a97f5e9d952 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 01:59:58.898881 9375 ts_tablet_manager.cc:1367] T 51c0137930464635be1e6a97f5e9d952 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20240503 01:59:58.899466 9375 raft_consensus.cc:348] T 51c0137930464635be1e6a97f5e9d952 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:58.899595 9375 raft_consensus.cc:374] T 51c0137930464635be1e6a97f5e9d952 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:58.899636 9375 raft_consensus.cc:729] T 51c0137930464635be1e6a97f5e9d952 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 01:59:58.899745 9375 consensus_queue.cc:260] T 51c0137930464635be1e6a97f5e9d952 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:58.900030 9375 ts_tablet_manager.cc:1398] T 51c0137930464635be1e6a97f5e9d952 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 01:59:58.900211 9375 tablet_bootstrap.cc:492] T f5ed01740019467d8c1d55cc2fa18ca3 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 01:59:58.900691 9472 tablet.cc:1620] T e0921a7de46f4385969a72d9088c3bcb P 80fd18ef4d604c86b909d5c008a46b4f: MemRowSet was empty: no flush needed.
I20240503 01:59:58.901166 9375 tablet_bootstrap.cc:654] T f5ed01740019467d8c1d55cc2fa18ca3 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:58.901428 9483 tablet_bootstrap.cc:492] T 51c0137930464635be1e6a97f5e9d952 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 01:59:58.902500 9476 tablet_replica.cc:330] T 481e1968fa634172a4312352e1a1b6ec P 492448570cc94ed1879feb2ec3b9051d: stopping tablet replica
I20240503 01:59:58.902518 6315 catalog_manager.cc:5576] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863) does not have the latest schema for tablet e0921a7de46f4385969a72d9088c3bcb (table rebalancer_dynamic_table [id=c7c1d119289f48aab21886443f61f742]). Expected version 1 got 0
I20240503 01:59:58.902745 9476 raft_consensus.cc:2229] T 481e1968fa634172a4312352e1a1b6ec P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:58.903064 6312 catalog_manager.cc:5576] TS 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157) does not have the latest schema for tablet e0921a7de46f4385969a72d9088c3bcb (table rebalancer_dynamic_table [id=c7c1d119289f48aab21886443f61f742]). Expected version 1 got 0
I20240503 01:59:58.903993 9374 tablet_bootstrap.cc:492] T f44b59277c3b47adbd4dd6aff9350af1 P e1a9251ce1cf4361b8b0ede9fbfc4023: No bootstrap required, opened a new log
I20240503 01:59:58.904086 9374 ts_tablet_manager.cc:1367] T f44b59277c3b47adbd4dd6aff9350af1 P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 01:59:58.904248 9483 tablet_bootstrap.cc:654] T 51c0137930464635be1e6a97f5e9d952 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:58.904868 9374 raft_consensus.cc:348] T f44b59277c3b47adbd4dd6aff9350af1 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 01:59:58.904995 9374 raft_consensus.cc:374] T f44b59277c3b47adbd4dd6aff9350af1 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:58.905030 9374 raft_consensus.cc:729] T f44b59277c3b47adbd4dd6aff9350af1 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1a9251ce1cf4361b8b0ede9fbfc4023, State: Initialized, Role: FOLLOWER
I20240503 01:59:58.905153 9374 consensus_queue.cc:260] T f44b59277c3b47adbd4dd6aff9350af1 P e1a9251ce1cf4361b8b0ede9fbfc4023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 01:59:58.905426 9374 ts_tablet_manager.cc:1398] T f44b59277c3b47adbd4dd6aff9350af1 P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 01:59:58.903400 9476 raft_consensus.cc:2258] T 481e1968fa634172a4312352e1a1b6ec P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:58.905488 9480 tablet_bootstrap.cc:492] T f44b59277c3b47adbd4dd6aff9350af1 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 01:59:58.907176 9375 tablet_bootstrap.cc:492] T f5ed01740019467d8c1d55cc2fa18ca3 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 01:59:58.907269 9375 ts_tablet_manager.cc:1367] T f5ed01740019467d8c1d55cc2fa18ca3 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 01:59:58.907737 9375 raft_consensus.cc:348] T f5ed01740019467d8c1d55cc2fa18ca3 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 01:59:58.907855 9375 raft_consensus.cc:374] T f5ed01740019467d8c1d55cc2fa18ca3 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:58.907889 9375 raft_consensus.cc:729] T f5ed01740019467d8c1d55cc2fa18ca3 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 01:59:58.908002 9375 consensus_queue.cc:260] T f5ed01740019467d8c1d55cc2fa18ca3 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 01:59:58.908257 9375 ts_tablet_manager.cc:1398] T f5ed01740019467d8c1d55cc2fa18ca3 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:58.908424 9469 mvcc.cc:205] Tried to move back new op lower bound from 7023417748963176448 to 7023417748326043648. Current Snapshot: MvccSnapshot[applied={T|T < 7023417748963176448}]
I20240503 01:59:58.908892 9474 ts_tablet_manager.cc:1888] T 771f069dd030449489008509f9794d7e P dec3e4a04d604f7eaaa3f077d081218a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:58.908991 9474 log.cc:1196] T 771f069dd030449489008509f9794d7e P dec3e4a04d604f7eaaa3f077d081218a: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/771f069dd030449489008509f9794d7e
I20240503 01:59:58.908999 9473 ts_tablet_manager.cc:1888] T 37b626b17ee242f6ac73689212b7474b P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:58.909073 9473 log.cc:1196] T 37b626b17ee242f6ac73689212b7474b P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/37b626b17ee242f6ac73689212b7474b
I20240503 01:59:58.909287 9474 ts_tablet_manager.cc:1909] T 771f069dd030449489008509f9794d7e P dec3e4a04d604f7eaaa3f077d081218a: Deleting consensus metadata
I20240503 01:59:58.909468 9473 ts_tablet_manager.cc:1909] T 37b626b17ee242f6ac73689212b7474b P 80fd18ef4d604c86b909d5c008a46b4f: Deleting consensus metadata
I20240503 01:59:58.910315 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet 37b626b17ee242f6ac73689212b7474b (table rebalance_test_table_0 [id=cce4a4befe484b7f83338cb366b6106d]) successfully deleted
I20240503 01:59:58.910207 9485 tablet.cc:1722] T e0921a7de46f4385969a72d9088c3bcb P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:58.911841 9485 tablet.cc:1620] T e0921a7de46f4385969a72d9088c3bcb P 99ead02915954b4fab23b033ad6d546b: MemRowSet was empty: no flush needed.
I20240503 01:59:58.912024 6315 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=c7c1d119289f48aab21886443f61f742] alter complete (version 1)
I20240503 01:59:58.919246 9474 tablet_replica.cc:330] T 37b626b17ee242f6ac73689212b7474b P dec3e4a04d604f7eaaa3f077d081218a: stopping tablet replica
I20240503 01:59:58.919307 6300 catalog_manager.cc:4830] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): tablet 771f069dd030449489008509f9794d7e (table rebalance_test_table_0 [id=cce4a4befe484b7f83338cb366b6106d]) successfully deleted
I20240503 01:59:58.919972 9474 raft_consensus.cc:2229] T 37b626b17ee242f6ac73689212b7474b P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:58.919706 8887 tablet_service.cc:1467] Processing CreateTablet for tablet f5ed01740019467d8c1d55cc2fa18ca3 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=296df7c9486144058745436d9b814a78]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:58.920492 9474 raft_consensus.cc:2258] T 37b626b17ee242f6ac73689212b7474b P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:58.920811 8880 tablet_service.cc:1467] Processing CreateTablet for tablet 51c0137930464635be1e6a97f5e9d952 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=296df7c9486144058745436d9b814a78]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:58.921522 8880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51c0137930464635be1e6a97f5e9d952. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:58.921780 9474 ts_tablet_manager.cc:1875] T 37b626b17ee242f6ac73689212b7474b P dec3e4a04d604f7eaaa3f077d081218a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:58.922412 9480 tablet_bootstrap.cc:654] T f44b59277c3b47adbd4dd6aff9350af1 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:58.924068 9474 ts_tablet_manager.cc:1888] T 37b626b17ee242f6ac73689212b7474b P dec3e4a04d604f7eaaa3f077d081218a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:58.924144 9474 log.cc:1196] T 37b626b17ee242f6ac73689212b7474b P dec3e4a04d604f7eaaa3f077d081218a: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/37b626b17ee242f6ac73689212b7474b
I20240503 01:59:58.924428 9474 ts_tablet_manager.cc:1909] T 37b626b17ee242f6ac73689212b7474b P dec3e4a04d604f7eaaa3f077d081218a: Deleting consensus metadata
I20240503 01:59:58.925379 9483 tablet_bootstrap.cc:492] T 51c0137930464635be1e6a97f5e9d952 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 01:59:58.925505 9483 ts_tablet_manager.cc:1367] T 51c0137930464635be1e6a97f5e9d952 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.000s
I20240503 01:59:58.925954 9483 raft_consensus.cc:348] T 51c0137930464635be1e6a97f5e9d952 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:58.926064 9483 raft_consensus.cc:374] T 51c0137930464635be1e6a97f5e9d952 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:58.926095 9483 raft_consensus.cc:729] T 51c0137930464635be1e6a97f5e9d952 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 01:59:58.926290 9483 consensus_queue.cc:260] T 51c0137930464635be1e6a97f5e9d952 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:58.926498 8887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5ed01740019467d8c1d55cc2fa18ca3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:58.906493 9476 ts_tablet_manager.cc:1875] T 481e1968fa634172a4312352e1a1b6ec P 492448570cc94ed1879feb2ec3b9051d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:58.928730 9475 tablet_replica.cc:330] T 771f069dd030449489008509f9794d7e P c2ac2eb7cc264047b9ab32fe3a4fc025: stopping tablet replica
I20240503 01:59:58.929101 9483 ts_tablet_manager.cc:1398] T 51c0137930464635be1e6a97f5e9d952 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 01:59:58.929662 9475 raft_consensus.cc:2229] T 771f069dd030449489008509f9794d7e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:58.931226 9475 raft_consensus.cc:2258] T 771f069dd030449489008509f9794d7e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:58.931594 9476 ts_tablet_manager.cc:1888] T 481e1968fa634172a4312352e1a1b6ec P 492448570cc94ed1879feb2ec3b9051d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:58.931679 9476 log.cc:1196] T 481e1968fa634172a4312352e1a1b6ec P 492448570cc94ed1879feb2ec3b9051d: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/481e1968fa634172a4312352e1a1b6ec
I20240503 01:59:58.931962 9476 ts_tablet_manager.cc:1909] T 481e1968fa634172a4312352e1a1b6ec P 492448570cc94ed1879feb2ec3b9051d: Deleting consensus metadata
I20240503 01:59:58.932988 6303 catalog_manager.cc:4830] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719): tablet 481e1968fa634172a4312352e1a1b6ec (table rebalance_test_table_0 [id=cce4a4befe484b7f83338cb366b6106d]) successfully deleted
I20240503 01:59:58.933852 6300 catalog_manager.cc:4830] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): tablet 37b626b17ee242f6ac73689212b7474b (table rebalance_test_table_0 [id=cce4a4befe484b7f83338cb366b6106d]) successfully deleted
I20240503 01:59:58.934092 9474 tablet_replica.cc:330] T 481e1968fa634172a4312352e1a1b6ec P dec3e4a04d604f7eaaa3f077d081218a: stopping tablet replica
I20240503 01:59:58.934234 9474 raft_consensus.cc:2229] T 481e1968fa634172a4312352e1a1b6ec P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:58.934370 9474 raft_consensus.cc:2258] T 481e1968fa634172a4312352e1a1b6ec P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:58.934994 9474 ts_tablet_manager.cc:1875] T 481e1968fa634172a4312352e1a1b6ec P dec3e4a04d604f7eaaa3f077d081218a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:58.934684 9159 tablet_service.cc:1467] Processing CreateTablet for tablet f5ed01740019467d8c1d55cc2fa18ca3 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=296df7c9486144058745436d9b814a78]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:58.935747 9159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5ed01740019467d8c1d55cc2fa18ca3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:58.936690 9158 tablet_service.cc:1467] Processing CreateTablet for tablet f44b59277c3b47adbd4dd6aff9350af1 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=296df7c9486144058745436d9b814a78]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:58.936957 9158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f44b59277c3b47adbd4dd6aff9350af1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:58.937307 9474 ts_tablet_manager.cc:1888] T 481e1968fa634172a4312352e1a1b6ec P dec3e4a04d604f7eaaa3f077d081218a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:58.937386 9474 log.cc:1196] T 481e1968fa634172a4312352e1a1b6ec P dec3e4a04d604f7eaaa3f077d081218a: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/481e1968fa634172a4312352e1a1b6ec
I20240503 01:59:58.937646 9474 ts_tablet_manager.cc:1909] T 481e1968fa634172a4312352e1a1b6ec P dec3e4a04d604f7eaaa3f077d081218a: Deleting consensus metadata
I20240503 01:59:58.938524 6300 catalog_manager.cc:4830] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): tablet 481e1968fa634172a4312352e1a1b6ec (table rebalance_test_table_0 [id=cce4a4befe484b7f83338cb366b6106d]) successfully deleted
I20240503 01:59:58.943509 9475 ts_tablet_manager.cc:1875] T 771f069dd030449489008509f9794d7e P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:58.946192 9528 tablet_bootstrap.cc:492] T f5ed01740019467d8c1d55cc2fa18ca3 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 01:59:58.946552 9475 ts_tablet_manager.cc:1888] T 771f069dd030449489008509f9794d7e P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:58.946658 9475 log.cc:1196] T 771f069dd030449489008509f9794d7e P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/771f069dd030449489008509f9794d7e
I20240503 01:59:58.946990 9475 ts_tablet_manager.cc:1909] T 771f069dd030449489008509f9794d7e P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting consensus metadata
I20240503 01:59:58.948251 9475 tablet_replica.cc:330] T 37b626b17ee242f6ac73689212b7474b P c2ac2eb7cc264047b9ab32fe3a4fc025: stopping tablet replica
I20240503 01:59:58.948311 6303 catalog_manager.cc:4830] TS c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655): tablet 771f069dd030449489008509f9794d7e (table rebalance_test_table_0 [id=cce4a4befe484b7f83338cb366b6106d]) successfully deleted
I20240503 01:59:58.949057 9509 tablet_bootstrap.cc:492] T f5ed01740019467d8c1d55cc2fa18ca3 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 01:59:58.949158 9480 tablet_bootstrap.cc:492] T f44b59277c3b47adbd4dd6aff9350af1 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 01:59:58.949259 9480 ts_tablet_manager.cc:1367] T f44b59277c3b47adbd4dd6aff9350af1 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.044s user 0.002s sys 0.000s
I20240503 01:59:58.949702 9480 raft_consensus.cc:348] T f44b59277c3b47adbd4dd6aff9350af1 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 01:59:58.949822 9480 raft_consensus.cc:374] T f44b59277c3b47adbd4dd6aff9350af1 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:58.949851 9480 raft_consensus.cc:729] T f44b59277c3b47adbd4dd6aff9350af1 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 01:59:58.949980 9480 consensus_queue.cc:260] T f44b59277c3b47adbd4dd6aff9350af1 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 01:59:58.950258 9480 ts_tablet_manager.cc:1398] T f44b59277c3b47adbd4dd6aff9350af1 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:58.951702 9528 tablet_bootstrap.cc:654] T f5ed01740019467d8c1d55cc2fa18ca3 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:58.951675 9509 tablet_bootstrap.cc:654] T f5ed01740019467d8c1d55cc2fa18ca3 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:58.954861 9475 raft_consensus.cc:2229] T 37b626b17ee242f6ac73689212b7474b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:58.955160 9528 log.cc:824] T f5ed01740019467d8c1d55cc2fa18ca3 P 2a01ace80baf4057b6bf1995cf88468b: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:58.955219 9475 raft_consensus.cc:2258] T 37b626b17ee242f6ac73689212b7474b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:58.955952 9475 ts_tablet_manager.cc:1875] T 37b626b17ee242f6ac73689212b7474b P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:58.957435 9528 tablet_bootstrap.cc:492] T f5ed01740019467d8c1d55cc2fa18ca3 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 01:59:58.957638 9528 ts_tablet_manager.cc:1367] T f5ed01740019467d8c1d55cc2fa18ca3 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20240503 01:59:58.958302 9475 ts_tablet_manager.cc:1888] T 37b626b17ee242f6ac73689212b7474b P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:58.958379 9475 log.cc:1196] T 37b626b17ee242f6ac73689212b7474b P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/37b626b17ee242f6ac73689212b7474b
I20240503 01:59:58.958609 9475 ts_tablet_manager.cc:1909] T 37b626b17ee242f6ac73689212b7474b P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting consensus metadata
I20240503 01:59:58.959858 9509 log.cc:824] T f5ed01740019467d8c1d55cc2fa18ca3 P 9942306855404cfd891adf227dd65eb2: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:58.961030 9528 raft_consensus.cc:348] T f5ed01740019467d8c1d55cc2fa18ca3 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 01:59:58.961258 9528 raft_consensus.cc:374] T f5ed01740019467d8c1d55cc2fa18ca3 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:58.961314 9528 raft_consensus.cc:729] T f5ed01740019467d8c1d55cc2fa18ca3 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 01:59:58.962365 6303 catalog_manager.cc:4830] TS c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655): tablet 37b626b17ee242f6ac73689212b7474b (table rebalance_test_table_0 [id=cce4a4befe484b7f83338cb366b6106d]) successfully deleted
I20240503 01:59:58.963711 9528 consensus_queue.cc:260] T f5ed01740019467d8c1d55cc2fa18ca3 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 01:59:58.965070 9528 ts_tablet_manager.cc:1398] T f5ed01740019467d8c1d55cc2fa18ca3 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240503 01:59:58.965518 9225 heartbeater.cc:503] Master 127.0.99.190:35423 was elected leader, sending a full tablet report...
I20240503 01:59:58.968005 9509 tablet_bootstrap.cc:492] T f5ed01740019467d8c1d55cc2fa18ca3 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 01:59:58.968216 9528 tablet_bootstrap.cc:492] T f44b59277c3b47adbd4dd6aff9350af1 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 01:59:58.968228 9509 ts_tablet_manager.cc:1367] T f5ed01740019467d8c1d55cc2fa18ca3 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20240503 01:59:58.970388 9528 tablet_bootstrap.cc:654] T f44b59277c3b47adbd4dd6aff9350af1 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:58.979638 9509 raft_consensus.cc:348] T f5ed01740019467d8c1d55cc2fa18ca3 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 01:59:58.979885 9509 raft_consensus.cc:374] T f5ed01740019467d8c1d55cc2fa18ca3 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:58.979952 9509 raft_consensus.cc:729] T f5ed01740019467d8c1d55cc2fa18ca3 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 01:59:58.980535 9509 consensus_queue.cc:260] T f5ed01740019467d8c1d55cc2fa18ca3 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 01:59:58.984869 9528 tablet_bootstrap.cc:492] T f44b59277c3b47adbd4dd6aff9350af1 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 01:59:58.984997 9528 ts_tablet_manager.cc:1367] T f44b59277c3b47adbd4dd6aff9350af1 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.001s
I20240503 01:59:58.985672 9528 raft_consensus.cc:348] T f44b59277c3b47adbd4dd6aff9350af1 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 01:59:58.985802 9528 raft_consensus.cc:374] T f44b59277c3b47adbd4dd6aff9350af1 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:58.985839 9528 raft_consensus.cc:729] T f44b59277c3b47adbd4dd6aff9350af1 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 01:59:58.985839 9509 ts_tablet_manager.cc:1398] T f5ed01740019467d8c1d55cc2fa18ca3 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.018s user 0.007s sys 0.000s
I20240503 01:59:58.985961 9528 consensus_queue.cc:260] T f44b59277c3b47adbd4dd6aff9350af1 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 01:59:58.986263 9509 tablet_bootstrap.cc:492] T 51c0137930464635be1e6a97f5e9d952 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 01:59:58.986251 9528 ts_tablet_manager.cc:1398] T f44b59277c3b47adbd4dd6aff9350af1 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:58.987692 9509 tablet_bootstrap.cc:654] T 51c0137930464635be1e6a97f5e9d952 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:58.988965 9509 tablet_bootstrap.cc:492] T 51c0137930464635be1e6a97f5e9d952 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 01:59:58.989068 9509 ts_tablet_manager.cc:1367] T 51c0137930464635be1e6a97f5e9d952 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:58.989665 9509 raft_consensus.cc:348] T 51c0137930464635be1e6a97f5e9d952 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:58.989786 9509 raft_consensus.cc:374] T 51c0137930464635be1e6a97f5e9d952 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:58.989818 9509 raft_consensus.cc:729] T 51c0137930464635be1e6a97f5e9d952 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 01:59:58.989938 9509 consensus_queue.cc:260] T 51c0137930464635be1e6a97f5e9d952 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:58.990314 9509 ts_tablet_manager.cc:1398] T 51c0137930464635be1e6a97f5e9d952 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 01:59:59.013732 6307 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 01:59:59.019726 9472 tablet.cc:1722] T e0921a7de46f4385969a72d9088c3bcb P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:59.020057 9472 tablet.cc:1620] T e0921a7de46f4385969a72d9088c3bcb P 80fd18ef4d604c86b909d5c008a46b4f: MemRowSet was empty: no flush needed.
I20240503 01:59:59.020892 9477 tablet.cc:1722] T e0921a7de46f4385969a72d9088c3bcb P e1a9251ce1cf4361b8b0ede9fbfc4023: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:59.021207 9477 tablet.cc:1620] T e0921a7de46f4385969a72d9088c3bcb P e1a9251ce1cf4361b8b0ede9fbfc4023: MemRowSet was empty: no flush needed.
W20240503 01:59:59.022989 8964 tablet.cc:2369] T 51c0137930464635be1e6a97f5e9d952 P 9942306855404cfd891adf227dd65eb2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 01:59:59.027614 6313 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=c7c1d119289f48aab21886443f61f742] alter complete (version 2)
I20240503 01:59:59.030660 9485 tablet.cc:1722] T e0921a7de46f4385969a72d9088c3bcb P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:59.031020 9485 tablet.cc:1620] T e0921a7de46f4385969a72d9088c3bcb P 99ead02915954b4fab23b033ad6d546b: MemRowSet was empty: no flush needed.
I20240503 01:59:59.043179 8679 raft_consensus.cc:482] T 51c0137930464635be1e6a97f5e9d952 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:59.043318 8679 raft_consensus.cc:504] T 51c0137930464635be1e6a97f5e9d952 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:59.043949 8679 leader_election.cc:288] T 51c0137930464635be1e6a97f5e9d952 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 01:59:59.048341 8917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51c0137930464635be1e6a97f5e9d952" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 01:59:59.048744 8917 raft_consensus.cc:2454] T 51c0137930464635be1e6a97f5e9d952 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 01:59:59.048895 9039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51c0137930464635be1e6a97f5e9d952" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 01:59:59.049052 9039 raft_consensus.cc:2454] T 51c0137930464635be1e6a97f5e9d952 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 01:59:59.049381 6392 leader_election.cc:302] T 51c0137930464635be1e6a97f5e9d952 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99ead02915954b4fab23b033ad6d546b, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:59.049604 8679 raft_consensus.cc:2781] T 51c0137930464635be1e6a97f5e9d952 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:59.049664 8679 raft_consensus.cc:482] T 51c0137930464635be1e6a97f5e9d952 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:59.049702 8679 raft_consensus.cc:3037] T 51c0137930464635be1e6a97f5e9d952 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:59.050570 8679 raft_consensus.cc:504] T 51c0137930464635be1e6a97f5e9d952 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:59.050940 8679 leader_election.cc:288] T 51c0137930464635be1e6a97f5e9d952 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Requested vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 01:59:59.051334 9039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51c0137930464635be1e6a97f5e9d952" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 01:59:59.051491 9039 raft_consensus.cc:3037] T 51c0137930464635be1e6a97f5e9d952 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:59.051334 8917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51c0137930464635be1e6a97f5e9d952" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 01:59:59.051936 8917 raft_consensus.cc:3037] T 51c0137930464635be1e6a97f5e9d952 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:59.052628 9039 raft_consensus.cc:2454] T 51c0137930464635be1e6a97f5e9d952 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 01:59:59.053045 8917 raft_consensus.cc:2454] T 51c0137930464635be1e6a97f5e9d952 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 01:59:59.053112 6392 leader_election.cc:302] T 51c0137930464635be1e6a97f5e9d952 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99ead02915954b4fab23b033ad6d546b, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:59.053404 8679 raft_consensus.cc:2781] T 51c0137930464635be1e6a97f5e9d952 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:59.053503 8679 raft_consensus.cc:686] T 51c0137930464635be1e6a97f5e9d952 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Becoming Leader. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Running, Role: LEADER
I20240503 01:59:59.053640 8679 consensus_queue.cc:237] T 51c0137930464635be1e6a97f5e9d952 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 01:59:59.055809 6307 catalog_manager.cc:5484] T 51c0137930464635be1e6a97f5e9d952 P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: term changed from 0 to 1, leader changed from <none> to dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129). New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:59.068140 9531 raft_consensus.cc:482] T f5ed01740019467d8c1d55cc2fa18ca3 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:59.068331 9531 raft_consensus.cc:504] T f5ed01740019467d8c1d55cc2fa18ca3 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 01:59:59.069451 9531 leader_election.cc:288] T f5ed01740019467d8c1d55cc2fa18ca3 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 01:59:59.073648 9039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5ed01740019467d8c1d55cc2fa18ca3" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 01:59:59.073843 9039 raft_consensus.cc:2454] T f5ed01740019467d8c1d55cc2fa18ca3 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 01:59:59.073832 8917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5ed01740019467d8c1d55cc2fa18ca3" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 01:59:59.073951 8917 raft_consensus.cc:2454] T f5ed01740019467d8c1d55cc2fa18ca3 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 01:59:59.074301 9113 leader_election.cc:302] T f5ed01740019467d8c1d55cc2fa18ca3 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 01:59:59.074589 9531 raft_consensus.cc:2781] T f5ed01740019467d8c1d55cc2fa18ca3 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:59.074676 9531 raft_consensus.cc:482] T f5ed01740019467d8c1d55cc2fa18ca3 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:59.074721 9531 raft_consensus.cc:3037] T f5ed01740019467d8c1d55cc2fa18ca3 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:59.075649 9531 raft_consensus.cc:504] T f5ed01740019467d8c1d55cc2fa18ca3 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 01:59:59.076025 9531 leader_election.cc:288] T f5ed01740019467d8c1d55cc2fa18ca3 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Requested vote from peers 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 01:59:59.076506 9039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5ed01740019467d8c1d55cc2fa18ca3" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 01:59:59.076516 8917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5ed01740019467d8c1d55cc2fa18ca3" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 01:59:59.076638 9039 raft_consensus.cc:3037] T f5ed01740019467d8c1d55cc2fa18ca3 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:59.076645 8917 raft_consensus.cc:3037] T f5ed01740019467d8c1d55cc2fa18ca3 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:59.077629 8917 raft_consensus.cc:2454] T f5ed01740019467d8c1d55cc2fa18ca3 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 01:59:59.077636 9039 raft_consensus.cc:2454] T f5ed01740019467d8c1d55cc2fa18ca3 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 01:59:59.077965 9113 leader_election.cc:302] T f5ed01740019467d8c1d55cc2fa18ca3 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 01:59:59.078264 9531 raft_consensus.cc:2781] T f5ed01740019467d8c1d55cc2fa18ca3 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:59.078495 9531 raft_consensus.cc:686] T f5ed01740019467d8c1d55cc2fa18ca3 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Becoming Leader. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Running, Role: LEADER
I20240503 01:59:59.078884 9531 consensus_queue.cc:237] T f5ed01740019467d8c1d55cc2fa18ca3 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 01:59:59.082252 6307 catalog_manager.cc:5484] T f5ed01740019467d8c1d55cc2fa18ca3 P 2a01ace80baf4057b6bf1995cf88468b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135). New cstate: current_term: 1 leader_uuid: "2a01ace80baf4057b6bf1995cf88468b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:59.130199 6307 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 01:59:59.136364 9472 tablet.cc:1722] T e0921a7de46f4385969a72d9088c3bcb P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:59.137136 9485 tablet.cc:1722] T e0921a7de46f4385969a72d9088c3bcb P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:59.137480 9477 tablet.cc:1722] T e0921a7de46f4385969a72d9088c3bcb P e1a9251ce1cf4361b8b0ede9fbfc4023: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:59.142050 6307 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=c7c1d119289f48aab21886443f61f742] alter complete (version 3)
W20240503 01:59:59.156334 9226 tablet.cc:2369] T f44b59277c3b47adbd4dd6aff9350af1 P 2a01ace80baf4057b6bf1995cf88468b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 01:59:59.216724 9531 raft_consensus.cc:482] T f44b59277c3b47adbd4dd6aff9350af1 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:59.216897 9531 raft_consensus.cc:504] T f44b59277c3b47adbd4dd6aff9350af1 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 01:59:59.217382 9531 leader_election.cc:288] T f44b59277c3b47adbd4dd6aff9350af1 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311)
I20240503 01:59:59.221882 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f44b59277c3b47adbd4dd6aff9350af1" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 01:59:59.221926 9310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f44b59277c3b47adbd4dd6aff9350af1" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" is_pre_election: true
I20240503 01:59:59.222031 6849 raft_consensus.cc:2454] T f44b59277c3b47adbd4dd6aff9350af1 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 01:59:59.222064 9310 raft_consensus.cc:2454] T f44b59277c3b47adbd4dd6aff9350af1 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 01:59:59.222328 9115 leader_election.cc:302] T f44b59277c3b47adbd4dd6aff9350af1 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 492448570cc94ed1879feb2ec3b9051d; no voters:
I20240503 01:59:59.222504 9531 raft_consensus.cc:2781] T f44b59277c3b47adbd4dd6aff9350af1 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:59.222566 9531 raft_consensus.cc:482] T f44b59277c3b47adbd4dd6aff9350af1 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:59.222604 9531 raft_consensus.cc:3037] T f44b59277c3b47adbd4dd6aff9350af1 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:59.223485 9531 raft_consensus.cc:504] T f44b59277c3b47adbd4dd6aff9350af1 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 01:59:59.223870 9531 leader_election.cc:288] T f44b59277c3b47adbd4dd6aff9350af1 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Requested vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311)
I20240503 01:59:59.224221 9310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f44b59277c3b47adbd4dd6aff9350af1" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023"
I20240503 01:59:59.224364 9310 raft_consensus.cc:3037] T f44b59277c3b47adbd4dd6aff9350af1 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:59.224424 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f44b59277c3b47adbd4dd6aff9350af1" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 01:59:59.224529 6849 raft_consensus.cc:3037] T f44b59277c3b47adbd4dd6aff9350af1 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:59.225344 9310 raft_consensus.cc:2454] T f44b59277c3b47adbd4dd6aff9350af1 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 01:59:59.225345 6849 raft_consensus.cc:2454] T f44b59277c3b47adbd4dd6aff9350af1 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 01:59:59.225677 9113 leader_election.cc:302] T f44b59277c3b47adbd4dd6aff9350af1 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, e1a9251ce1cf4361b8b0ede9fbfc4023; no voters:
I20240503 01:59:59.225936 9531 raft_consensus.cc:2781] T f44b59277c3b47adbd4dd6aff9350af1 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:59.226016 9531 raft_consensus.cc:686] T f44b59277c3b47adbd4dd6aff9350af1 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Becoming Leader. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Running, Role: LEADER
I20240503 01:59:59.226156 9531 consensus_queue.cc:237] T f44b59277c3b47adbd4dd6aff9350af1 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 01:59:59.228030 6307 catalog_manager.cc:5484] T f44b59277c3b47adbd4dd6aff9350af1 P 2a01ace80baf4057b6bf1995cf88468b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135). New cstate: current_term: 1 leader_uuid: "2a01ace80baf4057b6bf1995cf88468b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:59.244599 6307 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:59.244760 6307 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:59.246989 6307 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet e0921a7de46f4385969a72d9088c3bcb
I20240503 01:59:59.247525 6568 tablet_service.cc:1514] Processing DeleteTablet for tablet e0921a7de46f4385969a72d9088c3bcb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:59 UTC) from {username='slave'} at 127.0.0.1:59846
I20240503 01:59:59.247675 9285 tablet_service.cc:1514] Processing DeleteTablet for tablet e0921a7de46f4385969a72d9088c3bcb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:59 UTC) from {username='slave'} at 127.0.0.1:36004
I20240503 01:59:59.247766 9473 tablet_replica.cc:330] T e0921a7de46f4385969a72d9088c3bcb P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 01:59:59.247903 9473 raft_consensus.cc:2229] T e0921a7de46f4385969a72d9088c3bcb P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:59.247979 9025 tablet_service.cc:1514] Processing DeleteTablet for tablet e0921a7de46f4385969a72d9088c3bcb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:59 UTC) from {username='slave'} at 127.0.0.1:52584
I20240503 01:59:59.248284 9473 raft_consensus.cc:2258] T e0921a7de46f4385969a72d9088c3bcb P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:59.248421 9545 tablet_replica.cc:330] T e0921a7de46f4385969a72d9088c3bcb P 99ead02915954b4fab23b033ad6d546b: stopping tablet replica
I20240503 01:59:59.248754 9545 raft_consensus.cc:2229] T e0921a7de46f4385969a72d9088c3bcb P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:59.248837 9544 tablet_replica.cc:330] T e0921a7de46f4385969a72d9088c3bcb P e1a9251ce1cf4361b8b0ede9fbfc4023: stopping tablet replica
I20240503 01:59:59.248953 9473 ts_tablet_manager.cc:1875] T e0921a7de46f4385969a72d9088c3bcb P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:59.249001 9545 raft_consensus.cc:2258] T e0921a7de46f4385969a72d9088c3bcb P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:59.249119 9544 raft_consensus.cc:2229] T e0921a7de46f4385969a72d9088c3bcb P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:59.249495 9544 raft_consensus.cc:2258] T e0921a7de46f4385969a72d9088c3bcb P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:59.250497 9544 ts_tablet_manager.cc:1875] T e0921a7de46f4385969a72d9088c3bcb P e1a9251ce1cf4361b8b0ede9fbfc4023: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:59.250542 9545 ts_tablet_manager.cc:1875] T e0921a7de46f4385969a72d9088c3bcb P 99ead02915954b4fab23b033ad6d546b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:59.253235 9544 ts_tablet_manager.cc:1888] T e0921a7de46f4385969a72d9088c3bcb P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:59.253329 9544 log.cc:1196] T e0921a7de46f4385969a72d9088c3bcb P e1a9251ce1cf4361b8b0ede9fbfc4023: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/e0921a7de46f4385969a72d9088c3bcb
I20240503 01:59:59.253440 9545 ts_tablet_manager.cc:1888] T e0921a7de46f4385969a72d9088c3bcb P 99ead02915954b4fab23b033ad6d546b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:59.253563 9545 log.cc:1196] T e0921a7de46f4385969a72d9088c3bcb P 99ead02915954b4fab23b033ad6d546b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/e0921a7de46f4385969a72d9088c3bcb
I20240503 01:59:59.253724 9544 ts_tablet_manager.cc:1909] T e0921a7de46f4385969a72d9088c3bcb P e1a9251ce1cf4361b8b0ede9fbfc4023: Deleting consensus metadata
I20240503 01:59:59.253918 9545 ts_tablet_manager.cc:1909] T e0921a7de46f4385969a72d9088c3bcb P 99ead02915954b4fab23b033ad6d546b: Deleting consensus metadata
I20240503 01:59:59.254252 9473 ts_tablet_manager.cc:1888] T e0921a7de46f4385969a72d9088c3bcb P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:59.254333 9473 log.cc:1196] T e0921a7de46f4385969a72d9088c3bcb P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/e0921a7de46f4385969a72d9088c3bcb
I20240503 01:59:59.254556 9473 ts_tablet_manager.cc:1909] T e0921a7de46f4385969a72d9088c3bcb P 80fd18ef4d604c86b909d5c008a46b4f: Deleting consensus metadata
I20240503 01:59:59.255077 6301 catalog_manager.cc:4830] TS 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157): tablet e0921a7de46f4385969a72d9088c3bcb (table rebalancer_dynamic_table_new_name [id=c7c1d119289f48aab21886443f61f742]) successfully deleted
I20240503 01:59:59.256143 6301 catalog_manager.cc:4830] TS e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): tablet e0921a7de46f4385969a72d9088c3bcb (table rebalancer_dynamic_table_new_name [id=c7c1d119289f48aab21886443f61f742]) successfully deleted
I20240503 01:59:59.258718 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet e0921a7de46f4385969a72d9088c3bcb (table rebalancer_dynamic_table_new_name [id=c7c1d119289f48aab21886443f61f742]) successfully deleted
I20240503 01:59:59.262974 9158 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:59.264756 8880 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:59.266093 6700 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 01:59:59.279666 6830 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:59.283051 9025 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:59.283995 9285 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:59.285741 6568 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 01:59:59.288586 6437 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 01:59:59.349285 6307 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 01:59:59.356931 9025 tablet_service.cc:1467] Processing CreateTablet for tablet f758d325e8e946deb97be30469844809 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8b571be2b4d14eab84dc1bc1e5b8d53a]), partition=
I20240503 01:59:59.356943 9285 tablet_service.cc:1467] Processing CreateTablet for tablet f758d325e8e946deb97be30469844809 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8b571be2b4d14eab84dc1bc1e5b8d53a]), partition=
I20240503 01:59:59.357239 9285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f758d325e8e946deb97be30469844809. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:59.357326 9025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f758d325e8e946deb97be30469844809. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:59.359136 9375 tablet_bootstrap.cc:492] T f758d325e8e946deb97be30469844809 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 01:59:59.359408 6568 tablet_service.cc:1467] Processing CreateTablet for tablet f758d325e8e946deb97be30469844809 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8b571be2b4d14eab84dc1bc1e5b8d53a]), partition=
I20240503 01:59:59.359680 6568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f758d325e8e946deb97be30469844809. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:59.360080 9375 tablet_bootstrap.cc:654] T f758d325e8e946deb97be30469844809 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:59.362010 9374 tablet_bootstrap.cc:492] T f758d325e8e946deb97be30469844809 P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap starting.
I20240503 01:59:59.362008 9565 tablet_bootstrap.cc:492] T f758d325e8e946deb97be30469844809 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 01:59:59.362807 9374 tablet_bootstrap.cc:654] T f758d325e8e946deb97be30469844809 P e1a9251ce1cf4361b8b0ede9fbfc4023: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:59.362882 9565 tablet_bootstrap.cc:654] T f758d325e8e946deb97be30469844809 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:59.364171 9374 tablet_bootstrap.cc:492] T f758d325e8e946deb97be30469844809 P e1a9251ce1cf4361b8b0ede9fbfc4023: No bootstrap required, opened a new log
I20240503 01:59:59.364270 9374 ts_tablet_manager.cc:1367] T f758d325e8e946deb97be30469844809 P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:59.364740 9374 raft_consensus.cc:348] T f758d325e8e946deb97be30469844809 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:59.364864 9374 raft_consensus.cc:374] T f758d325e8e946deb97be30469844809 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:59.364895 9374 raft_consensus.cc:729] T f758d325e8e946deb97be30469844809 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1a9251ce1cf4361b8b0ede9fbfc4023, State: Initialized, Role: FOLLOWER
I20240503 01:59:59.365028 9374 consensus_queue.cc:260] T f758d325e8e946deb97be30469844809 P e1a9251ce1cf4361b8b0ede9fbfc4023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:59.365314 9374 ts_tablet_manager.cc:1398] T f758d325e8e946deb97be30469844809 P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:59.367169 9375 tablet_bootstrap.cc:492] T f758d325e8e946deb97be30469844809 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 01:59:59.367285 9375 ts_tablet_manager.cc:1367] T f758d325e8e946deb97be30469844809 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20240503 01:59:59.367837 9375 raft_consensus.cc:348] T f758d325e8e946deb97be30469844809 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:59.367954 9375 raft_consensus.cc:374] T f758d325e8e946deb97be30469844809 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:59.367985 9375 raft_consensus.cc:729] T f758d325e8e946deb97be30469844809 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 01:59:59.368108 9375 consensus_queue.cc:260] T f758d325e8e946deb97be30469844809 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:59.368371 9375 ts_tablet_manager.cc:1398] T f758d325e8e946deb97be30469844809 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:59.375305 9565 tablet_bootstrap.cc:492] T f758d325e8e946deb97be30469844809 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 01:59:59.375427 9565 ts_tablet_manager.cc:1367] T f758d325e8e946deb97be30469844809 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 01:59:59.375878 9565 raft_consensus.cc:348] T f758d325e8e946deb97be30469844809 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:59.375998 9565 raft_consensus.cc:374] T f758d325e8e946deb97be30469844809 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:59.376019 9565 raft_consensus.cc:729] T f758d325e8e946deb97be30469844809 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 01:59:59.376176 9565 consensus_queue.cc:260] T f758d325e8e946deb97be30469844809 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:59.376693 9565 ts_tablet_manager.cc:1398] T f758d325e8e946deb97be30469844809 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 01:59:59.379664 6307 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240503 01:59:59.379828 6307 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240503 01:59:59.383021 6307 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet ba8b8fdffeeb423a9d89dca999b25904
I20240503 01:59:59.383479 6307 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet 714c076813ff4c4f84cb16d256f7b28c
I20240503 01:59:59.383682 6307 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet 7a44d033ab014466bd2100c0ca250ec8
I20240503 01:59:59.384541 6437 tablet_service.cc:1514] Processing DeleteTablet for tablet ba8b8fdffeeb423a9d89dca999b25904 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:59 UTC) from {username='slave'} at 127.0.0.1:51338
I20240503 01:59:59.384917 9474 tablet_replica.cc:330] T ba8b8fdffeeb423a9d89dca999b25904 P dec3e4a04d604f7eaaa3f077d081218a: stopping tablet replica
I20240503 01:59:59.385118 9474 raft_consensus.cc:2229] T ba8b8fdffeeb423a9d89dca999b25904 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:59.385357 9474 raft_consensus.cc:2258] T ba8b8fdffeeb423a9d89dca999b25904 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:59.385856 6436 tablet_service.cc:1514] Processing DeleteTablet for tablet 714c076813ff4c4f84cb16d256f7b28c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:59 UTC) from {username='slave'} at 127.0.0.1:51338
I20240503 01:59:59.385950 9474 ts_tablet_manager.cc:1875] T ba8b8fdffeeb423a9d89dca999b25904 P dec3e4a04d604f7eaaa3f077d081218a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:59.386060 6438 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a44d033ab014466bd2100c0ca250ec8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:59 UTC) from {username='slave'} at 127.0.0.1:51338
I20240503 01:59:59.388922 6568 tablet_service.cc:1514] Processing DeleteTablet for tablet ba8b8fdffeeb423a9d89dca999b25904 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:59 UTC) from {username='slave'} at 127.0.0.1:59846
I20240503 01:59:59.389078 6831 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a44d033ab014466bd2100c0ca250ec8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:59 UTC) from {username='slave'} at 127.0.0.1:52812
I20240503 01:59:59.389189 6567 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a44d033ab014466bd2100c0ca250ec8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:59 UTC) from {username='slave'} at 127.0.0.1:59846
I20240503 01:59:59.389290 9476 tablet_replica.cc:330] T 7a44d033ab014466bd2100c0ca250ec8 P 492448570cc94ed1879feb2ec3b9051d: stopping tablet replica
I20240503 01:59:59.389339 6829 tablet_service.cc:1514] Processing DeleteTablet for tablet 714c076813ff4c4f84cb16d256f7b28c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:59 UTC) from {username='slave'} at 127.0.0.1:52812
I20240503 01:59:59.389453 9476 raft_consensus.cc:2229] T 7a44d033ab014466bd2100c0ca250ec8 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:59.389477 9473 tablet_replica.cc:330] T ba8b8fdffeeb423a9d89dca999b25904 P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 01:59:59.389626 9476 raft_consensus.cc:2258] T 7a44d033ab014466bd2100c0ca250ec8 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:59.389648 9473 raft_consensus.cc:2229] T ba8b8fdffeeb423a9d89dca999b25904 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:59.389762 9473 raft_consensus.cc:2258] T ba8b8fdffeeb423a9d89dca999b25904 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:59.390193 9476 ts_tablet_manager.cc:1875] T 7a44d033ab014466bd2100c0ca250ec8 P 492448570cc94ed1879feb2ec3b9051d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:59.390239 9473 ts_tablet_manager.cc:1875] T ba8b8fdffeeb423a9d89dca999b25904 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:59.392428 9473 ts_tablet_manager.cc:1888] T ba8b8fdffeeb423a9d89dca999b25904 P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:59.392542 9473 log.cc:1196] T ba8b8fdffeeb423a9d89dca999b25904 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/ba8b8fdffeeb423a9d89dca999b25904
I20240503 01:59:59.392427 9476 ts_tablet_manager.cc:1888] T 7a44d033ab014466bd2100c0ca250ec8 P 492448570cc94ed1879feb2ec3b9051d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:59.392805 9473 ts_tablet_manager.cc:1909] T ba8b8fdffeeb423a9d89dca999b25904 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting consensus metadata
I20240503 01:59:59.392809 9476 log.cc:1196] T 7a44d033ab014466bd2100c0ca250ec8 P 492448570cc94ed1879feb2ec3b9051d: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/7a44d033ab014466bd2100c0ca250ec8
I20240503 01:59:59.389111 6830 tablet_service.cc:1514] Processing DeleteTablet for tablet ba8b8fdffeeb423a9d89dca999b25904 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:59 UTC) from {username='slave'} at 127.0.0.1:52812
I20240503 01:59:59.393064 9476 ts_tablet_manager.cc:1909] T 7a44d033ab014466bd2100c0ca250ec8 P 492448570cc94ed1879feb2ec3b9051d: Deleting consensus metadata
I20240503 01:59:59.393404 6700 tablet_service.cc:1514] Processing DeleteTablet for tablet 714c076813ff4c4f84cb16d256f7b28c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:59 UTC) from {username='slave'} at 127.0.0.1:40336
I20240503 01:59:59.393635 9475 tablet_replica.cc:330] T 714c076813ff4c4f84cb16d256f7b28c P c2ac2eb7cc264047b9ab32fe3a4fc025: stopping tablet replica
I20240503 01:59:59.393740 9476 tablet_replica.cc:330] T 714c076813ff4c4f84cb16d256f7b28c P 492448570cc94ed1879feb2ec3b9051d: stopping tablet replica
I20240503 01:59:59.393795 9475 raft_consensus.cc:2229] T 714c076813ff4c4f84cb16d256f7b28c P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:59.393857 9476 raft_consensus.cc:2229] T 714c076813ff4c4f84cb16d256f7b28c P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:59.393893 6303 catalog_manager.cc:4830] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719): tablet 7a44d033ab014466bd2100c0ca250ec8 (table rebalance_test_table_2 [id=5a03be9f319a4f58a441327c9bbe6ad9]) successfully deleted
I20240503 01:59:59.393935 9475 raft_consensus.cc:2258] T 714c076813ff4c4f84cb16d256f7b28c P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:59.394026 9476 raft_consensus.cc:2258] T 714c076813ff4c4f84cb16d256f7b28c P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:59.394443 9475 ts_tablet_manager.cc:1875] T 714c076813ff4c4f84cb16d256f7b28c P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:59.394785 9476 ts_tablet_manager.cc:1875] T 714c076813ff4c4f84cb16d256f7b28c P 492448570cc94ed1879feb2ec3b9051d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:59.388336 9474 ts_tablet_manager.cc:1888] T ba8b8fdffeeb423a9d89dca999b25904 P dec3e4a04d604f7eaaa3f077d081218a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:59.395599 9474 log.cc:1196] T ba8b8fdffeeb423a9d89dca999b25904 P dec3e4a04d604f7eaaa3f077d081218a: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/ba8b8fdffeeb423a9d89dca999b25904
I20240503 01:59:59.396149 9474 ts_tablet_manager.cc:1909] T ba8b8fdffeeb423a9d89dca999b25904 P dec3e4a04d604f7eaaa3f077d081218a: Deleting consensus metadata
I20240503 01:59:59.396773 9475 ts_tablet_manager.cc:1888] T 714c076813ff4c4f84cb16d256f7b28c P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:59.396899 6300 catalog_manager.cc:4830] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): tablet ba8b8fdffeeb423a9d89dca999b25904 (table rebalance_test_table_2 [id=5a03be9f319a4f58a441327c9bbe6ad9]) successfully deleted
I20240503 01:59:59.397166 9476 ts_tablet_manager.cc:1888] T 714c076813ff4c4f84cb16d256f7b28c P 492448570cc94ed1879feb2ec3b9051d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:59.397231 9476 log.cc:1196] T 714c076813ff4c4f84cb16d256f7b28c P 492448570cc94ed1879feb2ec3b9051d: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/714c076813ff4c4f84cb16d256f7b28c
I20240503 01:59:59.397457 9473 tablet_replica.cc:330] T 7a44d033ab014466bd2100c0ca250ec8 P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 01:59:59.396854 9475 log.cc:1196] T 714c076813ff4c4f84cb16d256f7b28c P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/714c076813ff4c4f84cb16d256f7b28c
I20240503 01:59:59.397614 9473 raft_consensus.cc:2229] T 7a44d033ab014466bd2100c0ca250ec8 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:59.397830 9475 ts_tablet_manager.cc:1909] T 714c076813ff4c4f84cb16d256f7b28c P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting consensus metadata
I20240503 01:59:59.397840 9476 ts_tablet_manager.cc:1909] T 714c076813ff4c4f84cb16d256f7b28c P 492448570cc94ed1879feb2ec3b9051d: Deleting consensus metadata
I20240503 01:59:59.397818 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet ba8b8fdffeeb423a9d89dca999b25904 (table rebalance_test_table_2 [id=5a03be9f319a4f58a441327c9bbe6ad9]) successfully deleted
I20240503 01:59:59.398046 9473 raft_consensus.cc:2258] T 7a44d033ab014466bd2100c0ca250ec8 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:59.398376 9476 tablet_replica.cc:330] T ba8b8fdffeeb423a9d89dca999b25904 P 492448570cc94ed1879feb2ec3b9051d: stopping tablet replica
I20240503 01:59:59.398476 6303 catalog_manager.cc:4830] TS c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655): tablet 714c076813ff4c4f84cb16d256f7b28c (table rebalance_test_table_2 [id=5a03be9f319a4f58a441327c9bbe6ad9]) successfully deleted
I20240503 01:59:59.398528 9473 ts_tablet_manager.cc:1875] T 7a44d033ab014466bd2100c0ca250ec8 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:59.398762 9476 raft_consensus.cc:2229] T ba8b8fdffeeb423a9d89dca999b25904 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:59.398836 6303 catalog_manager.cc:4830] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719): tablet 714c076813ff4c4f84cb16d256f7b28c (table rebalance_test_table_2 [id=5a03be9f319a4f58a441327c9bbe6ad9]) successfully deleted
I20240503 01:59:59.398988 9476 raft_consensus.cc:2258] T ba8b8fdffeeb423a9d89dca999b25904 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:59.399446 9476 ts_tablet_manager.cc:1875] T ba8b8fdffeeb423a9d89dca999b25904 P 492448570cc94ed1879feb2ec3b9051d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:59.400578 9473 ts_tablet_manager.cc:1888] T 7a44d033ab014466bd2100c0ca250ec8 P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:59.400652 9473 log.cc:1196] T 7a44d033ab014466bd2100c0ca250ec8 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/7a44d033ab014466bd2100c0ca250ec8
I20240503 01:59:59.400873 9473 ts_tablet_manager.cc:1909] T 7a44d033ab014466bd2100c0ca250ec8 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting consensus metadata
I20240503 01:59:59.401136 9474 tablet_replica.cc:330] T 714c076813ff4c4f84cb16d256f7b28c P dec3e4a04d604f7eaaa3f077d081218a: stopping tablet replica
I20240503 01:59:59.401329 9474 raft_consensus.cc:2229] T 714c076813ff4c4f84cb16d256f7b28c P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:59.401387 9476 ts_tablet_manager.cc:1888] T ba8b8fdffeeb423a9d89dca999b25904 P 492448570cc94ed1879feb2ec3b9051d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:59.401496 9476 log.cc:1196] T ba8b8fdffeeb423a9d89dca999b25904 P 492448570cc94ed1879feb2ec3b9051d: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/ba8b8fdffeeb423a9d89dca999b25904
I20240503 01:59:59.401615 9474 raft_consensus.cc:2258] T 714c076813ff4c4f84cb16d256f7b28c P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:59.401742 9476 ts_tablet_manager.cc:1909] T ba8b8fdffeeb423a9d89dca999b25904 P 492448570cc94ed1879feb2ec3b9051d: Deleting consensus metadata
I20240503 01:59:59.402158 9474 ts_tablet_manager.cc:1875] T 714c076813ff4c4f84cb16d256f7b28c P dec3e4a04d604f7eaaa3f077d081218a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:59.402597 6303 catalog_manager.cc:4830] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719): tablet ba8b8fdffeeb423a9d89dca999b25904 (table rebalance_test_table_2 [id=5a03be9f319a4f58a441327c9bbe6ad9]) successfully deleted
I20240503 01:59:59.404072 9474 ts_tablet_manager.cc:1888] T 714c076813ff4c4f84cb16d256f7b28c P dec3e4a04d604f7eaaa3f077d081218a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:59.404139 9474 log.cc:1196] T 714c076813ff4c4f84cb16d256f7b28c P dec3e4a04d604f7eaaa3f077d081218a: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/714c076813ff4c4f84cb16d256f7b28c
I20240503 01:59:59.404362 9474 ts_tablet_manager.cc:1909] T 714c076813ff4c4f84cb16d256f7b28c P dec3e4a04d604f7eaaa3f077d081218a: Deleting consensus metadata
I20240503 01:59:59.405054 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet 7a44d033ab014466bd2100c0ca250ec8 (table rebalance_test_table_2 [id=5a03be9f319a4f58a441327c9bbe6ad9]) successfully deleted
I20240503 01:59:59.405349 6300 catalog_manager.cc:4830] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): tablet 714c076813ff4c4f84cb16d256f7b28c (table rebalance_test_table_2 [id=5a03be9f319a4f58a441327c9bbe6ad9]) successfully deleted
I20240503 01:59:59.405546 9474 tablet_replica.cc:330] T 7a44d033ab014466bd2100c0ca250ec8 P dec3e4a04d604f7eaaa3f077d081218a: stopping tablet replica
I20240503 01:59:59.405699 9474 raft_consensus.cc:2229] T 7a44d033ab014466bd2100c0ca250ec8 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:59.405936 9474 raft_consensus.cc:2258] T 7a44d033ab014466bd2100c0ca250ec8 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:59.406551 9474 ts_tablet_manager.cc:1875] T 7a44d033ab014466bd2100c0ca250ec8 P dec3e4a04d604f7eaaa3f077d081218a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:59.408345 9474 ts_tablet_manager.cc:1888] T 7a44d033ab014466bd2100c0ca250ec8 P dec3e4a04d604f7eaaa3f077d081218a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:59.408490 9474 log.cc:1196] T 7a44d033ab014466bd2100c0ca250ec8 P dec3e4a04d604f7eaaa3f077d081218a: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/7a44d033ab014466bd2100c0ca250ec8
I20240503 01:59:59.408762 9474 ts_tablet_manager.cc:1909] T 7a44d033ab014466bd2100c0ca250ec8 P dec3e4a04d604f7eaaa3f077d081218a: Deleting consensus metadata
I20240503 01:59:59.409457 6300 catalog_manager.cc:4830] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): tablet 7a44d033ab014466bd2100c0ca250ec8 (table rebalance_test_table_2 [id=5a03be9f319a4f58a441327c9bbe6ad9]) successfully deleted
I20240503 01:59:59.434422 8833 raft_consensus.cc:482] T f758d325e8e946deb97be30469844809 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:59.434548 8833 raft_consensus.cc:504] T f758d325e8e946deb97be30469844809 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:59.434993 8833 leader_election.cc:288] T f758d325e8e946deb97be30469844809 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157), e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311)
I20240503 01:59:59.435472 9310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f758d325e8e946deb97be30469844809" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" is_pre_election: true
I20240503 01:59:59.435487 9039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f758d325e8e946deb97be30469844809" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 01:59:59.435623 9310 raft_consensus.cc:2454] T f758d325e8e946deb97be30469844809 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 01:59:59.435638 9039 raft_consensus.cc:2454] T f758d325e8e946deb97be30469844809 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 01:59:59.435897 6523 leader_election.cc:302] T f758d325e8e946deb97be30469844809 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 01:59:59.436125 8833 raft_consensus.cc:2781] T f758d325e8e946deb97be30469844809 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:59.436178 8833 raft_consensus.cc:482] T f758d325e8e946deb97be30469844809 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:59.436214 8833 raft_consensus.cc:3037] T f758d325e8e946deb97be30469844809 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:59.437072 8833 raft_consensus.cc:504] T f758d325e8e946deb97be30469844809 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:59.437381 8833 leader_election.cc:288] T f758d325e8e946deb97be30469844809 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Requested vote from peers 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157), e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311)
I20240503 01:59:59.437759 9039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f758d325e8e946deb97be30469844809" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 01:59:59.437826 9310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f758d325e8e946deb97be30469844809" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023"
I20240503 01:59:59.437878 9039 raft_consensus.cc:3037] T f758d325e8e946deb97be30469844809 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:59.437942 9310 raft_consensus.cc:3037] T f758d325e8e946deb97be30469844809 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:59.438660 9310 raft_consensus.cc:2454] T f758d325e8e946deb97be30469844809 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 01:59:59.438678 9039 raft_consensus.cc:2454] T f758d325e8e946deb97be30469844809 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 01:59:59.438974 6523 leader_election.cc:302] T f758d325e8e946deb97be30469844809 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 01:59:59.439116 8833 raft_consensus.cc:2781] T f758d325e8e946deb97be30469844809 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:59.439205 8833 raft_consensus.cc:686] T f758d325e8e946deb97be30469844809 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Becoming Leader. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Running, Role: LEADER
I20240503 01:59:59.439345 8833 consensus_queue.cc:237] T f758d325e8e946deb97be30469844809 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 01:59:59.441414 6307 catalog_manager.cc:5484] T f758d325e8e946deb97be30469844809 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:59.444293 8679 consensus_queue.cc:1035] T 51c0137930464635be1e6a97f5e9d952 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:59.449935 8633 consensus_queue.cc:1035] T 51c0137930464635be1e6a97f5e9d952 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:59.516038 6567 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 01:59:59.516444 8880 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:59.517987 9285 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:59.520823 6830 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:59.531962 9025 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:59.538545 9158 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:59.538578 6438 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:59.554816 6312 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 01:59:59.559566 9039 raft_consensus.cc:1261] T f758d325e8e946deb97be30469844809 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:59.560171 9484 consensus_queue.cc:1035] T f758d325e8e946deb97be30469844809 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:59.561121 9310 raft_consensus.cc:1261] T f758d325e8e946deb97be30469844809 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:59.563266 9484 consensus_queue.cc:1035] T f758d325e8e946deb97be30469844809 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:59.563863 6312 catalog_manager.cc:5576] TS 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157) does not have the latest schema for tablet f758d325e8e946deb97be30469844809 (table rebalancer_dynamic_table [id=8b571be2b4d14eab84dc1bc1e5b8d53a]). Expected version 1 got 0
I20240503 01:59:59.567068 6307 catalog_manager.cc:5576] TS e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311) does not have the latest schema for tablet f758d325e8e946deb97be30469844809 (table rebalancer_dynamic_table [id=8b571be2b4d14eab84dc1bc1e5b8d53a]). Expected version 1 got 0
I20240503 01:59:59.568213 6700 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 01:59:59.570053 9472 tablet.cc:1722] T f758d325e8e946deb97be30469844809 P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:59.570398 9472 tablet.cc:1620] T f758d325e8e946deb97be30469844809 P 80fd18ef4d604c86b909d5c008a46b4f: MemRowSet was empty: no flush needed.
I20240503 01:59:59.570578 9485 tablet.cc:1722] T f758d325e8e946deb97be30469844809 P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:59.570890 9485 tablet.cc:1620] T f758d325e8e946deb97be30469844809 P 99ead02915954b4fab23b033ad6d546b: MemRowSet was empty: no flush needed.
I20240503 01:59:59.572093 6312 catalog_manager.cc:5576] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863) does not have the latest schema for tablet f758d325e8e946deb97be30469844809 (table rebalancer_dynamic_table [id=8b571be2b4d14eab84dc1bc1e5b8d53a]). Expected version 1 got 0
I20240503 01:59:59.575414 6312 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=8b571be2b4d14eab84dc1bc1e5b8d53a] alter complete (version 1)
I20240503 01:59:59.586673 9531 consensus_queue.cc:1035] T f5ed01740019467d8c1d55cc2fa18ca3 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:59.589768 9477 tablet.cc:1722] T f758d325e8e946deb97be30469844809 P e1a9251ce1cf4361b8b0ede9fbfc4023: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:59.590121 9477 tablet.cc:1620] T f758d325e8e946deb97be30469844809 P e1a9251ce1cf4361b8b0ede9fbfc4023: MemRowSet was empty: no flush needed.
I20240503 01:59:59.598779 9541 consensus_queue.cc:1035] T f5ed01740019467d8c1d55cc2fa18ca3 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:59.638597 6851 consensus_queue.cc:237] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } }
I20240503 01:59:59.640235 6582 raft_consensus.cc:1261] T 1fa274c98fc147f29344ed537e2ccf17 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:59.640831 6720 raft_consensus.cc:1261] T 1fa274c98fc147f29344ed537e2ccf17 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:59.641227 9631 consensus_queue.cc:1035] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:59.641460 9632 consensus_queue.cc:1035] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:59.643545 9461 raft_consensus.cc:2932] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } } }
I20240503 01:59:59.644871 6720 raft_consensus.cc:2932] T 1fa274c98fc147f29344ed537e2ccf17 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } } }
W20240503 01:59:59.645339 6785 consensus_peers.cc:485] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1fa274c98fc147f29344ed537e2ccf17. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:59.645750 6582 raft_consensus.cc:2932] T 1fa274c98fc147f29344ed537e2ccf17 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } } }
I20240503 01:59:59.647547 6312 catalog_manager.cc:5484] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d reported cstate change: config changed from index -1 to 4, NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:59.656020 6720 consensus_queue.cc:237] T ceca04391cd446ae932b3485d958237a P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } }
I20240503 01:59:59.658094 6851 raft_consensus.cc:1261] T ceca04391cd446ae932b3485d958237a P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:59.658187 6582 raft_consensus.cc:1261] T ceca04391cd446ae932b3485d958237a P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:59.658601 8826 consensus_queue.cc:1035] T ceca04391cd446ae932b3485d958237a P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:59.658794 9642 consensus_queue.cc:1035] T ceca04391cd446ae932b3485d958237a P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:59.660719 8696 raft_consensus.cc:2932] T ceca04391cd446ae932b3485d958237a P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } } }
I20240503 01:59:59.661172 6582 raft_consensus.cc:2932] T ceca04391cd446ae932b3485d958237a P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } } }
I20240503 01:59:59.661283 6851 raft_consensus.cc:2932] T ceca04391cd446ae932b3485d958237a P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } } }
W20240503 01:59:59.663863 6656 consensus_peers.cc:485] T ceca04391cd446ae932b3485d958237a P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211): Couldn't send request to peer 9942306855404cfd891adf227dd65eb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ceca04391cd446ae932b3485d958237a. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:59.664603 6310 catalog_manager.cc:5484] T ceca04391cd446ae932b3485d958237a P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: config changed from index -1 to 4, NON_VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } } }
I20240503 01:59:59.667292 9541 consensus_queue.cc:1035] T f44b59277c3b47adbd4dd6aff9350af1 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:59.671433 9652 consensus_queue.cc:1035] T f44b59277c3b47adbd4dd6aff9350af1 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:59.676000 6582 consensus_queue.cc:237] T 1e445a3cc928429caf9e3c6193a63010 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } }
I20240503 01:59:59.677892 6851 raft_consensus.cc:1261] T 1e445a3cc928429caf9e3c6193a63010 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:59.678186 6458 raft_consensus.cc:1261] T 1e445a3cc928429caf9e3c6193a63010 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:59.678304 9484 consensus_queue.cc:1035] T 1e445a3cc928429caf9e3c6193a63010 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:59.678660 8833 consensus_queue.cc:1035] T 1e445a3cc928429caf9e3c6193a63010 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:59.678833 6316 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 01:59:59.681720 8833 raft_consensus.cc:2932] T 1e445a3cc928429caf9e3c6193a63010 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } } }
I20240503 01:59:59.682384 6851 raft_consensus.cc:2932] T 1e445a3cc928429caf9e3c6193a63010 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } } }
I20240503 01:59:59.683897 6458 raft_consensus.cc:2932] T 1e445a3cc928429caf9e3c6193a63010 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } } }
I20240503 01:59:59.684909 6316 catalog_manager.cc:5484] T 1e445a3cc928429caf9e3c6193a63010 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: config changed from index -1 to 4, NON_VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:59.687847 9472 tablet.cc:1722] T f758d325e8e946deb97be30469844809 P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:59.688207 9472 tablet.cc:1620] T f758d325e8e946deb97be30469844809 P 80fd18ef4d604c86b909d5c008a46b4f: MemRowSet was empty: no flush needed.
W20240503 01:59:59.690699 6522 consensus_peers.cc:485] T 1e445a3cc928429caf9e3c6193a63010 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079): Couldn't send request to peer 2a01ace80baf4057b6bf1995cf88468b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1e445a3cc928429caf9e3c6193a63010. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:59.691709 9477 tablet.cc:1722] T f758d325e8e946deb97be30469844809 P e1a9251ce1cf4361b8b0ede9fbfc4023: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:59.692051 9477 tablet.cc:1620] T f758d325e8e946deb97be30469844809 P e1a9251ce1cf4361b8b0ede9fbfc4023: MemRowSet was empty: no flush needed.
I20240503 01:59:59.693219 9485 tablet.cc:1722] T f758d325e8e946deb97be30469844809 P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:59.693584 9485 tablet.cc:1620] T f758d325e8e946deb97be30469844809 P 99ead02915954b4fab23b033ad6d546b: MemRowSet was empty: no flush needed.
I20240503 01:59:59.697295 6312 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=8b571be2b4d14eab84dc1bc1e5b8d53a] alter complete (version 2)
I20240503 01:59:59.701970 6720 consensus_queue.cc:237] T d82268925a934c559dfcf51a84f03d2e P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } }
I20240503 01:59:59.703418 6582 raft_consensus.cc:1261] T d82268925a934c559dfcf51a84f03d2e P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:59.703472 6458 raft_consensus.cc:1261] T d82268925a934c559dfcf51a84f03d2e P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:59.703936 9642 consensus_queue.cc:1035] T d82268925a934c559dfcf51a84f03d2e P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:59.704141 8826 consensus_queue.cc:1035] T d82268925a934c559dfcf51a84f03d2e P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:59.706401 9642 raft_consensus.cc:2932] T d82268925a934c559dfcf51a84f03d2e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } } }
I20240503 01:59:59.706761 6458 raft_consensus.cc:2932] T d82268925a934c559dfcf51a84f03d2e P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } } }
I20240503 01:59:59.708312 6582 raft_consensus.cc:2932] T d82268925a934c559dfcf51a84f03d2e P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } } }
W20240503 01:59:59.708869 6654 consensus_peers.cc:485] T d82268925a934c559dfcf51a84f03d2e P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157): Couldn't send request to peer 99ead02915954b4fab23b033ad6d546b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d82268925a934c559dfcf51a84f03d2e. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:59.710234 6316 catalog_manager.cc:5484] T d82268925a934c559dfcf51a84f03d2e P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: config changed from index -1 to 4, NON_VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } } }
I20240503 01:59:59.718911 6582 consensus_queue.cc:237] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } }
I20240503 01:59:59.720409 6851 raft_consensus.cc:1261] T 3416b6107e1848f299ac49ccd88ae7c8 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:59.720405 6720 raft_consensus.cc:1261] T 3416b6107e1848f299ac49ccd88ae7c8 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 01:59:59.720759 6523 consensus_peers.cc:485] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3416b6107e1848f299ac49ccd88ae7c8. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:59.720805 8598 consensus_queue.cc:1035] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:59.721021 9484 consensus_queue.cc:1035] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:59.723121 8833 raft_consensus.cc:2932] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } } }
I20240503 01:59:59.723722 6851 raft_consensus.cc:2932] T 3416b6107e1848f299ac49ccd88ae7c8 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } } }
I20240503 01:59:59.723870 6720 raft_consensus.cc:2932] T 3416b6107e1848f299ac49ccd88ae7c8 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } } }
I20240503 01:59:59.727121 6312 catalog_manager.cc:5484] T 3416b6107e1848f299ac49ccd88ae7c8 P 492448570cc94ed1879feb2ec3b9051d reported cstate change: config changed from index -1 to 4, NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } } }
I20240503 01:59:59.736441 6582 consensus_queue.cc:237] T 5f176537159143e8a770d65ba63ad62e P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } }
I20240503 01:59:59.738262 6458 raft_consensus.cc:1261] T 5f176537159143e8a770d65ba63ad62e P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:59.738629 6851 raft_consensus.cc:1261] T 5f176537159143e8a770d65ba63ad62e P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:59.738662 8598 consensus_queue.cc:1035] T 5f176537159143e8a770d65ba63ad62e P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:59.739022 8598 consensus_queue.cc:1035] T 5f176537159143e8a770d65ba63ad62e P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:59.740617 8833 raft_consensus.cc:2932] T 5f176537159143e8a770d65ba63ad62e P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } } }
I20240503 01:59:59.741117 6458 raft_consensus.cc:2932] T 5f176537159143e8a770d65ba63ad62e P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } } }
W20240503 01:59:59.741891 6525 consensus_peers.cc:485] T 5f176537159143e8a770d65ba63ad62e P 80fd18ef4d604c86b909d5c008a46b4f -> Peer 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211): Couldn't send request to peer 9942306855404cfd891adf227dd65eb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5f176537159143e8a770d65ba63ad62e. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:59.743983 6312 catalog_manager.cc:5484] T 5f176537159143e8a770d65ba63ad62e P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: config changed from index -1 to 4, NON_VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } } }
I20240503 01:59:59.744560 6851 raft_consensus.cc:2932] T 5f176537159143e8a770d65ba63ad62e P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } } }
I20240503 01:59:59.753065 6720 consensus_queue.cc:237] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } }
I20240503 01:59:59.754829 6851 raft_consensus.cc:1261] T 1feaf29d8c6b4842829ce66e0f4ec23b P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:59.755223 6458 raft_consensus.cc:1261] T 1feaf29d8c6b4842829ce66e0f4ec23b P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:59.755362 8696 consensus_queue.cc:1035] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:59.755782 8696 consensus_queue.cc:1035] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:59.757426 9667 raft_consensus.cc:2932] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } } }
I20240503 01:59:59.758257 6851 raft_consensus.cc:2932] T 1feaf29d8c6b4842829ce66e0f4ec23b P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } } }
W20240503 01:59:59.760620 6653 consensus_peers.cc:485] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079): Couldn't send request to peer 2a01ace80baf4057b6bf1995cf88468b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1feaf29d8c6b4842829ce66e0f4ec23b. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:59.760515 6316 catalog_manager.cc:5484] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: config changed from index -1 to 4, NON_VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:59.760923 6458 raft_consensus.cc:2932] T 1feaf29d8c6b4842829ce66e0f4ec23b P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } } }
I20240503 01:59:59.775216 6720 consensus_queue.cc:237] T 3cf186a0901f420390c8fda54316c5af P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } }
I20240503 01:59:59.776932 6307 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_extra_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:59.777316 6851 raft_consensus.cc:1261] T 3cf186a0901f420390c8fda54316c5af P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:59.777410 6458 raft_consensus.cc:1261] T 3cf186a0901f420390c8fda54316c5af P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:59.777765 8696 consensus_queue.cc:1035] T 3cf186a0901f420390c8fda54316c5af P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:59.778524 8696 consensus_queue.cc:1035] T 3cf186a0901f420390c8fda54316c5af P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 01:59:59.780018 6654 consensus_peers.cc:485] T 3cf186a0901f420390c8fda54316c5af P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157): Couldn't send request to peer 99ead02915954b4fab23b033ad6d546b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3cf186a0901f420390c8fda54316c5af. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:59.782033 8826 raft_consensus.cc:2932] T 3cf186a0901f420390c8fda54316c5af P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } } }
I20240503 01:59:59.784348 9158 tablet_service.cc:1467] Processing CreateTablet for tablet 0099654461ca41efa4323fc9f5adab4a (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=54f1891be3f54f19821aad5d1dfbd85d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:59.784694 9158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0099654461ca41efa4323fc9f5adab4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:59.784890 9285 tablet_service.cc:1467] Processing CreateTablet for tablet 067f54011a37432d99157884e70fc601 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=54f1891be3f54f19821aad5d1dfbd85d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:59.784904 9025 tablet_service.cc:1467] Processing CreateTablet for tablet 9c8a53980b00421d8c2bd3ae73f0c8cc (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=54f1891be3f54f19821aad5d1dfbd85d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:59.785128 9285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 067f54011a37432d99157884e70fc601. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:59.785171 9025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c8a53980b00421d8c2bd3ae73f0c8cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:59.787225 9374 tablet_bootstrap.cc:492] T 067f54011a37432d99157884e70fc601 P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap starting.
I20240503 01:59:59.787300 9375 tablet_bootstrap.cc:492] T 9c8a53980b00421d8c2bd3ae73f0c8cc P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 01:59:59.788007 9374 tablet_bootstrap.cc:654] T 067f54011a37432d99157884e70fc601 P e1a9251ce1cf4361b8b0ede9fbfc4023: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:59.788120 9375 tablet_bootstrap.cc:654] T 9c8a53980b00421d8c2bd3ae73f0c8cc P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:59.788887 6851 raft_consensus.cc:2932] T 3cf186a0901f420390c8fda54316c5af P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } } }
I20240503 01:59:59.789281 9375 tablet_bootstrap.cc:492] T 9c8a53980b00421d8c2bd3ae73f0c8cc P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 01:59:59.789372 9375 ts_tablet_manager.cc:1367] T 9c8a53980b00421d8c2bd3ae73f0c8cc P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:59.789767 9375 raft_consensus.cc:348] T 9c8a53980b00421d8c2bd3ae73f0c8cc P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 01:59:59.789757 6438 tablet_service.cc:1467] Processing CreateTablet for tablet 9c8a53980b00421d8c2bd3ae73f0c8cc (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=54f1891be3f54f19821aad5d1dfbd85d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:59.789861 9375 raft_consensus.cc:374] T 9c8a53980b00421d8c2bd3ae73f0c8cc P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:59.789888 9375 raft_consensus.cc:729] T 9c8a53980b00421d8c2bd3ae73f0c8cc P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 01:59:59.790004 9375 consensus_queue.cc:260] T 9c8a53980b00421d8c2bd3ae73f0c8cc P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 01:59:59.790055 6438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c8a53980b00421d8c2bd3ae73f0c8cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:59.790254 9375 ts_tablet_manager.cc:1398] T 9c8a53980b00421d8c2bd3ae73f0c8cc P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:59.790828 8887 tablet_service.cc:1467] Processing CreateTablet for tablet 0099654461ca41efa4323fc9f5adab4a (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=54f1891be3f54f19821aad5d1dfbd85d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:59.791105 8887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0099654461ca41efa4323fc9f5adab4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:59.792196 9704 tablet_bootstrap.cc:492] T 9c8a53980b00421d8c2bd3ae73f0c8cc P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 01:59:59.792356 6436 tablet_service.cc:1467] Processing CreateTablet for tablet 0099654461ca41efa4323fc9f5adab4a (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=54f1891be3f54f19821aad5d1dfbd85d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:59.792644 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0099654461ca41efa4323fc9f5adab4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:59.792730 8897 tablet_service.cc:1467] Processing CreateTablet for tablet 067f54011a37432d99157884e70fc601 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=54f1891be3f54f19821aad5d1dfbd85d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:59.792956 8897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 067f54011a37432d99157884e70fc601. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:59.793149 9704 tablet_bootstrap.cc:654] T 9c8a53980b00421d8c2bd3ae73f0c8cc P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:59.793216 9705 tablet_bootstrap.cc:492] T 0099654461ca41efa4323fc9f5adab4a P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 01:59:59.794106 9704 tablet_bootstrap.cc:492] T 9c8a53980b00421d8c2bd3ae73f0c8cc P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 01:59:59.794203 9704 ts_tablet_manager.cc:1367] T 9c8a53980b00421d8c2bd3ae73f0c8cc P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:59.794374 9705 tablet_bootstrap.cc:654] T 0099654461ca41efa4323fc9f5adab4a P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:59.794605 9704 raft_consensus.cc:348] T 9c8a53980b00421d8c2bd3ae73f0c8cc P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 01:59:59.794708 9704 raft_consensus.cc:374] T 9c8a53980b00421d8c2bd3ae73f0c8cc P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:59.794736 9704 raft_consensus.cc:729] T 9c8a53980b00421d8c2bd3ae73f0c8cc P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 01:59:59.794781 8880 tablet_service.cc:1467] Processing CreateTablet for tablet 9c8a53980b00421d8c2bd3ae73f0c8cc (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=54f1891be3f54f19821aad5d1dfbd85d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:59.794839 9704 consensus_queue.cc:260] T 9c8a53980b00421d8c2bd3ae73f0c8cc P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 01:59:59.795020 8880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c8a53980b00421d8c2bd3ae73f0c8cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:59.795205 9704 ts_tablet_manager.cc:1398] T 9c8a53980b00421d8c2bd3ae73f0c8cc P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:59.795365 9704 tablet_bootstrap.cc:492] T 0099654461ca41efa4323fc9f5adab4a P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 01:59:59.796192 9704 tablet_bootstrap.cc:654] T 0099654461ca41efa4323fc9f5adab4a P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:59.796197 9374 tablet_bootstrap.cc:492] T 067f54011a37432d99157884e70fc601 P e1a9251ce1cf4361b8b0ede9fbfc4023: No bootstrap required, opened a new log
I20240503 01:59:59.796277 9374 ts_tablet_manager.cc:1367] T 067f54011a37432d99157884e70fc601 P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240503 01:59:59.796689 9374 raft_consensus.cc:348] T 067f54011a37432d99157884e70fc601 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 01:59:59.796792 9374 raft_consensus.cc:374] T 067f54011a37432d99157884e70fc601 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:59.796818 9374 raft_consensus.cc:729] T 067f54011a37432d99157884e70fc601 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1a9251ce1cf4361b8b0ede9fbfc4023, State: Initialized, Role: FOLLOWER
I20240503 01:59:59.796753 9158 tablet_service.cc:1467] Processing CreateTablet for tablet 067f54011a37432d99157884e70fc601 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=54f1891be3f54f19821aad5d1dfbd85d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:59.796939 9374 consensus_queue.cc:260] T 067f54011a37432d99157884e70fc601 P e1a9251ce1cf4361b8b0ede9fbfc4023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 01:59:59.797201 9158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 067f54011a37432d99157884e70fc601. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:59.797220 9374 ts_tablet_manager.cc:1398] T 067f54011a37432d99157884e70fc601 P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:59.797246 9704 tablet_bootstrap.cc:492] T 0099654461ca41efa4323fc9f5adab4a P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 01:59:59.797325 9704 ts_tablet_manager.cc:1367] T 0099654461ca41efa4323fc9f5adab4a P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:59.797730 9704 raft_consensus.cc:348] T 0099654461ca41efa4323fc9f5adab4a P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 01:59:59.797830 9704 raft_consensus.cc:374] T 0099654461ca41efa4323fc9f5adab4a P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:59.797863 9704 raft_consensus.cc:729] T 0099654461ca41efa4323fc9f5adab4a P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 01:59:59.797973 9704 consensus_queue.cc:260] T 0099654461ca41efa4323fc9f5adab4a P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 01:59:59.798254 9704 ts_tablet_manager.cc:1398] T 0099654461ca41efa4323fc9f5adab4a P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:59.799212 9705 tablet_bootstrap.cc:492] T 0099654461ca41efa4323fc9f5adab4a P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 01:59:59.799352 9705 ts_tablet_manager.cc:1367] T 0099654461ca41efa4323fc9f5adab4a P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 01:59:59.799854 9705 raft_consensus.cc:348] T 0099654461ca41efa4323fc9f5adab4a P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 01:59:59.799993 9705 raft_consensus.cc:374] T 0099654461ca41efa4323fc9f5adab4a P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:59.800046 9705 raft_consensus.cc:729] T 0099654461ca41efa4323fc9f5adab4a P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 01:59:59.800012 6314 catalog_manager.cc:5484] T 3cf186a0901f420390c8fda54316c5af P 492448570cc94ed1879feb2ec3b9051d reported cstate change: config changed from index -1 to 4, NON_VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } } }
I20240503 01:59:59.800194 9705 consensus_queue.cc:260] T 0099654461ca41efa4323fc9f5adab4a P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 01:59:59.800631 9705 ts_tablet_manager.cc:1398] T 0099654461ca41efa4323fc9f5adab4a P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:59.800837 9705 tablet_bootstrap.cc:492] T 067f54011a37432d99157884e70fc601 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 01:59:59.801970 9705 tablet_bootstrap.cc:654] T 067f54011a37432d99157884e70fc601 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:59.806047 9701 tablet_bootstrap.cc:492] T 0099654461ca41efa4323fc9f5adab4a P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 01:59:59.806861 6310 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 01:59:59.807089 9701 tablet_bootstrap.cc:654] T 0099654461ca41efa4323fc9f5adab4a P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:59.807016 6458 raft_consensus.cc:2932] T 3cf186a0901f420390c8fda54316c5af P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } } }
I20240503 01:59:59.809861 9705 tablet_bootstrap.cc:492] T 067f54011a37432d99157884e70fc601 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 01:59:59.809963 9705 ts_tablet_manager.cc:1367] T 067f54011a37432d99157884e70fc601 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 01:59:59.810853 9705 raft_consensus.cc:348] T 067f54011a37432d99157884e70fc601 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 01:59:59.811008 9705 raft_consensus.cc:374] T 067f54011a37432d99157884e70fc601 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:59.811064 9705 raft_consensus.cc:729] T 067f54011a37432d99157884e70fc601 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 01:59:59.811203 9705 consensus_queue.cc:260] T 067f54011a37432d99157884e70fc601 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 01:59:59.811511 9705 ts_tablet_manager.cc:1398] T 067f54011a37432d99157884e70fc601 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:59.812008 9705 tablet_bootstrap.cc:492] T 9c8a53980b00421d8c2bd3ae73f0c8cc P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 01:59:59.813144 9705 tablet_bootstrap.cc:654] T 9c8a53980b00421d8c2bd3ae73f0c8cc P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:59.814635 9472 tablet.cc:1722] T f758d325e8e946deb97be30469844809 P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:59.815023 9701 tablet_bootstrap.cc:492] T 0099654461ca41efa4323fc9f5adab4a P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 01:59:59.815138 9701 ts_tablet_manager.cc:1367] T 0099654461ca41efa4323fc9f5adab4a P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 01:59:59.815618 9701 raft_consensus.cc:348] T 0099654461ca41efa4323fc9f5adab4a P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 01:59:59.815768 9701 raft_consensus.cc:374] T 0099654461ca41efa4323fc9f5adab4a P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:59.815809 9701 raft_consensus.cc:729] T 0099654461ca41efa4323fc9f5adab4a P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 01:59:59.815934 9701 consensus_queue.cc:260] T 0099654461ca41efa4323fc9f5adab4a P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 01:59:59.816242 9701 ts_tablet_manager.cc:1398] T 0099654461ca41efa4323fc9f5adab4a P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:59.816484 9701 tablet_bootstrap.cc:492] T 067f54011a37432d99157884e70fc601 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 01:59:59.817584 9477 tablet.cc:1722] T f758d325e8e946deb97be30469844809 P e1a9251ce1cf4361b8b0ede9fbfc4023: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:59.818157 9485 tablet.cc:1722] T f758d325e8e946deb97be30469844809 P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:59.821370 9701 tablet_bootstrap.cc:654] T 067f54011a37432d99157884e70fc601 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:59.822683 9705 tablet_bootstrap.cc:492] T 9c8a53980b00421d8c2bd3ae73f0c8cc P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 01:59:59.822777 9705 ts_tablet_manager.cc:1367] T 9c8a53980b00421d8c2bd3ae73f0c8cc P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 01:59:59.823518 6312 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=8b571be2b4d14eab84dc1bc1e5b8d53a] alter complete (version 3)
I20240503 01:59:59.824242 9705 raft_consensus.cc:348] T 9c8a53980b00421d8c2bd3ae73f0c8cc P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 01:59:59.824370 9701 tablet_bootstrap.cc:492] T 067f54011a37432d99157884e70fc601 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 01:59:59.824494 9701 ts_tablet_manager.cc:1367] T 067f54011a37432d99157884e70fc601 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 01:59:59.824532 9705 raft_consensus.cc:374] T 9c8a53980b00421d8c2bd3ae73f0c8cc P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:59.824582 9705 raft_consensus.cc:729] T 9c8a53980b00421d8c2bd3ae73f0c8cc P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 01:59:59.824720 9705 consensus_queue.cc:260] T 9c8a53980b00421d8c2bd3ae73f0c8cc P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 01:59:59.824892 9701 raft_consensus.cc:348] T 067f54011a37432d99157884e70fc601 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 01:59:59.824998 9701 raft_consensus.cc:374] T 067f54011a37432d99157884e70fc601 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:59.825006 9705 ts_tablet_manager.cc:1398] T 9c8a53980b00421d8c2bd3ae73f0c8cc P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 01:59:59.825028 9701 raft_consensus.cc:729] T 067f54011a37432d99157884e70fc601 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 01:59:59.825166 9701 consensus_queue.cc:260] T 067f54011a37432d99157884e70fc601 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 01:59:59.825441 9701 ts_tablet_manager.cc:1398] T 067f54011a37432d99157884e70fc601 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:59.832919 6458 consensus_queue.cc:237] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } }
I20240503 01:59:59.834461 6720 raft_consensus.cc:1261] T 46ad540deb624a25b97d3b967fe1f6f4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:59.834923 6849 raft_consensus.cc:1261] T 46ad540deb624a25b97d3b967fe1f6f4 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:59.834960 9721 consensus_queue.cc:1035] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:59.835340 9721 consensus_queue.cc:1035] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:59.837110 9721 raft_consensus.cc:2932] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } } }
I20240503 01:59:59.837626 6720 raft_consensus.cc:2932] T 46ad540deb624a25b97d3b967fe1f6f4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } } }
W20240503 01:59:59.839078 6392 consensus_peers.cc:485] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 46ad540deb624a25b97d3b967fe1f6f4. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:59.840694 6316 catalog_manager.cc:5484] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: config changed from index -1 to 4, NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:59.841378 6849 raft_consensus.cc:2932] T 46ad540deb624a25b97d3b967fe1f6f4 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } } }
I20240503 01:59:59.850956 6720 consensus_queue.cc:237] T 7a44b6d65bfa43e291b06b0dfd38fab3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } }
I20240503 01:59:59.852452 6582 raft_consensus.cc:1261] T 7a44b6d65bfa43e291b06b0dfd38fab3 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:59.852615 6458 raft_consensus.cc:1261] T 7a44b6d65bfa43e291b06b0dfd38fab3 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 01:59:59.852660 6656 consensus_peers.cc:485] T 7a44b6d65bfa43e291b06b0dfd38fab3 P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211): Couldn't send request to peer 9942306855404cfd891adf227dd65eb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7a44b6d65bfa43e291b06b0dfd38fab3. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:59.852995 8696 consensus_queue.cc:1035] T 7a44b6d65bfa43e291b06b0dfd38fab3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:59.853210 9711 consensus_queue.cc:1035] T 7a44b6d65bfa43e291b06b0dfd38fab3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:59.855012 8826 raft_consensus.cc:2932] T 7a44b6d65bfa43e291b06b0dfd38fab3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } } }
I20240503 01:59:59.855509 6582 raft_consensus.cc:2932] T 7a44b6d65bfa43e291b06b0dfd38fab3 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } } }
I20240503 01:59:59.857007 6458 raft_consensus.cc:2932] T 7a44b6d65bfa43e291b06b0dfd38fab3 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } } }
I20240503 01:59:59.859476 6316 catalog_manager.cc:5484] T 7a44b6d65bfa43e291b06b0dfd38fab3 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: config changed from index -1 to 4, NON_VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } } }
I20240503 01:59:59.869308 6720 consensus_queue.cc:237] T 485d1d89a8b94505969d663be92124cd P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } }
I20240503 01:59:59.870765 6849 raft_consensus.cc:1261] T 485d1d89a8b94505969d663be92124cd P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 01:59:59.870966 6654 consensus_peers.cc:485] T 485d1d89a8b94505969d663be92124cd P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157): Couldn't send request to peer 99ead02915954b4fab23b033ad6d546b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 485d1d89a8b94505969d663be92124cd. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:59.871014 6458 raft_consensus.cc:1261] T 485d1d89a8b94505969d663be92124cd P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:59.871223 8696 consensus_queue.cc:1035] T 485d1d89a8b94505969d663be92124cd P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:59.871860 8696 consensus_queue.cc:1035] T 485d1d89a8b94505969d663be92124cd P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:59.873131 8826 raft_consensus.cc:2932] T 485d1d89a8b94505969d663be92124cd P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } } }
I20240503 01:59:59.873654 6849 raft_consensus.cc:2932] T 485d1d89a8b94505969d663be92124cd P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } } }
I20240503 01:59:59.874183 6458 raft_consensus.cc:2932] T 485d1d89a8b94505969d663be92124cd P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } } }
I20240503 01:59:59.876471 6307 catalog_manager.cc:5484] T 485d1d89a8b94505969d663be92124cd P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: config changed from index -1 to 4, NON_VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:59.885578 6849 consensus_queue.cc:237] T 90ef3584830240acb0d8aad14ae851a6 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } }
I20240503 01:59:59.887434 6458 raft_consensus.cc:1261] T 90ef3584830240acb0d8aad14ae851a6 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:59.887614 6720 raft_consensus.cc:1261] T 90ef3584830240acb0d8aad14ae851a6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:59.887977 9461 consensus_queue.cc:1035] T 90ef3584830240acb0d8aad14ae851a6 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:59.888532 9461 consensus_queue.cc:1035] T 90ef3584830240acb0d8aad14ae851a6 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:59.890182 6308 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240503 01:59:59.890321 6308 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240503 01:59:59.890468 9461 raft_consensus.cc:2932] T 90ef3584830240acb0d8aad14ae851a6 P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } } }
W20240503 01:59:59.890992 6784 consensus_peers.cc:485] T 90ef3584830240acb0d8aad14ae851a6 P 492448570cc94ed1879feb2ec3b9051d -> Peer 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079): Couldn't send request to peer 2a01ace80baf4057b6bf1995cf88468b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 90ef3584830240acb0d8aad14ae851a6. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:59.891536 6458 raft_consensus.cc:2932] T 90ef3584830240acb0d8aad14ae851a6 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } } }
I20240503 01:59:59.892566 6720 raft_consensus.cc:2932] T 90ef3584830240acb0d8aad14ae851a6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } } }
I20240503 01:59:59.894486 9533 raft_consensus.cc:482] T 9c8a53980b00421d8c2bd3ae73f0c8cc P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:59.894623 9533 raft_consensus.cc:504] T 9c8a53980b00421d8c2bd3ae73f0c8cc P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 01:59:59.895360 6307 catalog_manager.cc:5484] T 90ef3584830240acb0d8aad14ae851a6 P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: config changed from index -1 to 4, NON_VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } } }
I20240503 01:59:59.895860 9533 leader_election.cc:288] T 9c8a53980b00421d8c2bd3ae73f0c8cc P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 01:59:59.896093 6308 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet fa51b5a29d10491a8437bd5cb84f1479
I20240503 01:59:59.896672 6308 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 4 replicas of tablet 5f176537159143e8a770d65ba63ad62e
I20240503 01:59:59.896976 6436 tablet_service.cc:1514] Processing DeleteTablet for tablet fa51b5a29d10491a8437bd5cb84f1479 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:59 UTC) from {username='slave'} at 127.0.0.1:51338
I20240503 01:59:59.897017 6308 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet 396a2a8083d240529a2db48519f6139e
I20240503 01:59:59.897141 6438 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f176537159143e8a770d65ba63ad62e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:59 UTC) from {username='slave'} at 127.0.0.1:51338
I20240503 01:59:59.897327 9474 tablet_replica.cc:330] T fa51b5a29d10491a8437bd5cb84f1479 P dec3e4a04d604f7eaaa3f077d081218a: stopping tablet replica
I20240503 01:59:59.897483 9474 raft_consensus.cc:2229] T fa51b5a29d10491a8437bd5cb84f1479 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:59.897624 9474 raft_consensus.cc:2258] T fa51b5a29d10491a8437bd5cb84f1479 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:59.898170 9474 ts_tablet_manager.cc:1875] T fa51b5a29d10491a8437bd5cb84f1479 P dec3e4a04d604f7eaaa3f077d081218a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:59.898361 8880 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f176537159143e8a770d65ba63ad62e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:59 UTC) from {username='slave'} at 127.0.0.1:40296
I20240503 01:59:59.898947 6569 tablet_service.cc:1514] Processing DeleteTablet for tablet 396a2a8083d240529a2db48519f6139e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:59 UTC) from {username='slave'} at 127.0.0.1:59846
I20240503 01:59:59.899116 9473 tablet_replica.cc:330] T 396a2a8083d240529a2db48519f6139e P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 01:59:59.899169 6699 tablet_service.cc:1514] Processing DeleteTablet for tablet 396a2a8083d240529a2db48519f6139e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:59 UTC) from {username='slave'} at 127.0.0.1:40336
I20240503 01:59:59.899245 9473 raft_consensus.cc:2229] T 396a2a8083d240529a2db48519f6139e P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:59.899462 9473 raft_consensus.cc:2258] T 396a2a8083d240529a2db48519f6139e P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:59.899976 9473 ts_tablet_manager.cc:1875] T 396a2a8083d240529a2db48519f6139e P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:59.900530 6830 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f176537159143e8a770d65ba63ad62e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:59 UTC) from {username='slave'} at 127.0.0.1:52812
I20240503 01:59:59.900681 6829 tablet_service.cc:1514] Processing DeleteTablet for tablet 396a2a8083d240529a2db48519f6139e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:59 UTC) from {username='slave'} at 127.0.0.1:52812
I20240503 01:59:59.900719 9476 tablet_replica.cc:330] T 5f176537159143e8a770d65ba63ad62e P 492448570cc94ed1879feb2ec3b9051d: stopping tablet replica
I20240503 01:59:59.900800 6567 tablet_service.cc:1514] Processing DeleteTablet for tablet fa51b5a29d10491a8437bd5cb84f1479 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:59 UTC) from {username='slave'} at 127.0.0.1:59846
I20240503 01:59:59.900873 9476 raft_consensus.cc:2229] T 5f176537159143e8a770d65ba63ad62e P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:59.900913 6568 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f176537159143e8a770d65ba63ad62e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:59 UTC) from {username='slave'} at 127.0.0.1:59846
I20240503 01:59:59.901041 9476 raft_consensus.cc:2258] T 5f176537159143e8a770d65ba63ad62e P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:59.901044 6700 tablet_service.cc:1514] Processing DeleteTablet for tablet fa51b5a29d10491a8437bd5cb84f1479 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:59 UTC) from {username='slave'} at 127.0.0.1:40336
I20240503 01:59:59.901773 9476 ts_tablet_manager.cc:1875] T 5f176537159143e8a770d65ba63ad62e P 492448570cc94ed1879feb2ec3b9051d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:59.902293 9473 ts_tablet_manager.cc:1888] T 396a2a8083d240529a2db48519f6139e P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:59.902357 9473 log.cc:1196] T 396a2a8083d240529a2db48519f6139e P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/396a2a8083d240529a2db48519f6139e
I20240503 01:59:59.902585 9473 ts_tablet_manager.cc:1909] T 396a2a8083d240529a2db48519f6139e P 80fd18ef4d604c86b909d5c008a46b4f: Deleting consensus metadata
I20240503 01:59:59.902837 6457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c8a53980b00421d8c2bd3ae73f0c8cc" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 01:59:59.902976 6457 raft_consensus.cc:2454] T 9c8a53980b00421d8c2bd3ae73f0c8cc P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 0.
I20240503 01:59:59.903398 8850 leader_election.cc:302] T 9c8a53980b00421d8c2bd3ae73f0c8cc P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9942306855404cfd891adf227dd65eb2, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:59.903669 9533 raft_consensus.cc:2781] T 9c8a53980b00421d8c2bd3ae73f0c8cc P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:59.903717 9533 raft_consensus.cc:482] T 9c8a53980b00421d8c2bd3ae73f0c8cc P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:59.903762 9533 raft_consensus.cc:3037] T 9c8a53980b00421d8c2bd3ae73f0c8cc P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:59.903991 9476 ts_tablet_manager.cc:1888] T 5f176537159143e8a770d65ba63ad62e P 492448570cc94ed1879feb2ec3b9051d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:59.904063 9476 log.cc:1196] T 5f176537159143e8a770d65ba63ad62e P 492448570cc94ed1879feb2ec3b9051d: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/5f176537159143e8a770d65ba63ad62e
I20240503 01:59:59.904304 9476 ts_tablet_manager.cc:1909] T 5f176537159143e8a770d65ba63ad62e P 492448570cc94ed1879feb2ec3b9051d: Deleting consensus metadata
I20240503 01:59:59.904634 9474 ts_tablet_manager.cc:1888] T fa51b5a29d10491a8437bd5cb84f1479 P dec3e4a04d604f7eaaa3f077d081218a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:59.904709 9474 log.cc:1196] T fa51b5a29d10491a8437bd5cb84f1479 P dec3e4a04d604f7eaaa3f077d081218a: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/fa51b5a29d10491a8437bd5cb84f1479
I20240503 01:59:59.904942 9474 ts_tablet_manager.cc:1909] T fa51b5a29d10491a8437bd5cb84f1479 P dec3e4a04d604f7eaaa3f077d081218a: Deleting consensus metadata
I20240503 01:59:59.904946 9533 raft_consensus.cc:504] T 9c8a53980b00421d8c2bd3ae73f0c8cc P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 01:59:59.905400 9533 leader_election.cc:288] T 9c8a53980b00421d8c2bd3ae73f0c8cc P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 election: Requested vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 01:59:59.907064 9476 tablet_replica.cc:330] T 396a2a8083d240529a2db48519f6139e P 492448570cc94ed1879feb2ec3b9051d: stopping tablet replica
I20240503 01:59:59.907147 6303 catalog_manager.cc:4830] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719): tablet 5f176537159143e8a770d65ba63ad62e (table rebalance_test_table_4 [id=d930c636ed124c0eb64e1d6ab7c8d106]) successfully deleted
I20240503 01:59:59.907240 9473 tablet_replica.cc:330] T fa51b5a29d10491a8437bd5cb84f1479 P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 01:59:59.907387 9473 raft_consensus.cc:2229] T fa51b5a29d10491a8437bd5cb84f1479 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:59.907413 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet 396a2a8083d240529a2db48519f6139e (table rebalance_test_table_4 [id=d930c636ed124c0eb64e1d6ab7c8d106]) successfully deleted
I20240503 01:59:59.907570 9473 raft_consensus.cc:2258] T fa51b5a29d10491a8437bd5cb84f1479 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 01:59:59.907814 6303 catalog_manager.cc:4797] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211): delete failed for tablet 5f176537159143e8a770d65ba63ad62e because the tablet was not found. No further retry: Not found: Tablet not found: 5f176537159143e8a770d65ba63ad62e
I20240503 01:59:59.905124 9721 raft_consensus.cc:482] T 0099654461ca41efa4323fc9f5adab4a P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:59.908200 9721 raft_consensus.cc:504] T 0099654461ca41efa4323fc9f5adab4a P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 01:59:59.908525 9473 ts_tablet_manager.cc:1875] T fa51b5a29d10491a8437bd5cb84f1479 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:59.908495 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c8a53980b00421d8c2bd3ae73f0c8cc" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 01:59:59.908612 6458 raft_consensus.cc:3037] T 9c8a53980b00421d8c2bd3ae73f0c8cc P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:59.908707 9721 leader_election.cc:288] T 0099654461ca41efa4323fc9f5adab4a P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 01:59:59.909042 8917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0099654461ca41efa4323fc9f5adab4a" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 01:59:59.909189 8917 raft_consensus.cc:2454] T 0099654461ca41efa4323fc9f5adab4a P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 01:59:59.909423 9039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c8a53980b00421d8c2bd3ae73f0c8cc" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 01:59:59.909559 9039 raft_consensus.cc:2454] T 9c8a53980b00421d8c2bd3ae73f0c8cc P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 0.
I20240503 01:59:59.909811 6394 leader_election.cc:302] T 0099654461ca41efa4323fc9f5adab4a P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9942306855404cfd891adf227dd65eb2, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:59.910122 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c8a53980b00421d8c2bd3ae73f0c8cc" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 01:59:59.910234 9048 raft_consensus.cc:3037] T 9c8a53980b00421d8c2bd3ae73f0c8cc P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:59.910275 6458 raft_consensus.cc:2454] T 9c8a53980b00421d8c2bd3ae73f0c8cc P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 1.
I20240503 01:59:59.910548 9756 tablet_replica.cc:330] T 396a2a8083d240529a2db48519f6139e P c2ac2eb7cc264047b9ab32fe3a4fc025: stopping tablet replica
I20240503 01:59:59.910574 8850 leader_election.cc:302] T 9c8a53980b00421d8c2bd3ae73f0c8cc P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9942306855404cfd891adf227dd65eb2, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:59.910732 9756 raft_consensus.cc:2229] T 396a2a8083d240529a2db48519f6139e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:59.910750 9533 raft_consensus.cc:2781] T 9c8a53980b00421d8c2bd3ae73f0c8cc P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:59.910897 9756 raft_consensus.cc:2258] T 396a2a8083d240529a2db48519f6139e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:59.910969 9473 ts_tablet_manager.cc:1888] T fa51b5a29d10491a8437bd5cb84f1479 P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:59.911042 9473 log.cc:1196] T fa51b5a29d10491a8437bd5cb84f1479 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/fa51b5a29d10491a8437bd5cb84f1479
I20240503 01:59:59.911170 9533 raft_consensus.cc:686] T 9c8a53980b00421d8c2bd3ae73f0c8cc P 9942306855404cfd891adf227dd65eb2 [term 1 LEADER]: Becoming Leader. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Running, Role: LEADER
I20240503 01:59:59.911235 9048 raft_consensus.cc:2454] T 9c8a53980b00421d8c2bd3ae73f0c8cc P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 1.
I20240503 01:59:59.911285 9473 ts_tablet_manager.cc:1909] T fa51b5a29d10491a8437bd5cb84f1479 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting consensus metadata
I20240503 01:59:59.911480 9756 ts_tablet_manager.cc:1875] T 396a2a8083d240529a2db48519f6139e P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:59.911523 9533 consensus_queue.cc:237] T 9c8a53980b00421d8c2bd3ae73f0c8cc P 9942306855404cfd891adf227dd65eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 01:59:59.912045 9473 tablet_replica.cc:330] T 5f176537159143e8a770d65ba63ad62e P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 01:59:59.912202 9473 raft_consensus.cc:2229] T 5f176537159143e8a770d65ba63ad62e P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:59.912164 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet fa51b5a29d10491a8437bd5cb84f1479 (table rebalance_test_table_4 [id=d930c636ed124c0eb64e1d6ab7c8d106]) successfully deleted
I20240503 01:59:59.912575 9473 raft_consensus.cc:2258] T 5f176537159143e8a770d65ba63ad62e P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:59.913401 9473 ts_tablet_manager.cc:1875] T 5f176537159143e8a770d65ba63ad62e P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:59.915858 9473 ts_tablet_manager.cc:1888] T 5f176537159143e8a770d65ba63ad62e P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:59.915942 9473 log.cc:1196] T 5f176537159143e8a770d65ba63ad62e P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/5f176537159143e8a770d65ba63ad62e
I20240503 01:59:59.916179 9473 ts_tablet_manager.cc:1909] T 5f176537159143e8a770d65ba63ad62e P 80fd18ef4d604c86b909d5c008a46b4f: Deleting consensus metadata
I20240503 01:59:59.916271 6312 catalog_manager.cc:5484] T 9c8a53980b00421d8c2bd3ae73f0c8cc P 9942306855404cfd891adf227dd65eb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9942306855404cfd891adf227dd65eb2 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "9942306855404cfd891adf227dd65eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:59.916502 9721 raft_consensus.cc:2781] T 0099654461ca41efa4323fc9f5adab4a P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:59.916563 9721 raft_consensus.cc:482] T 0099654461ca41efa4323fc9f5adab4a P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:59.916599 9721 raft_consensus.cc:3037] T 0099654461ca41efa4323fc9f5adab4a P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:59.916671 9756 ts_tablet_manager.cc:1888] T 396a2a8083d240529a2db48519f6139e P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:59.917773 9756 log.cc:1196] T 396a2a8083d240529a2db48519f6139e P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/396a2a8083d240529a2db48519f6139e
I20240503 01:59:59.918066 9756 ts_tablet_manager.cc:1909] T 396a2a8083d240529a2db48519f6139e P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting consensus metadata
I20240503 01:59:59.918944 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet 5f176537159143e8a770d65ba63ad62e (table rebalance_test_table_4 [id=d930c636ed124c0eb64e1d6ab7c8d106]) successfully deleted
I20240503 01:59:59.919025 9756 tablet_replica.cc:330] T fa51b5a29d10491a8437bd5cb84f1479 P c2ac2eb7cc264047b9ab32fe3a4fc025: stopping tablet replica
I20240503 01:59:59.919165 9756 raft_consensus.cc:2229] T fa51b5a29d10491a8437bd5cb84f1479 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:59.919196 6303 catalog_manager.cc:4830] TS c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655): tablet 396a2a8083d240529a2db48519f6139e (table rebalance_test_table_4 [id=d930c636ed124c0eb64e1d6ab7c8d106]) successfully deleted
I20240503 01:59:59.919291 9756 raft_consensus.cc:2258] T fa51b5a29d10491a8437bd5cb84f1479 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:59.919731 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0099654461ca41efa4323fc9f5adab4a" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 01:59:59.919834 9756 ts_tablet_manager.cc:1875] T fa51b5a29d10491a8437bd5cb84f1479 P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:59.919965 9179 raft_consensus.cc:2454] T 0099654461ca41efa4323fc9f5adab4a P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 01:59:59.920684 9721 raft_consensus.cc:504] T 0099654461ca41efa4323fc9f5adab4a P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 01:59:59.920706 6582 consensus_queue.cc:237] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } }
I20240503 01:59:59.921047 9721 leader_election.cc:288] T 0099654461ca41efa4323fc9f5adab4a P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Requested vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 01:59:59.921403 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0099654461ca41efa4323fc9f5adab4a" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 01:59:59.921532 9179 raft_consensus.cc:3037] T 0099654461ca41efa4323fc9f5adab4a P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:59.921963 9474 tablet_replica.cc:330] T 5f176537159143e8a770d65ba63ad62e P dec3e4a04d604f7eaaa3f077d081218a: stopping tablet replica
I20240503 01:59:59.922112 9474 raft_consensus.cc:2229] T 5f176537159143e8a770d65ba63ad62e P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:59.922114 8917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0099654461ca41efa4323fc9f5adab4a" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 01:59:59.922235 8917 raft_consensus.cc:3037] T 0099654461ca41efa4323fc9f5adab4a P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:59.922293 9474 raft_consensus.cc:2258] T 5f176537159143e8a770d65ba63ad62e P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:59.922441 9756 ts_tablet_manager.cc:1888] T fa51b5a29d10491a8437bd5cb84f1479 P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:59.922504 9756 log.cc:1196] T fa51b5a29d10491a8437bd5cb84f1479 P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/fa51b5a29d10491a8437bd5cb84f1479
I20240503 01:59:59.922519 9476 raft_consensus.cc:2229] T 396a2a8083d240529a2db48519f6139e P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:59.922659 9476 raft_consensus.cc:2258] T 396a2a8083d240529a2db48519f6139e P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:59.922727 9756 ts_tablet_manager.cc:1909] T fa51b5a29d10491a8437bd5cb84f1479 P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting consensus metadata
I20240503 01:59:59.922824 9179 raft_consensus.cc:2454] T 0099654461ca41efa4323fc9f5adab4a P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 01:59:59.923147 8917 raft_consensus.cc:2454] T 0099654461ca41efa4323fc9f5adab4a P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 01:59:59.923200 6300 catalog_manager.cc:4830] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): tablet fa51b5a29d10491a8437bd5cb84f1479 (table rebalance_test_table_4 [id=d930c636ed124c0eb64e1d6ab7c8d106]) successfully deleted
I20240503 01:59:59.923283 9476 ts_tablet_manager.cc:1875] T 396a2a8083d240529a2db48519f6139e P 492448570cc94ed1879feb2ec3b9051d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:59.923660 6391 leader_election.cc:302] T 0099654461ca41efa4323fc9f5adab4a P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 01:59:59.924260 6720 raft_consensus.cc:1261] T 915028b3478341d588ed6031e657c259 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:59.924283 6303 catalog_manager.cc:4830] TS c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655): tablet fa51b5a29d10491a8437bd5cb84f1479 (table rebalance_test_table_4 [id=d930c636ed124c0eb64e1d6ab7c8d106]) successfully deleted
I20240503 01:59:59.924556 9721 raft_consensus.cc:2781] T 0099654461ca41efa4323fc9f5adab4a P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:59.924638 9721 raft_consensus.cc:686] T 0099654461ca41efa4323fc9f5adab4a P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Becoming Leader. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Running, Role: LEADER
I20240503 01:59:59.924961 6849 raft_consensus.cc:1261] T 915028b3478341d588ed6031e657c259 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 01:59:59.925094 6525 consensus_peers.cc:485] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211): Couldn't send request to peer 9942306855404cfd891adf227dd65eb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 915028b3478341d588ed6031e657c259. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:59.925810 8598 consensus_queue.cc:1035] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:59.926088 8598 consensus_queue.cc:1035] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:59.927832 9476 ts_tablet_manager.cc:1888] T 396a2a8083d240529a2db48519f6139e P 492448570cc94ed1879feb2ec3b9051d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:59.927971 9476 log.cc:1196] T 396a2a8083d240529a2db48519f6139e P 492448570cc94ed1879feb2ec3b9051d: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/396a2a8083d240529a2db48519f6139e
I20240503 01:59:59.928093 8598 raft_consensus.cc:2932] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } } }
I20240503 01:59:59.928349 6312 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:59.928488 6312 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:59.928434 6849 raft_consensus.cc:2932] T 915028b3478341d588ed6031e657c259 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } } }
I20240503 01:59:59.928956 9476 ts_tablet_manager.cc:1909] T 396a2a8083d240529a2db48519f6139e P 492448570cc94ed1879feb2ec3b9051d: Deleting consensus metadata
I20240503 01:59:59.929795 6303 catalog_manager.cc:4830] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719): tablet 396a2a8083d240529a2db48519f6139e (table rebalance_test_table_4 [id=d930c636ed124c0eb64e1d6ab7c8d106]) successfully deleted
I20240503 01:59:59.930747 6312 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet f758d325e8e946deb97be30469844809
I20240503 01:59:59.931334 9025 tablet_service.cc:1514] Processing DeleteTablet for tablet f758d325e8e946deb97be30469844809 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:59 UTC) from {username='slave'} at 127.0.0.1:52584
I20240503 01:59:59.931392 9285 tablet_service.cc:1514] Processing DeleteTablet for tablet f758d325e8e946deb97be30469844809 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:59 UTC) from {username='slave'} at 127.0.0.1:36004
I20240503 01:59:59.931737 6568 tablet_service.cc:1514] Processing DeleteTablet for tablet f758d325e8e946deb97be30469844809 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:59 UTC) from {username='slave'} at 127.0.0.1:59846
I20240503 01:59:59.932520 9473 tablet_replica.cc:330] T f758d325e8e946deb97be30469844809 P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 01:59:59.932713 9473 raft_consensus.cc:2229] T f758d325e8e946deb97be30469844809 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:59.932718 6718 raft_consensus.cc:2932] T 915028b3478341d588ed6031e657c259 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } } }
I20240503 01:59:59.932941 9473 raft_consensus.cc:2258] T f758d325e8e946deb97be30469844809 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:59.924787 9721 consensus_queue.cc:237] T 0099654461ca41efa4323fc9f5adab4a P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 01:59:59.933714 9473 ts_tablet_manager.cc:1875] T f758d325e8e946deb97be30469844809 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:59.926522 9474 ts_tablet_manager.cc:1875] T 5f176537159143e8a770d65ba63ad62e P dec3e4a04d604f7eaaa3f077d081218a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:59.936286 9474 ts_tablet_manager.cc:1888] T 5f176537159143e8a770d65ba63ad62e P dec3e4a04d604f7eaaa3f077d081218a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:59.936394 9474 log.cc:1196] T 5f176537159143e8a770d65ba63ad62e P dec3e4a04d604f7eaaa3f077d081218a: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/5f176537159143e8a770d65ba63ad62e
I20240503 01:59:59.936637 9474 ts_tablet_manager.cc:1909] T 5f176537159143e8a770d65ba63ad62e P dec3e4a04d604f7eaaa3f077d081218a: Deleting consensus metadata
I20240503 01:59:59.937171 9473 ts_tablet_manager.cc:1888] T f758d325e8e946deb97be30469844809 P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:59.937315 9473 log.cc:1196] T f758d325e8e946deb97be30469844809 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/f758d325e8e946deb97be30469844809
I20240503 01:59:59.937624 9473 ts_tablet_manager.cc:1909] T f758d325e8e946deb97be30469844809 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting consensus metadata
I20240503 01:59:59.932163 6312 catalog_manager.cc:5484] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: config changed from index -1 to 4, NON_VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:59.938074 6300 catalog_manager.cc:4830] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): tablet 5f176537159143e8a770d65ba63ad62e (table rebalance_test_table_4 [id=d930c636ed124c0eb64e1d6ab7c8d106]) successfully deleted
I20240503 01:59:59.939168 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet f758d325e8e946deb97be30469844809 (table rebalancer_dynamic_table_new_name [id=8b571be2b4d14eab84dc1bc1e5b8d53a]) successfully deleted
I20240503 01:59:59.939664 9764 tablet_replica.cc:330] T f758d325e8e946deb97be30469844809 P e1a9251ce1cf4361b8b0ede9fbfc4023: stopping tablet replica
I20240503 01:59:59.939849 9764 raft_consensus.cc:2229] T f758d325e8e946deb97be30469844809 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:59.940070 9764 raft_consensus.cc:2258] T f758d325e8e946deb97be30469844809 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:59.940971 6316 catalog_manager.cc:5484] T 0099654461ca41efa4323fc9f5adab4a P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: term changed from 0 to 1, leader changed from <none> to dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129). New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:59.941402 9764 ts_tablet_manager.cc:1875] T f758d325e8e946deb97be30469844809 P e1a9251ce1cf4361b8b0ede9fbfc4023: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:59.942440 9763 tablet_replica.cc:330] T f758d325e8e946deb97be30469844809 P 99ead02915954b4fab23b033ad6d546b: stopping tablet replica
I20240503 01:59:59.943126 9763 raft_consensus.cc:2229] T f758d325e8e946deb97be30469844809 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:59.943338 9763 raft_consensus.cc:2258] T f758d325e8e946deb97be30469844809 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:59.944609 9763 ts_tablet_manager.cc:1875] T f758d325e8e946deb97be30469844809 P 99ead02915954b4fab23b033ad6d546b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:59.945591 6849 consensus_queue.cc:237] T 548f43b381f1445c8d3d55705ae5a48a P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } }
I20240503 01:59:59.946261 9764 ts_tablet_manager.cc:1888] T f758d325e8e946deb97be30469844809 P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:59.946347 9764 log.cc:1196] T f758d325e8e946deb97be30469844809 P e1a9251ce1cf4361b8b0ede9fbfc4023: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/f758d325e8e946deb97be30469844809
I20240503 01:59:59.946735 9764 ts_tablet_manager.cc:1909] T f758d325e8e946deb97be30469844809 P e1a9251ce1cf4361b8b0ede9fbfc4023: Deleting consensus metadata
I20240503 01:59:59.946803 9763 ts_tablet_manager.cc:1888] T f758d325e8e946deb97be30469844809 P 99ead02915954b4fab23b033ad6d546b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:59.946914 9763 log.cc:1196] T f758d325e8e946deb97be30469844809 P 99ead02915954b4fab23b033ad6d546b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/f758d325e8e946deb97be30469844809
I20240503 01:59:59.947268 6582 raft_consensus.cc:1261] T 548f43b381f1445c8d3d55705ae5a48a P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:59.947373 6458 raft_consensus.cc:1261] T 548f43b381f1445c8d3d55705ae5a48a P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:59.947485 9763 ts_tablet_manager.cc:1909] T f758d325e8e946deb97be30469844809 P 99ead02915954b4fab23b033ad6d546b: Deleting consensus metadata
I20240503 01:59:59.947659 6301 catalog_manager.cc:4830] TS e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): tablet f758d325e8e946deb97be30469844809 (table rebalancer_dynamic_table_new_name [id=8b571be2b4d14eab84dc1bc1e5b8d53a]) successfully deleted
I20240503 01:59:59.947810 9631 consensus_queue.cc:1035] T 548f43b381f1445c8d3d55705ae5a48a P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 01:59:59.948035 6785 consensus_peers.cc:485] T 548f43b381f1445c8d3d55705ae5a48a P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 548f43b381f1445c8d3d55705ae5a48a. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:59.948036 9461 consensus_queue.cc:1035] T 548f43b381f1445c8d3d55705ae5a48a P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:59.948729 6301 catalog_manager.cc:4830] TS 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157): tablet f758d325e8e946deb97be30469844809 (table rebalancer_dynamic_table_new_name [id=8b571be2b4d14eab84dc1bc1e5b8d53a]) successfully deleted
I20240503 01:59:59.950134 9632 raft_consensus.cc:2932] T 548f43b381f1445c8d3d55705ae5a48a P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } } }
I20240503 01:59:59.950568 6458 raft_consensus.cc:2932] T 548f43b381f1445c8d3d55705ae5a48a P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } } }
I20240503 01:59:59.950775 6582 raft_consensus.cc:2932] T 548f43b381f1445c8d3d55705ae5a48a P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } } }
I20240503 01:59:59.953583 6316 catalog_manager.cc:5484] T 548f43b381f1445c8d3d55705ae5a48a P 492448570cc94ed1879feb2ec3b9051d reported cstate change: config changed from index -1 to 4, NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:59.963413 6582 consensus_queue.cc:237] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } }
I20240503 01:59:59.964890 6849 raft_consensus.cc:1261] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:59.964890 6718 raft_consensus.cc:1261] T 74ebf5d0fb9a42e6a7ec743a98960e13 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 01:59:59.965169 6523 consensus_peers.cc:485] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157): Couldn't send request to peer 99ead02915954b4fab23b033ad6d546b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 74ebf5d0fb9a42e6a7ec743a98960e13. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:59.965320 8833 consensus_queue.cc:1035] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:59.965471 9484 consensus_queue.cc:1035] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:59.967862 9484 raft_consensus.cc:2932] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } } }
I20240503 01:59:59.968570 6718 raft_consensus.cc:2932] T 74ebf5d0fb9a42e6a7ec743a98960e13 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } } }
I20240503 01:59:59.968675 6849 raft_consensus.cc:2932] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } } }
I20240503 01:59:59.971570 6307 catalog_manager.cc:5484] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: config changed from index -1 to 4, NON_VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:59.981611 6458 consensus_queue.cc:237] T 8a76d3e2d4264687aa2972c179341b5e P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } }
I20240503 01:59:59.983197 6718 raft_consensus.cc:1261] T 8a76d3e2d4264687aa2972c179341b5e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:59.983335 6849 raft_consensus.cc:1261] T 8a76d3e2d4264687aa2972c179341b5e P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 01:59:59.983515 6391 consensus_peers.cc:485] T 8a76d3e2d4264687aa2972c179341b5e P dec3e4a04d604f7eaaa3f077d081218a -> Peer 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079): Couldn't send request to peer 2a01ace80baf4057b6bf1995cf88468b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8a76d3e2d4264687aa2972c179341b5e. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:59.983666 8633 consensus_queue.cc:1035] T 8a76d3e2d4264687aa2972c179341b5e P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:59.984081 8633 consensus_queue.cc:1035] T 8a76d3e2d4264687aa2972c179341b5e P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:59.985427 8679 raft_consensus.cc:2932] T 8a76d3e2d4264687aa2972c179341b5e P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } } }
I20240503 01:59:59.985883 6720 raft_consensus.cc:2932] T 8a76d3e2d4264687aa2972c179341b5e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } } }
I20240503 01:59:59.987365 6849 raft_consensus.cc:2932] T 8a76d3e2d4264687aa2972c179341b5e P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } } }
I20240503 01:59:59.988723 6316 catalog_manager.cc:5484] T 8a76d3e2d4264687aa2972c179341b5e P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: config changed from index -1 to 4, NON_VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } } }
I20240503 01:59:59.996479 6458 consensus_queue.cc:237] T 2b1d700d09534fa3adcf51bb3c97c8f5 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } }
I20240503 01:59:59.998142 6849 raft_consensus.cc:1261] T 2b1d700d09534fa3adcf51bb3c97c8f5 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:59.998180 6720 raft_consensus.cc:1261] T 2b1d700d09534fa3adcf51bb3c97c8f5 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 01:59:59.998307 6394 consensus_peers.cc:485] T 2b1d700d09534fa3adcf51bb3c97c8f5 P dec3e4a04d604f7eaaa3f077d081218a -> Peer 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211): Couldn't send request to peer 9942306855404cfd891adf227dd65eb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2b1d700d09534fa3adcf51bb3c97c8f5. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:59.998546 9721 consensus_queue.cc:1035] T 2b1d700d09534fa3adcf51bb3c97c8f5 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:59.998742 8633 consensus_queue.cc:1035] T 2b1d700d09534fa3adcf51bb3c97c8f5 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:00.000686 8679 raft_consensus.cc:2932] T 2b1d700d09534fa3adcf51bb3c97c8f5 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } } }
I20240503 02:00:00.001197 6720 raft_consensus.cc:2932] T 2b1d700d09534fa3adcf51bb3c97c8f5 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } } }
I20240503 02:00:00.001211 6849 raft_consensus.cc:2932] T 2b1d700d09534fa3adcf51bb3c97c8f5 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } } }
I20240503 02:00:00.003948 6307 catalog_manager.cc:5484] T 2b1d700d09534fa3adcf51bb3c97c8f5 P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: config changed from index -1 to 4, NON_VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:00.032572 6312 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:00:00.034511 9758 raft_consensus.cc:482] T 067f54011a37432d99157884e70fc601 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:00.034642 9758 raft_consensus.cc:504] T 067f54011a37432d99157884e70fc601 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:00.035203 9758 leader_election.cc:288] T 067f54011a37432d99157884e70fc601 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:00.040042 6568 tablet_service.cc:1467] Processing CreateTablet for tablet 9a9e4df681694aa6994b0020acd466f5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6aaa25ca43754a899cf8589522ff71da]), partition=
I20240503 02:00:00.040042 9285 tablet_service.cc:1467] Processing CreateTablet for tablet 9a9e4df681694aa6994b0020acd466f5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6aaa25ca43754a899cf8589522ff71da]), partition=
I20240503 02:00:00.040318 6568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a9e4df681694aa6994b0020acd466f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:00.040321 9285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a9e4df681694aa6994b0020acd466f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:00.042754 6829 tablet_service.cc:1467] Processing CreateTablet for tablet 9a9e4df681694aa6994b0020acd466f5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6aaa25ca43754a899cf8589522ff71da]), partition=
I20240503 02:00:00.043015 6829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a9e4df681694aa6994b0020acd466f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:00.044544 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "067f54011a37432d99157884e70fc601" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:00.044693 9179 raft_consensus.cc:2454] T 067f54011a37432d99157884e70fc601 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 0.
I20240503 02:00:00.045007 8850 leader_election.cc:302] T 067f54011a37432d99157884e70fc601 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:00.045202 9758 raft_consensus.cc:2781] T 067f54011a37432d99157884e70fc601 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:00.045251 9758 raft_consensus.cc:482] T 067f54011a37432d99157884e70fc601 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:00.045269 9374 tablet_bootstrap.cc:492] T 9a9e4df681694aa6994b0020acd466f5 P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap starting.
I20240503 02:00:00.045286 9758 raft_consensus.cc:3037] T 067f54011a37432d99157884e70fc601 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:00.046243 9374 tablet_bootstrap.cc:654] T 9a9e4df681694aa6994b0020acd466f5 P e1a9251ce1cf4361b8b0ede9fbfc4023: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:00.046262 9758 raft_consensus.cc:504] T 067f54011a37432d99157884e70fc601 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:00.046694 9758 leader_election.cc:288] T 067f54011a37432d99157884e70fc601 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 election: Requested vote from peers e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:00.047194 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "067f54011a37432d99157884e70fc601" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:00.047400 9179 raft_consensus.cc:3037] T 067f54011a37432d99157884e70fc601 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:00.047686 9816 tablet_bootstrap.cc:492] T 9a9e4df681694aa6994b0020acd466f5 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:00.048499 9179 raft_consensus.cc:2454] T 067f54011a37432d99157884e70fc601 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 1.
I20240503 02:00:00.048614 9816 tablet_bootstrap.cc:654] T 9a9e4df681694aa6994b0020acd466f5 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:00.048769 9310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "067f54011a37432d99157884e70fc601" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" is_pre_election: true
I20240503 02:00:00.048919 9310 raft_consensus.cc:2454] T 067f54011a37432d99157884e70fc601 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 0.
I20240503 02:00:00.049074 8850 leader_election.cc:302] T 067f54011a37432d99157884e70fc601 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:00.049139 9309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "067f54011a37432d99157884e70fc601" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023"
I20240503 02:00:00.049221 9309 raft_consensus.cc:3037] T 067f54011a37432d99157884e70fc601 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:00.049280 9815 tablet_bootstrap.cc:492] T 9a9e4df681694aa6994b0020acd466f5 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:00.049283 9758 raft_consensus.cc:2781] T 067f54011a37432d99157884e70fc601 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:00.049358 9758 raft_consensus.cc:686] T 067f54011a37432d99157884e70fc601 P 9942306855404cfd891adf227dd65eb2 [term 1 LEADER]: Becoming Leader. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Running, Role: LEADER
I20240503 02:00:00.049489 9758 consensus_queue.cc:237] T 067f54011a37432d99157884e70fc601 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:00.050206 9309 raft_consensus.cc:2454] T 067f54011a37432d99157884e70fc601 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 1.
I20240503 02:00:00.050710 9815 tablet_bootstrap.cc:654] T 9a9e4df681694aa6994b0020acd466f5 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:00.051983 9816 tablet_bootstrap.cc:492] T 9a9e4df681694aa6994b0020acd466f5 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:00.051908 6312 catalog_manager.cc:5484] T 067f54011a37432d99157884e70fc601 P 9942306855404cfd891adf227dd65eb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9942306855404cfd891adf227dd65eb2 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "9942306855404cfd891adf227dd65eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:00.052083 9816 ts_tablet_manager.cc:1367] T 9a9e4df681694aa6994b0020acd466f5 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:00.052595 9816 raft_consensus.cc:348] T 9a9e4df681694aa6994b0020acd466f5 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:00.052692 9816 raft_consensus.cc:374] T 9a9e4df681694aa6994b0020acd466f5 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:00.052721 9816 raft_consensus.cc:729] T 9a9e4df681694aa6994b0020acd466f5 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:00.052863 9816 consensus_queue.cc:260] T 9a9e4df681694aa6994b0020acd466f5 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:00.053143 9816 ts_tablet_manager.cc:1398] T 9a9e4df681694aa6994b0020acd466f5 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:00.056654 9374 tablet_bootstrap.cc:492] T 9a9e4df681694aa6994b0020acd466f5 P e1a9251ce1cf4361b8b0ede9fbfc4023: No bootstrap required, opened a new log
I20240503 02:00:00.056768 9374 ts_tablet_manager.cc:1367] T 9a9e4df681694aa6994b0020acd466f5 P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 02:00:00.057516 9374 raft_consensus.cc:348] T 9a9e4df681694aa6994b0020acd466f5 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:00.057638 9374 raft_consensus.cc:374] T 9a9e4df681694aa6994b0020acd466f5 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:00.057667 9374 raft_consensus.cc:729] T 9a9e4df681694aa6994b0020acd466f5 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1a9251ce1cf4361b8b0ede9fbfc4023, State: Initialized, Role: FOLLOWER
I20240503 02:00:00.057776 9374 consensus_queue.cc:260] T 9a9e4df681694aa6994b0020acd466f5 P e1a9251ce1cf4361b8b0ede9fbfc4023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:00.058065 9374 ts_tablet_manager.cc:1398] T 9a9e4df681694aa6994b0020acd466f5 P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:00.061308 9815 tablet_bootstrap.cc:492] T 9a9e4df681694aa6994b0020acd466f5 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:00.061498 9815 ts_tablet_manager.cc:1367] T 9a9e4df681694aa6994b0020acd466f5 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 02:00:00.061966 9815 raft_consensus.cc:348] T 9a9e4df681694aa6994b0020acd466f5 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:00.062100 9815 raft_consensus.cc:374] T 9a9e4df681694aa6994b0020acd466f5 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:00.062127 9815 raft_consensus.cc:729] T 9a9e4df681694aa6994b0020acd466f5 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:00.062255 9815 consensus_queue.cc:260] T 9a9e4df681694aa6994b0020acd466f5 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:00.062633 9815 ts_tablet_manager.cc:1398] T 9a9e4df681694aa6994b0020acd466f5 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:00.066655 9631 raft_consensus.cc:482] T 9a9e4df681694aa6994b0020acd466f5 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:00.066769 9631 raft_consensus.cc:504] T 9a9e4df681694aa6994b0020acd466f5 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:00.067193 9631 leader_election.cc:288] T 9a9e4df681694aa6994b0020acd466f5 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:00.067567 9309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a9e4df681694aa6994b0020acd466f5" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" is_pre_election: true
I20240503 02:00:00.067699 9309 raft_consensus.cc:2454] T 9a9e4df681694aa6994b0020acd466f5 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 02:00:00.068102 6582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a9e4df681694aa6994b0020acd466f5" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 02:00:00.068135 6785 leader_election.cc:302] T 9a9e4df681694aa6994b0020acd466f5 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, e1a9251ce1cf4361b8b0ede9fbfc4023; no voters:
I20240503 02:00:00.068226 6582 raft_consensus.cc:2454] T 9a9e4df681694aa6994b0020acd466f5 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 02:00:00.068322 9631 raft_consensus.cc:2781] T 9a9e4df681694aa6994b0020acd466f5 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:00.068423 9631 raft_consensus.cc:482] T 9a9e4df681694aa6994b0020acd466f5 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:00.068526 9631 raft_consensus.cc:3037] T 9a9e4df681694aa6994b0020acd466f5 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:00.069535 9631 raft_consensus.cc:504] T 9a9e4df681694aa6994b0020acd466f5 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:00.069981 9631 leader_election.cc:288] T 9a9e4df681694aa6994b0020acd466f5 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Requested vote from peers e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:00.070801 9309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a9e4df681694aa6994b0020acd466f5" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023"
I20240503 02:00:00.070935 9309 raft_consensus.cc:3037] T 9a9e4df681694aa6994b0020acd466f5 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:00.071616 6582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a9e4df681694aa6994b0020acd466f5" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 02:00:00.071724 6582 raft_consensus.cc:3037] T 9a9e4df681694aa6994b0020acd466f5 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:00.071995 9309 raft_consensus.cc:2454] T 9a9e4df681694aa6994b0020acd466f5 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 02:00:00.072497 6785 leader_election.cc:302] T 9a9e4df681694aa6994b0020acd466f5 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, e1a9251ce1cf4361b8b0ede9fbfc4023; no voters:
I20240503 02:00:00.072695 9631 raft_consensus.cc:2781] T 9a9e4df681694aa6994b0020acd466f5 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:00.072772 9631 raft_consensus.cc:686] T 9a9e4df681694aa6994b0020acd466f5 P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Becoming Leader. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Running, Role: LEADER
I20240503 02:00:00.072762 6582 raft_consensus.cc:2454] T 9a9e4df681694aa6994b0020acd466f5 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 02:00:00.072916 9631 consensus_queue.cc:237] T 9a9e4df681694aa6994b0020acd466f5 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:00.077039 6314 catalog_manager.cc:5484] T 9a9e4df681694aa6994b0020acd466f5 P 492448570cc94ed1879feb2ec3b9051d reported cstate change: term changed from 0 to 1, leader changed from <none> to 492448570cc94ed1879feb2ec3b9051d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:00.133222 9853 ts_tablet_manager.cc:902] T 3416b6107e1848f299ac49ccd88ae7c8 P e1a9251ce1cf4361b8b0ede9fbfc4023: Initiating tablet copy from peer 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:00.134493 6568 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:00:00.134522 9853 tablet_copy_client.cc:308] T 3416b6107e1848f299ac49ccd88ae7c8 P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43863
I20240503 02:00:00.135157 9158 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:00:00.136277 6829 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 02:00:00.146135 6700 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:00:00.146260 9025 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:00.147102 8880 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:00:00.151422 9859 ts_tablet_manager.cc:902] T ceca04391cd446ae932b3485d958237a P 9942306855404cfd891adf227dd65eb2: Initiating tablet copy from peer c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:00.155057 6438 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:00:00.157078 9859 tablet_copy_client.cc:308] T ceca04391cd446ae932b3485d958237a P 9942306855404cfd891adf227dd65eb2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:42655
I20240503 02:00:00.167726 6609 tablet_copy_service.cc:140] P 80fd18ef4d604c86b909d5c008a46b4f: Received BeginTabletCopySession request for tablet 3416b6107e1848f299ac49ccd88ae7c8 from peer e1a9251ce1cf4361b8b0ede9fbfc4023 ({username='slave'} at 127.0.99.136:60117)
I20240503 02:00:00.167912 6609 tablet_copy_service.cc:161] P 80fd18ef4d604c86b909d5c008a46b4f: Beginning new tablet copy session on tablet 3416b6107e1848f299ac49ccd88ae7c8 from peer e1a9251ce1cf4361b8b0ede9fbfc4023 at {username='slave'} at 127.0.99.136:60117: session id = e1a9251ce1cf4361b8b0ede9fbfc4023-3416b6107e1848f299ac49ccd88ae7c8
I20240503 02:00:00.169442 6609 tablet_copy_source_session.cc:205] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:00.171233 9853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3416b6107e1848f299ac49ccd88ae7c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:00.176908 6740 tablet_copy_service.cc:140] P c2ac2eb7cc264047b9ab32fe3a4fc025: Received BeginTabletCopySession request for tablet ceca04391cd446ae932b3485d958237a from peer 9942306855404cfd891adf227dd65eb2 ({username='slave'} at 127.0.99.133:46077)
I20240503 02:00:00.177057 6740 tablet_copy_service.cc:161] P c2ac2eb7cc264047b9ab32fe3a4fc025: Beginning new tablet copy session on tablet ceca04391cd446ae932b3485d958237a from peer 9942306855404cfd891adf227dd65eb2 at {username='slave'} at 127.0.99.133:46077: session id = 9942306855404cfd891adf227dd65eb2-ceca04391cd446ae932b3485d958237a
I20240503 02:00:00.178447 6740 tablet_copy_source_session.cc:205] T ceca04391cd446ae932b3485d958237a P c2ac2eb7cc264047b9ab32fe3a4fc025: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:00.178685 9853 tablet_copy_client.cc:783] T 3416b6107e1848f299ac49ccd88ae7c8 P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:00.178949 9853 tablet_copy_client.cc:647] T 3416b6107e1848f299ac49ccd88ae7c8 P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:00.180456 9859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ceca04391cd446ae932b3485d958237a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:00.184111 9859 tablet_copy_client.cc:783] T ceca04391cd446ae932b3485d958237a P 9942306855404cfd891adf227dd65eb2: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:00.184463 9859 tablet_copy_client.cc:647] T ceca04391cd446ae932b3485d958237a P 9942306855404cfd891adf227dd65eb2: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:00.189165 6307 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:00.182765 9864 ts_tablet_manager.cc:902] T 1fa274c98fc147f29344ed537e2ccf17 P e1a9251ce1cf4361b8b0ede9fbfc4023: Initiating tablet copy from peer 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:00.197376 9285 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:00.197933 9864 tablet_copy_client.cc:308] T 1fa274c98fc147f29344ed537e2ccf17 P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:42719
I20240503 02:00:00.198002 9853 tablet_copy_client.cc:523] T 3416b6107e1848f299ac49ccd88ae7c8 P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:00.199839 9853 tablet_bootstrap.cc:492] T 3416b6107e1848f299ac49ccd88ae7c8 P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap starting.
I20240503 02:00:00.207854 6871 tablet_copy_service.cc:140] P 492448570cc94ed1879feb2ec3b9051d: Received BeginTabletCopySession request for tablet 1fa274c98fc147f29344ed537e2ccf17 from peer e1a9251ce1cf4361b8b0ede9fbfc4023 ({username='slave'} at 127.0.99.136:48421)
I20240503 02:00:00.208035 6871 tablet_copy_service.cc:161] P 492448570cc94ed1879feb2ec3b9051d: Beginning new tablet copy session on tablet 1fa274c98fc147f29344ed537e2ccf17 from peer e1a9251ce1cf4361b8b0ede9fbfc4023 at {username='slave'} at 127.0.99.136:48421: session id = e1a9251ce1cf4361b8b0ede9fbfc4023-1fa274c98fc147f29344ed537e2ccf17
I20240503 02:00:00.209493 6871 tablet_copy_source_session.cc:205] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:00.211197 9864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fa274c98fc147f29344ed537e2ccf17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:00.213850 9859 tablet_copy_client.cc:523] T ceca04391cd446ae932b3485d958237a P 9942306855404cfd891adf227dd65eb2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:00.214044 9864 tablet_copy_client.cc:783] T 1fa274c98fc147f29344ed537e2ccf17 P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:00.214218 9864 tablet_copy_client.cc:647] T 1fa274c98fc147f29344ed537e2ccf17 P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:00.215963 9864 tablet_copy_client.cc:523] T 1fa274c98fc147f29344ed537e2ccf17 P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:00.216755 9859 tablet_bootstrap.cc:492] T ceca04391cd446ae932b3485d958237a P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:00.218812 9864 tablet_bootstrap.cc:492] T 1fa274c98fc147f29344ed537e2ccf17 P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap starting.
I20240503 02:00:00.228037 6582 raft_consensus.cc:1261] T 9a9e4df681694aa6994b0020acd466f5 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:00.229329 9309 raft_consensus.cc:1261] T 9a9e4df681694aa6994b0020acd466f5 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:00.230144 9461 consensus_queue.cc:1035] T 9a9e4df681694aa6994b0020acd466f5 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:00.231482 9461 consensus_queue.cc:1035] T 9a9e4df681694aa6994b0020acd466f5 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:00.237145 6307 catalog_manager.cc:5576] TS e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311) does not have the latest schema for tablet 9a9e4df681694aa6994b0020acd466f5 (table rebalancer_dynamic_table [id=6aaa25ca43754a899cf8589522ff71da]). Expected version 1 got 0
I20240503 02:00:00.238431 6307 catalog_manager.cc:5576] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863) does not have the latest schema for tablet 9a9e4df681694aa6994b0020acd466f5 (table rebalancer_dynamic_table [id=6aaa25ca43754a899cf8589522ff71da]). Expected version 1 got 0
I20240503 02:00:00.245075 9876 ts_tablet_manager.cc:902] T 1feaf29d8c6b4842829ce66e0f4ec23b P 2a01ace80baf4057b6bf1995cf88468b: Initiating tablet copy from peer c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:00.246852 9876 tablet_copy_client.cc:308] T 1feaf29d8c6b4842829ce66e0f4ec23b P 2a01ace80baf4057b6bf1995cf88468b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:42655
I20240503 02:00:00.246011 9873 ts_tablet_manager.cc:902] T d82268925a934c559dfcf51a84f03d2e P 99ead02915954b4fab23b033ad6d546b: Initiating tablet copy from peer c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:00.251729 9873 tablet_copy_client.cc:308] T d82268925a934c559dfcf51a84f03d2e P 99ead02915954b4fab23b033ad6d546b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:42655
I20240503 02:00:00.252249 9477 tablet.cc:1722] T 9a9e4df681694aa6994b0020acd466f5 P e1a9251ce1cf4361b8b0ede9fbfc4023: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:00.252677 9477 tablet.cc:1620] T 9a9e4df681694aa6994b0020acd466f5 P e1a9251ce1cf4361b8b0ede9fbfc4023: MemRowSet was empty: no flush needed.
I20240503 02:00:00.253291 6307 catalog_manager.cc:5576] TS e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311) does not have the latest schema for tablet 9a9e4df681694aa6994b0020acd466f5 (table rebalancer_dynamic_table [id=6aaa25ca43754a899cf8589522ff71da]). Expected version 1 got 0
I20240503 02:00:00.253803 6315 catalog_manager.cc:5576] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719) does not have the latest schema for tablet 9a9e4df681694aa6994b0020acd466f5 (table rebalancer_dynamic_table [id=6aaa25ca43754a899cf8589522ff71da]). Expected version 1 got 0
I20240503 02:00:00.256016 9859 tablet_bootstrap.cc:492] T ceca04391cd446ae932b3485d958237a P 9942306855404cfd891adf227dd65eb2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:00.258096 9859 tablet_bootstrap.cc:492] T ceca04391cd446ae932b3485d958237a P 9942306855404cfd891adf227dd65eb2: Bootstrap complete.
I20240503 02:00:00.258478 9859 ts_tablet_manager.cc:1367] T ceca04391cd446ae932b3485d958237a P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.042s user 0.026s sys 0.005s
I20240503 02:00:00.258610 6315 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=6aaa25ca43754a899cf8589522ff71da] alter complete (version 1)
I20240503 02:00:00.257576 9889 ts_tablet_manager.cc:902] T 7a44b6d65bfa43e291b06b0dfd38fab3 P 9942306855404cfd891adf227dd65eb2: Initiating tablet copy from peer c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:00.259168 9859 raft_consensus.cc:348] T ceca04391cd446ae932b3485d958237a P 9942306855404cfd891adf227dd65eb2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } }
I20240503 02:00:00.259444 9859 raft_consensus.cc:729] T ceca04391cd446ae932b3485d958237a P 9942306855404cfd891adf227dd65eb2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: LEARNER
I20240503 02:00:00.259369 9884 tablet.cc:1722] T 9a9e4df681694aa6994b0020acd466f5 P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:00.259445 9889 tablet_copy_client.cc:308] T 7a44b6d65bfa43e291b06b0dfd38fab3 P 9942306855404cfd891adf227dd65eb2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:42655
I20240503 02:00:00.259721 9859 consensus_queue.cc:260] T ceca04391cd446ae932b3485d958237a P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } }
I20240503 02:00:00.260030 6740 tablet_copy_service.cc:140] P c2ac2eb7cc264047b9ab32fe3a4fc025: Received BeginTabletCopySession request for tablet 7a44b6d65bfa43e291b06b0dfd38fab3 from peer 9942306855404cfd891adf227dd65eb2 ({username='slave'} at 127.0.99.133:46077)
I20240503 02:00:00.260152 6740 tablet_copy_service.cc:161] P c2ac2eb7cc264047b9ab32fe3a4fc025: Beginning new tablet copy session on tablet 7a44b6d65bfa43e291b06b0dfd38fab3 from peer 9942306855404cfd891adf227dd65eb2 at {username='slave'} at 127.0.99.133:46077: session id = 9942306855404cfd891adf227dd65eb2-7a44b6d65bfa43e291b06b0dfd38fab3
I20240503 02:00:00.260159 9859 ts_tablet_manager.cc:1398] T ceca04391cd446ae932b3485d958237a P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:00.261207 6740 tablet_copy_source_session.cc:205] T 7a44b6d65bfa43e291b06b0dfd38fab3 P c2ac2eb7cc264047b9ab32fe3a4fc025: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:00.261641 6739 tablet_copy_service.cc:342] P c2ac2eb7cc264047b9ab32fe3a4fc025: Request end of tablet copy session 9942306855404cfd891adf227dd65eb2-ceca04391cd446ae932b3485d958237a received from {username='slave'} at 127.0.99.133:46077
I20240503 02:00:00.261749 6739 tablet_copy_service.cc:433] P c2ac2eb7cc264047b9ab32fe3a4fc025: ending tablet copy session 9942306855404cfd891adf227dd65eb2-ceca04391cd446ae932b3485d958237a on tablet ceca04391cd446ae932b3485d958237a with peer 9942306855404cfd891adf227dd65eb2
I20240503 02:00:00.262578 9884 tablet.cc:1620] T 9a9e4df681694aa6994b0020acd466f5 P 492448570cc94ed1879feb2ec3b9051d: MemRowSet was empty: no flush needed.
I20240503 02:00:00.264447 9889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a44b6d65bfa43e291b06b0dfd38fab3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:00.272454 6739 tablet_copy_service.cc:140] P c2ac2eb7cc264047b9ab32fe3a4fc025: Received BeginTabletCopySession request for tablet d82268925a934c559dfcf51a84f03d2e from peer 99ead02915954b4fab23b033ad6d546b ({username='slave'} at 127.0.99.134:48173)
I20240503 02:00:00.272598 6739 tablet_copy_service.cc:161] P c2ac2eb7cc264047b9ab32fe3a4fc025: Beginning new tablet copy session on tablet d82268925a934c559dfcf51a84f03d2e from peer 99ead02915954b4fab23b033ad6d546b at {username='slave'} at 127.0.99.134:48173: session id = 99ead02915954b4fab23b033ad6d546b-d82268925a934c559dfcf51a84f03d2e
I20240503 02:00:00.272945 9853 tablet_bootstrap.cc:492] T 3416b6107e1848f299ac49ccd88ae7c8 P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:00.273249 9853 tablet_bootstrap.cc:492] T 3416b6107e1848f299ac49ccd88ae7c8 P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap complete.
I20240503 02:00:00.273449 9853 ts_tablet_manager.cc:1367] T 3416b6107e1848f299ac49ccd88ae7c8 P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent bootstrapping tablet: real 0.074s user 0.025s sys 0.005s
I20240503 02:00:00.273591 6739 tablet_copy_source_session.cc:205] T d82268925a934c559dfcf51a84f03d2e P c2ac2eb7cc264047b9ab32fe3a4fc025: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:00.273965 9853 raft_consensus.cc:348] T 3416b6107e1848f299ac49ccd88ae7c8 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } }
I20240503 02:00:00.274199 9853 raft_consensus.cc:729] T 3416b6107e1848f299ac49ccd88ae7c8 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e1a9251ce1cf4361b8b0ede9fbfc4023, State: Initialized, Role: LEARNER
I20240503 02:00:00.274382 9853 consensus_queue.cc:260] T 3416b6107e1848f299ac49ccd88ae7c8 P e1a9251ce1cf4361b8b0ede9fbfc4023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } }
I20240503 02:00:00.275417 9853 ts_tablet_manager.cc:1398] T 3416b6107e1848f299ac49ccd88ae7c8 P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:00:00.276130 6609 tablet_copy_service.cc:342] P 80fd18ef4d604c86b909d5c008a46b4f: Request end of tablet copy session e1a9251ce1cf4361b8b0ede9fbfc4023-3416b6107e1848f299ac49ccd88ae7c8 received from {username='slave'} at 127.0.99.136:60117
I20240503 02:00:00.276228 6609 tablet_copy_service.cc:433] P 80fd18ef4d604c86b909d5c008a46b4f: ending tablet copy session e1a9251ce1cf4361b8b0ede9fbfc4023-3416b6107e1848f299ac49ccd88ae7c8 on tablet 3416b6107e1848f299ac49ccd88ae7c8 with peer e1a9251ce1cf4361b8b0ede9fbfc4023
I20240503 02:00:00.276583 9873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d82268925a934c559dfcf51a84f03d2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:00.276862 9864 tablet_bootstrap.cc:492] T 1fa274c98fc147f29344ed537e2ccf17 P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:00.277936 9864 tablet_bootstrap.cc:492] T 1fa274c98fc147f29344ed537e2ccf17 P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap complete.
I20240503 02:00:00.278311 9864 ts_tablet_manager.cc:1367] T 1fa274c98fc147f29344ed537e2ccf17 P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent bootstrapping tablet: real 0.060s user 0.028s sys 0.000s
I20240503 02:00:00.278860 9864 raft_consensus.cc:348] T 1fa274c98fc147f29344ed537e2ccf17 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } }
I20240503 02:00:00.279021 9864 raft_consensus.cc:729] T 1fa274c98fc147f29344ed537e2ccf17 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e1a9251ce1cf4361b8b0ede9fbfc4023, State: Initialized, Role: LEARNER
I20240503 02:00:00.279129 9864 consensus_queue.cc:260] T 1fa274c98fc147f29344ed537e2ccf17 P e1a9251ce1cf4361b8b0ede9fbfc4023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } }
I20240503 02:00:00.279404 9864 ts_tablet_manager.cc:1398] T 1fa274c98fc147f29344ed537e2ccf17 P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:00.279953 9873 tablet_copy_client.cc:783] T d82268925a934c559dfcf51a84f03d2e P 99ead02915954b4fab23b033ad6d546b: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:00.280002 6871 tablet_copy_service.cc:342] P 492448570cc94ed1879feb2ec3b9051d: Request end of tablet copy session e1a9251ce1cf4361b8b0ede9fbfc4023-1fa274c98fc147f29344ed537e2ccf17 received from {username='slave'} at 127.0.99.136:48421
I20240503 02:00:00.280107 6871 tablet_copy_service.cc:433] P 492448570cc94ed1879feb2ec3b9051d: ending tablet copy session e1a9251ce1cf4361b8b0ede9fbfc4023-1fa274c98fc147f29344ed537e2ccf17 on tablet 1fa274c98fc147f29344ed537e2ccf17 with peer e1a9251ce1cf4361b8b0ede9fbfc4023
I20240503 02:00:00.280316 9873 tablet_copy_client.cc:647] T d82268925a934c559dfcf51a84f03d2e P 99ead02915954b4fab23b033ad6d546b: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:00.283175 9472 tablet.cc:1722] T 9a9e4df681694aa6994b0020acd466f5 P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:00.283566 9472 tablet.cc:1620] T 9a9e4df681694aa6994b0020acd466f5 P 80fd18ef4d604c86b909d5c008a46b4f: MemRowSet was empty: no flush needed.
I20240503 02:00:00.284426 9889 tablet_copy_client.cc:783] T 7a44b6d65bfa43e291b06b0dfd38fab3 P 9942306855404cfd891adf227dd65eb2: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:00.284672 9889 tablet_copy_client.cc:647] T 7a44b6d65bfa43e291b06b0dfd38fab3 P 9942306855404cfd891adf227dd65eb2: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:00.285208 6307 catalog_manager.cc:5576] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863) does not have the latest schema for tablet 9a9e4df681694aa6994b0020acd466f5 (table rebalancer_dynamic_table [id=6aaa25ca43754a899cf8589522ff71da]). Expected version 1 got 0
I20240503 02:00:00.286520 9895 ts_tablet_manager.cc:902] T 90ef3584830240acb0d8aad14ae851a6 P 2a01ace80baf4057b6bf1995cf88468b: Initiating tablet copy from peer 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:00.287228 9895 tablet_copy_client.cc:308] T 90ef3584830240acb0d8aad14ae851a6 P 2a01ace80baf4057b6bf1995cf88468b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:42719
I20240503 02:00:00.286655 9894 ts_tablet_manager.cc:902] T 1e445a3cc928429caf9e3c6193a63010 P 2a01ace80baf4057b6bf1995cf88468b: Initiating tablet copy from peer 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:00.287652 6739 tablet_copy_service.cc:140] P c2ac2eb7cc264047b9ab32fe3a4fc025: Received BeginTabletCopySession request for tablet 1feaf29d8c6b4842829ce66e0f4ec23b from peer 2a01ace80baf4057b6bf1995cf88468b ({username='slave'} at 127.0.99.135:52621)
I20240503 02:00:00.287762 6739 tablet_copy_service.cc:161] P c2ac2eb7cc264047b9ab32fe3a4fc025: Beginning new tablet copy session on tablet 1feaf29d8c6b4842829ce66e0f4ec23b from peer 2a01ace80baf4057b6bf1995cf88468b at {username='slave'} at 127.0.99.135:52621: session id = 2a01ace80baf4057b6bf1995cf88468b-1feaf29d8c6b4842829ce66e0f4ec23b
I20240503 02:00:00.287940 9894 tablet_copy_client.cc:308] T 1e445a3cc928429caf9e3c6193a63010 P 2a01ace80baf4057b6bf1995cf88468b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43863
I20240503 02:00:00.288067 9873 tablet_copy_client.cc:523] T d82268925a934c559dfcf51a84f03d2e P 99ead02915954b4fab23b033ad6d546b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:00.288506 6871 tablet_copy_service.cc:140] P 492448570cc94ed1879feb2ec3b9051d: Received BeginTabletCopySession request for tablet 90ef3584830240acb0d8aad14ae851a6 from peer 2a01ace80baf4057b6bf1995cf88468b ({username='slave'} at 127.0.99.135:38375)
I20240503 02:00:00.288609 6871 tablet_copy_service.cc:161] P 492448570cc94ed1879feb2ec3b9051d: Beginning new tablet copy session on tablet 90ef3584830240acb0d8aad14ae851a6 from peer 2a01ace80baf4057b6bf1995cf88468b at {username='slave'} at 127.0.99.135:38375: session id = 2a01ace80baf4057b6bf1995cf88468b-90ef3584830240acb0d8aad14ae851a6
I20240503 02:00:00.288847 6739 tablet_copy_source_session.cc:205] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:00.289498 6871 tablet_copy_source_session.cc:205] T 90ef3584830240acb0d8aad14ae851a6 P 492448570cc94ed1879feb2ec3b9051d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:00.290023 9873 tablet_bootstrap.cc:492] T d82268925a934c559dfcf51a84f03d2e P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:00.290783 9876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1feaf29d8c6b4842829ce66e0f4ec23b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:00.294678 9904 ts_tablet_manager.cc:902] T 3cf186a0901f420390c8fda54316c5af P 99ead02915954b4fab23b033ad6d546b: Initiating tablet copy from peer c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:00.295096 9895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90ef3584830240acb0d8aad14ae851a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:00.295293 9904 tablet_copy_client.cc:308] T 3cf186a0901f420390c8fda54316c5af P 99ead02915954b4fab23b033ad6d546b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:42655
I20240503 02:00:00.295354 6609 tablet_copy_service.cc:140] P 80fd18ef4d604c86b909d5c008a46b4f: Received BeginTabletCopySession request for tablet 1e445a3cc928429caf9e3c6193a63010 from peer 2a01ace80baf4057b6bf1995cf88468b ({username='slave'} at 127.0.99.135:40117)
I20240503 02:00:00.295444 6609 tablet_copy_service.cc:161] P 80fd18ef4d604c86b909d5c008a46b4f: Beginning new tablet copy session on tablet 1e445a3cc928429caf9e3c6193a63010 from peer 2a01ace80baf4057b6bf1995cf88468b at {username='slave'} at 127.0.99.135:40117: session id = 2a01ace80baf4057b6bf1995cf88468b-1e445a3cc928429caf9e3c6193a63010
I20240503 02:00:00.296417 6609 tablet_copy_source_session.cc:205] T 1e445a3cc928429caf9e3c6193a63010 P 80fd18ef4d604c86b909d5c008a46b4f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:00.297530 9894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e445a3cc928429caf9e3c6193a63010. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:00.298208 6740 tablet_copy_service.cc:140] P c2ac2eb7cc264047b9ab32fe3a4fc025: Received BeginTabletCopySession request for tablet 3cf186a0901f420390c8fda54316c5af from peer 99ead02915954b4fab23b033ad6d546b ({username='slave'} at 127.0.99.134:48173)
I20240503 02:00:00.298314 6740 tablet_copy_service.cc:161] P c2ac2eb7cc264047b9ab32fe3a4fc025: Beginning new tablet copy session on tablet 3cf186a0901f420390c8fda54316c5af from peer 99ead02915954b4fab23b033ad6d546b at {username='slave'} at 127.0.99.134:48173: session id = 99ead02915954b4fab23b033ad6d546b-3cf186a0901f420390c8fda54316c5af
I20240503 02:00:00.299194 6740 tablet_copy_source_session.cc:205] T 3cf186a0901f420390c8fda54316c5af P c2ac2eb7cc264047b9ab32fe3a4fc025: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:00.299772 9894 tablet_copy_client.cc:783] T 1e445a3cc928429caf9e3c6193a63010 P 2a01ace80baf4057b6bf1995cf88468b: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:00.299985 9894 tablet_copy_client.cc:647] T 1e445a3cc928429caf9e3c6193a63010 P 2a01ace80baf4057b6bf1995cf88468b: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:00.297537 9895 tablet_copy_client.cc:783] T 90ef3584830240acb0d8aad14ae851a6 P 2a01ace80baf4057b6bf1995cf88468b: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:00.300195 9895 tablet_copy_client.cc:647] T 90ef3584830240acb0d8aad14ae851a6 P 2a01ace80baf4057b6bf1995cf88468b: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:00.300451 9904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cf186a0901f420390c8fda54316c5af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:00.300997 9889 tablet_copy_client.cc:523] T 7a44b6d65bfa43e291b06b0dfd38fab3 P 9942306855404cfd891adf227dd65eb2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:00.301900 9895 tablet_copy_client.cc:523] T 90ef3584830240acb0d8aad14ae851a6 P 2a01ace80baf4057b6bf1995cf88468b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:00.302443 9889 tablet_bootstrap.cc:492] T 7a44b6d65bfa43e291b06b0dfd38fab3 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:00.302732 9904 tablet_copy_client.cc:783] T 3cf186a0901f420390c8fda54316c5af P 99ead02915954b4fab23b033ad6d546b: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:00.302891 9904 tablet_copy_client.cc:647] T 3cf186a0901f420390c8fda54316c5af P 99ead02915954b4fab23b033ad6d546b: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:00.303748 9895 tablet_bootstrap.cc:492] T 90ef3584830240acb0d8aad14ae851a6 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:00.301914 9894 tablet_copy_client.cc:523] T 1e445a3cc928429caf9e3c6193a63010 P 2a01ace80baf4057b6bf1995cf88468b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:00.306685 9876 tablet_copy_client.cc:783] T 1feaf29d8c6b4842829ce66e0f4ec23b P 2a01ace80baf4057b6bf1995cf88468b: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:00.306849 9876 tablet_copy_client.cc:647] T 1feaf29d8c6b4842829ce66e0f4ec23b P 2a01ace80baf4057b6bf1995cf88468b: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:00.307315 9894 tablet_bootstrap.cc:492] T 1e445a3cc928429caf9e3c6193a63010 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:00.309047 9876 tablet_copy_client.cc:523] T 1feaf29d8c6b4842829ce66e0f4ec23b P 2a01ace80baf4057b6bf1995cf88468b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:00.310465 9876 tablet_bootstrap.cc:492] T 1feaf29d8c6b4842829ce66e0f4ec23b P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:00.310976 9904 tablet_copy_client.cc:523] T 3cf186a0901f420390c8fda54316c5af P 99ead02915954b4fab23b033ad6d546b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:00.312645 9904 tablet_bootstrap.cc:492] T 3cf186a0901f420390c8fda54316c5af P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:00.317677 9859 ts_tablet_manager.cc:902] T 915028b3478341d588ed6031e657c259 P 9942306855404cfd891adf227dd65eb2: Initiating tablet copy from peer 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:00.318142 9859 tablet_copy_client.cc:308] T 915028b3478341d588ed6031e657c259 P 9942306855404cfd891adf227dd65eb2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43863
I20240503 02:00:00.325287 6609 tablet_copy_service.cc:140] P 80fd18ef4d604c86b909d5c008a46b4f: Received BeginTabletCopySession request for tablet 915028b3478341d588ed6031e657c259 from peer 9942306855404cfd891adf227dd65eb2 ({username='slave'} at 127.0.99.133:50723)
I20240503 02:00:00.325438 6609 tablet_copy_service.cc:161] P 80fd18ef4d604c86b909d5c008a46b4f: Beginning new tablet copy session on tablet 915028b3478341d588ed6031e657c259 from peer 9942306855404cfd891adf227dd65eb2 at {username='slave'} at 127.0.99.133:50723: session id = 9942306855404cfd891adf227dd65eb2-915028b3478341d588ed6031e657c259
I20240503 02:00:00.326423 6609 tablet_copy_source_session.cc:205] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:00.327454 9859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 915028b3478341d588ed6031e657c259. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:00.330003 9859 tablet_copy_client.cc:783] T 915028b3478341d588ed6031e657c259 P 9942306855404cfd891adf227dd65eb2: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:00.330181 9859 tablet_copy_client.cc:647] T 915028b3478341d588ed6031e657c259 P 9942306855404cfd891adf227dd65eb2: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:00.340162 9889 tablet_bootstrap.cc:492] T 7a44b6d65bfa43e291b06b0dfd38fab3 P 9942306855404cfd891adf227dd65eb2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:00.340667 9889 tablet_bootstrap.cc:492] T 7a44b6d65bfa43e291b06b0dfd38fab3 P 9942306855404cfd891adf227dd65eb2: Bootstrap complete.
I20240503 02:00:00.340982 9889 ts_tablet_manager.cc:1367] T 7a44b6d65bfa43e291b06b0dfd38fab3 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.039s user 0.027s sys 0.003s
I20240503 02:00:00.341559 9889 raft_consensus.cc:348] T 7a44b6d65bfa43e291b06b0dfd38fab3 P 9942306855404cfd891adf227dd65eb2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } }
I20240503 02:00:00.341876 9889 raft_consensus.cc:729] T 7a44b6d65bfa43e291b06b0dfd38fab3 P 9942306855404cfd891adf227dd65eb2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: LEARNER
I20240503 02:00:00.342042 9889 consensus_queue.cc:260] T 7a44b6d65bfa43e291b06b0dfd38fab3 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } }
I20240503 02:00:00.342458 9889 ts_tablet_manager.cc:1398] T 7a44b6d65bfa43e291b06b0dfd38fab3 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:00.343336 6739 tablet_copy_service.cc:342] P c2ac2eb7cc264047b9ab32fe3a4fc025: Request end of tablet copy session 9942306855404cfd891adf227dd65eb2-7a44b6d65bfa43e291b06b0dfd38fab3 received from {username='slave'} at 127.0.99.133:46077
I20240503 02:00:00.343438 6739 tablet_copy_service.cc:433] P c2ac2eb7cc264047b9ab32fe3a4fc025: ending tablet copy session 9942306855404cfd891adf227dd65eb2-7a44b6d65bfa43e291b06b0dfd38fab3 on tablet 7a44b6d65bfa43e291b06b0dfd38fab3 with peer 9942306855404cfd891adf227dd65eb2
I20240503 02:00:00.349823 9859 tablet_copy_client.cc:523] T 915028b3478341d588ed6031e657c259 P 9942306855404cfd891adf227dd65eb2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:00.350147 9533 consensus_queue.cc:1035] T 9c8a53980b00421d8c2bd3ae73f0c8cc P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:00.351393 9859 tablet_bootstrap.cc:492] T 915028b3478341d588ed6031e657c259 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:00.352599 9873 tablet_bootstrap.cc:492] T d82268925a934c559dfcf51a84f03d2e P 99ead02915954b4fab23b033ad6d546b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:00.352933 9873 tablet_bootstrap.cc:492] T d82268925a934c559dfcf51a84f03d2e P 99ead02915954b4fab23b033ad6d546b: Bootstrap complete.
I20240503 02:00:00.353139 9873 ts_tablet_manager.cc:1367] T d82268925a934c559dfcf51a84f03d2e P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.063s user 0.028s sys 0.004s
I20240503 02:00:00.358863 9904 tablet_bootstrap.cc:492] T 3cf186a0901f420390c8fda54316c5af P 99ead02915954b4fab23b033ad6d546b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:00.359287 9904 tablet_bootstrap.cc:492] T 3cf186a0901f420390c8fda54316c5af P 99ead02915954b4fab23b033ad6d546b: Bootstrap complete.
I20240503 02:00:00.359515 9904 ts_tablet_manager.cc:1367] T 3cf186a0901f420390c8fda54316c5af P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.047s user 0.022s sys 0.004s
I20240503 02:00:00.360018 9904 raft_consensus.cc:348] T 3cf186a0901f420390c8fda54316c5af P 99ead02915954b4fab23b033ad6d546b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } }
I20240503 02:00:00.360235 9904 raft_consensus.cc:729] T 3cf186a0901f420390c8fda54316c5af P 99ead02915954b4fab23b033ad6d546b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: LEARNER
I20240503 02:00:00.360419 9904 consensus_queue.cc:260] T 3cf186a0901f420390c8fda54316c5af P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } }
I20240503 02:00:00.360889 9904 ts_tablet_manager.cc:1398] T 3cf186a0901f420390c8fda54316c5af P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:00.362988 6739 tablet_copy_service.cc:342] P c2ac2eb7cc264047b9ab32fe3a4fc025: Request end of tablet copy session 99ead02915954b4fab23b033ad6d546b-3cf186a0901f420390c8fda54316c5af received from {username='slave'} at 127.0.99.134:48173
I20240503 02:00:00.363111 6739 tablet_copy_service.cc:433] P c2ac2eb7cc264047b9ab32fe3a4fc025: ending tablet copy session 99ead02915954b4fab23b033ad6d546b-3cf186a0901f420390c8fda54316c5af on tablet 3cf186a0901f420390c8fda54316c5af with peer 99ead02915954b4fab23b033ad6d546b
I20240503 02:00:00.365038 9758 consensus_queue.cc:1035] T 9c8a53980b00421d8c2bd3ae73f0c8cc P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:00.365037 9873 raft_consensus.cc:348] T d82268925a934c559dfcf51a84f03d2e P 99ead02915954b4fab23b033ad6d546b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } }
I20240503 02:00:00.365186 9873 raft_consensus.cc:729] T d82268925a934c559dfcf51a84f03d2e P 99ead02915954b4fab23b033ad6d546b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: LEARNER
I20240503 02:00:00.365314 9873 consensus_queue.cc:260] T d82268925a934c559dfcf51a84f03d2e P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } }
I20240503 02:00:00.365589 9873 ts_tablet_manager.cc:1398] T d82268925a934c559dfcf51a84f03d2e P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.012s user 0.000s sys 0.000s
I20240503 02:00:00.368481 6739 tablet_copy_service.cc:342] P c2ac2eb7cc264047b9ab32fe3a4fc025: Request end of tablet copy session 99ead02915954b4fab23b033ad6d546b-d82268925a934c559dfcf51a84f03d2e received from {username='slave'} at 127.0.99.134:48173
I20240503 02:00:00.368603 6739 tablet_copy_service.cc:433] P c2ac2eb7cc264047b9ab32fe3a4fc025: ending tablet copy session 99ead02915954b4fab23b033ad6d546b-d82268925a934c559dfcf51a84f03d2e on tablet d82268925a934c559dfcf51a84f03d2e with peer 99ead02915954b4fab23b033ad6d546b
I20240503 02:00:00.371357 9721 consensus_queue.cc:1035] T 0099654461ca41efa4323fc9f5adab4a P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:00.373883 9876 tablet_bootstrap.cc:492] T 1feaf29d8c6b4842829ce66e0f4ec23b P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:00.376127 9876 tablet_bootstrap.cc:492] T 1feaf29d8c6b4842829ce66e0f4ec23b P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap complete.
I20240503 02:00:00.377200 6307 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:00.380815 8633 consensus_queue.cc:1035] T 0099654461ca41efa4323fc9f5adab4a P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:00.385811 6458 consensus_queue.cc:237] T 0b659e9d57db4ebf93c9143938970090 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } }
I20240503 02:00:00.387598 6582 raft_consensus.cc:1261] T 0b659e9d57db4ebf93c9143938970090 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:00:00.388063 6392 consensus_peers.cc:485] T 0b659e9d57db4ebf93c9143938970090 P dec3e4a04d604f7eaaa3f077d081218a -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0b659e9d57db4ebf93c9143938970090. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:00.388067 8679 consensus_queue.cc:1035] T 0b659e9d57db4ebf93c9143938970090 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:00.390450 6720 raft_consensus.cc:1261] T 0b659e9d57db4ebf93c9143938970090 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:00.390925 8679 consensus_queue.cc:1035] T 0b659e9d57db4ebf93c9143938970090 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:00.392217 9721 raft_consensus.cc:2932] T 0b659e9d57db4ebf93c9143938970090 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } } }
I20240503 02:00:00.392923 9873 ts_tablet_manager.cc:902] T 485d1d89a8b94505969d663be92124cd P 99ead02915954b4fab23b033ad6d546b: Initiating tablet copy from peer c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:00.392760 6587 raft_consensus.cc:2932] T 0b659e9d57db4ebf93c9143938970090 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } } }
I20240503 02:00:00.393381 9873 tablet_copy_client.cc:308] T 485d1d89a8b94505969d663be92124cd P 99ead02915954b4fab23b033ad6d546b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:42655
I20240503 02:00:00.393720 9477 tablet.cc:1722] T 9a9e4df681694aa6994b0020acd466f5 P e1a9251ce1cf4361b8b0ede9fbfc4023: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:00.394084 9477 tablet.cc:1620] T 9a9e4df681694aa6994b0020acd466f5 P e1a9251ce1cf4361b8b0ede9fbfc4023: MemRowSet was empty: no flush needed.
I20240503 02:00:00.394217 9884 tablet.cc:1722] T 9a9e4df681694aa6994b0020acd466f5 P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:00.394517 9884 tablet.cc:1620] T 9a9e4df681694aa6994b0020acd466f5 P 492448570cc94ed1879feb2ec3b9051d: MemRowSet was empty: no flush needed.
I20240503 02:00:00.395537 6739 tablet_copy_service.cc:140] P c2ac2eb7cc264047b9ab32fe3a4fc025: Received BeginTabletCopySession request for tablet 485d1d89a8b94505969d663be92124cd from peer 99ead02915954b4fab23b033ad6d546b ({username='slave'} at 127.0.99.134:48173)
I20240503 02:00:00.395664 6739 tablet_copy_service.cc:161] P c2ac2eb7cc264047b9ab32fe3a4fc025: Beginning new tablet copy session on tablet 485d1d89a8b94505969d663be92124cd from peer 99ead02915954b4fab23b033ad6d546b at {username='slave'} at 127.0.99.134:48173: session id = 99ead02915954b4fab23b033ad6d546b-485d1d89a8b94505969d663be92124cd
I20240503 02:00:00.395762 9876 ts_tablet_manager.cc:1367] T 1feaf29d8c6b4842829ce66e0f4ec23b P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.085s user 0.030s sys 0.001s
I20240503 02:00:00.396301 9876 raft_consensus.cc:348] T 1feaf29d8c6b4842829ce66e0f4ec23b P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } }
I20240503 02:00:00.396497 9876 raft_consensus.cc:729] T 1feaf29d8c6b4842829ce66e0f4ec23b P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: LEARNER
I20240503 02:00:00.396613 9876 consensus_queue.cc:260] T 1feaf29d8c6b4842829ce66e0f4ec23b P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } }
I20240503 02:00:00.396725 6739 tablet_copy_source_session.cc:205] T 485d1d89a8b94505969d663be92124cd P c2ac2eb7cc264047b9ab32fe3a4fc025: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:00.397730 9904 ts_tablet_manager.cc:902] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 99ead02915954b4fab23b033ad6d546b: Initiating tablet copy from peer 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:00.397936 6307 catalog_manager.cc:5484] T 0b659e9d57db4ebf93c9143938970090 P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: config changed from index -1 to 4, NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:00.398309 9904 tablet_copy_client.cc:308] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 99ead02915954b4fab23b033ad6d546b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43863
I20240503 02:00:00.399677 9873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 485d1d89a8b94505969d663be92124cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:00.400882 6315 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240503 02:00:00.401006 6315 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240503 02:00:00.401999 9873 tablet_copy_client.cc:783] T 485d1d89a8b94505969d663be92124cd P 99ead02915954b4fab23b033ad6d546b: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:00.402151 9873 tablet_copy_client.cc:647] T 485d1d89a8b94505969d663be92124cd P 99ead02915954b4fab23b033ad6d546b: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:00.407971 6315 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet f14c8008ab914d2091e6cc3f712cb1f0
I20240503 02:00:00.408314 6315 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet 6fc09cd6e45f449fa15e486d018751ae
I20240503 02:00:00.408730 9472 tablet.cc:1722] T 9a9e4df681694aa6994b0020acd466f5 P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:00.409054 9472 tablet.cc:1620] T 9a9e4df681694aa6994b0020acd466f5 P 80fd18ef4d604c86b909d5c008a46b4f: MemRowSet was empty: no flush needed.
I20240503 02:00:00.409063 6568 tablet_service.cc:1514] Processing DeleteTablet for tablet f14c8008ab914d2091e6cc3f712cb1f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:00 UTC) from {username='slave'} at 127.0.0.1:59846
I20240503 02:00:00.409267 9473 tablet_replica.cc:330] T f14c8008ab914d2091e6cc3f712cb1f0 P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 02:00:00.409417 9473 raft_consensus.cc:2229] T f14c8008ab914d2091e6cc3f712cb1f0 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:00.409564 9473 raft_consensus.cc:2258] T f14c8008ab914d2091e6cc3f712cb1f0 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:00.409644 6829 tablet_service.cc:1514] Processing DeleteTablet for tablet f14c8008ab914d2091e6cc3f712cb1f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:00 UTC) from {username='slave'} at 127.0.0.1:52812
I20240503 02:00:00.409762 6315 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet a9795c5271c548b9867bcdaeed3414e5
I20240503 02:00:00.410063 9476 tablet_replica.cc:330] T f14c8008ab914d2091e6cc3f712cb1f0 P 492448570cc94ed1879feb2ec3b9051d: stopping tablet replica
I20240503 02:00:00.410064 9473 ts_tablet_manager.cc:1875] T f14c8008ab914d2091e6cc3f712cb1f0 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:00.410198 9476 raft_consensus.cc:2229] T f14c8008ab914d2091e6cc3f712cb1f0 P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:00.410391 9476 raft_consensus.cc:2258] T f14c8008ab914d2091e6cc3f712cb1f0 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:00.410868 9476 ts_tablet_manager.cc:1875] T f14c8008ab914d2091e6cc3f712cb1f0 P 492448570cc94ed1879feb2ec3b9051d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:00.411975 6700 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fc09cd6e45f449fa15e486d018751ae with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:00 UTC) from {username='slave'} at 127.0.0.1:40336
I20240503 02:00:00.412156 9756 tablet_replica.cc:330] T 6fc09cd6e45f449fa15e486d018751ae P c2ac2eb7cc264047b9ab32fe3a4fc025: stopping tablet replica
I20240503 02:00:00.412122 6438 tablet_service.cc:1514] Processing DeleteTablet for tablet f14c8008ab914d2091e6cc3f712cb1f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:00 UTC) from {username='slave'} at 127.0.0.1:51338
I20240503 02:00:00.412274 9756 raft_consensus.cc:2229] T 6fc09cd6e45f449fa15e486d018751ae P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:00.412281 6436 tablet_service.cc:1514] Processing DeleteTablet for tablet a9795c5271c548b9867bcdaeed3414e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:00 UTC) from {username='slave'} at 127.0.0.1:51338
I20240503 02:00:00.412495 9756 raft_consensus.cc:2258] T 6fc09cd6e45f449fa15e486d018751ae P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:00.412988 9756 ts_tablet_manager.cc:1875] T 6fc09cd6e45f449fa15e486d018751ae P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:00.413933 9476 ts_tablet_manager.cc:1888] T f14c8008ab914d2091e6cc3f712cb1f0 P 492448570cc94ed1879feb2ec3b9051d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:00:00.414005 9476 log.cc:1196] T f14c8008ab914d2091e6cc3f712cb1f0 P 492448570cc94ed1879feb2ec3b9051d: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/f14c8008ab914d2091e6cc3f712cb1f0
I20240503 02:00:00.414242 9476 ts_tablet_manager.cc:1909] T f14c8008ab914d2091e6cc3f712cb1f0 P 492448570cc94ed1879feb2ec3b9051d: Deleting consensus metadata
I20240503 02:00:00.414959 9474 tablet_replica.cc:330] T f14c8008ab914d2091e6cc3f712cb1f0 P dec3e4a04d604f7eaaa3f077d081218a: stopping tablet replica
I20240503 02:00:00.415036 9756 ts_tablet_manager.cc:1888] T 6fc09cd6e45f449fa15e486d018751ae P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:00:00.415094 9756 log.cc:1196] T 6fc09cd6e45f449fa15e486d018751ae P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/6fc09cd6e45f449fa15e486d018751ae
I20240503 02:00:00.415119 9474 raft_consensus.cc:2229] T f14c8008ab914d2091e6cc3f712cb1f0 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:00.415266 9474 raft_consensus.cc:2258] T f14c8008ab914d2091e6cc3f712cb1f0 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:00.415328 9756 ts_tablet_manager.cc:1909] T 6fc09cd6e45f449fa15e486d018751ae P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting consensus metadata
I20240503 02:00:00.415793 9474 ts_tablet_manager.cc:1875] T f14c8008ab914d2091e6cc3f712cb1f0 P dec3e4a04d604f7eaaa3f077d081218a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:00.415974 6303 catalog_manager.cc:4830] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719): tablet f14c8008ab914d2091e6cc3f712cb1f0 (table rebalance_test_table_6 [id=65ed012903ed4deabf9e7b0dfe803452]) successfully deleted
I20240503 02:00:00.416164 6829 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fc09cd6e45f449fa15e486d018751ae with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:00 UTC) from {username='slave'} at 127.0.0.1:52812
I20240503 02:00:00.416185 6830 tablet_service.cc:1514] Processing DeleteTablet for tablet a9795c5271c548b9867bcdaeed3414e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:00 UTC) from {username='slave'} at 127.0.0.1:52812
I20240503 02:00:00.416343 9476 tablet_replica.cc:330] T 6fc09cd6e45f449fa15e486d018751ae P 492448570cc94ed1879feb2ec3b9051d: stopping tablet replica
I20240503 02:00:00.416589 9476 raft_consensus.cc:2229] T 6fc09cd6e45f449fa15e486d018751ae P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:00.416767 9476 raft_consensus.cc:2258] T 6fc09cd6e45f449fa15e486d018751ae P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:00.417070 6568 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fc09cd6e45f449fa15e486d018751ae with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:00 UTC) from {username='slave'} at 127.0.0.1:59846
I20240503 02:00:00.417120 6567 tablet_service.cc:1514] Processing DeleteTablet for tablet a9795c5271c548b9867bcdaeed3414e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:00 UTC) from {username='slave'} at 127.0.0.1:59846
I20240503 02:00:00.417256 9476 ts_tablet_manager.cc:1875] T 6fc09cd6e45f449fa15e486d018751ae P 492448570cc94ed1879feb2ec3b9051d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:00.418481 9474 ts_tablet_manager.cc:1888] T f14c8008ab914d2091e6cc3f712cb1f0 P dec3e4a04d604f7eaaa3f077d081218a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:00:00.418574 9474 log.cc:1196] T f14c8008ab914d2091e6cc3f712cb1f0 P dec3e4a04d604f7eaaa3f077d081218a: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/f14c8008ab914d2091e6cc3f712cb1f0
I20240503 02:00:00.418886 9474 ts_tablet_manager.cc:1909] T f14c8008ab914d2091e6cc3f712cb1f0 P dec3e4a04d604f7eaaa3f077d081218a: Deleting consensus metadata
I20240503 02:00:00.419574 9474 tablet_replica.cc:330] T a9795c5271c548b9867bcdaeed3414e5 P dec3e4a04d604f7eaaa3f077d081218a: stopping tablet replica
I20240503 02:00:00.420004 6300 catalog_manager.cc:4830] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): tablet f14c8008ab914d2091e6cc3f712cb1f0 (table rebalance_test_table_6 [id=65ed012903ed4deabf9e7b0dfe803452]) successfully deleted
I20240503 02:00:00.420053 9476 ts_tablet_manager.cc:1888] T 6fc09cd6e45f449fa15e486d018751ae P 492448570cc94ed1879feb2ec3b9051d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:00:00.420167 9476 log.cc:1196] T 6fc09cd6e45f449fa15e486d018751ae P 492448570cc94ed1879feb2ec3b9051d: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/6fc09cd6e45f449fa15e486d018751ae
I20240503 02:00:00.420643 9476 ts_tablet_manager.cc:1909] T 6fc09cd6e45f449fa15e486d018751ae P 492448570cc94ed1879feb2ec3b9051d: Deleting consensus metadata
I20240503 02:00:00.421480 9476 tablet_replica.cc:330] T a9795c5271c548b9867bcdaeed3414e5 P 492448570cc94ed1879feb2ec3b9051d: stopping tablet replica
I20240503 02:00:00.421622 6609 tablet_copy_service.cc:140] P 80fd18ef4d604c86b909d5c008a46b4f: Received BeginTabletCopySession request for tablet 74ebf5d0fb9a42e6a7ec743a98960e13 from peer 99ead02915954b4fab23b033ad6d546b ({username='slave'} at 127.0.99.134:40601)
I20240503 02:00:00.421721 6609 tablet_copy_service.cc:161] P 80fd18ef4d604c86b909d5c008a46b4f: Beginning new tablet copy session on tablet 74ebf5d0fb9a42e6a7ec743a98960e13 from peer 99ead02915954b4fab23b033ad6d546b at {username='slave'} at 127.0.99.134:40601: session id = 99ead02915954b4fab23b033ad6d546b-74ebf5d0fb9a42e6a7ec743a98960e13
I20240503 02:00:00.421803 9474 raft_consensus.cc:2229] T a9795c5271c548b9867bcdaeed3414e5 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:00.422008 9474 raft_consensus.cc:2258] T a9795c5271c548b9867bcdaeed3414e5 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:00.422103 9473 ts_tablet_manager.cc:1888] T f14c8008ab914d2091e6cc3f712cb1f0 P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:00:00.422164 9473 log.cc:1196] T f14c8008ab914d2091e6cc3f712cb1f0 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/f14c8008ab914d2091e6cc3f712cb1f0
I20240503 02:00:00.422379 6609 tablet_copy_source_session.cc:205] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 80fd18ef4d604c86b909d5c008a46b4f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:00.422410 9473 ts_tablet_manager.cc:1909] T f14c8008ab914d2091e6cc3f712cb1f0 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting consensus metadata
I20240503 02:00:00.422506 9474 ts_tablet_manager.cc:1875] T a9795c5271c548b9867bcdaeed3414e5 P dec3e4a04d604f7eaaa3f077d081218a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:00.422863 9476 raft_consensus.cc:2229] T a9795c5271c548b9867bcdaeed3414e5 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:00.423056 9476 raft_consensus.cc:2258] T a9795c5271c548b9867bcdaeed3414e5 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:00.423127 6303 catalog_manager.cc:4830] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719): tablet 6fc09cd6e45f449fa15e486d018751ae (table rebalance_test_table_6 [id=65ed012903ed4deabf9e7b0dfe803452]) successfully deleted
I20240503 02:00:00.423292 9904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74ebf5d0fb9a42e6a7ec743a98960e13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:00.423934 6303 catalog_manager.cc:4830] TS c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655): tablet 6fc09cd6e45f449fa15e486d018751ae (table rebalance_test_table_6 [id=65ed012903ed4deabf9e7b0dfe803452]) successfully deleted
I20240503 02:00:00.424121 9476 ts_tablet_manager.cc:1875] T a9795c5271c548b9867bcdaeed3414e5 P 492448570cc94ed1879feb2ec3b9051d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:00.424970 6313 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=6aaa25ca43754a899cf8589522ff71da] alter complete (version 2)
I20240503 02:00:00.424584 9473 tablet_replica.cc:330] T 6fc09cd6e45f449fa15e486d018751ae P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 02:00:00.425158 9473 raft_consensus.cc:2229] T 6fc09cd6e45f449fa15e486d018751ae P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:00.425346 9473 raft_consensus.cc:2258] T 6fc09cd6e45f449fa15e486d018751ae P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:00.425478 9904 tablet_copy_client.cc:783] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 99ead02915954b4fab23b033ad6d546b: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:00.425627 9904 tablet_copy_client.cc:647] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 99ead02915954b4fab23b033ad6d546b: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:00.425841 9473 ts_tablet_manager.cc:1875] T 6fc09cd6e45f449fa15e486d018751ae P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:00.425993 9474 ts_tablet_manager.cc:1888] T a9795c5271c548b9867bcdaeed3414e5 P dec3e4a04d604f7eaaa3f077d081218a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:00:00.426059 9474 log.cc:1196] T a9795c5271c548b9867bcdaeed3414e5 P dec3e4a04d604f7eaaa3f077d081218a: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/a9795c5271c548b9867bcdaeed3414e5
I20240503 02:00:00.426292 9474 ts_tablet_manager.cc:1909] T a9795c5271c548b9867bcdaeed3414e5 P dec3e4a04d604f7eaaa3f077d081218a: Deleting consensus metadata
I20240503 02:00:00.427094 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet f14c8008ab914d2091e6cc3f712cb1f0 (table rebalance_test_table_6 [id=65ed012903ed4deabf9e7b0dfe803452]) successfully deleted
I20240503 02:00:00.427459 9476 ts_tablet_manager.cc:1888] T a9795c5271c548b9867bcdaeed3414e5 P 492448570cc94ed1879feb2ec3b9051d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:00:00.427592 9476 log.cc:1196] T a9795c5271c548b9867bcdaeed3414e5 P 492448570cc94ed1879feb2ec3b9051d: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/a9795c5271c548b9867bcdaeed3414e5
I20240503 02:00:00.427971 9476 ts_tablet_manager.cc:1909] T a9795c5271c548b9867bcdaeed3414e5 P 492448570cc94ed1879feb2ec3b9051d: Deleting consensus metadata
I20240503 02:00:00.428769 6303 catalog_manager.cc:4830] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719): tablet a9795c5271c548b9867bcdaeed3414e5 (table rebalance_test_table_6 [id=65ed012903ed4deabf9e7b0dfe803452]) successfully deleted
I20240503 02:00:00.429716 9904 tablet_copy_client.cc:523] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 99ead02915954b4fab23b033ad6d546b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:00.430246 9473 ts_tablet_manager.cc:1888] T 6fc09cd6e45f449fa15e486d018751ae P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:00:00.430332 9473 log.cc:1196] T 6fc09cd6e45f449fa15e486d018751ae P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/6fc09cd6e45f449fa15e486d018751ae
I20240503 02:00:00.430744 9473 ts_tablet_manager.cc:1909] T 6fc09cd6e45f449fa15e486d018751ae P 80fd18ef4d604c86b909d5c008a46b4f: Deleting consensus metadata
I20240503 02:00:00.430881 9904 tablet_bootstrap.cc:492] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:00.431401 9473 tablet_replica.cc:330] T a9795c5271c548b9867bcdaeed3414e5 P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 02:00:00.431638 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet 6fc09cd6e45f449fa15e486d018751ae (table rebalance_test_table_6 [id=65ed012903ed4deabf9e7b0dfe803452]) successfully deleted
I20240503 02:00:00.431672 9473 raft_consensus.cc:2229] T a9795c5271c548b9867bcdaeed3414e5 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:00.431810 9473 raft_consensus.cc:2258] T a9795c5271c548b9867bcdaeed3414e5 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:00.432304 9473 ts_tablet_manager.cc:1875] T a9795c5271c548b9867bcdaeed3414e5 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:00.437309 6300 catalog_manager.cc:4830] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): tablet a9795c5271c548b9867bcdaeed3414e5 (table rebalance_test_table_6 [id=65ed012903ed4deabf9e7b0dfe803452]) successfully deleted
I20240503 02:00:00.438251 9876 ts_tablet_manager.cc:1398] T 1feaf29d8c6b4842829ce66e0f4ec23b P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.042s user 0.000s sys 0.001s
I20240503 02:00:00.440060 9473 ts_tablet_manager.cc:1888] T a9795c5271c548b9867bcdaeed3414e5 P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:00:00.440155 9473 log.cc:1196] T a9795c5271c548b9867bcdaeed3414e5 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/a9795c5271c548b9867bcdaeed3414e5
I20240503 02:00:00.441617 9873 tablet_copy_client.cc:523] T 485d1d89a8b94505969d663be92124cd P 99ead02915954b4fab23b033ad6d546b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:00.442919 6739 tablet_copy_service.cc:342] P c2ac2eb7cc264047b9ab32fe3a4fc025: Request end of tablet copy session 2a01ace80baf4057b6bf1995cf88468b-1feaf29d8c6b4842829ce66e0f4ec23b received from {username='slave'} at 127.0.99.135:52621
I20240503 02:00:00.442999 6739 tablet_copy_service.cc:433] P c2ac2eb7cc264047b9ab32fe3a4fc025: ending tablet copy session 2a01ace80baf4057b6bf1995cf88468b-1feaf29d8c6b4842829ce66e0f4ec23b on tablet 1feaf29d8c6b4842829ce66e0f4ec23b with peer 2a01ace80baf4057b6bf1995cf88468b
I20240503 02:00:00.443169 9873 tablet_bootstrap.cc:492] T 485d1d89a8b94505969d663be92124cd P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:00.443361 9473 ts_tablet_manager.cc:1909] T a9795c5271c548b9867bcdaeed3414e5 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting consensus metadata
I20240503 02:00:00.444108 9953 ts_tablet_manager.cc:902] T 8a76d3e2d4264687aa2972c179341b5e P 2a01ace80baf4057b6bf1995cf88468b: Initiating tablet copy from peer dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 02:00:00.444465 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet a9795c5271c548b9867bcdaeed3414e5 (table rebalance_test_table_6 [id=65ed012903ed4deabf9e7b0dfe803452]) successfully deleted
I20240503 02:00:00.445272 9953 tablet_copy_client.cc:308] T 8a76d3e2d4264687aa2972c179341b5e P 2a01ace80baf4057b6bf1995cf88468b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:33093
I20240503 02:00:00.445255 6720 raft_consensus.cc:2932] T 0b659e9d57db4ebf93c9143938970090 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } } }
I20240503 02:00:00.445757 9864 ts_tablet_manager.cc:902] T 46ad540deb624a25b97d3b967fe1f6f4 P e1a9251ce1cf4361b8b0ede9fbfc4023: Initiating tablet copy from peer dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 02:00:00.446156 9864 tablet_copy_client.cc:308] T 46ad540deb624a25b97d3b967fe1f6f4 P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:33093
I20240503 02:00:00.452618 6478 tablet_copy_service.cc:140] P dec3e4a04d604f7eaaa3f077d081218a: Received BeginTabletCopySession request for tablet 46ad540deb624a25b97d3b967fe1f6f4 from peer e1a9251ce1cf4361b8b0ede9fbfc4023 ({username='slave'} at 127.0.99.136:60209)
I20240503 02:00:00.452778 6478 tablet_copy_service.cc:161] P dec3e4a04d604f7eaaa3f077d081218a: Beginning new tablet copy session on tablet 46ad540deb624a25b97d3b967fe1f6f4 from peer e1a9251ce1cf4361b8b0ede9fbfc4023 at {username='slave'} at 127.0.99.136:60209: session id = e1a9251ce1cf4361b8b0ede9fbfc4023-46ad540deb624a25b97d3b967fe1f6f4
I20240503 02:00:00.454303 6478 tablet_copy_source_session.cc:205] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:00.455668 9864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46ad540deb624a25b97d3b967fe1f6f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:00.457979 9864 tablet_copy_client.cc:783] T 46ad540deb624a25b97d3b967fe1f6f4 P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:00.458137 9864 tablet_copy_client.cc:647] T 46ad540deb624a25b97d3b967fe1f6f4 P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:00.459707 9895 tablet_bootstrap.cc:492] T 90ef3584830240acb0d8aad14ae851a6 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:00.460109 9895 tablet_bootstrap.cc:492] T 90ef3584830240acb0d8aad14ae851a6 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap complete.
I20240503 02:00:00.460500 9895 ts_tablet_manager.cc:1367] T 90ef3584830240acb0d8aad14ae851a6 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.157s user 0.027s sys 0.004s
I20240503 02:00:00.461056 9895 raft_consensus.cc:348] T 90ef3584830240acb0d8aad14ae851a6 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } }
I20240503 02:00:00.461262 9895 raft_consensus.cc:729] T 90ef3584830240acb0d8aad14ae851a6 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: LEARNER
I20240503 02:00:00.461418 9895 consensus_queue.cc:260] T 90ef3584830240acb0d8aad14ae851a6 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } }
I20240503 02:00:00.461542 9864 tablet_copy_client.cc:523] T 46ad540deb624a25b97d3b967fe1f6f4 P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:00.462165 6478 tablet_copy_service.cc:140] P dec3e4a04d604f7eaaa3f077d081218a: Received BeginTabletCopySession request for tablet 8a76d3e2d4264687aa2972c179341b5e from peer 2a01ace80baf4057b6bf1995cf88468b ({username='slave'} at 127.0.99.135:56449)
I20240503 02:00:00.462289 6478 tablet_copy_service.cc:161] P dec3e4a04d604f7eaaa3f077d081218a: Beginning new tablet copy session on tablet 8a76d3e2d4264687aa2972c179341b5e from peer 2a01ace80baf4057b6bf1995cf88468b at {username='slave'} at 127.0.99.135:56449: session id = 2a01ace80baf4057b6bf1995cf88468b-8a76d3e2d4264687aa2972c179341b5e
I20240503 02:00:00.463196 9864 tablet_bootstrap.cc:492] T 46ad540deb624a25b97d3b967fe1f6f4 P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap starting.
I20240503 02:00:00.463197 6478 tablet_copy_source_session.cc:205] T 8a76d3e2d4264687aa2972c179341b5e P dec3e4a04d604f7eaaa3f077d081218a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:00.464707 9904 tablet_bootstrap.cc:492] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 99ead02915954b4fab23b033ad6d546b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:00.465023 9904 tablet_bootstrap.cc:492] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 99ead02915954b4fab23b033ad6d546b: Bootstrap complete.
I20240503 02:00:00.465226 9904 ts_tablet_manager.cc:1367] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.034s user 0.027s sys 0.000s
I20240503 02:00:00.465757 9904 raft_consensus.cc:348] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 99ead02915954b4fab23b033ad6d546b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } }
I20240503 02:00:00.465860 9859 tablet_bootstrap.cc:492] T 915028b3478341d588ed6031e657c259 P 9942306855404cfd891adf227dd65eb2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:00.465926 9904 raft_consensus.cc:729] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 99ead02915954b4fab23b033ad6d546b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: LEARNER
I20240503 02:00:00.466104 9859 tablet_bootstrap.cc:492] T 915028b3478341d588ed6031e657c259 P 9942306855404cfd891adf227dd65eb2: Bootstrap complete.
I20240503 02:00:00.466071 9904 consensus_queue.cc:260] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } }
I20240503 02:00:00.466374 9904 ts_tablet_manager.cc:1398] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:00.466414 9859 ts_tablet_manager.cc:1367] T 915028b3478341d588ed6031e657c259 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.115s user 0.025s sys 0.003s
I20240503 02:00:00.466924 9859 raft_consensus.cc:348] T 915028b3478341d588ed6031e657c259 P 9942306855404cfd891adf227dd65eb2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } }
I20240503 02:00:00.467242 6609 tablet_copy_service.cc:342] P 80fd18ef4d604c86b909d5c008a46b4f: Request end of tablet copy session 99ead02915954b4fab23b033ad6d546b-74ebf5d0fb9a42e6a7ec743a98960e13 received from {username='slave'} at 127.0.99.134:40601
I20240503 02:00:00.467336 6609 tablet_copy_service.cc:433] P 80fd18ef4d604c86b909d5c008a46b4f: ending tablet copy session 99ead02915954b4fab23b033ad6d546b-74ebf5d0fb9a42e6a7ec743a98960e13 on tablet 74ebf5d0fb9a42e6a7ec743a98960e13 with peer 99ead02915954b4fab23b033ad6d546b
I20240503 02:00:00.467159 9859 raft_consensus.cc:729] T 915028b3478341d588ed6031e657c259 P 9942306855404cfd891adf227dd65eb2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: LEARNER
I20240503 02:00:00.468317 9859 consensus_queue.cc:260] T 915028b3478341d588ed6031e657c259 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } }
I20240503 02:00:00.468796 9895 ts_tablet_manager.cc:1398] T 90ef3584830240acb0d8aad14ae851a6 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240503 02:00:00.469450 6871 tablet_copy_service.cc:342] P 492448570cc94ed1879feb2ec3b9051d: Request end of tablet copy session 2a01ace80baf4057b6bf1995cf88468b-90ef3584830240acb0d8aad14ae851a6 received from {username='slave'} at 127.0.99.135:38375
I20240503 02:00:00.469530 6871 tablet_copy_service.cc:433] P 492448570cc94ed1879feb2ec3b9051d: ending tablet copy session 2a01ace80baf4057b6bf1995cf88468b-90ef3584830240acb0d8aad14ae851a6 on tablet 90ef3584830240acb0d8aad14ae851a6 with peer 2a01ace80baf4057b6bf1995cf88468b
I20240503 02:00:00.464085 9953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a76d3e2d4264687aa2972c179341b5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:00.465656 9894 tablet_bootstrap.cc:492] T 1e445a3cc928429caf9e3c6193a63010 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:00.471374 9889 ts_tablet_manager.cc:902] T 2b1d700d09534fa3adcf51bb3c97c8f5 P 9942306855404cfd891adf227dd65eb2: Initiating tablet copy from peer dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 02:00:00.471518 9894 tablet_bootstrap.cc:492] T 1e445a3cc928429caf9e3c6193a63010 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap complete.
I20240503 02:00:00.471771 9894 ts_tablet_manager.cc:1367] T 1e445a3cc928429caf9e3c6193a63010 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.165s user 0.027s sys 0.005s
I20240503 02:00:00.471895 9889 tablet_copy_client.cc:308] T 2b1d700d09534fa3adcf51bb3c97c8f5 P 9942306855404cfd891adf227dd65eb2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:33093
I20240503 02:00:00.472476 6478 tablet_copy_service.cc:140] P dec3e4a04d604f7eaaa3f077d081218a: Received BeginTabletCopySession request for tablet 2b1d700d09534fa3adcf51bb3c97c8f5 from peer 9942306855404cfd891adf227dd65eb2 ({username='slave'} at 127.0.99.133:57637)
I20240503 02:00:00.472585 6478 tablet_copy_service.cc:161] P dec3e4a04d604f7eaaa3f077d081218a: Beginning new tablet copy session on tablet 2b1d700d09534fa3adcf51bb3c97c8f5 from peer 9942306855404cfd891adf227dd65eb2 at {username='slave'} at 127.0.99.133:57637: session id = 9942306855404cfd891adf227dd65eb2-2b1d700d09534fa3adcf51bb3c97c8f5
I20240503 02:00:00.472874 9894 raft_consensus.cc:348] T 1e445a3cc928429caf9e3c6193a63010 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } }
I20240503 02:00:00.473070 9894 raft_consensus.cc:729] T 1e445a3cc928429caf9e3c6193a63010 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: LEARNER
I20240503 02:00:00.473230 9894 consensus_queue.cc:260] T 1e445a3cc928429caf9e3c6193a63010 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } }
I20240503 02:00:00.473516 6478 tablet_copy_source_session.cc:205] T 2b1d700d09534fa3adcf51bb3c97c8f5 P dec3e4a04d604f7eaaa3f077d081218a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:00.473726 9894 ts_tablet_manager.cc:1398] T 1e445a3cc928429caf9e3c6193a63010 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:00.474272 6609 tablet_copy_service.cc:342] P 80fd18ef4d604c86b909d5c008a46b4f: Request end of tablet copy session 2a01ace80baf4057b6bf1995cf88468b-1e445a3cc928429caf9e3c6193a63010 received from {username='slave'} at 127.0.99.135:40117
I20240503 02:00:00.474359 6609 tablet_copy_service.cc:433] P 80fd18ef4d604c86b909d5c008a46b4f: ending tablet copy session 2a01ace80baf4057b6bf1995cf88468b-1e445a3cc928429caf9e3c6193a63010 on tablet 1e445a3cc928429caf9e3c6193a63010 with peer 2a01ace80baf4057b6bf1995cf88468b
I20240503 02:00:00.475118 9953 tablet_copy_client.cc:783] T 8a76d3e2d4264687aa2972c179341b5e P 2a01ace80baf4057b6bf1995cf88468b: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:00.475340 9953 tablet_copy_client.cc:647] T 8a76d3e2d4264687aa2972c179341b5e P 2a01ace80baf4057b6bf1995cf88468b: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:00.476500 9889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b1d700d09534fa3adcf51bb3c97c8f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:00.478832 9889 tablet_copy_client.cc:783] T 2b1d700d09534fa3adcf51bb3c97c8f5 P 9942306855404cfd891adf227dd65eb2: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:00.479667 9889 tablet_copy_client.cc:647] T 2b1d700d09534fa3adcf51bb3c97c8f5 P 9942306855404cfd891adf227dd65eb2: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:00.480888 9859 ts_tablet_manager.cc:1398] T 915028b3478341d588ed6031e657c259 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.014s user 0.001s sys 0.000s
I20240503 02:00:00.481812 9953 tablet_copy_client.cc:523] T 8a76d3e2d4264687aa2972c179341b5e P 2a01ace80baf4057b6bf1995cf88468b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:00.481818 6609 tablet_copy_service.cc:342] P 80fd18ef4d604c86b909d5c008a46b4f: Request end of tablet copy session 9942306855404cfd891adf227dd65eb2-915028b3478341d588ed6031e657c259 received from {username='slave'} at 127.0.99.133:50723
I20240503 02:00:00.481904 6609 tablet_copy_service.cc:433] P 80fd18ef4d604c86b909d5c008a46b4f: ending tablet copy session 9942306855404cfd891adf227dd65eb2-915028b3478341d588ed6031e657c259 on tablet 915028b3478341d588ed6031e657c259 with peer 9942306855404cfd891adf227dd65eb2
I20240503 02:00:00.482416 6587 consensus_queue.cc:237] T 7a4a910a8b9d4134adebb52a73dfae1d P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } }
I20240503 02:00:00.484786 9889 tablet_copy_client.cc:523] T 2b1d700d09534fa3adcf51bb3c97c8f5 P 9942306855404cfd891adf227dd65eb2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:00.485092 6849 raft_consensus.cc:1261] T 7a4a910a8b9d4134adebb52a73dfae1d P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:00.485543 9953 tablet_bootstrap.cc:492] T 8a76d3e2d4264687aa2972c179341b5e P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:00.485705 9873 tablet_bootstrap.cc:492] T 485d1d89a8b94505969d663be92124cd P 99ead02915954b4fab23b033ad6d546b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:00.485954 9873 tablet_bootstrap.cc:492] T 485d1d89a8b94505969d663be92124cd P 99ead02915954b4fab23b033ad6d546b: Bootstrap complete.
I20240503 02:00:00.486217 9889 tablet_bootstrap.cc:492] T 2b1d700d09534fa3adcf51bb3c97c8f5 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:00.486223 9873 ts_tablet_manager.cc:1367] T 485d1d89a8b94505969d663be92124cd P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.043s user 0.022s sys 0.005s
W20240503 02:00:00.486258 6523 consensus_peers.cc:485] T 7a4a910a8b9d4134adebb52a73dfae1d P 80fd18ef4d604c86b909d5c008a46b4f -> Peer 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157): Couldn't send request to peer 99ead02915954b4fab23b033ad6d546b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7a4a910a8b9d4134adebb52a73dfae1d. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:00.486482 6458 raft_consensus.cc:1261] T 7a4a910a8b9d4134adebb52a73dfae1d P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:00.486709 9873 raft_consensus.cc:348] T 485d1d89a8b94505969d663be92124cd P 99ead02915954b4fab23b033ad6d546b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } }
I20240503 02:00:00.486840 9873 raft_consensus.cc:729] T 485d1d89a8b94505969d663be92124cd P 99ead02915954b4fab23b033ad6d546b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: LEARNER
I20240503 02:00:00.486968 9873 consensus_queue.cc:260] T 485d1d89a8b94505969d663be92124cd P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } }
I20240503 02:00:00.487396 9873 ts_tablet_manager.cc:1398] T 485d1d89a8b94505969d663be92124cd P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:00.487618 9484 consensus_queue.cc:1035] T 7a4a910a8b9d4134adebb52a73dfae1d P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:00.488222 6739 tablet_copy_service.cc:342] P c2ac2eb7cc264047b9ab32fe3a4fc025: Request end of tablet copy session 99ead02915954b4fab23b033ad6d546b-485d1d89a8b94505969d663be92124cd received from {username='slave'} at 127.0.99.134:48173
I20240503 02:00:00.488297 6739 tablet_copy_service.cc:433] P c2ac2eb7cc264047b9ab32fe3a4fc025: ending tablet copy session 99ead02915954b4fab23b033ad6d546b-485d1d89a8b94505969d663be92124cd on tablet 485d1d89a8b94505969d663be92124cd with peer 99ead02915954b4fab23b033ad6d546b
I20240503 02:00:00.489301 9484 consensus_queue.cc:1035] T 7a4a910a8b9d4134adebb52a73dfae1d P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:00.491569 9484 raft_consensus.cc:2932] T 7a4a910a8b9d4134adebb52a73dfae1d P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } } }
I20240503 02:00:00.494742 6315 catalog_manager.cc:5484] T 7a4a910a8b9d4134adebb52a73dfae1d P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: config changed from index -1 to 4, NON_VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:00.497820 6458 raft_consensus.cc:2932] T 7a4a910a8b9d4134adebb52a73dfae1d P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } } }
I20240503 02:00:00.506260 6851 raft_consensus.cc:2932] T 7a4a910a8b9d4134adebb52a73dfae1d P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } } }
I20240503 02:00:00.520820 9889 tablet_bootstrap.cc:492] T 2b1d700d09534fa3adcf51bb3c97c8f5 P 9942306855404cfd891adf227dd65eb2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:00.521211 9889 tablet_bootstrap.cc:492] T 2b1d700d09534fa3adcf51bb3c97c8f5 P 9942306855404cfd891adf227dd65eb2: Bootstrap complete.
I20240503 02:00:00.521458 9889 ts_tablet_manager.cc:1367] T 2b1d700d09534fa3adcf51bb3c97c8f5 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.035s user 0.023s sys 0.002s
I20240503 02:00:00.521988 9889 raft_consensus.cc:348] T 2b1d700d09534fa3adcf51bb3c97c8f5 P 9942306855404cfd891adf227dd65eb2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } }
I20240503 02:00:00.522190 9889 raft_consensus.cc:729] T 2b1d700d09534fa3adcf51bb3c97c8f5 P 9942306855404cfd891adf227dd65eb2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: LEARNER
I20240503 02:00:00.522336 9889 consensus_queue.cc:260] T 2b1d700d09534fa3adcf51bb3c97c8f5 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } }
I20240503 02:00:00.523344 9889 ts_tablet_manager.cc:1398] T 2b1d700d09534fa3adcf51bb3c97c8f5 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:00.523952 6478 tablet_copy_service.cc:342] P dec3e4a04d604f7eaaa3f077d081218a: Request end of tablet copy session 9942306855404cfd891adf227dd65eb2-2b1d700d09534fa3adcf51bb3c97c8f5 received from {username='slave'} at 127.0.99.133:57637
I20240503 02:00:00.524058 6478 tablet_copy_service.cc:433] P dec3e4a04d604f7eaaa3f077d081218a: ending tablet copy session 9942306855404cfd891adf227dd65eb2-2b1d700d09534fa3adcf51bb3c97c8f5 on tablet 2b1d700d09534fa3adcf51bb3c97c8f5 with peer 9942306855404cfd891adf227dd65eb2
I20240503 02:00:00.524775 6720 consensus_queue.cc:237] T 7f637097ad06465cb93c6ff6d97c1d17 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } }
I20240503 02:00:00.526540 6587 raft_consensus.cc:1261] T 7f637097ad06465cb93c6ff6d97c1d17 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:00.526955 8696 consensus_queue.cc:1035] T 7f637097ad06465cb93c6ff6d97c1d17 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:00.527264 6458 raft_consensus.cc:1261] T 7f637097ad06465cb93c6ff6d97c1d17 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:00:00.527691 6653 consensus_peers.cc:485] T 7f637097ad06465cb93c6ff6d97c1d17 P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079): Couldn't send request to peer 2a01ace80baf4057b6bf1995cf88468b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7f637097ad06465cb93c6ff6d97c1d17. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:00.527841 8696 consensus_queue.cc:1035] T 7f637097ad06465cb93c6ff6d97c1d17 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:00.528191 9953 tablet_bootstrap.cc:492] T 8a76d3e2d4264687aa2972c179341b5e P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:00.528522 9953 tablet_bootstrap.cc:492] T 8a76d3e2d4264687aa2972c179341b5e P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap complete.
I20240503 02:00:00.528745 9953 ts_tablet_manager.cc:1367] T 8a76d3e2d4264687aa2972c179341b5e P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.043s user 0.012s sys 0.012s
I20240503 02:00:00.529295 9953 raft_consensus.cc:348] T 8a76d3e2d4264687aa2972c179341b5e P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } }
I20240503 02:00:00.529438 9953 raft_consensus.cc:729] T 8a76d3e2d4264687aa2972c179341b5e P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: LEARNER
I20240503 02:00:00.529469 6313 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:00.529557 9953 consensus_queue.cc:260] T 8a76d3e2d4264687aa2972c179341b5e P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } }
I20240503 02:00:00.530445 9953 ts_tablet_manager.cc:1398] T 8a76d3e2d4264687aa2972c179341b5e P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:00:00.531049 9984 raft_consensus.cc:2932] T 7f637097ad06465cb93c6ff6d97c1d17 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } } }
I20240503 02:00:00.532253 9864 tablet_bootstrap.cc:492] T 46ad540deb624a25b97d3b967fe1f6f4 P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:00.532364 9853 ts_tablet_manager.cc:902] T 548f43b381f1445c8d3d55705ae5a48a P e1a9251ce1cf4361b8b0ede9fbfc4023: Initiating tablet copy from peer 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:00.532629 9864 tablet_bootstrap.cc:492] T 46ad540deb624a25b97d3b967fe1f6f4 P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap complete.
I20240503 02:00:00.532801 9864 ts_tablet_manager.cc:1367] T 46ad540deb624a25b97d3b967fe1f6f4 P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent bootstrapping tablet: real 0.070s user 0.028s sys 0.000s
I20240503 02:00:00.533011 9853 tablet_copy_client.cc:308] T 548f43b381f1445c8d3d55705ae5a48a P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:42719
I20240503 02:00:00.533313 9864 raft_consensus.cc:348] T 46ad540deb624a25b97d3b967fe1f6f4 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } }
I20240503 02:00:00.533499 9864 raft_consensus.cc:729] T 46ad540deb624a25b97d3b967fe1f6f4 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e1a9251ce1cf4361b8b0ede9fbfc4023, State: Initialized, Role: LEARNER
I20240503 02:00:00.533476 6478 tablet_copy_service.cc:342] P dec3e4a04d604f7eaaa3f077d081218a: Request end of tablet copy session 2a01ace80baf4057b6bf1995cf88468b-8a76d3e2d4264687aa2972c179341b5e received from {username='slave'} at 127.0.99.135:56449
I20240503 02:00:00.533552 6478 tablet_copy_service.cc:433] P dec3e4a04d604f7eaaa3f077d081218a: ending tablet copy session 2a01ace80baf4057b6bf1995cf88468b-8a76d3e2d4264687aa2972c179341b5e on tablet 8a76d3e2d4264687aa2972c179341b5e with peer 2a01ace80baf4057b6bf1995cf88468b
I20240503 02:00:00.533675 9864 consensus_queue.cc:260] T 46ad540deb624a25b97d3b967fe1f6f4 P e1a9251ce1cf4361b8b0ede9fbfc4023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } }
I20240503 02:00:00.534572 9864 ts_tablet_manager.cc:1398] T 46ad540deb624a25b97d3b967fe1f6f4 P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:00.534703 6871 tablet_copy_service.cc:140] P 492448570cc94ed1879feb2ec3b9051d: Received BeginTabletCopySession request for tablet 548f43b381f1445c8d3d55705ae5a48a from peer e1a9251ce1cf4361b8b0ede9fbfc4023 ({username='slave'} at 127.0.99.136:48421)
I20240503 02:00:00.534813 6871 tablet_copy_service.cc:161] P 492448570cc94ed1879feb2ec3b9051d: Beginning new tablet copy session on tablet 548f43b381f1445c8d3d55705ae5a48a from peer e1a9251ce1cf4361b8b0ede9fbfc4023 at {username='slave'} at 127.0.99.136:48421: session id = e1a9251ce1cf4361b8b0ede9fbfc4023-548f43b381f1445c8d3d55705ae5a48a
I20240503 02:00:00.535069 6313 catalog_manager.cc:5484] T 7f637097ad06465cb93c6ff6d97c1d17 P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: config changed from index -1 to 4, NON_VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:00.535753 6871 tablet_copy_source_session.cc:205] T 548f43b381f1445c8d3d55705ae5a48a P 492448570cc94ed1879feb2ec3b9051d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:00.537230 6589 raft_consensus.cc:2932] T 7f637097ad06465cb93c6ff6d97c1d17 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } } }
I20240503 02:00:00.540452 9884 tablet.cc:1722] T 9a9e4df681694aa6994b0020acd466f5 P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:00.540513 6478 tablet_copy_service.cc:342] P dec3e4a04d604f7eaaa3f077d081218a: Request end of tablet copy session e1a9251ce1cf4361b8b0ede9fbfc4023-46ad540deb624a25b97d3b967fe1f6f4 received from {username='slave'} at 127.0.99.136:60209
I20240503 02:00:00.540606 6478 tablet_copy_service.cc:433] P dec3e4a04d604f7eaaa3f077d081218a: ending tablet copy session e1a9251ce1cf4361b8b0ede9fbfc4023-46ad540deb624a25b97d3b967fe1f6f4 on tablet 46ad540deb624a25b97d3b967fe1f6f4 with peer e1a9251ce1cf4361b8b0ede9fbfc4023
I20240503 02:00:00.540664 9853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 548f43b381f1445c8d3d55705ae5a48a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:00.543789 9853 tablet_copy_client.cc:783] T 548f43b381f1445c8d3d55705ae5a48a P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:00.544098 9853 tablet_copy_client.cc:647] T 548f43b381f1445c8d3d55705ae5a48a P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:00.545224 6458 raft_consensus.cc:2932] T 7f637097ad06465cb93c6ff6d97c1d17 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } } }
I20240503 02:00:00.545498 9477 tablet.cc:1722] T 9a9e4df681694aa6994b0020acd466f5 P e1a9251ce1cf4361b8b0ede9fbfc4023: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:00.548482 9853 tablet_copy_client.cc:523] T 548f43b381f1445c8d3d55705ae5a48a P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:00.549657 9472 tablet.cc:1722] T 9a9e4df681694aa6994b0020acd466f5 P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:00.549957 9853 tablet_bootstrap.cc:492] T 548f43b381f1445c8d3d55705ae5a48a P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap starting.
I20240503 02:00:00.552868 6309 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=6aaa25ca43754a899cf8589522ff71da] alter complete (version 3)
I20240503 02:00:00.557149 6315 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_extra_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:00.566344 9158 tablet_service.cc:1467] Processing CreateTablet for tablet 51b82ef8e4cc474387f6902f2e526ac0 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=c1969b96b61041869ddd81db8c84bf44]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:00.566679 9158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51b82ef8e4cc474387f6902f2e526ac0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:00.566833 9285 tablet_service.cc:1467] Processing CreateTablet for tablet 51b82ef8e4cc474387f6902f2e526ac0 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=c1969b96b61041869ddd81db8c84bf44]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:00.567091 9285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51b82ef8e4cc474387f6902f2e526ac0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:00.569923 9025 tablet_service.cc:1467] Processing CreateTablet for tablet 51b82ef8e4cc474387f6902f2e526ac0 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=c1969b96b61041869ddd81db8c84bf44]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:00.570215 9025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51b82ef8e4cc474387f6902f2e526ac0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:00.570336 9286 tablet_service.cc:1467] Processing CreateTablet for tablet f28586e70bd9473fbdc6b9126b666fa9 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=c1969b96b61041869ddd81db8c84bf44]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:00.570588 9286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f28586e70bd9473fbdc6b9126b666fa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:00.573652 9027 tablet_service.cc:1467] Processing CreateTablet for tablet 6a82dcc9be5b4c93b5fff03f789350ab (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=c1969b96b61041869ddd81db8c84bf44]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:00.573938 9027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a82dcc9be5b4c93b5fff03f789350ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:00.575322 9853 tablet_bootstrap.cc:492] T 548f43b381f1445c8d3d55705ae5a48a P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:00.575570 9853 tablet_bootstrap.cc:492] T 548f43b381f1445c8d3d55705ae5a48a P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap complete.
I20240503 02:00:00.575675 9285 tablet_service.cc:1467] Processing CreateTablet for tablet 6a82dcc9be5b4c93b5fff03f789350ab (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=c1969b96b61041869ddd81db8c84bf44]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:00.575790 9853 ts_tablet_manager.cc:1367] T 548f43b381f1445c8d3d55705ae5a48a P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent bootstrapping tablet: real 0.026s user 0.024s sys 0.001s
I20240503 02:00:00.575915 9285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a82dcc9be5b4c93b5fff03f789350ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:00.576220 9853 raft_consensus.cc:348] T 548f43b381f1445c8d3d55705ae5a48a P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } }
I20240503 02:00:00.576324 9853 raft_consensus.cc:729] T 548f43b381f1445c8d3d55705ae5a48a P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e1a9251ce1cf4361b8b0ede9fbfc4023, State: Initialized, Role: LEARNER
I20240503 02:00:00.576434 9853 consensus_queue.cc:260] T 548f43b381f1445c8d3d55705ae5a48a P e1a9251ce1cf4361b8b0ede9fbfc4023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } }
I20240503 02:00:00.576918 6830 tablet_service.cc:1467] Processing CreateTablet for tablet 6a82dcc9be5b4c93b5fff03f789350ab (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=c1969b96b61041869ddd81db8c84bf44]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:00.577188 6830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a82dcc9be5b4c93b5fff03f789350ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:00.577416 9853 ts_tablet_manager.cc:1398] T 548f43b381f1445c8d3d55705ae5a48a P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:00.578938 8880 tablet_service.cc:1467] Processing CreateTablet for tablet f28586e70bd9473fbdc6b9126b666fa9 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=c1969b96b61041869ddd81db8c84bf44]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:00.579236 8880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f28586e70bd9473fbdc6b9126b666fa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:00.580577 10003 tablet_bootstrap.cc:492] T 51b82ef8e4cc474387f6902f2e526ac0 P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap starting.
I20240503 02:00:00.580682 6871 tablet_copy_service.cc:342] P 492448570cc94ed1879feb2ec3b9051d: Request end of tablet copy session e1a9251ce1cf4361b8b0ede9fbfc4023-548f43b381f1445c8d3d55705ae5a48a received from {username='slave'} at 127.0.99.136:48421
I20240503 02:00:00.580766 6871 tablet_copy_service.cc:433] P 492448570cc94ed1879feb2ec3b9051d: ending tablet copy session e1a9251ce1cf4361b8b0ede9fbfc4023-548f43b381f1445c8d3d55705ae5a48a on tablet 548f43b381f1445c8d3d55705ae5a48a with peer e1a9251ce1cf4361b8b0ede9fbfc4023
I20240503 02:00:00.581791 10003 tablet_bootstrap.cc:654] T 51b82ef8e4cc474387f6902f2e526ac0 P e1a9251ce1cf4361b8b0ede9fbfc4023: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:00.583640 10003 tablet_bootstrap.cc:492] T 51b82ef8e4cc474387f6902f2e526ac0 P e1a9251ce1cf4361b8b0ede9fbfc4023: No bootstrap required, opened a new log
I20240503 02:00:00.583819 10003 ts_tablet_manager.cc:1367] T 51b82ef8e4cc474387f6902f2e526ac0 P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:00.584568 10007 tablet_bootstrap.cc:492] T f28586e70bd9473fbdc6b9126b666fa9 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:00.584483 10003 raft_consensus.cc:348] T 51b82ef8e4cc474387f6902f2e526ac0 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:00.584877 10003 raft_consensus.cc:374] T 51b82ef8e4cc474387f6902f2e526ac0 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:00.584957 10003 raft_consensus.cc:729] T 51b82ef8e4cc474387f6902f2e526ac0 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1a9251ce1cf4361b8b0ede9fbfc4023, State: Initialized, Role: FOLLOWER
I20240503 02:00:00.585186 10003 consensus_queue.cc:260] T 51b82ef8e4cc474387f6902f2e526ac0 P e1a9251ce1cf4361b8b0ede9fbfc4023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:00.586997 9309 raft_consensus.cc:1203] T 1fa274c98fc147f29344ed537e2ccf17 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:00:00.588140 10007 tablet_bootstrap.cc:654] T f28586e70bd9473fbdc6b9126b666fa9 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:00.588173 9158 tablet_service.cc:1467] Processing CreateTablet for tablet f28586e70bd9473fbdc6b9126b666fa9 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=c1969b96b61041869ddd81db8c84bf44]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:00.588594 9158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f28586e70bd9473fbdc6b9126b666fa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:00.589563 10007 tablet_bootstrap.cc:492] T f28586e70bd9473fbdc6b9126b666fa9 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:00.589663 10007 ts_tablet_manager.cc:1367] T f28586e70bd9473fbdc6b9126b666fa9 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:00:00.590162 10007 raft_consensus.cc:348] T f28586e70bd9473fbdc6b9126b666fa9 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:00.590298 10007 raft_consensus.cc:374] T f28586e70bd9473fbdc6b9126b666fa9 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:00.590333 10007 raft_consensus.cc:729] T f28586e70bd9473fbdc6b9126b666fa9 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:00.590456 10007 consensus_queue.cc:260] T f28586e70bd9473fbdc6b9126b666fa9 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:00.590728 10007 ts_tablet_manager.cc:1398] T f28586e70bd9473fbdc6b9126b666fa9 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:00.592059 10004 tablet_bootstrap.cc:492] T 51b82ef8e4cc474387f6902f2e526ac0 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:00.593372 10003 ts_tablet_manager.cc:1398] T 51b82ef8e4cc474387f6902f2e526ac0 P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20240503 02:00:00.593820 10006 tablet_bootstrap.cc:492] T 6a82dcc9be5b4c93b5fff03f789350ab P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:00.594305 10003 tablet_bootstrap.cc:492] T f28586e70bd9473fbdc6b9126b666fa9 P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap starting.
I20240503 02:00:00.594789 10006 tablet_bootstrap.cc:654] T 6a82dcc9be5b4c93b5fff03f789350ab P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:00.596685 10003 tablet_bootstrap.cc:654] T f28586e70bd9473fbdc6b9126b666fa9 P e1a9251ce1cf4361b8b0ede9fbfc4023: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:00.597237 10002 tablet_bootstrap.cc:492] T 51b82ef8e4cc474387f6902f2e526ac0 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:00.599562 10003 tablet_bootstrap.cc:492] T f28586e70bd9473fbdc6b9126b666fa9 P e1a9251ce1cf4361b8b0ede9fbfc4023: No bootstrap required, opened a new log
I20240503 02:00:00.599778 10002 tablet_bootstrap.cc:654] T 51b82ef8e4cc474387f6902f2e526ac0 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:00.600080 10003 ts_tablet_manager.cc:1367] T f28586e70bd9473fbdc6b9126b666fa9 P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240503 02:00:00.599112 10004 tablet_bootstrap.cc:654] T 51b82ef8e4cc474387f6902f2e526ac0 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:00.600908 10003 raft_consensus.cc:348] T f28586e70bd9473fbdc6b9126b666fa9 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:00.601034 10003 raft_consensus.cc:374] T f28586e70bd9473fbdc6b9126b666fa9 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:00.601094 10003 raft_consensus.cc:729] T f28586e70bd9473fbdc6b9126b666fa9 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1a9251ce1cf4361b8b0ede9fbfc4023, State: Initialized, Role: FOLLOWER
I20240503 02:00:00.601285 10003 consensus_queue.cc:260] T f28586e70bd9473fbdc6b9126b666fa9 P e1a9251ce1cf4361b8b0ede9fbfc4023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:00.601796 10002 tablet_bootstrap.cc:492] T 51b82ef8e4cc474387f6902f2e526ac0 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:00.601843 10004 tablet_bootstrap.cc:492] T 51b82ef8e4cc474387f6902f2e526ac0 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:00.601935 10002 ts_tablet_manager.cc:1367] T 51b82ef8e4cc474387f6902f2e526ac0 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:00:00.601958 10004 ts_tablet_manager.cc:1367] T 51b82ef8e4cc474387f6902f2e526ac0 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20240503 02:00:00.602375 10004 raft_consensus.cc:348] T 51b82ef8e4cc474387f6902f2e526ac0 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:00.602496 10004 raft_consensus.cc:374] T 51b82ef8e4cc474387f6902f2e526ac0 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:00.602536 10004 raft_consensus.cc:729] T 51b82ef8e4cc474387f6902f2e526ac0 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:00.602475 10002 raft_consensus.cc:348] T 51b82ef8e4cc474387f6902f2e526ac0 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:00.602622 10002 raft_consensus.cc:374] T 51b82ef8e4cc474387f6902f2e526ac0 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:00.602679 10002 raft_consensus.cc:729] T 51b82ef8e4cc474387f6902f2e526ac0 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:00.602684 10004 consensus_queue.cc:260] T 51b82ef8e4cc474387f6902f2e526ac0 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:00.602855 10002 consensus_queue.cc:260] T 51b82ef8e4cc474387f6902f2e526ac0 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:00.603291 10002 ts_tablet_manager.cc:1398] T 51b82ef8e4cc474387f6902f2e526ac0 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:00.603767 10004 ts_tablet_manager.cc:1398] T 51b82ef8e4cc474387f6902f2e526ac0 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:00.603969 10004 tablet_bootstrap.cc:492] T 6a82dcc9be5b4c93b5fff03f789350ab P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:00.604079 10002 tablet_bootstrap.cc:492] T f28586e70bd9473fbdc6b9126b666fa9 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:00.605181 10002 tablet_bootstrap.cc:654] T f28586e70bd9473fbdc6b9126b666fa9 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:00.605906 10004 tablet_bootstrap.cc:654] T 6a82dcc9be5b4c93b5fff03f789350ab P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:00.606452 10002 tablet_bootstrap.cc:492] T f28586e70bd9473fbdc6b9126b666fa9 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:00.606551 10002 ts_tablet_manager.cc:1367] T f28586e70bd9473fbdc6b9126b666fa9 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:00.606575 10003 ts_tablet_manager.cc:1398] T f28586e70bd9473fbdc6b9126b666fa9 P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240503 02:00:00.606835 10003 tablet_bootstrap.cc:492] T 6a82dcc9be5b4c93b5fff03f789350ab P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap starting.
I20240503 02:00:00.607023 10002 raft_consensus.cc:348] T f28586e70bd9473fbdc6b9126b666fa9 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:00.607175 10002 raft_consensus.cc:374] T f28586e70bd9473fbdc6b9126b666fa9 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:00.607216 10002 raft_consensus.cc:729] T f28586e70bd9473fbdc6b9126b666fa9 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:00.607321 10002 consensus_queue.cc:260] T f28586e70bd9473fbdc6b9126b666fa9 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:00.607591 10002 ts_tablet_manager.cc:1398] T f28586e70bd9473fbdc6b9126b666fa9 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:00.607764 10003 tablet_bootstrap.cc:654] T 6a82dcc9be5b4c93b5fff03f789350ab P e1a9251ce1cf4361b8b0ede9fbfc4023: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:00.608943 10006 tablet_bootstrap.cc:492] T 6a82dcc9be5b4c93b5fff03f789350ab P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:00.609046 10006 ts_tablet_manager.cc:1367] T 6a82dcc9be5b4c93b5fff03f789350ab P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240503 02:00:00.609397 10003 tablet_bootstrap.cc:492] T 6a82dcc9be5b4c93b5fff03f789350ab P e1a9251ce1cf4361b8b0ede9fbfc4023: No bootstrap required, opened a new log
I20240503 02:00:00.609476 10003 ts_tablet_manager.cc:1367] T 6a82dcc9be5b4c93b5fff03f789350ab P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:00:00.609537 10006 raft_consensus.cc:348] T 6a82dcc9be5b4c93b5fff03f789350ab P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:00.609647 10006 raft_consensus.cc:374] T 6a82dcc9be5b4c93b5fff03f789350ab P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:00.609675 10006 raft_consensus.cc:729] T 6a82dcc9be5b4c93b5fff03f789350ab P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:00.609807 10006 consensus_queue.cc:260] T 6a82dcc9be5b4c93b5fff03f789350ab P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:00.609908 10003 raft_consensus.cc:348] T 6a82dcc9be5b4c93b5fff03f789350ab P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:00.609992 10003 raft_consensus.cc:374] T 6a82dcc9be5b4c93b5fff03f789350ab P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:00.610021 10003 raft_consensus.cc:729] T 6a82dcc9be5b4c93b5fff03f789350ab P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1a9251ce1cf4361b8b0ede9fbfc4023, State: Initialized, Role: FOLLOWER
I20240503 02:00:00.610092 10006 ts_tablet_manager.cc:1398] T 6a82dcc9be5b4c93b5fff03f789350ab P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:00.610139 10003 consensus_queue.cc:260] T 6a82dcc9be5b4c93b5fff03f789350ab P e1a9251ce1cf4361b8b0ede9fbfc4023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:00.610400 10003 ts_tablet_manager.cc:1398] T 6a82dcc9be5b4c93b5fff03f789350ab P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:00.612779 10004 tablet_bootstrap.cc:492] T 6a82dcc9be5b4c93b5fff03f789350ab P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:00.612977 10004 ts_tablet_manager.cc:1367] T 6a82dcc9be5b4c93b5fff03f789350ab P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 02:00:00.613576 10004 raft_consensus.cc:348] T 6a82dcc9be5b4c93b5fff03f789350ab P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:00.613772 10004 raft_consensus.cc:374] T 6a82dcc9be5b4c93b5fff03f789350ab P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:00.613843 10004 raft_consensus.cc:729] T 6a82dcc9be5b4c93b5fff03f789350ab P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:00.614044 10004 consensus_queue.cc:260] T 6a82dcc9be5b4c93b5fff03f789350ab P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:00.614439 10004 ts_tablet_manager.cc:1398] T 6a82dcc9be5b4c93b5fff03f789350ab P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:00.620873 8917 raft_consensus.cc:1203] T ceca04391cd446ae932b3485d958237a P 9942306855404cfd891adf227dd65eb2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:00:00.627369 9929 raft_consensus.cc:482] T 6a82dcc9be5b4c93b5fff03f789350ab P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:00.627511 9929 raft_consensus.cc:504] T 6a82dcc9be5b4c93b5fff03f789350ab P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:00.628834 9929 leader_election.cc:288] T 6a82dcc9be5b4c93b5fff03f789350ab P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:00.632647 9309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a82dcc9be5b4c93b5fff03f789350ab" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" is_pre_election: true
I20240503 02:00:00.632844 9309 raft_consensus.cc:2454] T 6a82dcc9be5b4c93b5fff03f789350ab P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:00.633236 8982 leader_election.cc:302] T 6a82dcc9be5b4c93b5fff03f789350ab P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99ead02915954b4fab23b033ad6d546b, e1a9251ce1cf4361b8b0ede9fbfc4023; no voters:
I20240503 02:00:00.633510 9929 raft_consensus.cc:2781] T 6a82dcc9be5b4c93b5fff03f789350ab P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:00.633566 9929 raft_consensus.cc:482] T 6a82dcc9be5b4c93b5fff03f789350ab P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:00.633599 9929 raft_consensus.cc:3037] T 6a82dcc9be5b4c93b5fff03f789350ab P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:00.634042 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a82dcc9be5b4c93b5fff03f789350ab" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 02:00:00.634194 6851 raft_consensus.cc:2454] T 6a82dcc9be5b4c93b5fff03f789350ab P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:00.634682 9929 raft_consensus.cc:504] T 6a82dcc9be5b4c93b5fff03f789350ab P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:00.635102 9929 leader_election.cc:288] T 6a82dcc9be5b4c93b5fff03f789350ab P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Requested vote from peers e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:00.635450 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a82dcc9be5b4c93b5fff03f789350ab" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 02:00:00.635512 9309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a82dcc9be5b4c93b5fff03f789350ab" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023"
I20240503 02:00:00.635573 6849 raft_consensus.cc:3037] T 6a82dcc9be5b4c93b5fff03f789350ab P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:00.635648 9309 raft_consensus.cc:3037] T 6a82dcc9be5b4c93b5fff03f789350ab P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:00.636361 6849 raft_consensus.cc:2454] T 6a82dcc9be5b4c93b5fff03f789350ab P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:00.636521 9309 raft_consensus.cc:2454] T 6a82dcc9be5b4c93b5fff03f789350ab P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:00.636822 8984 leader_election.cc:302] T 6a82dcc9be5b4c93b5fff03f789350ab P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:00.637024 9929 raft_consensus.cc:2781] T 6a82dcc9be5b4c93b5fff03f789350ab P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:00.637145 9929 raft_consensus.cc:686] T 6a82dcc9be5b4c93b5fff03f789350ab P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Becoming Leader. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Running, Role: LEADER
I20240503 02:00:00.637540 9929 consensus_queue.cc:237] T 6a82dcc9be5b4c93b5fff03f789350ab P 99ead02915954b4fab23b033ad6d546b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:00.640041 6315 catalog_manager.cc:5484] T 6a82dcc9be5b4c93b5fff03f789350ab P 99ead02915954b4fab23b033ad6d546b reported cstate change: term changed from 0 to 1, leader changed from <none> to 99ead02915954b4fab23b033ad6d546b (127.0.99.134). New cstate: current_term: 1 leader_uuid: "99ead02915954b4fab23b033ad6d546b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:00.648408 9758 consensus_queue.cc:1035] T 067f54011a37432d99157884e70fc601 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:00.652055 9533 consensus_queue.cc:1035] T 067f54011a37432d99157884e70fc601 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:00.656216 6315 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:00.656371 6315 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:00.657135 8917 raft_consensus.cc:1203] T 7a44b6d65bfa43e291b06b0dfd38fab3 P 9942306855404cfd891adf227dd65eb2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:00:00.658532 6315 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet 9a9e4df681694aa6994b0020acd466f5
I20240503 02:00:00.659329 6830 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a9e4df681694aa6994b0020acd466f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:00 UTC) from {username='slave'} at 127.0.0.1:52812
I20240503 02:00:00.659618 9476 tablet_replica.cc:330] T 9a9e4df681694aa6994b0020acd466f5 P 492448570cc94ed1879feb2ec3b9051d: stopping tablet replica
I20240503 02:00:00.659768 9476 raft_consensus.cc:2229] T 9a9e4df681694aa6994b0020acd466f5 P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:00.659927 9476 raft_consensus.cc:2258] T 9a9e4df681694aa6994b0020acd466f5 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:00.659902 6567 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a9e4df681694aa6994b0020acd466f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:00 UTC) from {username='slave'} at 127.0.0.1:59846
I20240503 02:00:00.660079 9285 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a9e4df681694aa6994b0020acd466f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:00 UTC) from {username='slave'} at 127.0.0.1:36004
I20240503 02:00:00.660480 9473 tablet_replica.cc:330] T 9a9e4df681694aa6994b0020acd466f5 P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 02:00:00.660612 9473 raft_consensus.cc:2229] T 9a9e4df681694aa6994b0020acd466f5 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:00.660728 9473 raft_consensus.cc:2258] T 9a9e4df681694aa6994b0020acd466f5 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:00.661055 10051 tablet_replica.cc:330] T 9a9e4df681694aa6994b0020acd466f5 P e1a9251ce1cf4361b8b0ede9fbfc4023: stopping tablet replica
I20240503 02:00:00.661176 10051 raft_consensus.cc:2229] T 9a9e4df681694aa6994b0020acd466f5 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:00.661294 10051 raft_consensus.cc:2258] T 9a9e4df681694aa6994b0020acd466f5 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:00.662060 10051 ts_tablet_manager.cc:1875] T 9a9e4df681694aa6994b0020acd466f5 P e1a9251ce1cf4361b8b0ede9fbfc4023: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:00.662997 9473 ts_tablet_manager.cc:1875] T 9a9e4df681694aa6994b0020acd466f5 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:00.664203 10051 ts_tablet_manager.cc:1888] T 9a9e4df681694aa6994b0020acd466f5 P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:00.664284 10051 log.cc:1196] T 9a9e4df681694aa6994b0020acd466f5 P e1a9251ce1cf4361b8b0ede9fbfc4023: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/9a9e4df681694aa6994b0020acd466f5
I20240503 02:00:00.664705 10051 ts_tablet_manager.cc:1909] T 9a9e4df681694aa6994b0020acd466f5 P e1a9251ce1cf4361b8b0ede9fbfc4023: Deleting consensus metadata
I20240503 02:00:00.665376 9473 ts_tablet_manager.cc:1888] T 9a9e4df681694aa6994b0020acd466f5 P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:00.665566 6301 catalog_manager.cc:4830] TS e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): tablet 9a9e4df681694aa6994b0020acd466f5 (table rebalancer_dynamic_table_new_name [id=6aaa25ca43754a899cf8589522ff71da]) successfully deleted
I20240503 02:00:00.666033 9476 ts_tablet_manager.cc:1875] T 9a9e4df681694aa6994b0020acd466f5 P 492448570cc94ed1879feb2ec3b9051d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:00.667778 9476 ts_tablet_manager.cc:1888] T 9a9e4df681694aa6994b0020acd466f5 P 492448570cc94ed1879feb2ec3b9051d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:00.667876 9476 log.cc:1196] T 9a9e4df681694aa6994b0020acd466f5 P 492448570cc94ed1879feb2ec3b9051d: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/9a9e4df681694aa6994b0020acd466f5
I20240503 02:00:00.668069 9476 ts_tablet_manager.cc:1909] T 9a9e4df681694aa6994b0020acd466f5 P 492448570cc94ed1879feb2ec3b9051d: Deleting consensus metadata
I20240503 02:00:00.668709 9473 log.cc:1196] T 9a9e4df681694aa6994b0020acd466f5 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/9a9e4df681694aa6994b0020acd466f5
I20240503 02:00:00.669003 9473 ts_tablet_manager.cc:1909] T 9a9e4df681694aa6994b0020acd466f5 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting consensus metadata
I20240503 02:00:00.669139 6303 catalog_manager.cc:4830] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719): tablet 9a9e4df681694aa6994b0020acd466f5 (table rebalancer_dynamic_table_new_name [id=6aaa25ca43754a899cf8589522ff71da]) successfully deleted
I20240503 02:00:00.670203 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet 9a9e4df681694aa6994b0020acd466f5 (table rebalancer_dynamic_table_new_name [id=6aaa25ca43754a899cf8589522ff71da]) successfully deleted
I20240503 02:00:00.691051 9179 raft_consensus.cc:1203] T 1feaf29d8c6b4842829ce66e0f4ec23b P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:00:00.693336 9179 raft_consensus.cc:1203] T 90ef3584830240acb0d8aad14ae851a6 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:00:00.710078 10044 raft_consensus.cc:482] T f28586e70bd9473fbdc6b9126b666fa9 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:00.710228 10044 raft_consensus.cc:504] T f28586e70bd9473fbdc6b9126b666fa9 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:00.710693 10044 leader_election.cc:288] T f28586e70bd9473fbdc6b9126b666fa9 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:00.711302 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f28586e70bd9473fbdc6b9126b666fa9" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:00.711308 9309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f28586e70bd9473fbdc6b9126b666fa9" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" is_pre_election: true
I20240503 02:00:00.711443 9179 raft_consensus.cc:2454] T f28586e70bd9473fbdc6b9126b666fa9 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 0.
I20240503 02:00:00.711463 9309 raft_consensus.cc:2454] T f28586e70bd9473fbdc6b9126b666fa9 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 0.
I20240503 02:00:00.711851 8851 leader_election.cc:302] T f28586e70bd9473fbdc6b9126b666fa9 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9942306855404cfd891adf227dd65eb2, e1a9251ce1cf4361b8b0ede9fbfc4023; no voters:
I20240503 02:00:00.712036 10044 raft_consensus.cc:2781] T f28586e70bd9473fbdc6b9126b666fa9 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:00.712101 10044 raft_consensus.cc:482] T f28586e70bd9473fbdc6b9126b666fa9 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:00.712136 10044 raft_consensus.cc:3037] T f28586e70bd9473fbdc6b9126b666fa9 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:00.713266 10044 raft_consensus.cc:504] T f28586e70bd9473fbdc6b9126b666fa9 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:00.713651 10044 leader_election.cc:288] T f28586e70bd9473fbdc6b9126b666fa9 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 election: Requested vote from peers e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:00.713994 9309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f28586e70bd9473fbdc6b9126b666fa9" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023"
I20240503 02:00:00.714123 9309 raft_consensus.cc:3037] T f28586e70bd9473fbdc6b9126b666fa9 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:00.714282 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f28586e70bd9473fbdc6b9126b666fa9" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:00.714416 9179 raft_consensus.cc:3037] T f28586e70bd9473fbdc6b9126b666fa9 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:00.715435 9179 raft_consensus.cc:2454] T f28586e70bd9473fbdc6b9126b666fa9 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 1.
I20240503 02:00:00.715751 9309 raft_consensus.cc:2454] T f28586e70bd9473fbdc6b9126b666fa9 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 1.
I20240503 02:00:00.715888 8850 leader_election.cc:302] T f28586e70bd9473fbdc6b9126b666fa9 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:00.716135 10044 raft_consensus.cc:2781] T f28586e70bd9473fbdc6b9126b666fa9 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:00.716238 10044 raft_consensus.cc:686] T f28586e70bd9473fbdc6b9126b666fa9 P 9942306855404cfd891adf227dd65eb2 [term 1 LEADER]: Becoming Leader. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Running, Role: LEADER
I20240503 02:00:00.716418 10044 consensus_queue.cc:237] T f28586e70bd9473fbdc6b9126b666fa9 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:00.718336 6315 catalog_manager.cc:5484] T f28586e70bd9473fbdc6b9126b666fa9 P 9942306855404cfd891adf227dd65eb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9942306855404cfd891adf227dd65eb2 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "9942306855404cfd891adf227dd65eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:00.725437 9652 raft_consensus.cc:482] T 51b82ef8e4cc474387f6902f2e526ac0 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:00.725558 9652 raft_consensus.cc:504] T 51b82ef8e4cc474387f6902f2e526ac0 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:00.725947 9652 leader_election.cc:288] T 51b82ef8e4cc474387f6902f2e526ac0 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:00.726312 9309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51b82ef8e4cc474387f6902f2e526ac0" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" is_pre_election: true
I20240503 02:00:00.726378 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51b82ef8e4cc474387f6902f2e526ac0" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:00.726459 9309 raft_consensus.cc:2454] T 51b82ef8e4cc474387f6902f2e526ac0 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:00.726516 9048 raft_consensus.cc:2454] T 51b82ef8e4cc474387f6902f2e526ac0 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:00.726747 9113 leader_election.cc:302] T 51b82ef8e4cc474387f6902f2e526ac0 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, e1a9251ce1cf4361b8b0ede9fbfc4023; no voters:
I20240503 02:00:00.726903 9652 raft_consensus.cc:2781] T 51b82ef8e4cc474387f6902f2e526ac0 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:00.726959 9652 raft_consensus.cc:482] T 51b82ef8e4cc474387f6902f2e526ac0 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:00.727005 9652 raft_consensus.cc:3037] T 51b82ef8e4cc474387f6902f2e526ac0 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:00.728103 9652 raft_consensus.cc:504] T 51b82ef8e4cc474387f6902f2e526ac0 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:00.728502 9652 leader_election.cc:288] T 51b82ef8e4cc474387f6902f2e526ac0 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Requested vote from peers e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:00.728868 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51b82ef8e4cc474387f6902f2e526ac0" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:00.728992 9048 raft_consensus.cc:3037] T 51b82ef8e4cc474387f6902f2e526ac0 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:00.729014 9309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51b82ef8e4cc474387f6902f2e526ac0" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023"
I20240503 02:00:00.729117 9309 raft_consensus.cc:3037] T 51b82ef8e4cc474387f6902f2e526ac0 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:00.729944 9309 raft_consensus.cc:2454] T 51b82ef8e4cc474387f6902f2e526ac0 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:00.729982 9048 raft_consensus.cc:2454] T 51b82ef8e4cc474387f6902f2e526ac0 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:00.730315 9113 leader_election.cc:302] T 51b82ef8e4cc474387f6902f2e526ac0 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:00.730512 9652 raft_consensus.cc:2781] T 51b82ef8e4cc474387f6902f2e526ac0 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:00.730664 9652 raft_consensus.cc:686] T 51b82ef8e4cc474387f6902f2e526ac0 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Becoming Leader. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Running, Role: LEADER
I20240503 02:00:00.730824 9652 consensus_queue.cc:237] T 51b82ef8e4cc474387f6902f2e526ac0 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:00.733101 6309 catalog_manager.cc:5484] T 51b82ef8e4cc474387f6902f2e526ac0 P 2a01ace80baf4057b6bf1995cf88468b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135). New cstate: current_term: 1 leader_uuid: "2a01ace80baf4057b6bf1995cf88468b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:00.733672 9179 raft_consensus.cc:1203] T 1e445a3cc928429caf9e3c6193a63010 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:00:00.737342 9309 raft_consensus.cc:1203] T 3416b6107e1848f299ac49ccd88ae7c8 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:00:00.759342 9048 raft_consensus.cc:1203] T 3cf186a0901f420390c8fda54316c5af P 99ead02915954b4fab23b033ad6d546b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:00:00.760399 6315 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:00:00.770721 9853 ts_tablet_manager.cc:902] T 0b659e9d57db4ebf93c9143938970090 P e1a9251ce1cf4361b8b0ede9fbfc4023: Initiating tablet copy from peer dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 02:00:00.771178 9853 tablet_copy_client.cc:308] T 0b659e9d57db4ebf93c9143938970090 P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:33093
I20240503 02:00:00.771854 6478 tablet_copy_service.cc:140] P dec3e4a04d604f7eaaa3f077d081218a: Received BeginTabletCopySession request for tablet 0b659e9d57db4ebf93c9143938970090 from peer e1a9251ce1cf4361b8b0ede9fbfc4023 ({username='slave'} at 127.0.99.136:60209)
I20240503 02:00:00.771970 6478 tablet_copy_service.cc:161] P dec3e4a04d604f7eaaa3f077d081218a: Beginning new tablet copy session on tablet 0b659e9d57db4ebf93c9143938970090 from peer e1a9251ce1cf4361b8b0ede9fbfc4023 at {username='slave'} at 127.0.99.136:60209: session id = e1a9251ce1cf4361b8b0ede9fbfc4023-0b659e9d57db4ebf93c9143938970090
I20240503 02:00:00.772992 6478 tablet_copy_source_session.cc:205] T 0b659e9d57db4ebf93c9143938970090 P dec3e4a04d604f7eaaa3f077d081218a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:00.773844 6700 tablet_service.cc:1467] Processing CreateTablet for tablet e555f9ea8c7e409fbd7b0ee7d6d07277 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3295265fc41646e7942b5cd0d2d8ab20]), partition=
I20240503 02:00:00.773985 9027 tablet_service.cc:1467] Processing CreateTablet for tablet e555f9ea8c7e409fbd7b0ee7d6d07277 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3295265fc41646e7942b5cd0d2d8ab20]), partition=
I20240503 02:00:00.774014 9853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b659e9d57db4ebf93c9143938970090. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:00.774078 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e555f9ea8c7e409fbd7b0ee7d6d07277. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:00.774205 9027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e555f9ea8c7e409fbd7b0ee7d6d07277. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:00.776069 10004 tablet_bootstrap.cc:492] T e555f9ea8c7e409fbd7b0ee7d6d07277 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:00.776206 9853 tablet_copy_client.cc:783] T 0b659e9d57db4ebf93c9143938970090 P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:00.777050 10004 tablet_bootstrap.cc:654] T e555f9ea8c7e409fbd7b0ee7d6d07277 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:00.778048 6567 tablet_service.cc:1467] Processing CreateTablet for tablet e555f9ea8c7e409fbd7b0ee7d6d07277 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3295265fc41646e7942b5cd0d2d8ab20]), partition=
I20240503 02:00:00.778318 6567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e555f9ea8c7e409fbd7b0ee7d6d07277. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:00.776367 9853 tablet_copy_client.cc:647] T 0b659e9d57db4ebf93c9143938970090 P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:00.779863 10121 tablet_bootstrap.cc:492] T e555f9ea8c7e409fbd7b0ee7d6d07277 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:00.780550 9853 tablet_copy_client.cc:523] T 0b659e9d57db4ebf93c9143938970090 P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:00.782038 9853 tablet_bootstrap.cc:492] T 0b659e9d57db4ebf93c9143938970090 P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap starting.
I20240503 02:00:00.782141 10121 tablet_bootstrap.cc:654] T e555f9ea8c7e409fbd7b0ee7d6d07277 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:00.786337 10004 tablet_bootstrap.cc:492] T e555f9ea8c7e409fbd7b0ee7d6d07277 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:00.786455 10004 ts_tablet_manager.cc:1367] T e555f9ea8c7e409fbd7b0ee7d6d07277 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 02:00:00.786957 10004 raft_consensus.cc:348] T e555f9ea8c7e409fbd7b0ee7d6d07277 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:00.787076 10004 raft_consensus.cc:374] T e555f9ea8c7e409fbd7b0ee7d6d07277 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:00.787103 10004 raft_consensus.cc:729] T e555f9ea8c7e409fbd7b0ee7d6d07277 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:00.787220 10004 consensus_queue.cc:260] T e555f9ea8c7e409fbd7b0ee7d6d07277 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:00.787514 10004 ts_tablet_manager.cc:1398] T e555f9ea8c7e409fbd7b0ee7d6d07277 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:00.790277 8917 raft_consensus.cc:1203] T 915028b3478341d588ed6031e657c259 P 9942306855404cfd891adf227dd65eb2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:00:00.797215 9048 raft_consensus.cc:1203] T d82268925a934c559dfcf51a84f03d2e P 99ead02915954b4fab23b033ad6d546b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:00:00.799196 10121 tablet_bootstrap.cc:492] T e555f9ea8c7e409fbd7b0ee7d6d07277 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:00.799324 10121 ts_tablet_manager.cc:1367] T e555f9ea8c7e409fbd7b0ee7d6d07277 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20240503 02:00:00.799760 10121 raft_consensus.cc:348] T e555f9ea8c7e409fbd7b0ee7d6d07277 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:00.799887 10121 raft_consensus.cc:374] T e555f9ea8c7e409fbd7b0ee7d6d07277 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:00.799925 10121 raft_consensus.cc:729] T e555f9ea8c7e409fbd7b0ee7d6d07277 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:00.800034 10121 consensus_queue.cc:260] T e555f9ea8c7e409fbd7b0ee7d6d07277 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:00.800330 10121 ts_tablet_manager.cc:1398] T e555f9ea8c7e409fbd7b0ee7d6d07277 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:00.800846 10123 tablet_bootstrap.cc:492] T e555f9ea8c7e409fbd7b0ee7d6d07277 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:00.801931 10123 tablet_bootstrap.cc:654] T e555f9ea8c7e409fbd7b0ee7d6d07277 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:00.803448 10123 tablet_bootstrap.cc:492] T e555f9ea8c7e409fbd7b0ee7d6d07277 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:00.803553 10123 ts_tablet_manager.cc:1367] T e555f9ea8c7e409fbd7b0ee7d6d07277 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:00.803967 10123 raft_consensus.cc:348] T e555f9ea8c7e409fbd7b0ee7d6d07277 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:00.804082 10123 raft_consensus.cc:374] T e555f9ea8c7e409fbd7b0ee7d6d07277 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:00.804109 10123 raft_consensus.cc:729] T e555f9ea8c7e409fbd7b0ee7d6d07277 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:00.804301 10123 consensus_queue.cc:260] T e555f9ea8c7e409fbd7b0ee7d6d07277 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:00.804709 10123 ts_tablet_manager.cc:1398] T e555f9ea8c7e409fbd7b0ee7d6d07277 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:00.823429 9853 tablet_bootstrap.cc:492] T 0b659e9d57db4ebf93c9143938970090 P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:00.823729 9853 tablet_bootstrap.cc:492] T 0b659e9d57db4ebf93c9143938970090 P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap complete.
I20240503 02:00:00.823925 9853 ts_tablet_manager.cc:1367] T 0b659e9d57db4ebf93c9143938970090 P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent bootstrapping tablet: real 0.042s user 0.022s sys 0.001s
I20240503 02:00:00.824450 9853 raft_consensus.cc:348] T 0b659e9d57db4ebf93c9143938970090 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } }
I20240503 02:00:00.824582 9853 raft_consensus.cc:729] T 0b659e9d57db4ebf93c9143938970090 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e1a9251ce1cf4361b8b0ede9fbfc4023, State: Initialized, Role: LEARNER
I20240503 02:00:00.824685 9853 consensus_queue.cc:260] T 0b659e9d57db4ebf93c9143938970090 P e1a9251ce1cf4361b8b0ede9fbfc4023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } }
I20240503 02:00:00.824981 9853 ts_tablet_manager.cc:1398] T 0b659e9d57db4ebf93c9143938970090 P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:00.825465 6478 tablet_copy_service.cc:342] P dec3e4a04d604f7eaaa3f077d081218a: Request end of tablet copy session e1a9251ce1cf4361b8b0ede9fbfc4023-0b659e9d57db4ebf93c9143938970090 received from {username='slave'} at 127.0.99.136:60209
I20240503 02:00:00.825552 6478 tablet_copy_service.cc:433] P dec3e4a04d604f7eaaa3f077d081218a: ending tablet copy session e1a9251ce1cf4361b8b0ede9fbfc4023-0b659e9d57db4ebf93c9143938970090 on tablet 0b659e9d57db4ebf93c9143938970090 with peer e1a9251ce1cf4361b8b0ede9fbfc4023
I20240503 02:00:00.872570 9973 raft_consensus.cc:482] T e555f9ea8c7e409fbd7b0ee7d6d07277 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:00.872735 9973 raft_consensus.cc:504] T e555f9ea8c7e409fbd7b0ee7d6d07277 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:00.873204 9973 leader_election.cc:288] T e555f9ea8c7e409fbd7b0ee7d6d07277 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:00.873577 6718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e555f9ea8c7e409fbd7b0ee7d6d07277" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 02:00:00.873733 6718 raft_consensus.cc:2454] T e555f9ea8c7e409fbd7b0ee7d6d07277 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:00.874001 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e555f9ea8c7e409fbd7b0ee7d6d07277" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:00.874048 6525 leader_election.cc:302] T e555f9ea8c7e409fbd7b0ee7d6d07277 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:00.874132 9048 raft_consensus.cc:2454] T e555f9ea8c7e409fbd7b0ee7d6d07277 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:00.874224 9973 raft_consensus.cc:2781] T e555f9ea8c7e409fbd7b0ee7d6d07277 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:00.874284 9973 raft_consensus.cc:482] T e555f9ea8c7e409fbd7b0ee7d6d07277 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:00.874318 9973 raft_consensus.cc:3037] T e555f9ea8c7e409fbd7b0ee7d6d07277 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:00.875459 9973 raft_consensus.cc:504] T e555f9ea8c7e409fbd7b0ee7d6d07277 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:00.875861 9973 leader_election.cc:288] T e555f9ea8c7e409fbd7b0ee7d6d07277 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Requested vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:00.876195 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e555f9ea8c7e409fbd7b0ee7d6d07277" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:00.876330 9048 raft_consensus.cc:3037] T e555f9ea8c7e409fbd7b0ee7d6d07277 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:00.876513 6718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e555f9ea8c7e409fbd7b0ee7d6d07277" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 02:00:00.876629 6718 raft_consensus.cc:3037] T e555f9ea8c7e409fbd7b0ee7d6d07277 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:00.877383 9048 raft_consensus.cc:2454] T e555f9ea8c7e409fbd7b0ee7d6d07277 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:00.877619 6718 raft_consensus.cc:2454] T e555f9ea8c7e409fbd7b0ee7d6d07277 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:00.877703 6523 leader_election.cc:302] T e555f9ea8c7e409fbd7b0ee7d6d07277 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:00.878592 9973 raft_consensus.cc:2781] T e555f9ea8c7e409fbd7b0ee7d6d07277 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:00.878697 9973 raft_consensus.cc:686] T e555f9ea8c7e409fbd7b0ee7d6d07277 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Becoming Leader. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Running, Role: LEADER
I20240503 02:00:00.878861 9973 consensus_queue.cc:237] T e555f9ea8c7e409fbd7b0ee7d6d07277 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:00.880951 6312 catalog_manager.cc:5484] T e555f9ea8c7e409fbd7b0ee7d6d07277 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:00.890555 9179 raft_consensus.cc:1203] T 8a76d3e2d4264687aa2972c179341b5e P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:00:00.897863 6830 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:00:00.902482 9158 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:00:00.909111 6700 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:00:00.915001 9027 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:00.915522 8880 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:00:00.915820 8917 raft_consensus.cc:1203] T 2b1d700d09534fa3adcf51bb3c97c8f5 P 9942306855404cfd891adf227dd65eb2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:00:00.917590 6314 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240503 02:00:00.917762 6314 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240503 02:00:00.917896 6567 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:00:00.920632 6436 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:00:00.924751 9048 raft_consensus.cc:1203] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 99ead02915954b4fab23b033ad6d546b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:00:00.924958 6314 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 4 replicas of tablet 7f637097ad06465cb93c6ff6d97c1d17
I20240503 02:00:00.925333 6314 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 4 replicas of tablet 548f43b381f1445c8d3d55705ae5a48a
I20240503 02:00:00.925573 6314 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet ce8bfc835792444e8703e73252b1f638
I20240503 02:00:00.926259 9158 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f637097ad06465cb93c6ff6d97c1d17 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:00 UTC) from {username='slave'} at 127.0.0.1:51940
I20240503 02:00:00.926849 6436 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f637097ad06465cb93c6ff6d97c1d17 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:00 UTC) from {username='slave'} at 127.0.0.1:51338
I20240503 02:00:00.926999 6438 tablet_service.cc:1514] Processing DeleteTablet for tablet 548f43b381f1445c8d3d55705ae5a48a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:00 UTC) from {username='slave'} at 127.0.0.1:51338
I20240503 02:00:00.928346 6829 tablet_service.cc:1514] Processing DeleteTablet for tablet ce8bfc835792444e8703e73252b1f638 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:00 UTC) from {username='slave'} at 127.0.0.1:52812
I20240503 02:00:00.928452 6830 tablet_service.cc:1514] Processing DeleteTablet for tablet 548f43b381f1445c8d3d55705ae5a48a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:00 UTC) from {username='slave'} at 127.0.0.1:52812
I20240503 02:00:00.928637 9476 tablet_replica.cc:330] T ce8bfc835792444e8703e73252b1f638 P 492448570cc94ed1879feb2ec3b9051d: stopping tablet replica
I20240503 02:00:00.928752 9474 tablet_replica.cc:330] T 7f637097ad06465cb93c6ff6d97c1d17 P dec3e4a04d604f7eaaa3f077d081218a: stopping tablet replica
I20240503 02:00:00.928776 9476 raft_consensus.cc:2229] T ce8bfc835792444e8703e73252b1f638 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:00.928893 9474 raft_consensus.cc:2229] T 7f637097ad06465cb93c6ff6d97c1d17 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:00.928932 9476 raft_consensus.cc:2258] T ce8bfc835792444e8703e73252b1f638 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:00.929049 9474 raft_consensus.cc:2258] T 7f637097ad06465cb93c6ff6d97c1d17 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:00.929471 6700 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f637097ad06465cb93c6ff6d97c1d17 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:00 UTC) from {username='slave'} at 127.0.0.1:40336
I20240503 02:00:00.929508 9476 ts_tablet_manager.cc:1875] T ce8bfc835792444e8703e73252b1f638 P 492448570cc94ed1879feb2ec3b9051d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:00.929546 6699 tablet_service.cc:1514] Processing DeleteTablet for tablet ce8bfc835792444e8703e73252b1f638 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:00 UTC) from {username='slave'} at 127.0.0.1:40336
I20240503 02:00:00.930722 6568 tablet_service.cc:1514] Processing DeleteTablet for tablet 548f43b381f1445c8d3d55705ae5a48a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:00 UTC) from {username='slave'} at 127.0.0.1:59846
I20240503 02:00:00.930770 6567 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f637097ad06465cb93c6ff6d97c1d17 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:00 UTC) from {username='slave'} at 127.0.0.1:59846
I20240503 02:00:00.930930 9473 tablet_replica.cc:330] T 548f43b381f1445c8d3d55705ae5a48a P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 02:00:00.931051 9473 raft_consensus.cc:2229] T 548f43b381f1445c8d3d55705ae5a48a P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:00.931212 9473 raft_consensus.cc:2258] T 548f43b381f1445c8d3d55705ae5a48a P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:00.931850 9476 ts_tablet_manager.cc:1888] T ce8bfc835792444e8703e73252b1f638 P 492448570cc94ed1879feb2ec3b9051d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:00:00.931917 9476 log.cc:1196] T ce8bfc835792444e8703e73252b1f638 P 492448570cc94ed1879feb2ec3b9051d: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/ce8bfc835792444e8703e73252b1f638
I20240503 02:00:00.931893 9473 ts_tablet_manager.cc:1875] T 548f43b381f1445c8d3d55705ae5a48a P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:00.932168 9476 ts_tablet_manager.cc:1909] T ce8bfc835792444e8703e73252b1f638 P 492448570cc94ed1879feb2ec3b9051d: Deleting consensus metadata
I20240503 02:00:00.932670 6569 tablet_service.cc:1514] Processing DeleteTablet for tablet ce8bfc835792444e8703e73252b1f638 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:00 UTC) from {username='slave'} at 127.0.0.1:59846
I20240503 02:00:00.932936 9474 ts_tablet_manager.cc:1875] T 7f637097ad06465cb93c6ff6d97c1d17 P dec3e4a04d604f7eaaa3f077d081218a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:00.933401 9476 tablet_replica.cc:330] T 548f43b381f1445c8d3d55705ae5a48a P 492448570cc94ed1879feb2ec3b9051d: stopping tablet replica
I20240503 02:00:00.933527 9476 raft_consensus.cc:2229] T 548f43b381f1445c8d3d55705ae5a48a P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:00.933519 9285 tablet_service.cc:1514] Processing DeleteTablet for tablet 548f43b381f1445c8d3d55705ae5a48a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:00 UTC) from {username='slave'} at 127.0.0.1:36004
I20240503 02:00:00.933679 10051 tablet_replica.cc:330] T 548f43b381f1445c8d3d55705ae5a48a P e1a9251ce1cf4361b8b0ede9fbfc4023: stopping tablet replica
I20240503 02:00:00.933787 10051 raft_consensus.cc:2229] T 548f43b381f1445c8d3d55705ae5a48a P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 LEARNER]: Raft consensus shutting down.
I20240503 02:00:00.933820 9476 raft_consensus.cc:2258] T 548f43b381f1445c8d3d55705ae5a48a P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:00.933895 10051 raft_consensus.cc:2258] T 548f43b381f1445c8d3d55705ae5a48a P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 LEARNER]: Raft consensus is shut down!
I20240503 02:00:00.934052 9473 ts_tablet_manager.cc:1888] T 548f43b381f1445c8d3d55705ae5a48a P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:00.934115 9473 log.cc:1196] T 548f43b381f1445c8d3d55705ae5a48a P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/548f43b381f1445c8d3d55705ae5a48a
I20240503 02:00:00.934348 9473 ts_tablet_manager.cc:1909] T 548f43b381f1445c8d3d55705ae5a48a P 80fd18ef4d604c86b909d5c008a46b4f: Deleting consensus metadata
I20240503 02:00:00.934684 10051 ts_tablet_manager.cc:1875] T 548f43b381f1445c8d3d55705ae5a48a P e1a9251ce1cf4361b8b0ede9fbfc4023: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:00.935031 9474 ts_tablet_manager.cc:1888] T 7f637097ad06465cb93c6ff6d97c1d17 P dec3e4a04d604f7eaaa3f077d081218a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:00.935101 9474 log.cc:1196] T 7f637097ad06465cb93c6ff6d97c1d17 P dec3e4a04d604f7eaaa3f077d081218a: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/7f637097ad06465cb93c6ff6d97c1d17
I20240503 02:00:00.935362 9474 ts_tablet_manager.cc:1909] T 7f637097ad06465cb93c6ff6d97c1d17 P dec3e4a04d604f7eaaa3f077d081218a: Deleting consensus metadata
I20240503 02:00:00.935859 6303 catalog_manager.cc:4830] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719): tablet ce8bfc835792444e8703e73252b1f638 (table rebalance_test_table_8 [id=d93ca01e08fd40d886555f16d21192fc]) successfully deleted
I20240503 02:00:00.936553 9474 tablet_replica.cc:330] T 548f43b381f1445c8d3d55705ae5a48a P dec3e4a04d604f7eaaa3f077d081218a: stopping tablet replica
I20240503 02:00:00.936714 9474 raft_consensus.cc:2229] T 548f43b381f1445c8d3d55705ae5a48a P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:00.936743 6300 catalog_manager.cc:4830] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): tablet 7f637097ad06465cb93c6ff6d97c1d17 (table rebalance_test_table_8 [id=d93ca01e08fd40d886555f16d21192fc]) successfully deleted
I20240503 02:00:00.936956 9474 raft_consensus.cc:2258] T 548f43b381f1445c8d3d55705ae5a48a P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:00.937559 9476 ts_tablet_manager.cc:1875] T 548f43b381f1445c8d3d55705ae5a48a P 492448570cc94ed1879feb2ec3b9051d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:00.937687 9474 ts_tablet_manager.cc:1875] T 548f43b381f1445c8d3d55705ae5a48a P dec3e4a04d604f7eaaa3f077d081218a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:00.938884 9473 tablet_replica.cc:330] T 7f637097ad06465cb93c6ff6d97c1d17 P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 02:00:00.939272 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet 548f43b381f1445c8d3d55705ae5a48a (table rebalance_test_table_8 [id=d93ca01e08fd40d886555f16d21192fc]) successfully deleted
I20240503 02:00:00.939028 9473 raft_consensus.cc:2229] T 7f637097ad06465cb93c6ff6d97c1d17 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:00.939709 9473 raft_consensus.cc:2258] T 7f637097ad06465cb93c6ff6d97c1d17 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:00.939967 9474 ts_tablet_manager.cc:1888] T 548f43b381f1445c8d3d55705ae5a48a P dec3e4a04d604f7eaaa3f077d081218a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:00.940068 9474 log.cc:1196] T 548f43b381f1445c8d3d55705ae5a48a P dec3e4a04d604f7eaaa3f077d081218a: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/548f43b381f1445c8d3d55705ae5a48a
I20240503 02:00:00.940251 9476 ts_tablet_manager.cc:1888] T 548f43b381f1445c8d3d55705ae5a48a P 492448570cc94ed1879feb2ec3b9051d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:00.940353 9476 log.cc:1196] T 548f43b381f1445c8d3d55705ae5a48a P 492448570cc94ed1879feb2ec3b9051d: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/548f43b381f1445c8d3d55705ae5a48a
I20240503 02:00:00.940367 9474 ts_tablet_manager.cc:1909] T 548f43b381f1445c8d3d55705ae5a48a P dec3e4a04d604f7eaaa3f077d081218a: Deleting consensus metadata
I20240503 02:00:00.940629 9476 ts_tablet_manager.cc:1909] T 548f43b381f1445c8d3d55705ae5a48a P 492448570cc94ed1879feb2ec3b9051d: Deleting consensus metadata
I20240503 02:00:00.941174 6300 catalog_manager.cc:4830] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): tablet 548f43b381f1445c8d3d55705ae5a48a (table rebalance_test_table_8 [id=d93ca01e08fd40d886555f16d21192fc]) successfully deleted
I20240503 02:00:00.941421 6303 catalog_manager.cc:4830] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719): tablet 548f43b381f1445c8d3d55705ae5a48a (table rebalance_test_table_8 [id=d93ca01e08fd40d886555f16d21192fc]) successfully deleted
I20240503 02:00:00.942770 9473 ts_tablet_manager.cc:1875] T 7f637097ad06465cb93c6ff6d97c1d17 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:00.944808 9473 ts_tablet_manager.cc:1888] T 7f637097ad06465cb93c6ff6d97c1d17 P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:00.947543 9473 log.cc:1196] T 7f637097ad06465cb93c6ff6d97c1d17 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/7f637097ad06465cb93c6ff6d97c1d17
I20240503 02:00:00.948237 10051 ts_tablet_manager.cc:1888] T 548f43b381f1445c8d3d55705ae5a48a P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:00.948329 10051 log.cc:1196] T 548f43b381f1445c8d3d55705ae5a48a P e1a9251ce1cf4361b8b0ede9fbfc4023: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/548f43b381f1445c8d3d55705ae5a48a
I20240503 02:00:00.948647 10051 ts_tablet_manager.cc:1909] T 548f43b381f1445c8d3d55705ae5a48a P e1a9251ce1cf4361b8b0ede9fbfc4023: Deleting consensus metadata
I20240503 02:00:00.949398 6301 catalog_manager.cc:4830] TS e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): tablet 548f43b381f1445c8d3d55705ae5a48a (table rebalance_test_table_8 [id=d93ca01e08fd40d886555f16d21192fc]) successfully deleted
I20240503 02:00:00.949802 9473 ts_tablet_manager.cc:1909] T 7f637097ad06465cb93c6ff6d97c1d17 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting consensus metadata
I20240503 02:00:00.950454 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet 7f637097ad06465cb93c6ff6d97c1d17 (table rebalance_test_table_8 [id=d93ca01e08fd40d886555f16d21192fc]) successfully deleted
I20240503 02:00:00.950850 9473 tablet_replica.cc:330] T ce8bfc835792444e8703e73252b1f638 P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 02:00:00.950985 9473 raft_consensus.cc:2229] T ce8bfc835792444e8703e73252b1f638 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:00.951098 9473 raft_consensus.cc:2258] T ce8bfc835792444e8703e73252b1f638 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:00.951562 9473 ts_tablet_manager.cc:1875] T ce8bfc835792444e8703e73252b1f638 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:00.953766 9473 ts_tablet_manager.cc:1888] T ce8bfc835792444e8703e73252b1f638 P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
W20240503 02:00:00.953820 6300 catalog_manager.cc:4797] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079): delete failed for tablet 7f637097ad06465cb93c6ff6d97c1d17 because the tablet was not found. No further retry: Not found: Tablet not found: 7f637097ad06465cb93c6ff6d97c1d17
I20240503 02:00:00.954108 9285 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:00:00.959203 6653 consensus_peers.cc:485] T 7f637097ad06465cb93c6ff6d97c1d17 P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): Couldn't send request to peer dec3e4a04d604f7eaaa3f077d081218a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7f637097ad06465cb93c6ff6d97c1d17. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:00.967149 9473 log.cc:1196] T ce8bfc835792444e8703e73252b1f638 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/ce8bfc835792444e8703e73252b1f638
I20240503 02:00:00.967655 9473 ts_tablet_manager.cc:1909] T ce8bfc835792444e8703e73252b1f638 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting consensus metadata
I20240503 02:00:00.968700 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet ce8bfc835792444e8703e73252b1f638 (table rebalance_test_table_8 [id=d93ca01e08fd40d886555f16d21192fc]) successfully deleted
I20240503 02:00:00.976893 9048 raft_consensus.cc:1203] T 485d1d89a8b94505969d663be92124cd P 99ead02915954b4fab23b033ad6d546b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:00:00.977663 10173 tablet_replica.cc:330] T 7f637097ad06465cb93c6ff6d97c1d17 P c2ac2eb7cc264047b9ab32fe3a4fc025: stopping tablet replica
I20240503 02:00:00.977887 10173 raft_consensus.cc:2229] T 7f637097ad06465cb93c6ff6d97c1d17 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:00.978247 10173 raft_consensus.cc:2258] T 7f637097ad06465cb93c6ff6d97c1d17 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:00.979843 10173 ts_tablet_manager.cc:1875] T 7f637097ad06465cb93c6ff6d97c1d17 P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:00.984135 10173 ts_tablet_manager.cc:1888] T 7f637097ad06465cb93c6ff6d97c1d17 P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:00.984237 10173 log.cc:1196] T 7f637097ad06465cb93c6ff6d97c1d17 P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/7f637097ad06465cb93c6ff6d97c1d17
I20240503 02:00:00.984505 10173 ts_tablet_manager.cc:1909] T 7f637097ad06465cb93c6ff6d97c1d17 P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting consensus metadata
I20240503 02:00:00.993758 10173 tablet_replica.cc:330] T ce8bfc835792444e8703e73252b1f638 P c2ac2eb7cc264047b9ab32fe3a4fc025: stopping tablet replica
I20240503 02:00:00.993899 6303 catalog_manager.cc:4830] TS c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655): tablet 7f637097ad06465cb93c6ff6d97c1d17 (table rebalance_test_table_8 [id=d93ca01e08fd40d886555f16d21192fc]) successfully deleted
I20240503 02:00:00.994081 10173 raft_consensus.cc:2229] T ce8bfc835792444e8703e73252b1f638 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:00.994329 10173 raft_consensus.cc:2258] T ce8bfc835792444e8703e73252b1f638 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:00.994993 10173 ts_tablet_manager.cc:1875] T ce8bfc835792444e8703e73252b1f638 P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:00.995375 6312 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:01.000183 10173 ts_tablet_manager.cc:1888] T ce8bfc835792444e8703e73252b1f638 P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:00:01.000330 10173 log.cc:1196] T ce8bfc835792444e8703e73252b1f638 P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/ce8bfc835792444e8703e73252b1f638
I20240503 02:00:01.000633 10173 ts_tablet_manager.cc:1909] T ce8bfc835792444e8703e73252b1f638 P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting consensus metadata
I20240503 02:00:01.001683 9048 raft_consensus.cc:1261] T e555f9ea8c7e409fbd7b0ee7d6d07277 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:01.002538 6718 raft_consensus.cc:1261] T e555f9ea8c7e409fbd7b0ee7d6d07277 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:01.002760 8833 consensus_queue.cc:1035] T e555f9ea8c7e409fbd7b0ee7d6d07277 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:01.003065 6303 catalog_manager.cc:4830] TS c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655): tablet ce8bfc835792444e8703e73252b1f638 (table rebalance_test_table_8 [id=d93ca01e08fd40d886555f16d21192fc]) successfully deleted
I20240503 02:00:01.003551 8833 consensus_queue.cc:1035] T e555f9ea8c7e409fbd7b0ee7d6d07277 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:01.005256 10174 ts_tablet_manager.cc:902] T 7a4a910a8b9d4134adebb52a73dfae1d P 99ead02915954b4fab23b033ad6d546b: Initiating tablet copy from peer 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:01.006315 6312 catalog_manager.cc:5576] TS 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157) does not have the latest schema for tablet e555f9ea8c7e409fbd7b0ee7d6d07277 (table rebalancer_dynamic_table [id=3295265fc41646e7942b5cd0d2d8ab20]). Expected version 1 got 0
I20240503 02:00:01.007189 6314 catalog_manager.cc:5576] TS c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655) does not have the latest schema for tablet e555f9ea8c7e409fbd7b0ee7d6d07277 (table rebalancer_dynamic_table [id=3295265fc41646e7942b5cd0d2d8ab20]). Expected version 1 got 0
I20240503 02:00:01.008987 9472 tablet.cc:1722] T e555f9ea8c7e409fbd7b0ee7d6d07277 P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:01.009333 9472 tablet.cc:1620] T e555f9ea8c7e409fbd7b0ee7d6d07277 P 80fd18ef4d604c86b909d5c008a46b4f: MemRowSet was empty: no flush needed.
I20240503 02:00:01.010735 10174 tablet_copy_client.cc:308] T 7a4a910a8b9d4134adebb52a73dfae1d P 99ead02915954b4fab23b033ad6d546b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43863
I20240503 02:00:01.010753 6314 catalog_manager.cc:5576] TS 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157) does not have the latest schema for tablet e555f9ea8c7e409fbd7b0ee7d6d07277 (table rebalancer_dynamic_table [id=3295265fc41646e7942b5cd0d2d8ab20]). Expected version 1 got 0
I20240503 02:00:01.012235 6609 tablet_copy_service.cc:140] P 80fd18ef4d604c86b909d5c008a46b4f: Received BeginTabletCopySession request for tablet 7a4a910a8b9d4134adebb52a73dfae1d from peer 99ead02915954b4fab23b033ad6d546b ({username='slave'} at 127.0.99.134:40601)
I20240503 02:00:01.012362 6609 tablet_copy_service.cc:161] P 80fd18ef4d604c86b909d5c008a46b4f: Beginning new tablet copy session on tablet 7a4a910a8b9d4134adebb52a73dfae1d from peer 99ead02915954b4fab23b033ad6d546b at {username='slave'} at 127.0.99.134:40601: session id = 99ead02915954b4fab23b033ad6d546b-7a4a910a8b9d4134adebb52a73dfae1d
I20240503 02:00:01.012310 10179 tablet.cc:1722] T e555f9ea8c7e409fbd7b0ee7d6d07277 P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:01.013305 6609 tablet_copy_source_session.cc:205] T 7a4a910a8b9d4134adebb52a73dfae1d P 80fd18ef4d604c86b909d5c008a46b4f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:01.014684 10174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a4a910a8b9d4134adebb52a73dfae1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:01.014720 6312 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=3295265fc41646e7942b5cd0d2d8ab20] alter complete (version 1)
I20240503 02:00:01.017056 10184 tablet.cc:1722] T e555f9ea8c7e409fbd7b0ee7d6d07277 P c2ac2eb7cc264047b9ab32fe3a4fc025: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:01.017853 10184 tablet.cc:1620] T e555f9ea8c7e409fbd7b0ee7d6d07277 P c2ac2eb7cc264047b9ab32fe3a4fc025: MemRowSet was empty: no flush needed.
I20240503 02:00:01.020862 6312 catalog_manager.cc:5576] TS c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655) does not have the latest schema for tablet e555f9ea8c7e409fbd7b0ee7d6d07277 (table rebalancer_dynamic_table [id=3295265fc41646e7942b5cd0d2d8ab20]). Expected version 1 got 0
I20240503 02:00:01.021806 9309 raft_consensus.cc:1203] T 46ad540deb624a25b97d3b967fe1f6f4 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:00:01.025566 10174 tablet_copy_client.cc:783] T 7a4a910a8b9d4134adebb52a73dfae1d P 99ead02915954b4fab23b033ad6d546b: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:01.025847 10174 tablet_copy_client.cc:647] T 7a4a910a8b9d4134adebb52a73dfae1d P 99ead02915954b4fab23b033ad6d546b: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:01.026960 10174 tablet_copy_client.cc:523] T 7a4a910a8b9d4134adebb52a73dfae1d P 99ead02915954b4fab23b033ad6d546b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:01.027612 6457 consensus_queue.cc:237] T 0099654461ca41efa4323fc9f5adab4a P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: true } }
I20240503 02:00:01.029336 10174 tablet_bootstrap.cc:492] T 7a4a910a8b9d4134adebb52a73dfae1d P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:01.029417 8917 raft_consensus.cc:1261] T 0099654461ca41efa4323fc9f5adab4a P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:00:01.029572 6393 consensus_peers.cc:485] T 0099654461ca41efa4323fc9f5adab4a P dec3e4a04d604f7eaaa3f077d081218a -> Peer 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): Couldn't send request to peer 80fd18ef4d604c86b909d5c008a46b4f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0099654461ca41efa4323fc9f5adab4a. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:01.029886 8679 consensus_queue.cc:1035] T 0099654461ca41efa4323fc9f5adab4a P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:01.030089 9179 raft_consensus.cc:1261] T 0099654461ca41efa4323fc9f5adab4a P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:01.030637 8679 consensus_queue.cc:1035] T 0099654461ca41efa4323fc9f5adab4a P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:01.031715 9955 raft_consensus.cc:2932] T 0099654461ca41efa4323fc9f5adab4a P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: true } } }
I20240503 02:00:01.012823 10179 tablet.cc:1620] T e555f9ea8c7e409fbd7b0ee7d6d07277 P 99ead02915954b4fab23b033ad6d546b: MemRowSet was empty: no flush needed.
I20240503 02:00:01.033742 9179 raft_consensus.cc:2932] T 0099654461ca41efa4323fc9f5adab4a P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: true } } }
I20240503 02:00:01.037493 6314 catalog_manager.cc:5484] T 0099654461ca41efa4323fc9f5adab4a P 2a01ace80baf4057b6bf1995cf88468b reported cstate change: config changed from index -1 to 2, NON_VOTER 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: true } } }
I20240503 02:00:01.037555 8917 raft_consensus.cc:2932] T 0099654461ca41efa4323fc9f5adab4a P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: true } } }
I20240503 02:00:01.063970 10174 tablet_bootstrap.cc:492] T 7a4a910a8b9d4134adebb52a73dfae1d P 99ead02915954b4fab23b033ad6d546b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:01.069947 9461 raft_consensus.cc:1050] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d: attempting to promote NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 to VOTER
I20240503 02:00:01.070461 9461 consensus_queue.cc:237] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } }
I20240503 02:00:01.071915 9309 raft_consensus.cc:1261] T 1fa274c98fc147f29344ed537e2ccf17 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 LEARNER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:01.071949 6587 raft_consensus.cc:1261] T 1fa274c98fc147f29344ed537e2ccf17 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:01.071949 6718 raft_consensus.cc:1261] T 1fa274c98fc147f29344ed537e2ccf17 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:01.072412 9631 consensus_queue.cc:1035] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:01.072608 10212 consensus_queue.cc:1035] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:01.072870 9632 consensus_queue.cc:1035] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:01.075126 9631 raft_consensus.cc:2932] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:01.076743 6718 raft_consensus.cc:2932] T 1fa274c98fc147f29344ed537e2ccf17 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:01.078002 9309 raft_consensus.cc:2932] T 1fa274c98fc147f29344ed537e2ccf17 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:01.078253 6309 catalog_manager.cc:5484] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d reported cstate change: config changed from index 4 to 5, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:01.078312 6587 raft_consensus.cc:2932] T 1fa274c98fc147f29344ed537e2ccf17 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:01.080582 10174 tablet_bootstrap.cc:492] T 7a4a910a8b9d4134adebb52a73dfae1d P 99ead02915954b4fab23b033ad6d546b: Bootstrap complete.
I20240503 02:00:01.080823 10174 ts_tablet_manager.cc:1367] T 7a4a910a8b9d4134adebb52a73dfae1d P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.052s user 0.024s sys 0.000s
I20240503 02:00:01.081279 10174 raft_consensus.cc:348] T 7a4a910a8b9d4134adebb52a73dfae1d P 99ead02915954b4fab23b033ad6d546b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } }
I20240503 02:00:01.081393 10174 raft_consensus.cc:729] T 7a4a910a8b9d4134adebb52a73dfae1d P 99ead02915954b4fab23b033ad6d546b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: LEARNER
I20240503 02:00:01.081487 10174 consensus_queue.cc:260] T 7a4a910a8b9d4134adebb52a73dfae1d P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } }
I20240503 02:00:01.081764 10174 ts_tablet_manager.cc:1398] T 7a4a910a8b9d4134adebb52a73dfae1d P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:01.082437 6609 tablet_copy_service.cc:342] P 80fd18ef4d604c86b909d5c008a46b4f: Request end of tablet copy session 99ead02915954b4fab23b033ad6d546b-7a4a910a8b9d4134adebb52a73dfae1d received from {username='slave'} at 127.0.99.134:40601
I20240503 02:00:01.082521 6609 tablet_copy_service.cc:433] P 80fd18ef4d604c86b909d5c008a46b4f: ending tablet copy session 99ead02915954b4fab23b033ad6d546b-7a4a910a8b9d4134adebb52a73dfae1d on tablet 7a4a910a8b9d4134adebb52a73dfae1d with peer 99ead02915954b4fab23b033ad6d546b
I20240503 02:00:01.089831 6851 consensus_queue.cc:237] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } }
I20240503 02:00:01.092672 9309 raft_consensus.cc:1261] T 1fa274c98fc147f29344ed537e2ccf17 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:01.093156 6587 raft_consensus.cc:1261] T 1fa274c98fc147f29344ed537e2ccf17 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:01.093111 9632 consensus_queue.cc:1035] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:01.093528 9632 consensus_queue.cc:1035] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:01.095252 9632 raft_consensus.cc:2932] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:01.095876 9309 raft_consensus.cc:2932] T 1fa274c98fc147f29344ed537e2ccf17 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:01.096035 6589 raft_consensus.cc:2932] T 1fa274c98fc147f29344ed537e2ccf17 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:01.097399 6303 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1fa274c98fc147f29344ed537e2ccf17 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:01.098363 6312 catalog_manager.cc:5484] T 1fa274c98fc147f29344ed537e2ccf17 P e1a9251ce1cf4361b8b0ede9fbfc4023 reported cstate change: config changed from index 5 to 6, VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:01.101300 6700 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fa274c98fc147f29344ed537e2ccf17 with delete_type TABLET_DATA_TOMBSTONED (TS c2ac2eb7cc264047b9ab32fe3a4fc025 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:40336
I20240503 02:00:01.101662 10173 tablet_replica.cc:330] T 1fa274c98fc147f29344ed537e2ccf17 P c2ac2eb7cc264047b9ab32fe3a4fc025: stopping tablet replica
I20240503 02:00:01.101802 10173 raft_consensus.cc:2229] T 1fa274c98fc147f29344ed537e2ccf17 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:01.101933 10173 raft_consensus.cc:2258] T 1fa274c98fc147f29344ed537e2ccf17 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:01.103008 10173 ts_tablet_manager.cc:1875] T 1fa274c98fc147f29344ed537e2ccf17 P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:01.105367 10173 ts_tablet_manager.cc:1888] T 1fa274c98fc147f29344ed537e2ccf17 P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:00:01.105455 10173 log.cc:1196] T 1fa274c98fc147f29344ed537e2ccf17 P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/1fa274c98fc147f29344ed537e2ccf17
I20240503 02:00:01.106040 6303 catalog_manager.cc:4830] TS c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655): tablet 1fa274c98fc147f29344ed537e2ccf17 (table rebalance_test_table_1 [id=6e24d367bc1d434584d8a7363cf538b6]) successfully deleted
I20240503 02:00:01.119542 6307 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:01.125856 9472 tablet.cc:1722] T e555f9ea8c7e409fbd7b0ee7d6d07277 P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:01.126227 9472 tablet.cc:1620] T e555f9ea8c7e409fbd7b0ee7d6d07277 P 80fd18ef4d604c86b909d5c008a46b4f: MemRowSet was empty: no flush needed.
I20240503 02:00:01.128103 10184 tablet.cc:1722] T e555f9ea8c7e409fbd7b0ee7d6d07277 P c2ac2eb7cc264047b9ab32fe3a4fc025: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:01.128266 10179 tablet.cc:1722] T e555f9ea8c7e409fbd7b0ee7d6d07277 P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:01.128509 10184 tablet.cc:1620] T e555f9ea8c7e409fbd7b0ee7d6d07277 P c2ac2eb7cc264047b9ab32fe3a4fc025: MemRowSet was empty: no flush needed.
I20240503 02:00:01.128603 10179 tablet.cc:1620] T e555f9ea8c7e409fbd7b0ee7d6d07277 P 99ead02915954b4fab23b033ad6d546b: MemRowSet was empty: no flush needed.
I20240503 02:00:01.133296 6307 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=3295265fc41646e7942b5cd0d2d8ab20] alter complete (version 2)
I20240503 02:00:01.143491 10093 consensus_queue.cc:1035] T 51b82ef8e4cc474387f6902f2e526ac0 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:01.147317 9652 consensus_queue.cc:1035] T 51b82ef8e4cc474387f6902f2e526ac0 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:01.151047 9929 consensus_queue.cc:1035] T 6a82dcc9be5b4c93b5fff03f789350ab P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:01.159706 9984 raft_consensus.cc:1050] T 7a44b6d65bfa43e291b06b0dfd38fab3 P c2ac2eb7cc264047b9ab32fe3a4fc025: attempting to promote NON_VOTER 9942306855404cfd891adf227dd65eb2 to VOTER
I20240503 02:00:01.160262 9984 consensus_queue.cc:237] T 7a44b6d65bfa43e291b06b0dfd38fab3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } }
I20240503 02:00:01.161206 9929 consensus_queue.cc:1035] T 6a82dcc9be5b4c93b5fff03f789350ab P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:01.163115 8917 raft_consensus.cc:1261] T 7a44b6d65bfa43e291b06b0dfd38fab3 P 9942306855404cfd891adf227dd65eb2 [term 1 LEARNER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:01.163115 6587 raft_consensus.cc:1261] T 7a44b6d65bfa43e291b06b0dfd38fab3 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:01.163286 6457 raft_consensus.cc:1261] T 7a44b6d65bfa43e291b06b0dfd38fab3 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:01.163885 8696 consensus_queue.cc:1035] T 7a44b6d65bfa43e291b06b0dfd38fab3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:01.165711 8696 consensus_queue.cc:1035] T 7a44b6d65bfa43e291b06b0dfd38fab3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:01.166673 8696 consensus_queue.cc:1035] T 7a44b6d65bfa43e291b06b0dfd38fab3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:01.172583 9973 raft_consensus.cc:1050] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f: attempting to promote NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 to VOTER
I20240503 02:00:01.172969 9631 raft_consensus.cc:1050] T 90ef3584830240acb0d8aad14ae851a6 P 492448570cc94ed1879feb2ec3b9051d: attempting to promote NON_VOTER 2a01ace80baf4057b6bf1995cf88468b to VOTER
I20240503 02:00:01.173096 9973 consensus_queue.cc:237] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } }
I20240503 02:00:01.173362 9631 consensus_queue.cc:237] T 90ef3584830240acb0d8aad14ae851a6 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } }
I20240503 02:00:01.174741 6720 raft_consensus.cc:1261] T 90ef3584830240acb0d8aad14ae851a6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:01.174765 9179 raft_consensus.cc:1261] T 90ef3584830240acb0d8aad14ae851a6 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEARNER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:01.175104 6457 raft_consensus.cc:1261] T 90ef3584830240acb0d8aad14ae851a6 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:01.175575 9309 raft_consensus.cc:1261] T 3416b6107e1848f299ac49ccd88ae7c8 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 LEARNER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:01.175542 9632 consensus_queue.cc:1035] T 90ef3584830240acb0d8aad14ae851a6 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:01.176118 6851 raft_consensus.cc:1261] T 3416b6107e1848f299ac49ccd88ae7c8 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:01.176944 9484 consensus_queue.cc:1035] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:01.177243 8833 consensus_queue.cc:1035] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:01.177412 6720 raft_consensus.cc:1261] T 3416b6107e1848f299ac49ccd88ae7c8 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:01.178494 9632 consensus_queue.cc:1035] T 90ef3584830240acb0d8aad14ae851a6 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:01.178965 9632 consensus_queue.cc:1035] T 90ef3584830240acb0d8aad14ae851a6 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:01.180121 8833 consensus_queue.cc:1035] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:01.181700 8696 raft_consensus.cc:2932] T 7a44b6d65bfa43e291b06b0dfd38fab3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:01.185078 6457 raft_consensus.cc:2932] T 7a44b6d65bfa43e291b06b0dfd38fab3 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:01.186072 10278 raft_consensus.cc:2932] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:01.186854 6307 catalog_manager.cc:5484] T 7a44b6d65bfa43e291b06b0dfd38fab3 P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: config changed from index 4 to 5, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:01.188145 9632 raft_consensus.cc:2932] T 90ef3584830240acb0d8aad14ae851a6 P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:01.189751 6314 catalog_manager.cc:5484] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: config changed from index 4 to 5, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:01.190423 8917 raft_consensus.cc:2932] T 7a44b6d65bfa43e291b06b0dfd38fab3 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:01.190510 6457 raft_consensus.cc:2932] T 90ef3584830240acb0d8aad14ae851a6 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:01.193065 6315 catalog_manager.cc:5484] T 90ef3584830240acb0d8aad14ae851a6 P 492448570cc94ed1879feb2ec3b9051d reported cstate change: config changed from index 4 to 5, 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:01.194131 9310 raft_consensus.cc:2932] T 3416b6107e1848f299ac49ccd88ae7c8 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:01.197690 9179 raft_consensus.cc:2932] T 90ef3584830240acb0d8aad14ae851a6 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:01.198105 6587 raft_consensus.cc:2932] T 7a44b6d65bfa43e291b06b0dfd38fab3 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:01.198684 6720 raft_consensus.cc:2932] T 3416b6107e1848f299ac49ccd88ae7c8 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:01.199445 6719 raft_consensus.cc:2932] T 90ef3584830240acb0d8aad14ae851a6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:01.200107 6851 raft_consensus.cc:2932] T 3416b6107e1848f299ac49ccd88ae7c8 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:01.209010 6587 consensus_queue.cc:237] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } }
I20240503 02:00:01.211095 10268 raft_consensus.cc:1050] T ceca04391cd446ae932b3485d958237a P c2ac2eb7cc264047b9ab32fe3a4fc025: attempting to promote NON_VOTER 9942306855404cfd891adf227dd65eb2 to VOTER
I20240503 02:00:01.211548 10268 consensus_queue.cc:237] T ceca04391cd446ae932b3485d958237a P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } }
I20240503 02:00:01.213348 6849 raft_consensus.cc:1261] T ceca04391cd446ae932b3485d958237a P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:01.213807 9711 consensus_queue.cc:1035] T ceca04391cd446ae932b3485d958237a P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:01.213937 9310 raft_consensus.cc:1261] T 3416b6107e1848f299ac49ccd88ae7c8 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:01.214398 8833 consensus_queue.cc:1035] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:01.216226 8917 raft_consensus.cc:1261] T ceca04391cd446ae932b3485d958237a P 9942306855404cfd891adf227dd65eb2 [term 1 LEARNER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:01.216290 6719 consensus_queue.cc:237] T 7a44b6d65bfa43e291b06b0dfd38fab3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } }
I20240503 02:00:01.216653 9711 consensus_queue.cc:1035] T ceca04391cd446ae932b3485d958237a P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:01.217463 10044 consensus_queue.cc:1035] T f28586e70bd9473fbdc6b9126b666fa9 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:01.217569 6457 raft_consensus.cc:1261] T 7a44b6d65bfa43e291b06b0dfd38fab3 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:01.217614 8916 raft_consensus.cc:1261] T 7a44b6d65bfa43e291b06b0dfd38fab3 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:01.217963 8696 consensus_queue.cc:1035] T 7a44b6d65bfa43e291b06b0dfd38fab3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:01.218899 8696 consensus_queue.cc:1035] T 7a44b6d65bfa43e291b06b0dfd38fab3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:01.220714 9984 raft_consensus.cc:2932] T 7a44b6d65bfa43e291b06b0dfd38fab3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:01.221292 6849 raft_consensus.cc:1261] T 3416b6107e1848f299ac49ccd88ae7c8 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:01.221293 6587 raft_consensus.cc:1261] T ceca04391cd446ae932b3485d958237a P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:01.222183 9973 consensus_queue.cc:1035] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:01.222522 9973 raft_consensus.cc:1050] T 1e445a3cc928429caf9e3c6193a63010 P 80fd18ef4d604c86b909d5c008a46b4f: attempting to promote NON_VOTER 2a01ace80baf4057b6bf1995cf88468b to VOTER
I20240503 02:00:01.222908 9973 consensus_queue.cc:237] T 1e445a3cc928429caf9e3c6193a63010 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } }
I20240503 02:00:01.222983 6457 raft_consensus.cc:2932] T 7a44b6d65bfa43e291b06b0dfd38fab3 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:01.223412 6303 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7a44b6d65bfa43e291b06b0dfd38fab3 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:01.223628 8696 consensus_queue.cc:1035] T ceca04391cd446ae932b3485d958237a P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:01.224455 6458 raft_consensus.cc:1261] T 1e445a3cc928429caf9e3c6193a63010 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:01.224782 9179 raft_consensus.cc:1261] T 1e445a3cc928429caf9e3c6193a63010 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEARNER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:01.224705 8916 raft_consensus.cc:2932] T 7a44b6d65bfa43e291b06b0dfd38fab3 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:01.225346 10278 consensus_queue.cc:1035] T 1e445a3cc928429caf9e3c6193a63010 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:01.225370 6851 consensus_queue.cc:237] T 90ef3584830240acb0d8aad14ae851a6 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } }
I20240503 02:00:01.225684 10278 consensus_queue.cc:1035] T 1e445a3cc928429caf9e3c6193a63010 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:01.227780 6312 catalog_manager.cc:5484] T 7a44b6d65bfa43e291b06b0dfd38fab3 P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: config changed from index 5 to 6, VOTER 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:01.230232 9984 raft_consensus.cc:2932] T ceca04391cd446ae932b3485d958237a P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:01.231218 6569 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a44b6d65bfa43e291b06b0dfd38fab3 with delete_type TABLET_DATA_TOMBSTONED (TS 80fd18ef4d604c86b909d5c008a46b4f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59846
I20240503 02:00:01.231415 9179 raft_consensus.cc:1261] T 90ef3584830240acb0d8aad14ae851a6 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:01.231482 9473 tablet_replica.cc:330] T 7a44b6d65bfa43e291b06b0dfd38fab3 P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 02:00:01.231619 9473 raft_consensus.cc:2229] T 7a44b6d65bfa43e291b06b0dfd38fab3 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:01.231781 9473 raft_consensus.cc:2258] T 7a44b6d65bfa43e291b06b0dfd38fab3 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:01.231992 10212 consensus_queue.cc:1035] T 90ef3584830240acb0d8aad14ae851a6 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:01.232543 8833 raft_consensus.cc:2932] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:01.233850 9473 ts_tablet_manager.cc:1875] T 7a44b6d65bfa43e291b06b0dfd38fab3 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:01.235318 6302 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3416b6107e1848f299ac49ccd88ae7c8 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:01.236027 9473 ts_tablet_manager.cc:1888] T 7a44b6d65bfa43e291b06b0dfd38fab3 P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:00:01.236078 9473 log.cc:1196] T 7a44b6d65bfa43e291b06b0dfd38fab3 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/7a44b6d65bfa43e291b06b0dfd38fab3
I20240503 02:00:01.236757 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet 7a44b6d65bfa43e291b06b0dfd38fab3 (table rebalance_test_table_1 [id=6e24d367bc1d434584d8a7363cf538b6]) successfully deleted
I20240503 02:00:01.237303 6315 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:01.237612 10212 raft_consensus.cc:2932] T 90ef3584830240acb0d8aad14ae851a6 P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:01.237737 6587 raft_consensus.cc:2932] T ceca04391cd446ae932b3485d958237a P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:01.241349 9533 consensus_queue.cc:1035] T f28586e70bd9473fbdc6b9126b666fa9 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:01.241873 9973 raft_consensus.cc:2932] T 1e445a3cc928429caf9e3c6193a63010 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:01.242146 6312 catalog_manager.cc:5484] T 90ef3584830240acb0d8aad14ae851a6 P 492448570cc94ed1879feb2ec3b9051d reported cstate change: config changed from index 5 to 6, VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:01.242259 9309 raft_consensus.cc:2932] T 3416b6107e1848f299ac49ccd88ae7c8 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:01.242908 6310 catalog_manager.cc:5484] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: config changed from index 5 to 6, VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:01.243618 9178 raft_consensus.cc:2932] T 1e445a3cc928429caf9e3c6193a63010 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:01.243947 6851 raft_consensus.cc:2932] T ceca04391cd446ae932b3485d958237a P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:01.245879 6850 raft_consensus.cc:1261] T 1e445a3cc928429caf9e3c6193a63010 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:01.246737 6315 catalog_manager.cc:5484] T ceca04391cd446ae932b3485d958237a P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: config changed from index 4 to 5, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:01.246994 6303 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 90ef3584830240acb0d8aad14ae851a6 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:01.247941 6719 raft_consensus.cc:1261] T 90ef3584830240acb0d8aad14ae851a6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:01.248164 6458 raft_consensus.cc:2932] T 1e445a3cc928429caf9e3c6193a63010 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:01.248939 9179 raft_consensus.cc:2932] T 90ef3584830240acb0d8aad14ae851a6 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:01.249342 6308 catalog_manager.cc:5484] T 1e445a3cc928429caf9e3c6193a63010 P 2a01ace80baf4057b6bf1995cf88468b reported cstate change: config changed from index 4 to 5, 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:01.250743 9472 tablet.cc:1722] T e555f9ea8c7e409fbd7b0ee7d6d07277 P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:01.252079 10179 tablet.cc:1722] T e555f9ea8c7e409fbd7b0ee7d6d07277 P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:01.252369 10184 tablet.cc:1722] T e555f9ea8c7e409fbd7b0ee7d6d07277 P c2ac2eb7cc264047b9ab32fe3a4fc025: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:01.253759 9631 consensus_queue.cc:1035] T 90ef3584830240acb0d8aad14ae851a6 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:01.254249 10278 consensus_queue.cc:1035] T 1e445a3cc928429caf9e3c6193a63010 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:01.256639 6850 raft_consensus.cc:2932] T 3416b6107e1848f299ac49ccd88ae7c8 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:01.257668 6438 tablet_service.cc:1514] Processing DeleteTablet for tablet 90ef3584830240acb0d8aad14ae851a6 with delete_type TABLET_DATA_TOMBSTONED (TS dec3e4a04d604f7eaaa3f077d081218a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51338
I20240503 02:00:01.257913 9474 tablet_replica.cc:330] T 90ef3584830240acb0d8aad14ae851a6 P dec3e4a04d604f7eaaa3f077d081218a: stopping tablet replica
I20240503 02:00:01.258060 9474 raft_consensus.cc:2229] T 90ef3584830240acb0d8aad14ae851a6 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:01.258250 9474 raft_consensus.cc:2258] T 90ef3584830240acb0d8aad14ae851a6 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:01.258639 6849 raft_consensus.cc:2932] T 1e445a3cc928429caf9e3c6193a63010 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:01.259336 9474 ts_tablet_manager.cc:1875] T 90ef3584830240acb0d8aad14ae851a6 P dec3e4a04d604f7eaaa3f077d081218a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:01.260560 6700 tablet_service.cc:1514] Processing DeleteTablet for tablet 3416b6107e1848f299ac49ccd88ae7c8 with delete_type TABLET_DATA_TOMBSTONED (TS c2ac2eb7cc264047b9ab32fe3a4fc025 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:40336
I20240503 02:00:01.260838 10173 tablet_replica.cc:330] T 3416b6107e1848f299ac49ccd88ae7c8 P c2ac2eb7cc264047b9ab32fe3a4fc025: stopping tablet replica
I20240503 02:00:01.260972 10173 raft_consensus.cc:2229] T 3416b6107e1848f299ac49ccd88ae7c8 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:01.260891 6720 raft_consensus.cc:2932] T 90ef3584830240acb0d8aad14ae851a6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:01.261119 10173 raft_consensus.cc:2258] T 3416b6107e1848f299ac49ccd88ae7c8 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:01.261632 9474 ts_tablet_manager.cc:1888] T 90ef3584830240acb0d8aad14ae851a6 P dec3e4a04d604f7eaaa3f077d081218a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:00:01.261689 9474 log.cc:1196] T 90ef3584830240acb0d8aad14ae851a6 P dec3e4a04d604f7eaaa3f077d081218a: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/90ef3584830240acb0d8aad14ae851a6
I20240503 02:00:01.262182 6300 catalog_manager.cc:4830] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): tablet 90ef3584830240acb0d8aad14ae851a6 (table rebalance_test_table_5 [id=9b77b848294749e09f0854d24da5d9f5]) successfully deleted
I20240503 02:00:01.262794 10173 ts_tablet_manager.cc:1875] T 3416b6107e1848f299ac49ccd88ae7c8 P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:01.262867 6314 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=3295265fc41646e7942b5cd0d2d8ab20] alter complete (version 3)
I20240503 02:00:01.266263 10173 ts_tablet_manager.cc:1888] T 3416b6107e1848f299ac49ccd88ae7c8 P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:00:01.266463 10173 log.cc:1196] T 3416b6107e1848f299ac49ccd88ae7c8 P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/3416b6107e1848f299ac49ccd88ae7c8
I20240503 02:00:01.266405 8917 raft_consensus.cc:2932] T ceca04391cd446ae932b3485d958237a P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:01.268189 6587 consensus_queue.cc:237] T 1e445a3cc928429caf9e3c6193a63010 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } }
I20240503 02:00:01.269526 9178 raft_consensus.cc:1261] T 1e445a3cc928429caf9e3c6193a63010 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:01.269526 6458 raft_consensus.cc:1261] T 1e445a3cc928429caf9e3c6193a63010 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:01.269928 8833 consensus_queue.cc:1035] T 1e445a3cc928429caf9e3c6193a63010 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:01.270012 6303 catalog_manager.cc:4830] TS c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655): tablet 3416b6107e1848f299ac49ccd88ae7c8 (table rebalance_test_table_7 [id=e0e5e479de824fb18516cfe56d5892c6]) successfully deleted
I20240503 02:00:01.270512 9973 consensus_queue.cc:1035] T 1e445a3cc928429caf9e3c6193a63010 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:01.271701 6720 consensus_queue.cc:237] T ceca04391cd446ae932b3485d958237a P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } }
I20240503 02:00:01.271667 9973 raft_consensus.cc:2932] T 1e445a3cc928429caf9e3c6193a63010 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:01.273314 6849 raft_consensus.cc:1261] T ceca04391cd446ae932b3485d958237a P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:01.273291 9178 raft_consensus.cc:2932] T 1e445a3cc928429caf9e3c6193a63010 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:01.275372 8696 consensus_queue.cc:1035] T ceca04391cd446ae932b3485d958237a P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:01.275687 8917 raft_consensus.cc:1261] T ceca04391cd446ae932b3485d958237a P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:01.275821 6311 catalog_manager.cc:5484] T 1e445a3cc928429caf9e3c6193a63010 P 2a01ace80baf4057b6bf1995cf88468b reported cstate change: config changed from index 5 to 6, VOTER 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:01.276074 8696 consensus_queue.cc:1035] T ceca04391cd446ae932b3485d958237a P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:01.277287 8696 raft_consensus.cc:2932] T ceca04391cd446ae932b3485d958237a P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:01.279156 6849 raft_consensus.cc:2932] T ceca04391cd446ae932b3485d958237a P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:01.279548 6314 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_extra_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:01.280227 6458 raft_consensus.cc:2932] T 1e445a3cc928429caf9e3c6193a63010 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:01.281647 8917 raft_consensus.cc:2932] T ceca04391cd446ae932b3485d958237a P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:01.282059 6302 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1e445a3cc928429caf9e3c6193a63010 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:01.282312 6303 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ceca04391cd446ae932b3485d958237a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:01.283041 6310 catalog_manager.cc:5484] T ceca04391cd446ae932b3485d958237a P 492448570cc94ed1879feb2ec3b9051d reported cstate change: config changed from index 5 to 6, VOTER 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:01.287026 6569 tablet_service.cc:1514] Processing DeleteTablet for tablet ceca04391cd446ae932b3485d958237a with delete_type TABLET_DATA_TOMBSTONED (TS 80fd18ef4d604c86b909d5c008a46b4f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59846
I20240503 02:00:01.287269 9473 tablet_replica.cc:330] T ceca04391cd446ae932b3485d958237a P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 02:00:01.287446 9473 raft_consensus.cc:2229] T ceca04391cd446ae932b3485d958237a P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:01.287606 9473 raft_consensus.cc:2258] T ceca04391cd446ae932b3485d958237a P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:01.288673 9158 tablet_service.cc:1467] Processing CreateTablet for tablet 925612b660ec471da5b3c548802ede6e (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=0392495bb8a743099943f2bba5f15608]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:01.288728 6569 tablet_service.cc:1467] Processing CreateTablet for tablet cb873018448549ccbe3d14b784e47d26 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=0392495bb8a743099943f2bba5f15608]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:01.288975 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb873018448549ccbe3d14b784e47d26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:01.288975 9158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 925612b660ec471da5b3c548802ede6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:01.289047 9159 tablet_service.cc:1467] Processing CreateTablet for tablet 96c395771fd44056abacceb6c1acdb9f (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=0392495bb8a743099943f2bba5f15608]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:01.289281 9159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96c395771fd44056abacceb6c1acdb9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:01.290810 8880 tablet_service.cc:1467] Processing CreateTablet for tablet cb873018448549ccbe3d14b784e47d26 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=0392495bb8a743099943f2bba5f15608]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:01.291116 8880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb873018448549ccbe3d14b784e47d26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:01.291462 6567 tablet_service.cc:1467] Processing CreateTablet for tablet 925612b660ec471da5b3c548802ede6e (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=0392495bb8a743099943f2bba5f15608]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:01.291721 6567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 925612b660ec471da5b3c548802ede6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:01.291803 9027 tablet_service.cc:1467] Processing CreateTablet for tablet cb873018448549ccbe3d14b784e47d26 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=0392495bb8a743099943f2bba5f15608]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:01.292057 9027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb873018448549ccbe3d14b784e47d26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:01.293771 9473 ts_tablet_manager.cc:1875] T ceca04391cd446ae932b3485d958237a P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:01.293879 6830 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e445a3cc928429caf9e3c6193a63010 with delete_type TABLET_DATA_TOMBSTONED (TS 492448570cc94ed1879feb2ec3b9051d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52812
I20240503 02:00:01.294111 9476 tablet_replica.cc:330] T 1e445a3cc928429caf9e3c6193a63010 P 492448570cc94ed1879feb2ec3b9051d: stopping tablet replica
I20240503 02:00:01.294070 8880 tablet_service.cc:1467] Processing CreateTablet for tablet 925612b660ec471da5b3c548802ede6e (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=0392495bb8a743099943f2bba5f15608]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:01.294248 9476 raft_consensus.cc:2229] T 1e445a3cc928429caf9e3c6193a63010 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:01.294307 8880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 925612b660ec471da5b3c548802ede6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:01.294407 9476 raft_consensus.cc:2258] T 1e445a3cc928429caf9e3c6193a63010 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:01.294430 9025 tablet_service.cc:1467] Processing CreateTablet for tablet 96c395771fd44056abacceb6c1acdb9f (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=0392495bb8a743099943f2bba5f15608]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:01.294656 9025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96c395771fd44056abacceb6c1acdb9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:01.291527 10123 tablet_bootstrap.cc:492] T cb873018448549ccbe3d14b784e47d26 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:01.295857 9473 ts_tablet_manager.cc:1888] T ceca04391cd446ae932b3485d958237a P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:00:01.295926 9473 log.cc:1196] T ceca04391cd446ae932b3485d958237a P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/ceca04391cd446ae932b3485d958237a
I20240503 02:00:01.295958 10123 tablet_bootstrap.cc:654] T cb873018448549ccbe3d14b784e47d26 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:01.297089 9476 ts_tablet_manager.cc:1875] T 1e445a3cc928429caf9e3c6193a63010 P 492448570cc94ed1879feb2ec3b9051d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:01.297284 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet ceca04391cd446ae932b3485d958237a (table rebalance_test_table_14 [id=7455f5480e88494b83e1ac1d180ca4e6]) successfully deleted
I20240503 02:00:01.297588 10315 tablet_bootstrap.cc:492] T cb873018448549ccbe3d14b784e47d26 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:01.299567 9476 ts_tablet_manager.cc:1888] T 1e445a3cc928429caf9e3c6193a63010 P 492448570cc94ed1879feb2ec3b9051d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:00:01.299629 10316 tablet_bootstrap.cc:492] T cb873018448549ccbe3d14b784e47d26 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:01.299660 9476 log.cc:1196] T 1e445a3cc928429caf9e3c6193a63010 P 492448570cc94ed1879feb2ec3b9051d: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/1e445a3cc928429caf9e3c6193a63010
I20240503 02:00:01.300705 10316 tablet_bootstrap.cc:654] T cb873018448549ccbe3d14b784e47d26 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:01.300874 6303 catalog_manager.cc:4830] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719): tablet 1e445a3cc928429caf9e3c6193a63010 (table rebalance_test_table_19 [id=250f0720cafa4425b45a0360ebe04a03]) successfully deleted
I20240503 02:00:01.300952 10315 tablet_bootstrap.cc:654] T cb873018448549ccbe3d14b784e47d26 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:01.301457 10314 tablet_bootstrap.cc:492] T 925612b660ec471da5b3c548802ede6e P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:01.302240 9309 raft_consensus.cc:1203] T 0b659e9d57db4ebf93c9143938970090 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:00:01.302960 8880 tablet_service.cc:1467] Processing CreateTablet for tablet 96c395771fd44056abacceb6c1acdb9f (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=0392495bb8a743099943f2bba5f15608]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:01.303263 8880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96c395771fd44056abacceb6c1acdb9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:01.303396 10314 tablet_bootstrap.cc:654] T 925612b660ec471da5b3c548802ede6e P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:01.304945 10123 tablet_bootstrap.cc:492] T cb873018448549ccbe3d14b784e47d26 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:01.305044 10123 ts_tablet_manager.cc:1367] T cb873018448549ccbe3d14b784e47d26 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 02:00:01.305212 10316 tablet_bootstrap.cc:492] T cb873018448549ccbe3d14b784e47d26 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:01.305306 10316 ts_tablet_manager.cc:1367] T cb873018448549ccbe3d14b784e47d26 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:00:01.305650 10123 raft_consensus.cc:348] T cb873018448549ccbe3d14b784e47d26 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:01.305769 10123 raft_consensus.cc:374] T cb873018448549ccbe3d14b784e47d26 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:01.305799 10123 raft_consensus.cc:729] T cb873018448549ccbe3d14b784e47d26 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:01.305946 10123 consensus_queue.cc:260] T cb873018448549ccbe3d14b784e47d26 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:01.305833 10316 raft_consensus.cc:348] T cb873018448549ccbe3d14b784e47d26 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:01.306142 10316 raft_consensus.cc:374] T cb873018448549ccbe3d14b784e47d26 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:01.306177 10316 raft_consensus.cc:729] T cb873018448549ccbe3d14b784e47d26 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:01.306227 10123 ts_tablet_manager.cc:1398] T cb873018448549ccbe3d14b784e47d26 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:01.306325 10316 consensus_queue.cc:260] T cb873018448549ccbe3d14b784e47d26 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:01.306448 10123 tablet_bootstrap.cc:492] T 925612b660ec471da5b3c548802ede6e P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:01.307726 10315 tablet_bootstrap.cc:492] T cb873018448549ccbe3d14b784e47d26 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:01.307917 10315 ts_tablet_manager.cc:1367] T cb873018448549ccbe3d14b784e47d26 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 02:00:01.310139 10315 raft_consensus.cc:348] T cb873018448549ccbe3d14b784e47d26 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:01.310582 10315 raft_consensus.cc:374] T cb873018448549ccbe3d14b784e47d26 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:01.311019 9984 raft_consensus.cc:1050] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025: attempting to promote NON_VOTER 2a01ace80baf4057b6bf1995cf88468b to VOTER
I20240503 02:00:01.311328 10123 tablet_bootstrap.cc:654] T 925612b660ec471da5b3c548802ede6e P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:01.311471 10315 raft_consensus.cc:729] T cb873018448549ccbe3d14b784e47d26 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:01.311471 9984 consensus_queue.cc:237] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } }
I20240503 02:00:01.311802 10315 consensus_queue.cc:260] T cb873018448549ccbe3d14b784e47d26 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:01.312525 10316 ts_tablet_manager.cc:1398] T cb873018448549ccbe3d14b784e47d26 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:00:01.312857 10123 tablet_bootstrap.cc:492] T 925612b660ec471da5b3c548802ede6e P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:01.312932 10315 ts_tablet_manager.cc:1398] T cb873018448549ccbe3d14b784e47d26 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:00:01.312953 10123 ts_tablet_manager.cc:1367] T 925612b660ec471da5b3c548802ede6e P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:00:01.313110 10315 tablet_bootstrap.cc:492] T 925612b660ec471da5b3c548802ede6e P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:01.313429 10123 raft_consensus.cc:348] T 925612b660ec471da5b3c548802ede6e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:01.313534 10123 raft_consensus.cc:374] T 925612b660ec471da5b3c548802ede6e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:01.313565 10123 raft_consensus.cc:729] T 925612b660ec471da5b3c548802ede6e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:01.313673 10123 consensus_queue.cc:260] T 925612b660ec471da5b3c548802ede6e P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:01.313722 6849 raft_consensus.cc:1261] T 1feaf29d8c6b4842829ce66e0f4ec23b P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:01.313767 9178 raft_consensus.cc:1261] T 1feaf29d8c6b4842829ce66e0f4ec23b P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEARNER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:01.313946 10123 ts_tablet_manager.cc:1398] T 925612b660ec471da5b3c548802ede6e P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:01.314038 10315 tablet_bootstrap.cc:654] T 925612b660ec471da5b3c548802ede6e P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:01.314419 8696 consensus_queue.cc:1035] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:01.314850 6458 raft_consensus.cc:1261] T 1feaf29d8c6b4842829ce66e0f4ec23b P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:01.315281 8696 consensus_queue.cc:1035] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:01.315956 10315 tablet_bootstrap.cc:492] T 925612b660ec471da5b3c548802ede6e P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:01.316098 10315 ts_tablet_manager.cc:1367] T 925612b660ec471da5b3c548802ede6e P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:01.316895 10315 raft_consensus.cc:348] T 925612b660ec471da5b3c548802ede6e P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:01.317190 8696 consensus_queue.cc:1035] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:01.317309 10315 raft_consensus.cc:374] T 925612b660ec471da5b3c548802ede6e P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:01.317358 10315 raft_consensus.cc:729] T 925612b660ec471da5b3c548802ede6e P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:01.317521 10315 consensus_queue.cc:260] T 925612b660ec471da5b3c548802ede6e P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:01.318135 10315 ts_tablet_manager.cc:1398] T 925612b660ec471da5b3c548802ede6e P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:01.318387 10315 tablet_bootstrap.cc:492] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:01.319833 10316 tablet_bootstrap.cc:492] T 96c395771fd44056abacceb6c1acdb9f P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:01.319743 8696 raft_consensus.cc:2932] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:01.322943 6314 catalog_manager.cc:5484] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: config changed from index 4 to 5, 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:01.323164 6458 raft_consensus.cc:2932] T 1feaf29d8c6b4842829ce66e0f4ec23b P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:01.323173 9178 raft_consensus.cc:2932] T 1feaf29d8c6b4842829ce66e0f4ec23b P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:01.323746 10315 tablet_bootstrap.cc:654] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:01.326658 9484 raft_consensus.cc:1050] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f: attempting to promote NON_VOTER 9942306855404cfd891adf227dd65eb2 to VOTER
I20240503 02:00:01.327101 9484 consensus_queue.cc:237] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } }
I20240503 02:00:01.327662 10316 tablet_bootstrap.cc:654] T 96c395771fd44056abacceb6c1acdb9f P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:01.328841 6720 consensus_queue.cc:237] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } }
I20240503 02:00:01.329913 10314 tablet_bootstrap.cc:492] T 925612b660ec471da5b3c548802ede6e P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:01.330013 10314 ts_tablet_manager.cc:1367] T 925612b660ec471da5b3c548802ede6e P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.000s
I20240503 02:00:01.330430 6458 raft_consensus.cc:1261] T 1feaf29d8c6b4842829ce66e0f4ec23b P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:01.330457 10314 raft_consensus.cc:348] T 925612b660ec471da5b3c548802ede6e P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:01.330569 10314 raft_consensus.cc:374] T 925612b660ec471da5b3c548802ede6e P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:01.330600 10314 raft_consensus.cc:729] T 925612b660ec471da5b3c548802ede6e P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:01.330718 10314 consensus_queue.cc:260] T 925612b660ec471da5b3c548802ede6e P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:01.330834 9178 raft_consensus.cc:1261] T 1feaf29d8c6b4842829ce66e0f4ec23b P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:01.331032 10314 ts_tablet_manager.cc:1398] T 925612b660ec471da5b3c548802ede6e P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:01.331130 10315 tablet_bootstrap.cc:492] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:01.331229 10314 tablet_bootstrap.cc:492] T 96c395771fd44056abacceb6c1acdb9f P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:01.331254 10315 ts_tablet_manager.cc:1367] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.002s
I20240503 02:00:01.331208 10268 consensus_queue.cc:1035] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:01.331476 10268 consensus_queue.cc:1035] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:01.331713 10315 raft_consensus.cc:348] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:01.331815 10315 raft_consensus.cc:374] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:01.331841 10315 raft_consensus.cc:729] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:01.331964 10315 consensus_queue.cc:260] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:01.332101 10314 tablet_bootstrap.cc:654] T 96c395771fd44056abacceb6c1acdb9f P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:01.332283 10315 ts_tablet_manager.cc:1398] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:01.333364 8917 raft_consensus.cc:1261] T 915028b3478341d588ed6031e657c259 P 9942306855404cfd891adf227dd65eb2 [term 1 LEARNER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:01.333914 8833 consensus_queue.cc:1035] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:01.333976 6850 raft_consensus.cc:1261] T 915028b3478341d588ed6031e657c259 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:01.333976 6720 raft_consensus.cc:1261] T 915028b3478341d588ed6031e657c259 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:01.334403 8833 consensus_queue.cc:1035] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:01.334637 10278 consensus_queue.cc:1035] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:01.335281 9711 raft_consensus.cc:2932] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:01.335743 6458 raft_consensus.cc:2932] T 1feaf29d8c6b4842829ce66e0f4ec23b P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:01.337147 10316 tablet_bootstrap.cc:492] T 96c395771fd44056abacceb6c1acdb9f P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:01.337245 10316 ts_tablet_manager.cc:1367] T 96c395771fd44056abacceb6c1acdb9f P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240503 02:00:01.337735 10316 raft_consensus.cc:348] T 96c395771fd44056abacceb6c1acdb9f P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:01.337841 10316 raft_consensus.cc:374] T 96c395771fd44056abacceb6c1acdb9f P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:01.337868 10316 raft_consensus.cc:729] T 96c395771fd44056abacceb6c1acdb9f P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:01.338078 10316 consensus_queue.cc:260] T 96c395771fd44056abacceb6c1acdb9f P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:01.338289 6303 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1feaf29d8c6b4842829ce66e0f4ec23b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:01.338553 6311 catalog_manager.cc:5484] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: config changed from index 5 to 6, VOTER 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:01.339956 10316 ts_tablet_manager.cc:1398] T 96c395771fd44056abacceb6c1acdb9f P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:00:01.342357 6830 tablet_service.cc:1514] Processing DeleteTablet for tablet 1feaf29d8c6b4842829ce66e0f4ec23b with delete_type TABLET_DATA_TOMBSTONED (TS 492448570cc94ed1879feb2ec3b9051d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52812
I20240503 02:00:01.342717 9476 tablet_replica.cc:330] T 1feaf29d8c6b4842829ce66e0f4ec23b P 492448570cc94ed1879feb2ec3b9051d: stopping tablet replica
I20240503 02:00:01.342588 9178 raft_consensus.cc:2932] T 1feaf29d8c6b4842829ce66e0f4ec23b P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:01.342854 9476 raft_consensus.cc:2229] T 1feaf29d8c6b4842829ce66e0f4ec23b P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:01.342978 9476 pending_rounds.cc:62] T 1feaf29d8c6b4842829ce66e0f4ec23b P 492448570cc94ed1879feb2ec3b9051d: Trying to abort 1 pending ops.
I20240503 02:00:01.343040 9476 pending_rounds.cc:68] T 1feaf29d8c6b4842829ce66e0f4ec23b P 492448570cc94ed1879feb2ec3b9051d: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023417758970761216 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "1feaf29d8c6b4842829ce66e0f4ec23b" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } } }
I20240503 02:00:01.343187 9476 raft_consensus.cc:2894] T 1feaf29d8c6b4842829ce66e0f4ec23b P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240503 02:00:01.343245 9476 raft_consensus.cc:2866] T 1feaf29d8c6b4842829ce66e0f4ec23b P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 02:00:01.343286 9476 raft_consensus.cc:2258] T 1feaf29d8c6b4842829ce66e0f4ec23b P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:01.343796 10314 tablet_bootstrap.cc:492] T 96c395771fd44056abacceb6c1acdb9f P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:01.343890 10314 ts_tablet_manager.cc:1367] T 96c395771fd44056abacceb6c1acdb9f P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.002s
I20240503 02:00:01.344358 10314 raft_consensus.cc:348] T 96c395771fd44056abacceb6c1acdb9f P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:01.344498 10314 raft_consensus.cc:374] T 96c395771fd44056abacceb6c1acdb9f P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:01.344532 10314 raft_consensus.cc:729] T 96c395771fd44056abacceb6c1acdb9f P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:01.344599 10278 raft_consensus.cc:1050] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 80fd18ef4d604c86b909d5c008a46b4f: attempting to promote NON_VOTER 99ead02915954b4fab23b033ad6d546b to VOTER
I20240503 02:00:01.344651 10314 consensus_queue.cc:260] T 96c395771fd44056abacceb6c1acdb9f P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:01.344914 10314 ts_tablet_manager.cc:1398] T 96c395771fd44056abacceb6c1acdb9f P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:01.345046 10278 consensus_queue.cc:237] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } }
I20240503 02:00:01.345472 9484 raft_consensus.cc:2932] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:01.346612 8917 raft_consensus.cc:2932] T 915028b3478341d588ed6031e657c259 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:01.346804 6720 raft_consensus.cc:1261] T 74ebf5d0fb9a42e6a7ec743a98960e13 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:01.348577 6309 catalog_manager.cc:5484] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: config changed from index 4 to 5, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:01.349148 6849 raft_consensus.cc:2932] T 915028b3478341d588ed6031e657c259 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:01.349557 9048 raft_consensus.cc:1261] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 99ead02915954b4fab23b033ad6d546b [term 1 LEARNER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:01.350224 6720 raft_consensus.cc:2932] T 915028b3478341d588ed6031e657c259 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:01.352178 6850 raft_consensus.cc:1261] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:01.352541 9973 consensus_queue.cc:1035] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:01.352638 9711 raft_consensus.cc:1050] T 3cf186a0901f420390c8fda54316c5af P c2ac2eb7cc264047b9ab32fe3a4fc025: attempting to promote NON_VOTER 99ead02915954b4fab23b033ad6d546b to VOTER
I20240503 02:00:01.353034 9973 consensus_queue.cc:1035] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:01.353070 9711 consensus_queue.cc:237] T 3cf186a0901f420390c8fda54316c5af P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } }
I20240503 02:00:01.353358 9973 consensus_queue.cc:1035] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:01.354832 6458 raft_consensus.cc:1261] T 3cf186a0901f420390c8fda54316c5af P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:01.355284 8696 consensus_queue.cc:1035] T 3cf186a0901f420390c8fda54316c5af P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:01.355832 6850 raft_consensus.cc:1261] T 3cf186a0901f420390c8fda54316c5af P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:01.356762 8696 consensus_queue.cc:1035] T 3cf186a0901f420390c8fda54316c5af P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:01.358070 9039 raft_consensus.cc:1261] T 3cf186a0901f420390c8fda54316c5af P 99ead02915954b4fab23b033ad6d546b [term 1 LEARNER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:01.358541 8696 consensus_queue.cc:1035] T 3cf186a0901f420390c8fda54316c5af P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:01.359381 6589 consensus_queue.cc:237] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } }
I20240503 02:00:01.360720 6849 raft_consensus.cc:1261] T 915028b3478341d588ed6031e657c259 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:01.360800 8917 raft_consensus.cc:1261] T 915028b3478341d588ed6031e657c259 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:01.361107 9711 raft_consensus.cc:1050] T d82268925a934c559dfcf51a84f03d2e P c2ac2eb7cc264047b9ab32fe3a4fc025: attempting to promote NON_VOTER 99ead02915954b4fab23b033ad6d546b to VOTER
I20240503 02:00:01.361552 9711 consensus_queue.cc:237] T d82268925a934c559dfcf51a84f03d2e P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } }
I20240503 02:00:01.361490 9484 raft_consensus.cc:2932] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 99ead02915954b4fab23b033ad6d546b (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:01.364099 6456 raft_consensus.cc:1261] T d82268925a934c559dfcf51a84f03d2e P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:01.364255 9047 raft_consensus.cc:1261] T d82268925a934c559dfcf51a84f03d2e P 99ead02915954b4fab23b033ad6d546b [term 1 LEARNER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:01.364614 6589 raft_consensus.cc:1261] T d82268925a934c559dfcf51a84f03d2e P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:01.364735 9973 consensus_queue.cc:1035] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:01.365011 9711 consensus_queue.cc:1035] T d82268925a934c559dfcf51a84f03d2e P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:01.365072 9973 consensus_queue.cc:1035] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:01.365223 6314 catalog_manager.cc:5484] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: config changed from index 4 to 5, 99ead02915954b4fab23b033ad6d546b (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:01.366103 9984 consensus_queue.cc:1035] T d82268925a934c559dfcf51a84f03d2e P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:01.366343 6849 raft_consensus.cc:2932] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 99ead02915954b4fab23b033ad6d546b (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:01.366142 6720 raft_consensus.cc:2932] T 74ebf5d0fb9a42e6a7ec743a98960e13 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 99ead02915954b4fab23b033ad6d546b (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:01.377113 6311 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:01.377287 6311 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:01.378973 9476 ts_tablet_manager.cc:1875] T 1feaf29d8c6b4842829ce66e0f4ec23b P 492448570cc94ed1879feb2ec3b9051d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:01.379662 8833 raft_consensus.cc:2932] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:01.380455 9711 consensus_queue.cc:1035] T d82268925a934c559dfcf51a84f03d2e P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:01.381057 9711 raft_consensus.cc:2932] T d82268925a934c559dfcf51a84f03d2e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 99ead02915954b4fab23b033ad6d546b (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:01.381758 9476 ts_tablet_manager.cc:1888] T 1feaf29d8c6b4842829ce66e0f4ec23b P 492448570cc94ed1879feb2ec3b9051d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:00:01.381834 9476 log.cc:1196] T 1feaf29d8c6b4842829ce66e0f4ec23b P 492448570cc94ed1879feb2ec3b9051d: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/1feaf29d8c6b4842829ce66e0f4ec23b
I20240503 02:00:01.382592 6303 catalog_manager.cc:4830] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719): tablet 1feaf29d8c6b4842829ce66e0f4ec23b (table rebalance_test_table_13 [id=029e2476b8c44a8d8cdb98005eb84a4b]) successfully deleted
I20240503 02:00:01.383599 6302 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 915028b3478341d588ed6031e657c259 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:01.383850 6456 raft_consensus.cc:2932] T d82268925a934c559dfcf51a84f03d2e P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 99ead02915954b4fab23b033ad6d546b (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:01.384480 6850 raft_consensus.cc:2932] T 915028b3478341d588ed6031e657c259 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:01.385550 8917 raft_consensus.cc:2932] T 915028b3478341d588ed6031e657c259 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:01.385993 9047 raft_consensus.cc:2932] T d82268925a934c559dfcf51a84f03d2e P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 99ead02915954b4fab23b033ad6d546b (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:01.386509 6314 catalog_manager.cc:5484] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: config changed from index 5 to 6, VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:01.388645 6315 catalog_manager.cc:5484] T d82268925a934c559dfcf51a84f03d2e P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: config changed from index 4 to 5, 99ead02915954b4fab23b033ad6d546b (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:01.389560 6311 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet e555f9ea8c7e409fbd7b0ee7d6d07277
I20240503 02:00:01.390035 9025 tablet_service.cc:1514] Processing DeleteTablet for tablet e555f9ea8c7e409fbd7b0ee7d6d07277 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:01 UTC) from {username='slave'} at 127.0.0.1:52584
I20240503 02:00:01.390465 6567 tablet_service.cc:1514] Processing DeleteTablet for tablet e555f9ea8c7e409fbd7b0ee7d6d07277 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:01 UTC) from {username='slave'} at 127.0.0.1:59846
I20240503 02:00:01.390645 9473 tablet_replica.cc:330] T e555f9ea8c7e409fbd7b0ee7d6d07277 P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 02:00:01.390771 9473 raft_consensus.cc:2229] T e555f9ea8c7e409fbd7b0ee7d6d07277 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:01.390959 9473 raft_consensus.cc:2258] T e555f9ea8c7e409fbd7b0ee7d6d07277 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:01.391583 9473 ts_tablet_manager.cc:1875] T e555f9ea8c7e409fbd7b0ee7d6d07277 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:01.392410 6700 tablet_service.cc:1514] Processing DeleteTablet for tablet 915028b3478341d588ed6031e657c259 with delete_type TABLET_DATA_TOMBSTONED (TS c2ac2eb7cc264047b9ab32fe3a4fc025 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:40336
I20240503 02:00:01.392477 6699 tablet_service.cc:1514] Processing DeleteTablet for tablet e555f9ea8c7e409fbd7b0ee7d6d07277 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:01 UTC) from {username='slave'} at 127.0.0.1:40336
I20240503 02:00:01.392803 10173 tablet_replica.cc:330] T 915028b3478341d588ed6031e657c259 P c2ac2eb7cc264047b9ab32fe3a4fc025: stopping tablet replica
I20240503 02:00:01.392943 10173 raft_consensus.cc:2229] T 915028b3478341d588ed6031e657c259 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:01.393093 10173 raft_consensus.cc:2258] T 915028b3478341d588ed6031e657c259 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:01.393713 10173 ts_tablet_manager.cc:1875] T 915028b3478341d588ed6031e657c259 P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:01.395928 10173 ts_tablet_manager.cc:1888] T 915028b3478341d588ed6031e657c259 P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:00:01.396006 10173 log.cc:1196] T 915028b3478341d588ed6031e657c259 P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/915028b3478341d588ed6031e657c259
I20240503 02:00:01.396174 9473 ts_tablet_manager.cc:1888] T e555f9ea8c7e409fbd7b0ee7d6d07277 P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:01.396265 9473 log.cc:1196] T e555f9ea8c7e409fbd7b0ee7d6d07277 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/e555f9ea8c7e409fbd7b0ee7d6d07277
I20240503 02:00:01.396418 10173 tablet_replica.cc:330] T e555f9ea8c7e409fbd7b0ee7d6d07277 P c2ac2eb7cc264047b9ab32fe3a4fc025: stopping tablet replica
I20240503 02:00:01.396498 6303 catalog_manager.cc:4830] TS c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655): tablet 915028b3478341d588ed6031e657c259 (table rebalance_test_table_3 [id=72f29547da0b4a95974018c2782cb359]) successfully deleted
I20240503 02:00:01.396564 9473 ts_tablet_manager.cc:1909] T e555f9ea8c7e409fbd7b0ee7d6d07277 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting consensus metadata
I20240503 02:00:01.396557 10173 raft_consensus.cc:2229] T e555f9ea8c7e409fbd7b0ee7d6d07277 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:01.396683 10173 raft_consensus.cc:2258] T e555f9ea8c7e409fbd7b0ee7d6d07277 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:01.397234 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet e555f9ea8c7e409fbd7b0ee7d6d07277 (table rebalancer_dynamic_table_new_name [id=3295265fc41646e7942b5cd0d2d8ab20]) successfully deleted
I20240503 02:00:01.397487 10173 ts_tablet_manager.cc:1875] T e555f9ea8c7e409fbd7b0ee7d6d07277 P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:01.399405 10173 ts_tablet_manager.cc:1888] T e555f9ea8c7e409fbd7b0ee7d6d07277 P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:01.399559 10173 log.cc:1196] T e555f9ea8c7e409fbd7b0ee7d6d07277 P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/e555f9ea8c7e409fbd7b0ee7d6d07277
I20240503 02:00:01.399616 10349 raft_consensus.cc:2932] T 3cf186a0901f420390c8fda54316c5af P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 99ead02915954b4fab23b033ad6d546b (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:01.399960 10173 ts_tablet_manager.cc:1909] T e555f9ea8c7e409fbd7b0ee7d6d07277 P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting consensus metadata
I20240503 02:00:01.400836 6303 catalog_manager.cc:4830] TS c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655): tablet e555f9ea8c7e409fbd7b0ee7d6d07277 (table rebalancer_dynamic_table_new_name [id=3295265fc41646e7942b5cd0d2d8ab20]) successfully deleted
I20240503 02:00:01.401389 9039 raft_consensus.cc:2932] T 3cf186a0901f420390c8fda54316c5af P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 99ead02915954b4fab23b033ad6d546b (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:01.404600 6314 catalog_manager.cc:5484] T 3cf186a0901f420390c8fda54316c5af P 99ead02915954b4fab23b033ad6d546b reported cstate change: config changed from index 4 to 5, 99ead02915954b4fab23b033ad6d546b (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:01.406605 6850 raft_consensus.cc:2932] T 3cf186a0901f420390c8fda54316c5af P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 99ead02915954b4fab23b033ad6d546b (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:01.411371 6587 consensus_queue.cc:237] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } }
I20240503 02:00:01.411635 6720 consensus_queue.cc:237] T 3cf186a0901f420390c8fda54316c5af P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } }
W20240503 02:00:01.413015 6523 consensus_peers.cc:485] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9f7ade1e2af24e86b6e54e2b5769e4c0. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:01.413095 6456 raft_consensus.cc:1261] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:01.413484 9484 consensus_queue.cc:1035] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:01.413477 6850 raft_consensus.cc:1261] T 3cf186a0901f420390c8fda54316c5af P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:01.414156 10349 consensus_queue.cc:1035] T 3cf186a0901f420390c8fda54316c5af P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:01.414719 6720 raft_consensus.cc:1261] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:01.415071 9039 raft_consensus.cc:1261] T 3cf186a0901f420390c8fda54316c5af P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:01.415253 10360 tablet_replica.cc:330] T e555f9ea8c7e409fbd7b0ee7d6d07277 P 99ead02915954b4fab23b033ad6d546b: stopping tablet replica
I20240503 02:00:01.415385 10360 raft_consensus.cc:2229] T e555f9ea8c7e409fbd7b0ee7d6d07277 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:01.415403 9484 consensus_queue.cc:1035] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:01.415583 10347 consensus_queue.cc:1035] T 3cf186a0901f420390c8fda54316c5af P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:01.416445 10349 raft_consensus.cc:2932] T 3cf186a0901f420390c8fda54316c5af P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:01.416559 6850 raft_consensus.cc:2932] T 3cf186a0901f420390c8fda54316c5af P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:01.418886 6589 raft_consensus.cc:2932] T d82268925a934c559dfcf51a84f03d2e P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 99ead02915954b4fab23b033ad6d546b (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:01.420017 6309 catalog_manager.cc:5484] T 3cf186a0901f420390c8fda54316c5af P 492448570cc94ed1879feb2ec3b9051d reported cstate change: config changed from index 5 to 6, VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:01.421974 6303 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3cf186a0901f420390c8fda54316c5af with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:01.416256 9039 raft_consensus.cc:2932] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 99ead02915954b4fab23b033ad6d546b (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:01.422778 9484 raft_consensus.cc:2932] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } } }
I20240503 02:00:01.424400 6719 consensus_queue.cc:237] T d82268925a934c559dfcf51a84f03d2e P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } }
I20240503 02:00:01.425032 6456 raft_consensus.cc:2932] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } } }
I20240503 02:00:01.425312 8833 raft_consensus.cc:482] T 925612b660ec471da5b3c548802ede6e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:01.425396 8833 raft_consensus.cc:504] T 925612b660ec471da5b3c548802ede6e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:01.425737 8833 leader_election.cc:288] T 925612b660ec471da5b3c548802ede6e P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:01.419467 9048 raft_consensus.cc:2932] T 3cf186a0901f420390c8fda54316c5af P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:01.426244 8917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "925612b660ec471da5b3c548802ede6e" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:01.426375 8917 raft_consensus.cc:2454] T 925612b660ec471da5b3c548802ede6e P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:01.426983 6525 leader_election.cc:302] T 925612b660ec471da5b3c548802ede6e P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:01.427141 9484 raft_consensus.cc:2781] T 925612b660ec471da5b3c548802ede6e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:01.427197 9484 raft_consensus.cc:482] T 925612b660ec471da5b3c548802ede6e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:01.427232 9484 raft_consensus.cc:3037] T 925612b660ec471da5b3c548802ede6e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:01.428194 9484 raft_consensus.cc:504] T 925612b660ec471da5b3c548802ede6e P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:01.428597 9484 leader_election.cc:288] T 925612b660ec471da5b3c548802ede6e P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Requested vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:01.428622 6589 raft_consensus.cc:1261] T d82268925a934c559dfcf51a84f03d2e P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:01.428773 9048 raft_consensus.cc:1261] T d82268925a934c559dfcf51a84f03d2e P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:01.428813 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "925612b660ec471da5b3c548802ede6e" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:01.428947 9178 raft_consensus.cc:2454] T 925612b660ec471da5b3c548802ede6e P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:01.429055 10360 raft_consensus.cc:2258] T e555f9ea8c7e409fbd7b0ee7d6d07277 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:01.429319 10348 consensus_queue.cc:1035] T d82268925a934c559dfcf51a84f03d2e P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:01.430567 6312 catalog_manager.cc:5484] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: config changed from index -1 to 4, NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } } }
I20240503 02:00:01.430684 6438 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cf186a0901f420390c8fda54316c5af with delete_type TABLET_DATA_TOMBSTONED (TS dec3e4a04d604f7eaaa3f077d081218a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51338
I20240503 02:00:01.430904 9474 tablet_replica.cc:330] T 3cf186a0901f420390c8fda54316c5af P dec3e4a04d604f7eaaa3f077d081218a: stopping tablet replica
I20240503 02:00:01.431020 9474 raft_consensus.cc:2229] T 3cf186a0901f420390c8fda54316c5af P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:01.431145 9474 pending_rounds.cc:62] T 3cf186a0901f420390c8fda54316c5af P dec3e4a04d604f7eaaa3f077d081218a: Trying to abort 1 pending ops.
I20240503 02:00:01.431212 9474 pending_rounds.cc:68] T 3cf186a0901f420390c8fda54316c5af P dec3e4a04d604f7eaaa3f077d081218a: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023417759141130240 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "3cf186a0901f420390c8fda54316c5af" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } } }
I20240503 02:00:01.431353 9474 raft_consensus.cc:2894] T 3cf186a0901f420390c8fda54316c5af P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240503 02:00:01.431412 9474 raft_consensus.cc:2866] T 3cf186a0901f420390c8fda54316c5af P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 02:00:01.431452 9474 raft_consensus.cc:2258] T 3cf186a0901f420390c8fda54316c5af P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:01.432731 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "925612b660ec471da5b3c548802ede6e" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:01.432870 9178 raft_consensus.cc:3037] T 925612b660ec471da5b3c548802ede6e P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:01.433938 9178 raft_consensus.cc:2454] T 925612b660ec471da5b3c548802ede6e P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:01.434367 6522 leader_election.cc:302] T 925612b660ec471da5b3c548802ede6e P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 80fd18ef4d604c86b909d5c008a46b4f; no voters:
I20240503 02:00:01.434737 9474 ts_tablet_manager.cc:1875] T 3cf186a0901f420390c8fda54316c5af P dec3e4a04d604f7eaaa3f077d081218a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:01.435863 9484 raft_consensus.cc:2781] T 925612b660ec471da5b3c548802ede6e P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:01.435969 9484 raft_consensus.cc:686] T 925612b660ec471da5b3c548802ede6e P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Becoming Leader. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Running, Role: LEADER
I20240503 02:00:01.436094 9484 consensus_queue.cc:237] T 925612b660ec471da5b3c548802ede6e P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:01.436717 9474 ts_tablet_manager.cc:1888] T 3cf186a0901f420390c8fda54316c5af P dec3e4a04d604f7eaaa3f077d081218a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:00:01.436789 9474 log.cc:1196] T 3cf186a0901f420390c8fda54316c5af P dec3e4a04d604f7eaaa3f077d081218a: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/3cf186a0901f420390c8fda54316c5af
I20240503 02:00:01.437667 8917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "925612b660ec471da5b3c548802ede6e" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:01.437796 8917 raft_consensus.cc:3037] T 925612b660ec471da5b3c548802ede6e P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:01.437961 10360 ts_tablet_manager.cc:1875] T e555f9ea8c7e409fbd7b0ee7d6d07277 P 99ead02915954b4fab23b033ad6d546b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:01.438066 6582 consensus_queue.cc:237] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } }
I20240503 02:00:01.438257 10348 consensus_queue.cc:1035] T d82268925a934c559dfcf51a84f03d2e P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:01.438741 8917 raft_consensus.cc:2454] T 925612b660ec471da5b3c548802ede6e P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:01.439429 6720 raft_consensus.cc:2932] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } } }
I20240503 02:00:01.439771 6300 catalog_manager.cc:4830] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): tablet 3cf186a0901f420390c8fda54316c5af (table rebalance_test_table_11 [id=805c97ce87c74e019deaa2acee6a5329]) successfully deleted
I20240503 02:00:01.440165 10360 ts_tablet_manager.cc:1888] T e555f9ea8c7e409fbd7b0ee7d6d07277 P 99ead02915954b4fab23b033ad6d546b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:01.440235 10360 log.cc:1196] T e555f9ea8c7e409fbd7b0ee7d6d07277 P 99ead02915954b4fab23b033ad6d546b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/e555f9ea8c7e409fbd7b0ee7d6d07277
I20240503 02:00:01.440994 10360 ts_tablet_manager.cc:1909] T e555f9ea8c7e409fbd7b0ee7d6d07277 P 99ead02915954b4fab23b033ad6d546b: Deleting consensus metadata
I20240503 02:00:01.441133 9048 raft_consensus.cc:1261] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:01.441177 6720 raft_consensus.cc:1261] T 74ebf5d0fb9a42e6a7ec743a98960e13 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:01.441533 10278 consensus_queue.cc:1035] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:01.441825 6301 catalog_manager.cc:4830] TS 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157): tablet e555f9ea8c7e409fbd7b0ee7d6d07277 (table rebalancer_dynamic_table_new_name [id=3295265fc41646e7942b5cd0d2d8ab20]) successfully deleted
I20240503 02:00:01.441988 10345 consensus_queue.cc:1035] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:01.442051 8633 raft_consensus.cc:1050] T 8a76d3e2d4264687aa2972c179341b5e P dec3e4a04d604f7eaaa3f077d081218a: attempting to promote NON_VOTER 2a01ace80baf4057b6bf1995cf88468b to VOTER
I20240503 02:00:01.442534 8633 consensus_queue.cc:237] T 8a76d3e2d4264687aa2972c179341b5e P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } }
I20240503 02:00:01.443842 6311 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240503 02:00:01.444144 6719 raft_consensus.cc:1261] T 8a76d3e2d4264687aa2972c179341b5e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:01.444203 6850 raft_consensus.cc:1261] T 8a76d3e2d4264687aa2972c179341b5e P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:01.444109 10268 raft_consensus.cc:2932] T d82268925a934c559dfcf51a84f03d2e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:01.443976 6311 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240503 02:00:01.444682 9721 consensus_queue.cc:1035] T 8a76d3e2d4264687aa2972c179341b5e P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:01.445040 9721 consensus_queue.cc:1035] T 8a76d3e2d4264687aa2972c179341b5e P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:01.445703 9178 raft_consensus.cc:1261] T 8a76d3e2d4264687aa2972c179341b5e P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEARNER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:01.446101 6303 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d82268925a934c559dfcf51a84f03d2e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:01.446267 8679 consensus_queue.cc:1035] T 8a76d3e2d4264687aa2972c179341b5e P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:01.447031 6582 raft_consensus.cc:2932] T d82268925a934c559dfcf51a84f03d2e P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:01.447307 6312 catalog_manager.cc:5484] T 925612b660ec471da5b3c548802ede6e P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:01.449692 8833 raft_consensus.cc:2932] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:01.450513 6311 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet 7f61f9a345d94a00a0929d8ebd197c23
I20240503 02:00:01.450840 6311 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet 1832eb75b09d4823ad07c56868f3aede
I20240503 02:00:01.450855 9048 raft_consensus.cc:2932] T d82268925a934c559dfcf51a84f03d2e P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:01.451193 6311 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet 20b9d73abd6a424193fc102180f1dd4b
I20240503 02:00:01.451532 6302 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 74ebf5d0fb9a42e6a7ec743a98960e13 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:01.451767 9533 raft_consensus.cc:482] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:01.451855 9533 raft_consensus.cc:504] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:01.452227 9533 leader_election.cc:288] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:01.452723 6436 tablet_service.cc:1514] Processing DeleteTablet for tablet 20b9d73abd6a424193fc102180f1dd4b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:01 UTC) from {username='slave'} at 127.0.0.1:51338
I20240503 02:00:01.452847 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c395771fd44056abacceb6c1acdb9f" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:01.452965 9179 raft_consensus.cc:2454] T 96c395771fd44056abacceb6c1acdb9f P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 0.
I20240503 02:00:01.452945 9039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c395771fd44056abacceb6c1acdb9f" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:01.453061 9039 raft_consensus.cc:2454] T 96c395771fd44056abacceb6c1acdb9f P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 0.
I20240503 02:00:01.452716 6438 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f61f9a345d94a00a0929d8ebd197c23 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:01 UTC) from {username='slave'} at 127.0.0.1:51338
I20240503 02:00:01.453374 8851 leader_election.cc:302] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9942306855404cfd891adf227dd65eb2, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:01.453641 9474 tablet_replica.cc:330] T 20b9d73abd6a424193fc102180f1dd4b P dec3e4a04d604f7eaaa3f077d081218a: stopping tablet replica
I20240503 02:00:01.453771 9474 raft_consensus.cc:2229] T 20b9d73abd6a424193fc102180f1dd4b P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:01.453830 9533 raft_consensus.cc:2781] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:01.453886 9533 raft_consensus.cc:482] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:01.453919 9533 raft_consensus.cc:3037] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:01.453902 9474 raft_consensus.cc:2258] T 20b9d73abd6a424193fc102180f1dd4b P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:01.454464 9474 ts_tablet_manager.cc:1875] T 20b9d73abd6a424193fc102180f1dd4b P dec3e4a04d604f7eaaa3f077d081218a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:01.454847 9533 raft_consensus.cc:504] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:01.454784 6311 catalog_manager.cc:5484] T d82268925a934c559dfcf51a84f03d2e P 99ead02915954b4fab23b033ad6d546b reported cstate change: config changed from index 5 to 6, VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:01.455211 9533 leader_election.cc:288] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 election: Requested vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:01.455530 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c395771fd44056abacceb6c1acdb9f" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:01.455637 9048 raft_consensus.cc:3037] T 96c395771fd44056abacceb6c1acdb9f P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:01.455595 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c395771fd44056abacceb6c1acdb9f" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:01.455710 9179 raft_consensus.cc:3037] T 96c395771fd44056abacceb6c1acdb9f P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:01.456588 9048 raft_consensus.cc:2454] T 96c395771fd44056abacceb6c1acdb9f P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 1.
I20240503 02:00:01.456580 9179 raft_consensus.cc:2454] T 96c395771fd44056abacceb6c1acdb9f P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 1.
I20240503 02:00:01.456867 8851 leader_election.cc:302] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9942306855404cfd891adf227dd65eb2, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:01.457125 6830 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f61f9a345d94a00a0929d8ebd197c23 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:01 UTC) from {username='slave'} at 127.0.0.1:52812
I20240503 02:00:01.457229 6829 tablet_service.cc:1514] Processing DeleteTablet for tablet 1832eb75b09d4823ad07c56868f3aede with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:01 UTC) from {username='slave'} at 127.0.0.1:52812
I20240503 02:00:01.457280 6831 tablet_service.cc:1514] Processing DeleteTablet for tablet 20b9d73abd6a424193fc102180f1dd4b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:01 UTC) from {username='slave'} at 127.0.0.1:52812
I20240503 02:00:01.457321 9476 tablet_replica.cc:330] T 7f61f9a345d94a00a0929d8ebd197c23 P 492448570cc94ed1879feb2ec3b9051d: stopping tablet replica
I20240503 02:00:01.457386 6699 tablet_service.cc:1514] Processing DeleteTablet for tablet 1832eb75b09d4823ad07c56868f3aede with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:01 UTC) from {username='slave'} at 127.0.0.1:40336
I20240503 02:00:01.457464 9476 raft_consensus.cc:2229] T 7f61f9a345d94a00a0929d8ebd197c23 P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:01.457659 9476 raft_consensus.cc:2258] T 7f61f9a345d94a00a0929d8ebd197c23 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:01.457827 9533 raft_consensus.cc:2781] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:01.457870 10173 tablet_replica.cc:330] T 1832eb75b09d4823ad07c56868f3aede P c2ac2eb7cc264047b9ab32fe3a4fc025: stopping tablet replica
I20240503 02:00:01.457903 9533 raft_consensus.cc:686] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2 [term 1 LEADER]: Becoming Leader. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Running, Role: LEADER
I20240503 02:00:01.457994 10173 raft_consensus.cc:2229] T 1832eb75b09d4823ad07c56868f3aede P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:01.458032 9533 consensus_queue.cc:237] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:01.458125 10173 raft_consensus.cc:2258] T 1832eb75b09d4823ad07c56868f3aede P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:01.458186 9476 ts_tablet_manager.cc:1875] T 7f61f9a345d94a00a0929d8ebd197c23 P 492448570cc94ed1879feb2ec3b9051d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:01.458631 10173 ts_tablet_manager.cc:1875] T 1832eb75b09d4823ad07c56868f3aede P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:01.459339 9474 ts_tablet_manager.cc:1888] T 20b9d73abd6a424193fc102180f1dd4b P dec3e4a04d604f7eaaa3f077d081218a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:00:01.459411 9474 log.cc:1196] T 20b9d73abd6a424193fc102180f1dd4b P dec3e4a04d604f7eaaa3f077d081218a: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/20b9d73abd6a424193fc102180f1dd4b
I20240503 02:00:01.459585 6569 tablet_service.cc:1514] Processing DeleteTablet for tablet 1832eb75b09d4823ad07c56868f3aede with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:01 UTC) from {username='slave'} at 127.0.0.1:59846
I20240503 02:00:01.459673 9474 ts_tablet_manager.cc:1909] T 20b9d73abd6a424193fc102180f1dd4b P dec3e4a04d604f7eaaa3f077d081218a: Deleting consensus metadata
I20240503 02:00:01.459875 6569 tablet_service.cc:1514] Processing DeleteTablet for tablet 20b9d73abd6a424193fc102180f1dd4b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:01 UTC) from {username='slave'} at 127.0.0.1:59846
I20240503 02:00:01.460464 6567 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f61f9a345d94a00a0929d8ebd197c23 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:01 UTC) from {username='slave'} at 127.0.0.1:59846
I20240503 02:00:01.460633 9473 tablet_replica.cc:330] T 1832eb75b09d4823ad07c56868f3aede P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 02:00:01.460758 9473 raft_consensus.cc:2229] T 1832eb75b09d4823ad07c56868f3aede P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:01.460760 10173 ts_tablet_manager.cc:1888] T 1832eb75b09d4823ad07c56868f3aede P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:00:01.460829 10173 log.cc:1196] T 1832eb75b09d4823ad07c56868f3aede P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/1832eb75b09d4823ad07c56868f3aede
I20240503 02:00:01.460978 9473 raft_consensus.cc:2258] T 1832eb75b09d4823ad07c56868f3aede P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:01.461059 10173 ts_tablet_manager.cc:1909] T 1832eb75b09d4823ad07c56868f3aede P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting consensus metadata
I20240503 02:00:01.461515 9473 ts_tablet_manager.cc:1875] T 1832eb75b09d4823ad07c56868f3aede P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:01.462965 6314 catalog_manager.cc:5484] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9942306855404cfd891adf227dd65eb2 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "9942306855404cfd891adf227dd65eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:01.462932 8679 raft_consensus.cc:2932] T 8a76d3e2d4264687aa2972c179341b5e P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:01.463600 9048 raft_consensus.cc:2932] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:01.463943 9473 ts_tablet_manager.cc:1888] T 1832eb75b09d4823ad07c56868f3aede P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:00:01.464016 9473 log.cc:1196] T 1832eb75b09d4823ad07c56868f3aede P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/1832eb75b09d4823ad07c56868f3aede
I20240503 02:00:01.464251 9473 ts_tablet_manager.cc:1909] T 1832eb75b09d4823ad07c56868f3aede P 80fd18ef4d604c86b909d5c008a46b4f: Deleting consensus metadata
I20240503 02:00:01.465234 6719 raft_consensus.cc:2932] T 8a76d3e2d4264687aa2972c179341b5e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:01.465670 6315 catalog_manager.cc:5484] T 8a76d3e2d4264687aa2972c179341b5e P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: config changed from index 4 to 5, 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:01.466462 9476 ts_tablet_manager.cc:1888] T 7f61f9a345d94a00a0929d8ebd197c23 P 492448570cc94ed1879feb2ec3b9051d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:00:01.466547 9476 log.cc:1196] T 7f61f9a345d94a00a0929d8ebd197c23 P 492448570cc94ed1879feb2ec3b9051d: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/7f61f9a345d94a00a0929d8ebd197c23
I20240503 02:00:01.466794 9476 ts_tablet_manager.cc:1909] T 7f61f9a345d94a00a0929d8ebd197c23 P 492448570cc94ed1879feb2ec3b9051d: Deleting consensus metadata
I20240503 02:00:01.466692 6720 raft_consensus.cc:2932] T 74ebf5d0fb9a42e6a7ec743a98960e13 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:01.467025 6850 raft_consensus.cc:2932] T 8a76d3e2d4264687aa2972c179341b5e P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:01.467379 9476 tablet_replica.cc:330] T 1832eb75b09d4823ad07c56868f3aede P 492448570cc94ed1879feb2ec3b9051d: stopping tablet replica
I20240503 02:00:01.468323 6303 catalog_manager.cc:4830] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719): tablet 7f61f9a345d94a00a0929d8ebd197c23 (table rebalance_test_table_10 [id=14361c7197694d579466ed5d4f2922f7]) successfully deleted
I20240503 02:00:01.468850 10369 ts_tablet_manager.cc:902] T 0099654461ca41efa4323fc9f5adab4a P 80fd18ef4d604c86b909d5c008a46b4f: Initiating tablet copy from peer dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 02:00:01.469056 8633 raft_consensus.cc:1050] T 2b1d700d09534fa3adcf51bb3c97c8f5 P dec3e4a04d604f7eaaa3f077d081218a: attempting to promote NON_VOTER 9942306855404cfd891adf227dd65eb2 to VOTER
I20240503 02:00:01.469375 9473 tablet_replica.cc:330] T 20b9d73abd6a424193fc102180f1dd4b P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 02:00:01.469501 9473 raft_consensus.cc:2229] T 20b9d73abd6a424193fc102180f1dd4b P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:01.469460 8633 consensus_queue.cc:237] T 2b1d700d09534fa3adcf51bb3c97c8f5 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } }
I20240503 02:00:01.469585 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet 1832eb75b09d4823ad07c56868f3aede (table rebalance_test_table_10 [id=14361c7197694d579466ed5d4f2922f7]) successfully deleted
I20240503 02:00:01.469619 9473 raft_consensus.cc:2258] T 20b9d73abd6a424193fc102180f1dd4b P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:01.469928 10369 tablet_copy_client.cc:308] T 0099654461ca41efa4323fc9f5adab4a P 80fd18ef4d604c86b909d5c008a46b4f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:33093
I20240503 02:00:01.470089 9473 ts_tablet_manager.cc:1875] T 20b9d73abd6a424193fc102180f1dd4b P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:01.470902 6478 tablet_copy_service.cc:140] P dec3e4a04d604f7eaaa3f077d081218a: Received BeginTabletCopySession request for tablet 0099654461ca41efa4323fc9f5adab4a from peer 80fd18ef4d604c86b909d5c008a46b4f ({username='slave'} at 127.0.99.130:49797)
I20240503 02:00:01.470949 8917 raft_consensus.cc:1261] T 2b1d700d09534fa3adcf51bb3c97c8f5 P 9942306855404cfd891adf227dd65eb2 [term 1 LEARNER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:01.470991 6478 tablet_copy_service.cc:161] P dec3e4a04d604f7eaaa3f077d081218a: Beginning new tablet copy session on tablet 0099654461ca41efa4323fc9f5adab4a from peer 80fd18ef4d604c86b909d5c008a46b4f at {username='slave'} at 127.0.99.130:49797: session id = 80fd18ef4d604c86b909d5c008a46b4f-0099654461ca41efa4323fc9f5adab4a
I20240503 02:00:01.471037 6720 raft_consensus.cc:1261] T 2b1d700d09534fa3adcf51bb3c97c8f5 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:01.471277 6303 catalog_manager.cc:4830] TS c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655): tablet 1832eb75b09d4823ad07c56868f3aede (table rebalance_test_table_10 [id=14361c7197694d579466ed5d4f2922f7]) successfully deleted
I20240503 02:00:01.471572 9955 consensus_queue.cc:1035] T 2b1d700d09534fa3adcf51bb3c97c8f5 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:01.471861 9476 raft_consensus.cc:2229] T 1832eb75b09d4823ad07c56868f3aede P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:01.471894 6478 tablet_copy_source_session.cc:205] T 0099654461ca41efa4323fc9f5adab4a P dec3e4a04d604f7eaaa3f077d081218a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:01.471918 9955 consensus_queue.cc:1035] T 2b1d700d09534fa3adcf51bb3c97c8f5 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:01.472035 9476 raft_consensus.cc:2258] T 1832eb75b09d4823ad07c56868f3aede P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:01.472597 6850 raft_consensus.cc:1261] T 2b1d700d09534fa3adcf51bb3c97c8f5 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:01.472718 9476 ts_tablet_manager.cc:1875] T 1832eb75b09d4823ad07c56868f3aede P 492448570cc94ed1879feb2ec3b9051d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:01.473750 6309 catalog_manager.cc:5484] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: config changed from index 5 to 6, VOTER 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:01.473851 9721 consensus_queue.cc:1035] T 2b1d700d09534fa3adcf51bb3c97c8f5 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:01.475160 9476 ts_tablet_manager.cc:1888] T 1832eb75b09d4823ad07c56868f3aede P 492448570cc94ed1879feb2ec3b9051d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:00:01.475236 9476 log.cc:1196] T 1832eb75b09d4823ad07c56868f3aede P 492448570cc94ed1879feb2ec3b9051d: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/1832eb75b09d4823ad07c56868f3aede
I20240503 02:00:01.475616 9476 ts_tablet_manager.cc:1909] T 1832eb75b09d4823ad07c56868f3aede P 492448570cc94ed1879feb2ec3b9051d: Deleting consensus metadata
I20240503 02:00:01.475926 6438 tablet_service.cc:1514] Processing DeleteTablet for tablet d82268925a934c559dfcf51a84f03d2e with delete_type TABLET_DATA_TOMBSTONED (TS dec3e4a04d604f7eaaa3f077d081218a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51338
I20240503 02:00:01.476327 6303 catalog_manager.cc:4830] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719): tablet 1832eb75b09d4823ad07c56868f3aede (table rebalance_test_table_10 [id=14361c7197694d579466ed5d4f2922f7]) successfully deleted
I20240503 02:00:01.476821 6831 tablet_service.cc:1514] Processing DeleteTablet for tablet 74ebf5d0fb9a42e6a7ec743a98960e13 with delete_type TABLET_DATA_TOMBSTONED (TS 492448570cc94ed1879feb2ec3b9051d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52812
I20240503 02:00:01.478349 10369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0099654461ca41efa4323fc9f5adab4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:01.479959 9473 ts_tablet_manager.cc:1888] T 20b9d73abd6a424193fc102180f1dd4b P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:00:01.480796 9473 log.cc:1196] T 20b9d73abd6a424193fc102180f1dd4b P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/20b9d73abd6a424193fc102180f1dd4b
I20240503 02:00:01.480880 9178 raft_consensus.cc:2932] T 8a76d3e2d4264687aa2972c179341b5e P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:01.482430 10369 tablet_copy_client.cc:783] T 0099654461ca41efa4323fc9f5adab4a P 80fd18ef4d604c86b909d5c008a46b4f: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:01.482573 9473 ts_tablet_manager.cc:1909] T 20b9d73abd6a424193fc102180f1dd4b P 80fd18ef4d604c86b909d5c008a46b4f: Deleting consensus metadata
I20240503 02:00:01.482630 10369 tablet_copy_client.cc:647] T 0099654461ca41efa4323fc9f5adab4a P 80fd18ef4d604c86b909d5c008a46b4f: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:01.483157 9473 tablet_replica.cc:330] T 7f61f9a345d94a00a0929d8ebd197c23 P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 02:00:01.483295 9473 raft_consensus.cc:2229] T 7f61f9a345d94a00a0929d8ebd197c23 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:01.483319 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet 20b9d73abd6a424193fc102180f1dd4b (table rebalance_test_table_10 [id=14361c7197694d579466ed5d4f2922f7]) successfully deleted
I20240503 02:00:01.483692 9474 tablet_replica.cc:330] T 7f61f9a345d94a00a0929d8ebd197c23 P dec3e4a04d604f7eaaa3f077d081218a: stopping tablet replica
I20240503 02:00:01.483816 9474 raft_consensus.cc:2229] T 7f61f9a345d94a00a0929d8ebd197c23 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:01.483938 9474 raft_consensus.cc:2258] T 7f61f9a345d94a00a0929d8ebd197c23 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:01.484403 6300 catalog_manager.cc:4830] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): tablet 20b9d73abd6a424193fc102180f1dd4b (table rebalance_test_table_10 [id=14361c7197694d579466ed5d4f2922f7]) successfully deleted
I20240503 02:00:01.484655 9476 tablet_replica.cc:330] T 20b9d73abd6a424193fc102180f1dd4b P 492448570cc94ed1879feb2ec3b9051d: stopping tablet replica
I20240503 02:00:01.484807 9476 raft_consensus.cc:2229] T 20b9d73abd6a424193fc102180f1dd4b P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:01.485000 9476 raft_consensus.cc:2258] T 20b9d73abd6a424193fc102180f1dd4b P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:01.485249 9473 raft_consensus.cc:2258] T 7f61f9a345d94a00a0929d8ebd197c23 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:01.485738 9473 ts_tablet_manager.cc:1875] T 7f61f9a345d94a00a0929d8ebd197c23 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:01.486570 10382 raft_consensus.cc:2932] T 2b1d700d09534fa3adcf51bb3c97c8f5 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:01.487004 6720 raft_consensus.cc:2932] T 2b1d700d09534fa3adcf51bb3c97c8f5 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:01.488111 9473 ts_tablet_manager.cc:1888] T 7f61f9a345d94a00a0929d8ebd197c23 P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:00:01.484439 9474 ts_tablet_manager.cc:1875] T 7f61f9a345d94a00a0929d8ebd197c23 P dec3e4a04d604f7eaaa3f077d081218a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:01.489485 6315 catalog_manager.cc:5484] T 2b1d700d09534fa3adcf51bb3c97c8f5 P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: config changed from index 4 to 5, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:01.489565 6850 raft_consensus.cc:2932] T 2b1d700d09534fa3adcf51bb3c97c8f5 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:01.490702 9474 ts_tablet_manager.cc:1888] T 7f61f9a345d94a00a0929d8ebd197c23 P dec3e4a04d604f7eaaa3f077d081218a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:00:01.490785 9474 log.cc:1196] T 7f61f9a345d94a00a0929d8ebd197c23 P dec3e4a04d604f7eaaa3f077d081218a: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/7f61f9a345d94a00a0929d8ebd197c23
I20240503 02:00:01.491029 9474 ts_tablet_manager.cc:1909] T 7f61f9a345d94a00a0929d8ebd197c23 P dec3e4a04d604f7eaaa3f077d081218a: Deleting consensus metadata
I20240503 02:00:01.491588 9476 ts_tablet_manager.cc:1875] T 20b9d73abd6a424193fc102180f1dd4b P 492448570cc94ed1879feb2ec3b9051d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:01.492024 6310 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:00:01.493832 9476 ts_tablet_manager.cc:1888] T 20b9d73abd6a424193fc102180f1dd4b P 492448570cc94ed1879feb2ec3b9051d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:00:01.493904 9476 log.cc:1196] T 20b9d73abd6a424193fc102180f1dd4b P 492448570cc94ed1879feb2ec3b9051d: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/20b9d73abd6a424193fc102180f1dd4b
I20240503 02:00:01.494164 9476 ts_tablet_manager.cc:1909] T 20b9d73abd6a424193fc102180f1dd4b P 492448570cc94ed1879feb2ec3b9051d: Deleting consensus metadata
I20240503 02:00:01.494530 9473 log.cc:1196] T 7f61f9a345d94a00a0929d8ebd197c23 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/7f61f9a345d94a00a0929d8ebd197c23
I20240503 02:00:01.494786 9473 ts_tablet_manager.cc:1909] T 7f61f9a345d94a00a0929d8ebd197c23 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting consensus metadata
I20240503 02:00:01.495589 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet 7f61f9a345d94a00a0929d8ebd197c23 (table rebalance_test_table_10 [id=14361c7197694d579466ed5d4f2922f7]) successfully deleted
I20240503 02:00:01.496037 6303 catalog_manager.cc:4830] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719): tablet 20b9d73abd6a424193fc102180f1dd4b (table rebalance_test_table_10 [id=14361c7197694d579466ed5d4f2922f7]) successfully deleted
I20240503 02:00:01.496539 9476 tablet_replica.cc:330] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 492448570cc94ed1879feb2ec3b9051d: stopping tablet replica
I20240503 02:00:01.496721 9476 raft_consensus.cc:2229] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:01.496898 9474 tablet_replica.cc:330] T d82268925a934c559dfcf51a84f03d2e P dec3e4a04d604f7eaaa3f077d081218a: stopping tablet replica
I20240503 02:00:01.496920 9476 raft_consensus.cc:2258] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:01.496996 6300 catalog_manager.cc:4830] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): tablet 7f61f9a345d94a00a0929d8ebd197c23 (table rebalance_test_table_10 [id=14361c7197694d579466ed5d4f2922f7]) successfully deleted
I20240503 02:00:01.497036 9474 raft_consensus.cc:2229] T d82268925a934c559dfcf51a84f03d2e P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:01.497175 9474 raft_consensus.cc:2258] T d82268925a934c559dfcf51a84f03d2e P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:01.497893 6831 tablet_service.cc:1467] Processing CreateTablet for tablet 68e0ec26972c425c8f5c1bd94ae5e0d7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=281ac00372fe4c70b4cddfb02e3139b5]), partition=
I20240503 02:00:01.497881 8880 tablet_service.cc:1467] Processing CreateTablet for tablet 68e0ec26972c425c8f5c1bd94ae5e0d7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=281ac00372fe4c70b4cddfb02e3139b5]), partition=
I20240503 02:00:01.497881 9285 tablet_service.cc:1467] Processing CreateTablet for tablet 68e0ec26972c425c8f5c1bd94ae5e0d7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=281ac00372fe4c70b4cddfb02e3139b5]), partition=
I20240503 02:00:01.498121 9285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68e0ec26972c425c8f5c1bd94ae5e0d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:01.498122 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68e0ec26972c425c8f5c1bd94ae5e0d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:01.498132 8880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68e0ec26972c425c8f5c1bd94ae5e0d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:01.499203 9474 ts_tablet_manager.cc:1875] T d82268925a934c559dfcf51a84f03d2e P dec3e4a04d604f7eaaa3f077d081218a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:01.500336 10315 tablet_bootstrap.cc:492] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:01.501174 10387 tablet_bootstrap.cc:492] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap starting.
I20240503 02:00:01.501238 10315 tablet_bootstrap.cc:654] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:01.501427 9474 ts_tablet_manager.cc:1888] T d82268925a934c559dfcf51a84f03d2e P dec3e4a04d604f7eaaa3f077d081218a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:00:01.501513 9474 log.cc:1196] T d82268925a934c559dfcf51a84f03d2e P dec3e4a04d604f7eaaa3f077d081218a: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/d82268925a934c559dfcf51a84f03d2e
I20240503 02:00:01.502044 6300 catalog_manager.cc:4830] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): tablet d82268925a934c559dfcf51a84f03d2e (table rebalance_test_table_13 [id=029e2476b8c44a8d8cdb98005eb84a4b]) successfully deleted
I20240503 02:00:01.502226 10387 tablet_bootstrap.cc:654] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P e1a9251ce1cf4361b8b0ede9fbfc4023: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:01.502307 10315 tablet_bootstrap.cc:492] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:01.502413 10315 ts_tablet_manager.cc:1367] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:01.502893 10315 raft_consensus.cc:348] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:01.503016 10315 raft_consensus.cc:374] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:01.503046 10315 raft_consensus.cc:729] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:01.503170 10315 consensus_queue.cc:260] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:01.503270 10387 tablet_bootstrap.cc:492] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P e1a9251ce1cf4361b8b0ede9fbfc4023: No bootstrap required, opened a new log
I20240503 02:00:01.503334 10387 ts_tablet_manager.cc:1367] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:01.503448 10315 ts_tablet_manager.cc:1398] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:01.503692 10387 raft_consensus.cc:348] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:01.503790 10387 raft_consensus.cc:374] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:01.503818 10387 raft_consensus.cc:729] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1a9251ce1cf4361b8b0ede9fbfc4023, State: Initialized, Role: FOLLOWER
I20240503 02:00:01.503962 10387 consensus_queue.cc:260] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P e1a9251ce1cf4361b8b0ede9fbfc4023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:01.504513 10369 tablet_copy_client.cc:523] T 0099654461ca41efa4323fc9f5adab4a P 80fd18ef4d604c86b909d5c008a46b4f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:01.504969 6456 consensus_queue.cc:237] T 2b1d700d09534fa3adcf51bb3c97c8f5 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } }
I20240503 02:00:01.505410 10387 ts_tablet_manager.cc:1398] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:01.506351 9476 ts_tablet_manager.cc:1875] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 492448570cc94ed1879feb2ec3b9051d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:01.506723 8917 raft_consensus.cc:2932] T 2b1d700d09534fa3adcf51bb3c97c8f5 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:01.507480 10369 tablet_bootstrap.cc:492] T 0099654461ca41efa4323fc9f5adab4a P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:01.508113 8916 raft_consensus.cc:1261] T 2b1d700d09534fa3adcf51bb3c97c8f5 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:01.508463 6849 raft_consensus.cc:1261] T 2b1d700d09534fa3adcf51bb3c97c8f5 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:01.509001 10386 tablet_bootstrap.cc:492] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:01.509438 9476 ts_tablet_manager.cc:1888] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 492448570cc94ed1879feb2ec3b9051d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:00:01.509560 9476 log.cc:1196] T 74ebf5d0fb9a42e6a7ec743a98960e13 P 492448570cc94ed1879feb2ec3b9051d: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/74ebf5d0fb9a42e6a7ec743a98960e13
I20240503 02:00:01.510064 10386 tablet_bootstrap.cc:654] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:01.510128 6303 catalog_manager.cc:4830] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719): tablet 74ebf5d0fb9a42e6a7ec743a98960e13 (table rebalance_test_table_1 [id=6e24d367bc1d434584d8a7363cf538b6]) successfully deleted
I20240503 02:00:01.512837 10369 tablet_bootstrap.cc:492] T 0099654461ca41efa4323fc9f5adab4a P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:01.512027 10381 consensus_queue.cc:1035] T 2b1d700d09534fa3adcf51bb3c97c8f5 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:01.513253 10369 tablet_bootstrap.cc:492] T 0099654461ca41efa4323fc9f5adab4a P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap complete.
I20240503 02:00:01.513309 9721 consensus_queue.cc:1035] T 2b1d700d09534fa3adcf51bb3c97c8f5 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:01.513473 10369 ts_tablet_manager.cc:1367] T 0099654461ca41efa4323fc9f5adab4a P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240503 02:00:01.513813 10381 raft_consensus.cc:1050] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a: attempting to promote NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 to VOTER
I20240503 02:00:01.514107 10369 raft_consensus.cc:348] T 0099654461ca41efa4323fc9f5adab4a P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: true } }
I20240503 02:00:01.514273 10381 consensus_queue.cc:237] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } }
I20240503 02:00:01.514339 10369 raft_consensus.cc:729] T 0099654461ca41efa4323fc9f5adab4a P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: LEARNER
I20240503 02:00:01.514747 10369 consensus_queue.cc:260] T 0099654461ca41efa4323fc9f5adab4a P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: true } }
I20240503 02:00:01.515489 8679 raft_consensus.cc:2932] T 2b1d700d09534fa3adcf51bb3c97c8f5 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:01.516808 9309 raft_consensus.cc:1261] T 46ad540deb624a25b97d3b967fe1f6f4 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 LEARNER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:01.517200 9955 consensus_queue.cc:1035] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:01.518746 6300 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2b1d700d09534fa3adcf51bb3c97c8f5 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:01.519459 6310 catalog_manager.cc:5484] T 2b1d700d09534fa3adcf51bb3c97c8f5 P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: config changed from index 5 to 6, VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:01.520159 10369 ts_tablet_manager.cc:1398] T 0099654461ca41efa4323fc9f5adab4a P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240503 02:00:01.520550 8916 raft_consensus.cc:2932] T 2b1d700d09534fa3adcf51bb3c97c8f5 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:01.520874 6478 tablet_copy_service.cc:342] P dec3e4a04d604f7eaaa3f077d081218a: Request end of tablet copy session 80fd18ef4d604c86b909d5c008a46b4f-0099654461ca41efa4323fc9f5adab4a received from {username='slave'} at 127.0.99.130:49797
I20240503 02:00:01.520951 6478 tablet_copy_service.cc:433] P dec3e4a04d604f7eaaa3f077d081218a: ending tablet copy session 80fd18ef4d604c86b909d5c008a46b4f-0099654461ca41efa4323fc9f5adab4a on tablet 0099654461ca41efa4323fc9f5adab4a with peer 80fd18ef4d604c86b909d5c008a46b4f
I20240503 02:00:01.521592 6850 raft_consensus.cc:1261] T 46ad540deb624a25b97d3b967fe1f6f4 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:01.521833 10386 tablet_bootstrap.cc:492] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:01.521924 10386 ts_tablet_manager.cc:1367] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.000s
I20240503 02:00:01.522013 8679 consensus_queue.cc:1035] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:01.522364 10386 raft_consensus.cc:348] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:01.522480 10386 raft_consensus.cc:374] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:01.522512 10386 raft_consensus.cc:729] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:01.522624 10386 consensus_queue.cc:260] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:01.522902 10386 ts_tablet_manager.cc:1398] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:01.523247 6720 raft_consensus.cc:1261] T 46ad540deb624a25b97d3b967fe1f6f4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:01.522295 6849 raft_consensus.cc:2932] T 2b1d700d09534fa3adcf51bb3c97c8f5 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:01.525502 6699 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b1d700d09534fa3adcf51bb3c97c8f5 with delete_type TABLET_DATA_TOMBSTONED (TS c2ac2eb7cc264047b9ab32fe3a4fc025 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:40336
I20240503 02:00:01.526876 10173 tablet_replica.cc:330] T 2b1d700d09534fa3adcf51bb3c97c8f5 P c2ac2eb7cc264047b9ab32fe3a4fc025: stopping tablet replica
I20240503 02:00:01.527027 10173 raft_consensus.cc:2229] T 2b1d700d09534fa3adcf51bb3c97c8f5 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:01.527233 10173 raft_consensus.cc:2258] T 2b1d700d09534fa3adcf51bb3c97c8f5 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:01.528419 9955 consensus_queue.cc:1035] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:01.528687 8633 raft_consensus.cc:2932] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:01.530810 9309 raft_consensus.cc:2932] T 46ad540deb624a25b97d3b967fe1f6f4 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:01.531766 6720 raft_consensus.cc:2932] T 46ad540deb624a25b97d3b967fe1f6f4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:01.533566 6310 catalog_manager.cc:5484] T 46ad540deb624a25b97d3b967fe1f6f4 P e1a9251ce1cf4361b8b0ede9fbfc4023 reported cstate change: config changed from index 4 to 5, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:01.534451 6850 raft_consensus.cc:2932] T 46ad540deb624a25b97d3b967fe1f6f4 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:01.536821 6456 consensus_queue.cc:237] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } }
I20240503 02:00:01.537581 10173 ts_tablet_manager.cc:1875] T 2b1d700d09534fa3adcf51bb3c97c8f5 P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:01.538164 6850 raft_consensus.cc:1261] T 46ad540deb624a25b97d3b967fe1f6f4 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:01.538602 8679 consensus_queue.cc:1035] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:01.540005 10173 ts_tablet_manager.cc:1888] T 2b1d700d09534fa3adcf51bb3c97c8f5 P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:00:01.540096 10173 log.cc:1196] T 2b1d700d09534fa3adcf51bb3c97c8f5 P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/2b1d700d09534fa3adcf51bb3c97c8f5
I20240503 02:00:01.540966 9309 raft_consensus.cc:1261] T 46ad540deb624a25b97d3b967fe1f6f4 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:01.540971 6303 catalog_manager.cc:4830] TS c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655): tablet 2b1d700d09534fa3adcf51bb3c97c8f5 (table rebalance_test_table_13 [id=029e2476b8c44a8d8cdb98005eb84a4b]) successfully deleted
I20240503 02:00:01.541437 8679 consensus_queue.cc:1035] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:01.542446 9484 raft_consensus.cc:482] T cb873018448549ccbe3d14b784e47d26 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:01.542549 9484 raft_consensus.cc:504] T cb873018448549ccbe3d14b784e47d26 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:01.542960 9484 leader_election.cc:288] T cb873018448549ccbe3d14b784e47d26 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:01.543011 8633 raft_consensus.cc:2932] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:01.543685 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb873018448549ccbe3d14b784e47d26" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:01.543830 9048 raft_consensus.cc:2454] T cb873018448549ccbe3d14b784e47d26 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:01.544275 6523 leader_election.cc:302] T cb873018448549ccbe3d14b784e47d26 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:01.544464 9484 raft_consensus.cc:2781] T cb873018448549ccbe3d14b784e47d26 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:01.544514 9484 raft_consensus.cc:482] T cb873018448549ccbe3d14b784e47d26 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:01.544548 9484 raft_consensus.cc:3037] T cb873018448549ccbe3d14b784e47d26 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:01.544876 6300 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 46ad540deb624a25b97d3b967fe1f6f4 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:01.545140 6850 raft_consensus.cc:2932] T 46ad540deb624a25b97d3b967fe1f6f4 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:01.545590 9484 raft_consensus.cc:504] T cb873018448549ccbe3d14b784e47d26 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:01.545600 9309 raft_consensus.cc:2932] T 46ad540deb624a25b97d3b967fe1f6f4 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:01.545950 9484 leader_election.cc:288] T cb873018448549ccbe3d14b784e47d26 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Requested vote from peers 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:01.546500 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb873018448549ccbe3d14b784e47d26" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:01.546639 9048 raft_consensus.cc:3037] T cb873018448549ccbe3d14b784e47d26 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:01.547601 9048 raft_consensus.cc:2454] T cb873018448549ccbe3d14b784e47d26 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:01.548056 6523 leader_election.cc:302] T cb873018448549ccbe3d14b784e47d26 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:01.548631 9484 raft_consensus.cc:2781] T cb873018448549ccbe3d14b784e47d26 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:01.548720 9484 raft_consensus.cc:686] T cb873018448549ccbe3d14b784e47d26 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Becoming Leader. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Running, Role: LEADER
I20240503 02:00:01.548852 9484 consensus_queue.cc:237] T cb873018448549ccbe3d14b784e47d26 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:01.548908 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb873018448549ccbe3d14b784e47d26" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:01.549073 8916 raft_consensus.cc:2454] T cb873018448549ccbe3d14b784e47d26 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:01.549592 6309 catalog_manager.cc:5484] T 46ad540deb624a25b97d3b967fe1f6f4 P 492448570cc94ed1879feb2ec3b9051d reported cstate change: config changed from index 5 to 6, VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:01.548908 8917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb873018448549ccbe3d14b784e47d26" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:01.550352 8917 raft_consensus.cc:3037] T cb873018448549ccbe3d14b784e47d26 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:01.551321 8917 raft_consensus.cc:2454] T cb873018448549ccbe3d14b784e47d26 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:01.552865 6312 catalog_manager.cc:5484] T cb873018448549ccbe3d14b784e47d26 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:01.556366 6699 tablet_service.cc:1514] Processing DeleteTablet for tablet 46ad540deb624a25b97d3b967fe1f6f4 with delete_type TABLET_DATA_TOMBSTONED (TS c2ac2eb7cc264047b9ab32fe3a4fc025 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:40336
I20240503 02:00:01.556648 10173 tablet_replica.cc:330] T 46ad540deb624a25b97d3b967fe1f6f4 P c2ac2eb7cc264047b9ab32fe3a4fc025: stopping tablet replica
I20240503 02:00:01.556804 10173 raft_consensus.cc:2229] T 46ad540deb624a25b97d3b967fe1f6f4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:01.556943 10173 raft_consensus.cc:2258] T 46ad540deb624a25b97d3b967fe1f6f4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:01.557932 10173 ts_tablet_manager.cc:1875] T 46ad540deb624a25b97d3b967fe1f6f4 P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:01.560154 10173 ts_tablet_manager.cc:1888] T 46ad540deb624a25b97d3b967fe1f6f4 P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:00:01.560278 10173 log.cc:1196] T 46ad540deb624a25b97d3b967fe1f6f4 P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/46ad540deb624a25b97d3b967fe1f6f4
I20240503 02:00:01.560765 6303 catalog_manager.cc:4830] TS c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655): tablet 46ad540deb624a25b97d3b967fe1f6f4 (table rebalance_test_table_9 [id=c65135643094446abc8eba15be23cd99]) successfully deleted
I20240503 02:00:01.574216 9039 raft_consensus.cc:1203] T 7a4a910a8b9d4134adebb52a73dfae1d P 99ead02915954b4fab23b033ad6d546b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:00:01.574913 10349 raft_consensus.cc:1050] T 485d1d89a8b94505969d663be92124cd P c2ac2eb7cc264047b9ab32fe3a4fc025: attempting to promote NON_VOTER 99ead02915954b4fab23b033ad6d546b to VOTER
I20240503 02:00:01.575402 10349 consensus_queue.cc:237] T 485d1d89a8b94505969d663be92124cd P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } }
I20240503 02:00:01.576943 9039 raft_consensus.cc:1261] T 485d1d89a8b94505969d663be92124cd P 99ead02915954b4fab23b033ad6d546b [term 1 LEARNER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:01.577203 6850 raft_consensus.cc:1261] T 485d1d89a8b94505969d663be92124cd P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:01.577323 9711 consensus_queue.cc:1035] T 485d1d89a8b94505969d663be92124cd P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:01.577584 10268 consensus_queue.cc:1035] T 485d1d89a8b94505969d663be92124cd P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:01.579177 6456 raft_consensus.cc:1261] T 485d1d89a8b94505969d663be92124cd P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:01.579659 10348 consensus_queue.cc:1035] T 485d1d89a8b94505969d663be92124cd P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:01.581511 10268 raft_consensus.cc:2932] T 485d1d89a8b94505969d663be92124cd P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 99ead02915954b4fab23b033ad6d546b (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:01.581961 6456 raft_consensus.cc:2932] T 485d1d89a8b94505969d663be92124cd P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 99ead02915954b4fab23b033ad6d546b (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:01.584872 6310 catalog_manager.cc:5484] T 485d1d89a8b94505969d663be92124cd P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: config changed from index 4 to 5, 99ead02915954b4fab23b033ad6d546b (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:01.585189 6850 raft_consensus.cc:2932] T 485d1d89a8b94505969d663be92124cd P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 99ead02915954b4fab23b033ad6d546b (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:01.586917 9039 raft_consensus.cc:2932] T 485d1d89a8b94505969d663be92124cd P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 99ead02915954b4fab23b033ad6d546b (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:01.594285 6720 consensus_queue.cc:237] T 485d1d89a8b94505969d663be92124cd P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } }
I20240503 02:00:01.595508 9039 raft_consensus.cc:1261] T 485d1d89a8b94505969d663be92124cd P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:01.595585 6456 raft_consensus.cc:1261] T 485d1d89a8b94505969d663be92124cd P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:01.595942 9711 consensus_queue.cc:1035] T 485d1d89a8b94505969d663be92124cd P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:01.596298 9711 consensus_queue.cc:1035] T 485d1d89a8b94505969d663be92124cd P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:01.597934 10349 raft_consensus.cc:2932] T 485d1d89a8b94505969d663be92124cd P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:01.598325 6456 raft_consensus.cc:2932] T 485d1d89a8b94505969d663be92124cd P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:01.599583 6303 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 485d1d89a8b94505969d663be92124cd with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:01.601583 6310 catalog_manager.cc:5484] T 485d1d89a8b94505969d663be92124cd P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: config changed from index 5 to 6, VOTER 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:01.602715 9039 raft_consensus.cc:2932] T 485d1d89a8b94505969d663be92124cd P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:01.606237 6831 tablet_service.cc:1514] Processing DeleteTablet for tablet 485d1d89a8b94505969d663be92124cd with delete_type TABLET_DATA_TOMBSTONED (TS 492448570cc94ed1879feb2ec3b9051d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52812
I20240503 02:00:01.606534 9476 tablet_replica.cc:330] T 485d1d89a8b94505969d663be92124cd P 492448570cc94ed1879feb2ec3b9051d: stopping tablet replica
I20240503 02:00:01.606717 9476 raft_consensus.cc:2229] T 485d1d89a8b94505969d663be92124cd P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:01.606904 9476 raft_consensus.cc:2258] T 485d1d89a8b94505969d663be92124cd P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:01.607802 9476 ts_tablet_manager.cc:1875] T 485d1d89a8b94505969d663be92124cd P 492448570cc94ed1879feb2ec3b9051d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:01.610628 9476 ts_tablet_manager.cc:1888] T 485d1d89a8b94505969d663be92124cd P 492448570cc94ed1879feb2ec3b9051d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:00:01.610713 9476 log.cc:1196] T 485d1d89a8b94505969d663be92124cd P 492448570cc94ed1879feb2ec3b9051d: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/485d1d89a8b94505969d663be92124cd
I20240503 02:00:01.611258 6303 catalog_manager.cc:4830] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719): tablet 485d1d89a8b94505969d663be92124cd (table rebalance_test_table_15 [id=4e740b0167ed439291b74d77653ec386]) successfully deleted
I20240503 02:00:01.672509 9631 raft_consensus.cc:482] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:01.672632 9631 raft_consensus.cc:504] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:01.673166 9631 leader_election.cc:288] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:01.674427 9310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68e0ec26972c425c8f5c1bd94ae5e0d7" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" is_pre_election: true
I20240503 02:00:01.674589 9310 raft_consensus.cc:2454] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 02:00:01.674932 6785 leader_election.cc:302] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, e1a9251ce1cf4361b8b0ede9fbfc4023; no voters:
I20240503 02:00:01.675159 9631 raft_consensus.cc:2781] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:01.675221 9631 raft_consensus.cc:482] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:01.675271 9631 raft_consensus.cc:3037] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:01.677018 9631 raft_consensus.cc:504] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:01.677444 9631 leader_election.cc:288] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Requested vote from peers e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:01.678184 9310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68e0ec26972c425c8f5c1bd94ae5e0d7" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023"
I20240503 02:00:01.678314 9310 raft_consensus.cc:3037] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:01.679276 9310 raft_consensus.cc:2454] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 02:00:01.679610 6785 leader_election.cc:302] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, e1a9251ce1cf4361b8b0ede9fbfc4023; no voters:
I20240503 02:00:01.679870 9631 raft_consensus.cc:2781] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:01.679958 9631 raft_consensus.cc:686] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Becoming Leader. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Running, Role: LEADER
I20240503 02:00:01.680091 9631 consensus_queue.cc:237] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:01.680433 8917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68e0ec26972c425c8f5c1bd94ae5e0d7" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:01.680577 8917 raft_consensus.cc:2454] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 02:00:01.680457 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68e0ec26972c425c8f5c1bd94ae5e0d7" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:01.680738 8916 raft_consensus.cc:3037] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:01.681778 8916 raft_consensus.cc:2454] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 02:00:01.682653 6315 catalog_manager.cc:5484] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P 492448570cc94ed1879feb2ec3b9051d reported cstate change: term changed from 0 to 1, leader changed from <none> to 492448570cc94ed1879feb2ec3b9051d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:01.700352 6456 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8a76d3e2d4264687aa2972c179341b5e"
dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58654
I20240503 02:00:01.700625 6456 raft_consensus.cc:595] T 8a76d3e2d4264687aa2972c179341b5e P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Received request to transfer leadership
I20240503 02:00:01.710649 6582 consensus_queue.cc:237] T 7df4c083550046e883e5e61d3d4e1572 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } }
I20240503 02:00:01.712208 6720 raft_consensus.cc:1261] T 7df4c083550046e883e5e61d3d4e1572 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:01.712272 6456 raft_consensus.cc:1261] T 7df4c083550046e883e5e61d3d4e1572 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:00:01.712330 6525 consensus_peers.cc:485] T 7df4c083550046e883e5e61d3d4e1572 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211): Couldn't send request to peer 9942306855404cfd891adf227dd65eb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7df4c083550046e883e5e61d3d4e1572. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:01.712718 9973 consensus_queue.cc:1035] T 7df4c083550046e883e5e61d3d4e1572 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:01.712905 10345 consensus_queue.cc:1035] T 7df4c083550046e883e5e61d3d4e1572 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:01.715127 9484 raft_consensus.cc:2932] T 7df4c083550046e883e5e61d3d4e1572 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } } }
I20240503 02:00:01.715796 6720 raft_consensus.cc:2932] T 7df4c083550046e883e5e61d3d4e1572 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } } }
I20240503 02:00:01.715910 6456 raft_consensus.cc:2932] T 7df4c083550046e883e5e61d3d4e1572 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } } }
I20240503 02:00:01.718866 6315 catalog_manager.cc:5484] T 7df4c083550046e883e5e61d3d4e1572 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: config changed from index -1 to 4, NON_VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:01.729543 6850 consensus_queue.cc:237] T 34120f4948504cbabcd28e0c4f08f9f8 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } }
I20240503 02:00:01.731326 6456 raft_consensus.cc:1261] T 34120f4948504cbabcd28e0c4f08f9f8 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:01.731776 6582 raft_consensus.cc:1261] T 34120f4948504cbabcd28e0c4f08f9f8 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:01.731802 9461 consensus_queue.cc:1035] T 34120f4948504cbabcd28e0c4f08f9f8 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:01.732179 9461 consensus_queue.cc:1035] T 34120f4948504cbabcd28e0c4f08f9f8 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:01.735883 10212 raft_consensus.cc:2932] T 34120f4948504cbabcd28e0c4f08f9f8 P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } } }
W20240503 02:00:01.736199 6785 consensus_peers.cc:485] T 34120f4948504cbabcd28e0c4f08f9f8 P 492448570cc94ed1879feb2ec3b9051d -> Peer 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157): Couldn't send request to peer 99ead02915954b4fab23b033ad6d546b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34120f4948504cbabcd28e0c4f08f9f8. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:01.737008 6456 raft_consensus.cc:2932] T 34120f4948504cbabcd28e0c4f08f9f8 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } } }
I20240503 02:00:01.737334 6582 raft_consensus.cc:2932] T 34120f4948504cbabcd28e0c4f08f9f8 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } } }
I20240503 02:00:01.740271 6309 catalog_manager.cc:5484] T 34120f4948504cbabcd28e0c4f08f9f8 P 492448570cc94ed1879feb2ec3b9051d reported cstate change: config changed from index -1 to 4, NON_VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:01.748152 6719 consensus_queue.cc:237] T 2fc834abc6a94c638d6202b03320a2d2 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } }
I20240503 02:00:01.749998 6456 raft_consensus.cc:1261] T 2fc834abc6a94c638d6202b03320a2d2 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:01.750039 6850 raft_consensus.cc:1261] T 2fc834abc6a94c638d6202b03320a2d2 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:00:01.750488 6653 consensus_peers.cc:485] T 2fc834abc6a94c638d6202b03320a2d2 P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079): Couldn't send request to peer 2a01ace80baf4057b6bf1995cf88468b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2fc834abc6a94c638d6202b03320a2d2. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:01.750488 10348 consensus_queue.cc:1035] T 2fc834abc6a94c638d6202b03320a2d2 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:01.750731 9984 consensus_queue.cc:1035] T 2fc834abc6a94c638d6202b03320a2d2 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:01.752789 9984 raft_consensus.cc:2932] T 2fc834abc6a94c638d6202b03320a2d2 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } } }
I20240503 02:00:01.753401 6456 raft_consensus.cc:2932] T 2fc834abc6a94c638d6202b03320a2d2 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } } }
I20240503 02:00:01.753698 6850 raft_consensus.cc:2932] T 2fc834abc6a94c638d6202b03320a2d2 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } } }
I20240503 02:00:01.756044 6315 catalog_manager.cc:5484] T 2fc834abc6a94c638d6202b03320a2d2 P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: config changed from index -1 to 4, NON_VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:01.817304 6308 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:01.822250 8916 raft_consensus.cc:1261] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:01.822646 9310 raft_consensus.cc:1261] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:01.823187 9461 consensus_queue.cc:1035] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:01.825313 9461 consensus_queue.cc:1035] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:01.826328 6316 catalog_manager.cc:5576] TS e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311) does not have the latest schema for tablet 68e0ec26972c425c8f5c1bd94ae5e0d7 (table rebalancer_dynamic_table [id=281ac00372fe4c70b4cddfb02e3139b5]). Expected version 1 got 0
I20240503 02:00:01.828822 6312 catalog_manager.cc:5576] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211) does not have the latest schema for tablet 68e0ec26972c425c8f5c1bd94ae5e0d7 (table rebalancer_dynamic_table [id=281ac00372fe4c70b4cddfb02e3139b5]). Expected version 1 got 0
I20240503 02:00:01.833009 6308 catalog_manager.cc:5576] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719) does not have the latest schema for tablet 68e0ec26972c425c8f5c1bd94ae5e0d7 (table rebalancer_dynamic_table [id=281ac00372fe4c70b4cddfb02e3139b5]). Expected version 1 got 0
I20240503 02:00:01.834069 6312 catalog_manager.cc:5576] TS e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311) does not have the latest schema for tablet 68e0ec26972c425c8f5c1bd94ae5e0d7 (table rebalancer_dynamic_table [id=281ac00372fe4c70b4cddfb02e3139b5]). Expected version 1 got 0
I20240503 02:00:01.834937 10564 tablet.cc:1722] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:01.835867 10514 consensus_queue.cc:1035] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:01.837939 10566 ts_tablet_manager.cc:902] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P e1a9251ce1cf4361b8b0ede9fbfc4023: Initiating tablet copy from peer 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:01.838558 10566 tablet_copy_client.cc:308] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43863
I20240503 02:00:01.839136 10562 mvcc.cc:205] Tried to move back new op lower bound from 7023417761059889152 to 7023417760484626432. Current Snapshot: MvccSnapshot[applied={T|T < 7023417761059889152}]
I20240503 02:00:01.839219 6609 tablet_copy_service.cc:140] P 80fd18ef4d604c86b909d5c008a46b4f: Received BeginTabletCopySession request for tablet 9f7ade1e2af24e86b6e54e2b5769e4c0 from peer e1a9251ce1cf4361b8b0ede9fbfc4023 ({username='slave'} at 127.0.99.136:60117)
I20240503 02:00:01.839334 6609 tablet_copy_service.cc:161] P 80fd18ef4d604c86b909d5c008a46b4f: Beginning new tablet copy session on tablet 9f7ade1e2af24e86b6e54e2b5769e4c0 from peer e1a9251ce1cf4361b8b0ede9fbfc4023 at {username='slave'} at 127.0.99.136:60117: session id = e1a9251ce1cf4361b8b0ede9fbfc4023-9f7ade1e2af24e86b6e54e2b5769e4c0
I20240503 02:00:01.840023 6312 catalog_manager.cc:5576] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211) does not have the latest schema for tablet 68e0ec26972c425c8f5c1bd94ae5e0d7 (table rebalancer_dynamic_table [id=281ac00372fe4c70b4cddfb02e3139b5]). Expected version 1 got 0
I20240503 02:00:01.840303 6609 tablet_copy_source_session.cc:205] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P 80fd18ef4d604c86b909d5c008a46b4f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:01.840283 9758 consensus_queue.cc:1035] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:01.841508 10566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f7ade1e2af24e86b6e54e2b5769e4c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:01.841399 10570 tablet.cc:1722] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:01.842957 10570 tablet.cc:1620] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
I20240503 02:00:01.845222 10566 tablet_copy_client.cc:783] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:01.845548 10566 tablet_copy_client.cc:647] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:01.846844 10565 tablet.cc:1722] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P e1a9251ce1cf4361b8b0ede9fbfc4023: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:01.847237 10564 tablet.cc:1620] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P 492448570cc94ed1879feb2ec3b9051d: MemRowSet was empty: no flush needed.
I20240503 02:00:01.850087 10382 raft_consensus.cc:1050] T 0b659e9d57db4ebf93c9143938970090 P dec3e4a04d604f7eaaa3f077d081218a: attempting to promote NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 to VOTER
I20240503 02:00:01.850550 6312 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=281ac00372fe4c70b4cddfb02e3139b5] alter complete (version 1)
I20240503 02:00:01.850559 10382 consensus_queue.cc:237] T 0b659e9d57db4ebf93c9143938970090 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } }
I20240503 02:00:01.852504 6582 raft_consensus.cc:1261] T 0b659e9d57db4ebf93c9143938970090 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:01.852990 8633 consensus_queue.cc:1035] T 0b659e9d57db4ebf93c9143938970090 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:01.853176 9310 raft_consensus.cc:1261] T 0b659e9d57db4ebf93c9143938970090 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 LEARNER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:01.853267 10566 tablet_copy_client.cc:523] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:01.853921 10345 consensus_queue.cc:1035] T 925612b660ec471da5b3c548802ede6e P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:01.854064 6719 raft_consensus.cc:1261] T 0b659e9d57db4ebf93c9143938970090 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:01.854420 8633 consensus_queue.cc:1035] T 0b659e9d57db4ebf93c9143938970090 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:01.854578 9955 consensus_queue.cc:1035] T 0b659e9d57db4ebf93c9143938970090 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:01.854784 10566 tablet_bootstrap.cc:492] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap starting.
I20240503 02:00:01.858832 9973 consensus_queue.cc:1035] T 925612b660ec471da5b3c548802ede6e P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:01.859877 9955 raft_consensus.cc:979] T 8a76d3e2d4264687aa2972c179341b5e P dec3e4a04d604f7eaaa3f077d081218a: : Instructing follower c2ac2eb7cc264047b9ab32fe3a4fc025 to start an election
I20240503 02:00:01.859990 9955 raft_consensus.cc:1067] T 8a76d3e2d4264687aa2972c179341b5e P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Signalling peer c2ac2eb7cc264047b9ab32fe3a4fc025 to start an election
I20240503 02:00:01.860767 6720 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8a76d3e2d4264687aa2972c179341b5e"
dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
from {username='slave'} at 127.0.99.129:53433
I20240503 02:00:01.860934 6720 raft_consensus.cc:482] T 8a76d3e2d4264687aa2972c179341b5e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:01.860996 6720 raft_consensus.cc:3037] T 8a76d3e2d4264687aa2972c179341b5e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:01.862067 6720 raft_consensus.cc:504] T 8a76d3e2d4264687aa2972c179341b5e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } }
I20240503 02:00:01.862497 6720 leader_election.cc:288] T 8a76d3e2d4264687aa2972c179341b5e P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 2 election: Requested vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:01.862913 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a76d3e2d4264687aa2972c179341b5e" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 02:00:01.863025 6456 raft_consensus.cc:3032] T 8a76d3e2d4264687aa2972c179341b5e P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:00:01.863067 6456 raft_consensus.cc:729] T 8a76d3e2d4264687aa2972c179341b5e P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Running, Role: LEADER
I20240503 02:00:01.863212 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a76d3e2d4264687aa2972c179341b5e" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:01.863245 6456 consensus_queue.cc:260] T 8a76d3e2d4264687aa2972c179341b5e P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } }
I20240503 02:00:01.863327 9178 raft_consensus.cc:3037] T 8a76d3e2d4264687aa2972c179341b5e P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:01.863431 6456 raft_consensus.cc:3037] T 8a76d3e2d4264687aa2972c179341b5e P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:01.864246 9178 raft_consensus.cc:2454] T 8a76d3e2d4264687aa2972c179341b5e P 2a01ace80baf4057b6bf1995cf88468b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 2.
I20240503 02:00:01.862802 6850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a76d3e2d4264687aa2972c179341b5e" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 02:00:01.864460 6456 raft_consensus.cc:2454] T 8a76d3e2d4264687aa2972c179341b5e P dec3e4a04d604f7eaaa3f077d081218a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 2.
I20240503 02:00:01.864491 6850 raft_consensus.cc:3037] T 8a76d3e2d4264687aa2972c179341b5e P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:01.865146 6653 leader_election.cc:302] T 8a76d3e2d4264687aa2972c179341b5e P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, c2ac2eb7cc264047b9ab32fe3a4fc025, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:01.865346 9711 raft_consensus.cc:2781] T 8a76d3e2d4264687aa2972c179341b5e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:01.865432 9711 raft_consensus.cc:686] T 8a76d3e2d4264687aa2972c179341b5e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 2 LEADER]: Becoming Leader. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Running, Role: LEADER
I20240503 02:00:01.865494 6850 raft_consensus.cc:2454] T 8a76d3e2d4264687aa2972c179341b5e P 492448570cc94ed1879feb2ec3b9051d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 2.
I20240503 02:00:01.865581 9711 consensus_queue.cc:237] T 8a76d3e2d4264687aa2972c179341b5e P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } }
I20240503 02:00:01.868254 10565 tablet.cc:1620] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P e1a9251ce1cf4361b8b0ede9fbfc4023: MemRowSet was empty: no flush needed.
I20240503 02:00:01.868134 6312 catalog_manager.cc:5484] T 8a76d3e2d4264687aa2972c179341b5e P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: term changed from 1 to 2, leader changed from dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) to c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131). New cstate: current_term: 2 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:01.874648 8633 raft_consensus.cc:2932] T 0b659e9d57db4ebf93c9143938970090 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:01.875578 6719 raft_consensus.cc:2932] T 0b659e9d57db4ebf93c9143938970090 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:01.877048 6582 raft_consensus.cc:2932] T 0b659e9d57db4ebf93c9143938970090 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:01.877048 9310 raft_consensus.cc:2932] T 0b659e9d57db4ebf93c9143938970090 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:01.878504 6310 catalog_manager.cc:5484] T 0b659e9d57db4ebf93c9143938970090 P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: config changed from index 4 to 5, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:01.882915 6456 consensus_queue.cc:237] T 0b659e9d57db4ebf93c9143938970090 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } }
I20240503 02:00:01.884855 6719 raft_consensus.cc:1261] T 0b659e9d57db4ebf93c9143938970090 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:01.885388 9955 consensus_queue.cc:1035] T 0b659e9d57db4ebf93c9143938970090 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:01.887038 10382 raft_consensus.cc:2932] T 0b659e9d57db4ebf93c9143938970090 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:01.888528 6719 raft_consensus.cc:2932] T 0b659e9d57db4ebf93c9143938970090 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:01.889310 6300 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0b659e9d57db4ebf93c9143938970090 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:01.889526 9309 raft_consensus.cc:1261] T 0b659e9d57db4ebf93c9143938970090 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:01.891252 8633 consensus_queue.cc:1035] T 0b659e9d57db4ebf93c9143938970090 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:01.891594 6316 catalog_manager.cc:5484] T 0b659e9d57db4ebf93c9143938970090 P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: config changed from index 5 to 6, VOTER 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:01.894227 9309 raft_consensus.cc:2932] T 0b659e9d57db4ebf93c9143938970090 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:01.896279 6567 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b659e9d57db4ebf93c9143938970090 with delete_type TABLET_DATA_TOMBSTONED (TS 80fd18ef4d604c86b909d5c008a46b4f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59846
I20240503 02:00:01.896521 9473 tablet_replica.cc:330] T 0b659e9d57db4ebf93c9143938970090 P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 02:00:01.896659 9473 raft_consensus.cc:2229] T 0b659e9d57db4ebf93c9143938970090 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:01.896821 9473 raft_consensus.cc:2258] T 0b659e9d57db4ebf93c9143938970090 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:01.898960 9473 ts_tablet_manager.cc:1875] T 0b659e9d57db4ebf93c9143938970090 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:01.900851 6567 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:00:01.901006 9473 ts_tablet_manager.cc:1888] T 0b659e9d57db4ebf93c9143938970090 P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:00:01.901070 9473 log.cc:1196] T 0b659e9d57db4ebf93c9143938970090 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/0b659e9d57db4ebf93c9143938970090
I20240503 02:00:01.901078 9159 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:00:01.901535 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet 0b659e9d57db4ebf93c9143938970090 (table rebalance_test_table_16 [id=eecab7e70bfa4621832bb99bd9ef066c]) successfully deleted
I20240503 02:00:01.913571 9025 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:01.916561 6831 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:00:01.922226 10566 tablet_bootstrap.cc:492] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:01.922616 10566 tablet_bootstrap.cc:492] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap complete.
I20240503 02:00:01.922693 8880 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:01.922883 10566 ts_tablet_manager.cc:1367] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent bootstrapping tablet: real 0.068s user 0.023s sys 0.004s
I20240503 02:00:01.923352 10566 raft_consensus.cc:348] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } }
I20240503 02:00:01.923563 10566 raft_consensus.cc:729] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e1a9251ce1cf4361b8b0ede9fbfc4023, State: Initialized, Role: LEARNER
I20240503 02:00:01.923712 10566 consensus_queue.cc:260] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P e1a9251ce1cf4361b8b0ede9fbfc4023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } }
I20240503 02:00:01.924037 10566 ts_tablet_manager.cc:1398] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:01.925307 6609 tablet_copy_service.cc:342] P 80fd18ef4d604c86b909d5c008a46b4f: Request end of tablet copy session e1a9251ce1cf4361b8b0ede9fbfc4023-9f7ade1e2af24e86b6e54e2b5769e4c0 received from {username='slave'} at 127.0.99.136:60117
I20240503 02:00:01.925395 6609 tablet_copy_service.cc:433] P 80fd18ef4d604c86b909d5c008a46b4f: ending tablet copy session e1a9251ce1cf4361b8b0ede9fbfc4023-9f7ade1e2af24e86b6e54e2b5769e4c0 on tablet 9f7ade1e2af24e86b6e54e2b5769e4c0 with peer e1a9251ce1cf4361b8b0ede9fbfc4023
I20240503 02:00:01.928452 6438 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 02:00:01.929522 9973 consensus_queue.cc:1035] T cb873018448549ccbe3d14b784e47d26 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:01.934928 9973 consensus_queue.cc:1035] T cb873018448549ccbe3d14b784e47d26 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:01.948804 6699 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:01.960748 9285 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:01.972409 6313 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240503 02:00:01.972399 6310 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:01.972584 6313 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240503 02:00:01.976765 6313 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet e08ab73460e746fc86a9b77c405da86e
I20240503 02:00:01.977135 6313 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 4 replicas of tablet 34120f4948504cbabcd28e0c4f08f9f8
I20240503 02:00:01.977368 6313 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet 7b392142a4d948e7a36b49314ae15129
I20240503 02:00:01.977463 6699 tablet_service.cc:1514] Processing DeleteTablet for tablet e08ab73460e746fc86a9b77c405da86e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:01 UTC) from {username='slave'} at 127.0.0.1:40336
I20240503 02:00:01.977600 6567 tablet_service.cc:1514] Processing DeleteTablet for tablet e08ab73460e746fc86a9b77c405da86e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:01 UTC) from {username='slave'} at 127.0.0.1:59846
I20240503 02:00:01.977670 10173 tablet_replica.cc:330] T e08ab73460e746fc86a9b77c405da86e P c2ac2eb7cc264047b9ab32fe3a4fc025: stopping tablet replica
I20240503 02:00:01.977814 10173 raft_consensus.cc:2229] T e08ab73460e746fc86a9b77c405da86e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:01.977953 6569 tablet_service.cc:1514] Processing DeleteTablet for tablet 34120f4948504cbabcd28e0c4f08f9f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:01 UTC) from {username='slave'} at 127.0.0.1:59846
I20240503 02:00:01.978040 10173 raft_consensus.cc:2258] T e08ab73460e746fc86a9b77c405da86e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:01.978205 9473 tablet_replica.cc:330] T e08ab73460e746fc86a9b77c405da86e P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 02:00:01.978338 9473 raft_consensus.cc:2229] T e08ab73460e746fc86a9b77c405da86e P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:01.978327 6831 tablet_service.cc:1514] Processing DeleteTablet for tablet 34120f4948504cbabcd28e0c4f08f9f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:01 UTC) from {username='slave'} at 127.0.0.1:52812
I20240503 02:00:01.978466 9473 raft_consensus.cc:2258] T e08ab73460e746fc86a9b77c405da86e P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:01.978586 9476 tablet_replica.cc:330] T 34120f4948504cbabcd28e0c4f08f9f8 P 492448570cc94ed1879feb2ec3b9051d: stopping tablet replica
I20240503 02:00:01.978631 10173 ts_tablet_manager.cc:1875] T e08ab73460e746fc86a9b77c405da86e P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:01.978718 9476 raft_consensus.cc:2229] T 34120f4948504cbabcd28e0c4f08f9f8 P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:01.978955 9476 raft_consensus.cc:2258] T 34120f4948504cbabcd28e0c4f08f9f8 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:01.979004 9473 ts_tablet_manager.cc:1875] T e08ab73460e746fc86a9b77c405da86e P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:01.979311 6436 tablet_service.cc:1514] Processing DeleteTablet for tablet 34120f4948504cbabcd28e0c4f08f9f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:01 UTC) from {username='slave'} at 127.0.0.1:51338
I20240503 02:00:01.979497 9474 tablet_replica.cc:330] T 34120f4948504cbabcd28e0c4f08f9f8 P dec3e4a04d604f7eaaa3f077d081218a: stopping tablet replica
I20240503 02:00:01.979637 9474 raft_consensus.cc:2229] T 34120f4948504cbabcd28e0c4f08f9f8 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:01.979811 9474 raft_consensus.cc:2258] T 34120f4948504cbabcd28e0c4f08f9f8 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:01.980484 6829 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b392142a4d948e7a36b49314ae15129 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:01 UTC) from {username='slave'} at 127.0.0.1:52812
I20240503 02:00:01.980497 9025 tablet_service.cc:1514] Processing DeleteTablet for tablet 34120f4948504cbabcd28e0c4f08f9f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:01 UTC) from {username='slave'} at 127.0.0.1:52584
I20240503 02:00:01.980706 6438 tablet_service.cc:1514] Processing DeleteTablet for tablet e08ab73460e746fc86a9b77c405da86e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:01 UTC) from {username='slave'} at 127.0.0.1:51338
I20240503 02:00:01.981189 6699 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b392142a4d948e7a36b49314ae15129 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:01 UTC) from {username='slave'} at 127.0.0.1:40336
I20240503 02:00:01.981310 9473 ts_tablet_manager.cc:1888] T e08ab73460e746fc86a9b77c405da86e P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:00:01.981380 9473 log.cc:1196] T e08ab73460e746fc86a9b77c405da86e P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/e08ab73460e746fc86a9b77c405da86e
I20240503 02:00:01.981628 9473 ts_tablet_manager.cc:1909] T e08ab73460e746fc86a9b77c405da86e P 80fd18ef4d604c86b909d5c008a46b4f: Deleting consensus metadata
I20240503 02:00:01.981793 6569 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b392142a4d948e7a36b49314ae15129 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:01 UTC) from {username='slave'} at 127.0.0.1:59846
I20240503 02:00:01.982070 10173 ts_tablet_manager.cc:1888] T e08ab73460e746fc86a9b77c405da86e P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:00:01.982467 9473 tablet_replica.cc:330] T 34120f4948504cbabcd28e0c4f08f9f8 P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 02:00:01.982570 10173 log.cc:1196] T e08ab73460e746fc86a9b77c405da86e P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/e08ab73460e746fc86a9b77c405da86e
I20240503 02:00:01.982604 9473 raft_consensus.cc:2229] T 34120f4948504cbabcd28e0c4f08f9f8 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:01.982738 9473 raft_consensus.cc:2258] T 34120f4948504cbabcd28e0c4f08f9f8 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:01.982837 10173 ts_tablet_manager.cc:1909] T e08ab73460e746fc86a9b77c405da86e P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting consensus metadata
I20240503 02:00:01.982821 9476 ts_tablet_manager.cc:1875] T 34120f4948504cbabcd28e0c4f08f9f8 P 492448570cc94ed1879feb2ec3b9051d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:01.983381 10173 tablet_replica.cc:330] T 7b392142a4d948e7a36b49314ae15129 P c2ac2eb7cc264047b9ab32fe3a4fc025: stopping tablet replica
W20240503 02:00:01.983449 6301 catalog_manager.cc:4797] TS 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157): delete failed for tablet 34120f4948504cbabcd28e0c4f08f9f8 because the tablet was not found. No further retry: Not found: Tablet not found: 34120f4948504cbabcd28e0c4f08f9f8
I20240503 02:00:01.985074 9476 ts_tablet_manager.cc:1888] T 34120f4948504cbabcd28e0c4f08f9f8 P 492448570cc94ed1879feb2ec3b9051d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:01.985147 9476 log.cc:1196] T 34120f4948504cbabcd28e0c4f08f9f8 P 492448570cc94ed1879feb2ec3b9051d: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/34120f4948504cbabcd28e0c4f08f9f8
I20240503 02:00:01.985352 9476 ts_tablet_manager.cc:1909] T 34120f4948504cbabcd28e0c4f08f9f8 P 492448570cc94ed1879feb2ec3b9051d: Deleting consensus metadata
I20240503 02:00:01.986678 9473 ts_tablet_manager.cc:1875] T 34120f4948504cbabcd28e0c4f08f9f8 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:01.987458 9474 ts_tablet_manager.cc:1875] T 34120f4948504cbabcd28e0c4f08f9f8 P dec3e4a04d604f7eaaa3f077d081218a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:01.987526 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet e08ab73460e746fc86a9b77c405da86e (table rebalance_test_table_12 [id=043b44e4ca1745038882551026afda53]) successfully deleted
I20240503 02:00:01.988950 9473 ts_tablet_manager.cc:1888] T 34120f4948504cbabcd28e0c4f08f9f8 P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:01.989029 9473 log.cc:1196] T 34120f4948504cbabcd28e0c4f08f9f8 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/34120f4948504cbabcd28e0c4f08f9f8
I20240503 02:00:01.989012 10564 tablet.cc:1722] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:01.989262 9473 ts_tablet_manager.cc:1909] T 34120f4948504cbabcd28e0c4f08f9f8 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting consensus metadata
I20240503 02:00:01.989342 10564 tablet.cc:1620] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P 492448570cc94ed1879feb2ec3b9051d: MemRowSet was empty: no flush needed.
I20240503 02:00:01.989771 9474 ts_tablet_manager.cc:1888] T 34120f4948504cbabcd28e0c4f08f9f8 P dec3e4a04d604f7eaaa3f077d081218a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:01.989948 6303 catalog_manager.cc:4830] TS c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655): tablet e08ab73460e746fc86a9b77c405da86e (table rebalance_test_table_12 [id=043b44e4ca1745038882551026afda53]) successfully deleted
I20240503 02:00:01.989998 9476 tablet_replica.cc:330] T 7b392142a4d948e7a36b49314ae15129 P 492448570cc94ed1879feb2ec3b9051d: stopping tablet replica
I20240503 02:00:01.990067 9473 tablet_replica.cc:330] T 7b392142a4d948e7a36b49314ae15129 P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 02:00:01.990150 9476 raft_consensus.cc:2229] T 7b392142a4d948e7a36b49314ae15129 P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:01.990170 9473 raft_consensus.cc:2229] T 7b392142a4d948e7a36b49314ae15129 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:01.990285 9473 raft_consensus.cc:2258] T 7b392142a4d948e7a36b49314ae15129 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:01.990664 9476 raft_consensus.cc:2258] T 7b392142a4d948e7a36b49314ae15129 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:01.990824 9474 log.cc:1196] T 34120f4948504cbabcd28e0c4f08f9f8 P dec3e4a04d604f7eaaa3f077d081218a: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/34120f4948504cbabcd28e0c4f08f9f8
I20240503 02:00:01.990954 9473 ts_tablet_manager.cc:1875] T 7b392142a4d948e7a36b49314ae15129 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:01.990968 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet 34120f4948504cbabcd28e0c4f08f9f8 (table rebalance_test_table_12 [id=043b44e4ca1745038882551026afda53]) successfully deleted
I20240503 02:00:01.991111 9474 ts_tablet_manager.cc:1909] T 34120f4948504cbabcd28e0c4f08f9f8 P dec3e4a04d604f7eaaa3f077d081218a: Deleting consensus metadata
I20240503 02:00:01.991294 9476 ts_tablet_manager.cc:1875] T 7b392142a4d948e7a36b49314ae15129 P 492448570cc94ed1879feb2ec3b9051d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:01.991750 9474 tablet_replica.cc:330] T e08ab73460e746fc86a9b77c405da86e P dec3e4a04d604f7eaaa3f077d081218a: stopping tablet replica
I20240503 02:00:01.991813 6300 catalog_manager.cc:4830] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): tablet 34120f4948504cbabcd28e0c4f08f9f8 (table rebalance_test_table_12 [id=043b44e4ca1745038882551026afda53]) successfully deleted
I20240503 02:00:01.991899 9474 raft_consensus.cc:2229] T e08ab73460e746fc86a9b77c405da86e P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:01.992026 9474 raft_consensus.cc:2258] T e08ab73460e746fc86a9b77c405da86e P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:01.992729 10173 raft_consensus.cc:2229] T 7b392142a4d948e7a36b49314ae15129 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:01.993677 9473 ts_tablet_manager.cc:1888] T 7b392142a4d948e7a36b49314ae15129 P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:00:01.993801 9473 log.cc:1196] T 7b392142a4d948e7a36b49314ae15129 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/7b392142a4d948e7a36b49314ae15129
I20240503 02:00:01.994105 9473 ts_tablet_manager.cc:1909] T 7b392142a4d948e7a36b49314ae15129 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting consensus metadata
I20240503 02:00:01.994314 10565 tablet.cc:1722] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P e1a9251ce1cf4361b8b0ede9fbfc4023: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:01.994395 9476 ts_tablet_manager.cc:1888] T 7b392142a4d948e7a36b49314ae15129 P 492448570cc94ed1879feb2ec3b9051d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:00:01.994457 9476 log.cc:1196] T 7b392142a4d948e7a36b49314ae15129 P 492448570cc94ed1879feb2ec3b9051d: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/7b392142a4d948e7a36b49314ae15129
I20240503 02:00:01.994642 10565 tablet.cc:1620] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P e1a9251ce1cf4361b8b0ede9fbfc4023: MemRowSet was empty: no flush needed.
I20240503 02:00:01.994695 9476 ts_tablet_manager.cc:1909] T 7b392142a4d948e7a36b49314ae15129 P 492448570cc94ed1879feb2ec3b9051d: Deleting consensus metadata
I20240503 02:00:01.994992 10570 tablet.cc:1722] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:01.995270 10570 tablet.cc:1620] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
I20240503 02:00:01.995556 10173 raft_consensus.cc:2258] T 7b392142a4d948e7a36b49314ae15129 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:01.996105 10173 ts_tablet_manager.cc:1875] T 7b392142a4d948e7a36b49314ae15129 P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:01.996425 9474 ts_tablet_manager.cc:1875] T e08ab73460e746fc86a9b77c405da86e P dec3e4a04d604f7eaaa3f077d081218a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:01.997828 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet 7b392142a4d948e7a36b49314ae15129 (table rebalance_test_table_12 [id=043b44e4ca1745038882551026afda53]) successfully deleted
I20240503 02:00:01.998217 6303 catalog_manager.cc:4830] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719): tablet 34120f4948504cbabcd28e0c4f08f9f8 (table rebalance_test_table_12 [id=043b44e4ca1745038882551026afda53]) successfully deleted
I20240503 02:00:01.998384 6303 catalog_manager.cc:4830] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719): tablet 7b392142a4d948e7a36b49314ae15129 (table rebalance_test_table_12 [id=043b44e4ca1745038882551026afda53]) successfully deleted
I20240503 02:00:02.000165 6582 raft_consensus.cc:1203] T 0099654461ca41efa4323fc9f5adab4a P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:00:02.000875 6310 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=281ac00372fe4c70b4cddfb02e3139b5] alter complete (version 2)
I20240503 02:00:02.001801 10173 ts_tablet_manager.cc:1888] T 7b392142a4d948e7a36b49314ae15129 P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:00:02.001971 10173 log.cc:1196] T 7b392142a4d948e7a36b49314ae15129 P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/7b392142a4d948e7a36b49314ae15129
I20240503 02:00:02.002367 10173 ts_tablet_manager.cc:1909] T 7b392142a4d948e7a36b49314ae15129 P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting consensus metadata
I20240503 02:00:02.002663 9474 ts_tablet_manager.cc:1888] T e08ab73460e746fc86a9b77c405da86e P dec3e4a04d604f7eaaa3f077d081218a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:00:02.002754 9474 log.cc:1196] T e08ab73460e746fc86a9b77c405da86e P dec3e4a04d604f7eaaa3f077d081218a: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/e08ab73460e746fc86a9b77c405da86e
I20240503 02:00:02.003048 6303 catalog_manager.cc:4830] TS c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655): tablet 7b392142a4d948e7a36b49314ae15129 (table rebalance_test_table_12 [id=043b44e4ca1745038882551026afda53]) successfully deleted
I20240503 02:00:02.003399 9474 ts_tablet_manager.cc:1909] T e08ab73460e746fc86a9b77c405da86e P dec3e4a04d604f7eaaa3f077d081218a: Deleting consensus metadata
I20240503 02:00:02.004045 6300 catalog_manager.cc:4830] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): tablet e08ab73460e746fc86a9b77c405da86e (table rebalance_test_table_12 [id=043b44e4ca1745038882551026afda53]) successfully deleted
I20240503 02:00:02.034961 6719 consensus_queue.cc:237] T 4c13642db7fc40a5a934aa5e3c2b08fc P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } }
I20240503 02:00:02.036473 6850 raft_consensus.cc:1261] T 4c13642db7fc40a5a934aa5e3c2b08fc P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:00:02.036808 6653 consensus_peers.cc:485] T 4c13642db7fc40a5a934aa5e3c2b08fc P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079): Couldn't send request to peer 2a01ace80baf4057b6bf1995cf88468b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4c13642db7fc40a5a934aa5e3c2b08fc. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:02.036967 6582 raft_consensus.cc:1261] T 4c13642db7fc40a5a934aa5e3c2b08fc P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:02.037140 10349 consensus_queue.cc:1035] T 4c13642db7fc40a5a934aa5e3c2b08fc P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:02.037324 10348 consensus_queue.cc:1035] T 4c13642db7fc40a5a934aa5e3c2b08fc P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:02.039196 10349 raft_consensus.cc:2932] T 4c13642db7fc40a5a934aa5e3c2b08fc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } } }
I20240503 02:00:02.041206 6850 raft_consensus.cc:2932] T 4c13642db7fc40a5a934aa5e3c2b08fc P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } } }
I20240503 02:00:02.041718 6582 raft_consensus.cc:2932] T 4c13642db7fc40a5a934aa5e3c2b08fc P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } } }
I20240503 02:00:02.042485 6310 catalog_manager.cc:5484] T 4c13642db7fc40a5a934aa5e3c2b08fc P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: config changed from index -1 to 4, NON_VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:02.054713 6582 consensus_queue.cc:237] T 925612b660ec471da5b3c548802ede6e P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } }
I20240503 02:00:02.056106 9178 raft_consensus.cc:1261] T 925612b660ec471da5b3c548802ede6e P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:02.056248 8916 raft_consensus.cc:1261] T 925612b660ec471da5b3c548802ede6e P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:00:02.056337 6523 consensus_peers.cc:485] T 925612b660ec471da5b3c548802ede6e P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 925612b660ec471da5b3c548802ede6e. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:02.056694 10278 consensus_queue.cc:1035] T 925612b660ec471da5b3c548802ede6e P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:02.057125 10278 consensus_queue.cc:1035] T 925612b660ec471da5b3c548802ede6e P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:02.059032 9973 raft_consensus.cc:2932] T 925612b660ec471da5b3c548802ede6e P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } } }
I20240503 02:00:02.059425 9178 raft_consensus.cc:2932] T 925612b660ec471da5b3c548802ede6e P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } } }
I20240503 02:00:02.061470 8916 raft_consensus.cc:2932] T 925612b660ec471da5b3c548802ede6e P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } } }
I20240503 02:00:02.063657 6313 catalog_manager.cc:5484] T 925612b660ec471da5b3c548802ede6e P 2a01ace80baf4057b6bf1995cf88468b reported cstate change: config changed from index -1 to 2, NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } } }
I20240503 02:00:02.072783 6850 consensus_queue.cc:237] T 90ef3584830240acb0d8aad14ae851a6 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } }
I20240503 02:00:02.074311 6719 raft_consensus.cc:1261] T 90ef3584830240acb0d8aad14ae851a6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:02.074340 9178 raft_consensus.cc:1261] T 90ef3584830240acb0d8aad14ae851a6 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:02.074730 9461 consensus_queue.cc:1035] T 90ef3584830240acb0d8aad14ae851a6 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:02.074921 9632 consensus_queue.cc:1035] T 90ef3584830240acb0d8aad14ae851a6 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 02:00:02.075111 6785 consensus_peers.cc:485] T 90ef3584830240acb0d8aad14ae851a6 P 492448570cc94ed1879feb2ec3b9051d -> Peer 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157): Couldn't send request to peer 99ead02915954b4fab23b033ad6d546b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 90ef3584830240acb0d8aad14ae851a6. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:02.077142 9461 raft_consensus.cc:2932] T 90ef3584830240acb0d8aad14ae851a6 P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } } }
I20240503 02:00:02.077783 9178 raft_consensus.cc:2932] T 90ef3584830240acb0d8aad14ae851a6 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } } }
I20240503 02:00:02.080329 6315 catalog_manager.cc:5484] T 90ef3584830240acb0d8aad14ae851a6 P 2a01ace80baf4057b6bf1995cf88468b reported cstate change: config changed from index 6 to 7, NON_VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } } }
I20240503 02:00:02.081445 6719 raft_consensus.cc:2932] T 90ef3584830240acb0d8aad14ae851a6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } } }
I20240503 02:00:02.089877 6456 consensus_queue.cc:237] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } }
I20240503 02:00:02.090791 6315 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_extra_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:02.091336 9309 raft_consensus.cc:1261] T 46ad540deb624a25b97d3b967fe1f6f4 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:02.091382 6850 raft_consensus.cc:1261] T 46ad540deb624a25b97d3b967fe1f6f4 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 02:00:02.091449 6394 consensus_peers.cc:485] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a -> Peer 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211): Couldn't send request to peer 9942306855404cfd891adf227dd65eb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 46ad540deb624a25b97d3b967fe1f6f4. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:02.091754 8633 consensus_queue.cc:1035] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:02.092085 10382 consensus_queue.cc:1035] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:02.093902 8633 raft_consensus.cc:2932] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } } }
I20240503 02:00:02.097712 9309 raft_consensus.cc:2932] T 46ad540deb624a25b97d3b967fe1f6f4 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } } }
I20240503 02:00:02.098299 6310 catalog_manager.cc:5484] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: config changed from index 6 to 7, NON_VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:02.099953 9285 tablet_service.cc:1467] Processing CreateTablet for tablet cea1ea50fa434d31b42451f84f26bc76 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=404345ac1896484ca34a8c78c6770131]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:02.100000 6850 raft_consensus.cc:2932] T 46ad540deb624a25b97d3b967fe1f6f4 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } } }
I20240503 02:00:02.100261 9285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cea1ea50fa434d31b42451f84f26bc76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:02.101274 6569 tablet_service.cc:1467] Processing CreateTablet for tablet 88205f7c7ed4488e8fcfda3bfa0c4fca (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=404345ac1896484ca34a8c78c6770131]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:02.101553 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88205f7c7ed4488e8fcfda3bfa0c4fca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:02.102319 10630 tablet_bootstrap.cc:492] T cea1ea50fa434d31b42451f84f26bc76 P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap starting.
I20240503 02:00:02.102559 9025 tablet_service.cc:1467] Processing CreateTablet for tablet cea1ea50fa434d31b42451f84f26bc76 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=404345ac1896484ca34a8c78c6770131]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:02.102818 9025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cea1ea50fa434d31b42451f84f26bc76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:02.103240 10630 tablet_bootstrap.cc:654] T cea1ea50fa434d31b42451f84f26bc76 P e1a9251ce1cf4361b8b0ede9fbfc4023: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:02.105392 10633 tablet_bootstrap.cc:492] T cea1ea50fa434d31b42451f84f26bc76 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:02.105599 10631 tablet_bootstrap.cc:492] T 88205f7c7ed4488e8fcfda3bfa0c4fca P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:02.106189 10630 tablet_bootstrap.cc:492] T cea1ea50fa434d31b42451f84f26bc76 P e1a9251ce1cf4361b8b0ede9fbfc4023: No bootstrap required, opened a new log
I20240503 02:00:02.106277 10630 ts_tablet_manager.cc:1367] T cea1ea50fa434d31b42451f84f26bc76 P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 02:00:02.106451 9159 tablet_service.cc:1467] Processing CreateTablet for tablet 0ee05e70ccbb496ca3cff08dc38112e8 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=404345ac1896484ca34a8c78c6770131]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:02.106554 10631 tablet_bootstrap.cc:654] T 88205f7c7ed4488e8fcfda3bfa0c4fca P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:02.106715 9159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ee05e70ccbb496ca3cff08dc38112e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:02.106726 10630 raft_consensus.cc:348] T cea1ea50fa434d31b42451f84f26bc76 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:02.106827 10630 raft_consensus.cc:374] T cea1ea50fa434d31b42451f84f26bc76 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:02.106853 10630 raft_consensus.cc:729] T cea1ea50fa434d31b42451f84f26bc76 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1a9251ce1cf4361b8b0ede9fbfc4023, State: Initialized, Role: FOLLOWER
I20240503 02:00:02.106971 10633 tablet_bootstrap.cc:654] T cea1ea50fa434d31b42451f84f26bc76 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:02.106956 10630 consensus_queue.cc:260] T cea1ea50fa434d31b42451f84f26bc76 P e1a9251ce1cf4361b8b0ede9fbfc4023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:02.108054 10633 tablet_bootstrap.cc:492] T cea1ea50fa434d31b42451f84f26bc76 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:02.108120 10633 ts_tablet_manager.cc:1367] T cea1ea50fa434d31b42451f84f26bc76 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:00:02.108594 10633 raft_consensus.cc:348] T cea1ea50fa434d31b42451f84f26bc76 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:02.108682 10633 raft_consensus.cc:374] T cea1ea50fa434d31b42451f84f26bc76 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:02.108702 10633 raft_consensus.cc:729] T cea1ea50fa434d31b42451f84f26bc76 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:02.108789 10633 consensus_queue.cc:260] T cea1ea50fa434d31b42451f84f26bc76 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:02.109090 10633 ts_tablet_manager.cc:1398] T cea1ea50fa434d31b42451f84f26bc76 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:02.109246 9158 tablet_service.cc:1467] Processing CreateTablet for tablet 88205f7c7ed4488e8fcfda3bfa0c4fca (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=404345ac1896484ca34a8c78c6770131]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:02.109314 10630 ts_tablet_manager.cc:1398] T cea1ea50fa434d31b42451f84f26bc76 P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 02:00:02.109495 9158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88205f7c7ed4488e8fcfda3bfa0c4fca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:02.109567 10636 tablet_bootstrap.cc:492] T 0ee05e70ccbb496ca3cff08dc38112e8 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:02.110657 10636 tablet_bootstrap.cc:654] T 0ee05e70ccbb496ca3cff08dc38112e8 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:02.112351 9973 raft_consensus.cc:1050] T 7a4a910a8b9d4134adebb52a73dfae1d P 80fd18ef4d604c86b909d5c008a46b4f: attempting to promote NON_VOTER 99ead02915954b4fab23b033ad6d546b to VOTER
I20240503 02:00:02.112772 9973 consensus_queue.cc:237] T 7a4a910a8b9d4134adebb52a73dfae1d P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } }
I20240503 02:00:02.113518 9025 tablet_service.cc:1467] Processing CreateTablet for tablet 0ee05e70ccbb496ca3cff08dc38112e8 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=404345ac1896484ca34a8c78c6770131]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:02.113833 9025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ee05e70ccbb496ca3cff08dc38112e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:02.114586 9039 raft_consensus.cc:1261] T 7a4a910a8b9d4134adebb52a73dfae1d P 99ead02915954b4fab23b033ad6d546b [term 1 LEARNER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:02.115223 6850 raft_consensus.cc:1261] T 7a4a910a8b9d4134adebb52a73dfae1d P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:02.115643 6458 raft_consensus.cc:1261] T 7a4a910a8b9d4134adebb52a73dfae1d P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:02.116029 6310 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:02.116225 6699 tablet_service.cc:1467] Processing CreateTablet for tablet cea1ea50fa434d31b42451f84f26bc76 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=404345ac1896484ca34a8c78c6770131]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:02.116331 10278 consensus_queue.cc:1035] T 7a4a910a8b9d4134adebb52a73dfae1d P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:02.116537 6699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cea1ea50fa434d31b42451f84f26bc76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:02.117195 8880 tablet_service.cc:1467] Processing CreateTablet for tablet 88205f7c7ed4488e8fcfda3bfa0c4fca (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=404345ac1896484ca34a8c78c6770131]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:02.117290 6700 tablet_service.cc:1467] Processing CreateTablet for tablet 0ee05e70ccbb496ca3cff08dc38112e8 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=404345ac1896484ca34a8c78c6770131]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:02.117476 8880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88205f7c7ed4488e8fcfda3bfa0c4fca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:02.117487 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ee05e70ccbb496ca3cff08dc38112e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:02.118057 10278 consensus_queue.cc:1035] T 7a4a910a8b9d4134adebb52a73dfae1d P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:02.121251 10278 consensus_queue.cc:1035] T 7a4a910a8b9d4134adebb52a73dfae1d P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:02.122458 10633 tablet_bootstrap.cc:492] T 0ee05e70ccbb496ca3cff08dc38112e8 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:02.123565 10633 tablet_bootstrap.cc:654] T 0ee05e70ccbb496ca3cff08dc38112e8 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:02.124471 10642 tablet_bootstrap.cc:492] T 88205f7c7ed4488e8fcfda3bfa0c4fca P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:02.125380 10640 tablet_bootstrap.cc:492] T cea1ea50fa434d31b42451f84f26bc76 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:02.126451 10564 tablet.cc:1722] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:02.127346 10565 tablet.cc:1722] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P e1a9251ce1cf4361b8b0ede9fbfc4023: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:02.126916 10570 tablet.cc:1722] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:02.128980 10642 tablet_bootstrap.cc:654] T 88205f7c7ed4488e8fcfda3bfa0c4fca P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:02.131273 10640 tablet_bootstrap.cc:654] T cea1ea50fa434d31b42451f84f26bc76 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:02.133172 10640 tablet_bootstrap.cc:492] T cea1ea50fa434d31b42451f84f26bc76 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:02.132984 10345 raft_consensus.cc:2932] T 7a4a910a8b9d4134adebb52a73dfae1d P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 99ead02915954b4fab23b033ad6d546b (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:02.133275 10640 ts_tablet_manager.cc:1367] T cea1ea50fa434d31b42451f84f26bc76 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.001s
I20240503 02:00:02.133742 10640 raft_consensus.cc:348] T cea1ea50fa434d31b42451f84f26bc76 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:02.133846 10640 raft_consensus.cc:374] T cea1ea50fa434d31b42451f84f26bc76 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:02.133874 10640 raft_consensus.cc:729] T cea1ea50fa434d31b42451f84f26bc76 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:02.133984 10640 consensus_queue.cc:260] T cea1ea50fa434d31b42451f84f26bc76 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:02.134285 10640 ts_tablet_manager.cc:1398] T cea1ea50fa434d31b42451f84f26bc76 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:02.134512 10640 tablet_bootstrap.cc:492] T 0ee05e70ccbb496ca3cff08dc38112e8 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:02.135955 10640 tablet_bootstrap.cc:654] T 0ee05e70ccbb496ca3cff08dc38112e8 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:02.136265 6850 raft_consensus.cc:2932] T 7a4a910a8b9d4134adebb52a73dfae1d P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 99ead02915954b4fab23b033ad6d546b (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:02.137450 6316 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=281ac00372fe4c70b4cddfb02e3139b5] alter complete (version 3)
I20240503 02:00:02.137719 10640 tablet_bootstrap.cc:492] T 0ee05e70ccbb496ca3cff08dc38112e8 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:02.137815 10640 ts_tablet_manager.cc:1367] T 0ee05e70ccbb496ca3cff08dc38112e8 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:00:02.138298 10640 raft_consensus.cc:348] T 0ee05e70ccbb496ca3cff08dc38112e8 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:02.138413 10640 raft_consensus.cc:374] T 0ee05e70ccbb496ca3cff08dc38112e8 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:02.138444 10640 raft_consensus.cc:729] T 0ee05e70ccbb496ca3cff08dc38112e8 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:02.138304 9039 raft_consensus.cc:2932] T 7a4a910a8b9d4134adebb52a73dfae1d P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 99ead02915954b4fab23b033ad6d546b (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:02.138608 10640 consensus_queue.cc:260] T 0ee05e70ccbb496ca3cff08dc38112e8 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:02.138906 10640 ts_tablet_manager.cc:1398] T 0ee05e70ccbb496ca3cff08dc38112e8 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:02.141115 6316 catalog_manager.cc:5484] T 7a4a910a8b9d4134adebb52a73dfae1d P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: config changed from index 4 to 5, 99ead02915954b4fab23b033ad6d546b (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:02.143783 10642 tablet_bootstrap.cc:492] T 88205f7c7ed4488e8fcfda3bfa0c4fca P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:02.143980 10642 ts_tablet_manager.cc:1367] T 88205f7c7ed4488e8fcfda3bfa0c4fca P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.000s
I20240503 02:00:02.144686 10642 raft_consensus.cc:348] T 88205f7c7ed4488e8fcfda3bfa0c4fca P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:02.144855 10642 raft_consensus.cc:374] T 88205f7c7ed4488e8fcfda3bfa0c4fca P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:02.144919 10642 raft_consensus.cc:729] T 88205f7c7ed4488e8fcfda3bfa0c4fca P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:02.144848 6458 raft_consensus.cc:2932] T 7a4a910a8b9d4134adebb52a73dfae1d P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 99ead02915954b4fab23b033ad6d546b (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:02.145054 10642 consensus_queue.cc:260] T 88205f7c7ed4488e8fcfda3bfa0c4fca P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:02.145965 10642 ts_tablet_manager.cc:1398] T 88205f7c7ed4488e8fcfda3bfa0c4fca P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:02.147359 6582 consensus_queue.cc:237] T 7a4a910a8b9d4134adebb52a73dfae1d P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } }
I20240503 02:00:02.149099 6458 raft_consensus.cc:1261] T 7a4a910a8b9d4134adebb52a73dfae1d P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:02.149224 9039 raft_consensus.cc:1261] T 7a4a910a8b9d4134adebb52a73dfae1d P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:02.149528 10345 consensus_queue.cc:1035] T 7a4a910a8b9d4134adebb52a73dfae1d P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:02.149788 10278 consensus_queue.cc:1035] T 7a4a910a8b9d4134adebb52a73dfae1d P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:02.151011 10633 tablet_bootstrap.cc:492] T 0ee05e70ccbb496ca3cff08dc38112e8 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:02.151011 10631 tablet_bootstrap.cc:492] T 88205f7c7ed4488e8fcfda3bfa0c4fca P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:02.151109 10633 ts_tablet_manager.cc:1367] T 0ee05e70ccbb496ca3cff08dc38112e8 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.000s
I20240503 02:00:02.151110 10631 ts_tablet_manager.cc:1367] T 88205f7c7ed4488e8fcfda3bfa0c4fca P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.046s user 0.002s sys 0.000s
I20240503 02:00:02.151541 10633 raft_consensus.cc:348] T 0ee05e70ccbb496ca3cff08dc38112e8 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:02.151584 10631 raft_consensus.cc:348] T 88205f7c7ed4488e8fcfda3bfa0c4fca P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:02.151659 10633 raft_consensus.cc:374] T 0ee05e70ccbb496ca3cff08dc38112e8 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:02.151669 10631 raft_consensus.cc:374] T 88205f7c7ed4488e8fcfda3bfa0c4fca P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:02.151692 10633 raft_consensus.cc:729] T 0ee05e70ccbb496ca3cff08dc38112e8 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:02.151697 10631 raft_consensus.cc:729] T 88205f7c7ed4488e8fcfda3bfa0c4fca P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:02.151813 10631 consensus_queue.cc:260] T 88205f7c7ed4488e8fcfda3bfa0c4fca P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:02.151821 10633 consensus_queue.cc:260] T 0ee05e70ccbb496ca3cff08dc38112e8 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:02.152096 10631 ts_tablet_manager.cc:1398] T 88205f7c7ed4488e8fcfda3bfa0c4fca P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:02.152112 10633 ts_tablet_manager.cc:1398] T 0ee05e70ccbb496ca3cff08dc38112e8 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:02.152773 10278 raft_consensus.cc:2932] T 7a4a910a8b9d4134adebb52a73dfae1d P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:02.153206 9039 raft_consensus.cc:2932] T 7a4a910a8b9d4134adebb52a73dfae1d P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:02.154495 6302 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7a4a910a8b9d4134adebb52a73dfae1d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:02.156174 10636 tablet_bootstrap.cc:492] T 0ee05e70ccbb496ca3cff08dc38112e8 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:02.156458 10636 ts_tablet_manager.cc:1367] T 0ee05e70ccbb496ca3cff08dc38112e8 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.047s user 0.002s sys 0.000s
I20240503 02:00:02.157192 6313 catalog_manager.cc:5484] T 7a4a910a8b9d4134adebb52a73dfae1d P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: config changed from index 5 to 6, VOTER 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:02.158661 10636 raft_consensus.cc:348] T 0ee05e70ccbb496ca3cff08dc38112e8 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:02.158767 10636 raft_consensus.cc:374] T 0ee05e70ccbb496ca3cff08dc38112e8 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:02.158797 10636 raft_consensus.cc:729] T 0ee05e70ccbb496ca3cff08dc38112e8 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:02.160652 10636 consensus_queue.cc:260] T 0ee05e70ccbb496ca3cff08dc38112e8 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:02.161283 10636 ts_tablet_manager.cc:1398] T 0ee05e70ccbb496ca3cff08dc38112e8 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:00:02.161626 10636 tablet_bootstrap.cc:492] T 88205f7c7ed4488e8fcfda3bfa0c4fca P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:02.161823 6829 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a4a910a8b9d4134adebb52a73dfae1d with delete_type TABLET_DATA_TOMBSTONED (TS 492448570cc94ed1879feb2ec3b9051d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52812
I20240503 02:00:02.162086 9476 tablet_replica.cc:330] T 7a4a910a8b9d4134adebb52a73dfae1d P 492448570cc94ed1879feb2ec3b9051d: stopping tablet replica
I20240503 02:00:02.162247 9476 raft_consensus.cc:2229] T 7a4a910a8b9d4134adebb52a73dfae1d P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:02.162524 9476 raft_consensus.cc:2258] T 7a4a910a8b9d4134adebb52a73dfae1d P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:02.162871 6458 raft_consensus.cc:2932] T 7a4a910a8b9d4134adebb52a73dfae1d P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:02.163938 6719 consensus_queue.cc:237] T 3cf186a0901f420390c8fda54316c5af P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } }
W20240503 02:00:02.165688 6653 consensus_peers.cc:485] T 3cf186a0901f420390c8fda54316c5af P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079): Couldn't send request to peer 2a01ace80baf4057b6bf1995cf88468b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3cf186a0901f420390c8fda54316c5af. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:02.166308 6850 raft_consensus.cc:1261] T 3cf186a0901f420390c8fda54316c5af P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:02.166477 9039 raft_consensus.cc:1261] T 3cf186a0901f420390c8fda54316c5af P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:02.166787 10349 consensus_queue.cc:1035] T 3cf186a0901f420390c8fda54316c5af P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:02.167019 9984 consensus_queue.cc:1035] T 3cf186a0901f420390c8fda54316c5af P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:02.167613 9476 ts_tablet_manager.cc:1875] T 7a4a910a8b9d4134adebb52a73dfae1d P 492448570cc94ed1879feb2ec3b9051d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:02.168906 10349 raft_consensus.cc:2932] T 3cf186a0901f420390c8fda54316c5af P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } } }
I20240503 02:00:02.169342 9048 raft_consensus.cc:2932] T 3cf186a0901f420390c8fda54316c5af P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } } }
I20240503 02:00:02.170010 9476 ts_tablet_manager.cc:1888] T 7a4a910a8b9d4134adebb52a73dfae1d P 492448570cc94ed1879feb2ec3b9051d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:00:02.170079 9476 log.cc:1196] T 7a4a910a8b9d4134adebb52a73dfae1d P 492448570cc94ed1879feb2ec3b9051d: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/7a4a910a8b9d4134adebb52a73dfae1d
I20240503 02:00:02.170964 6303 catalog_manager.cc:4830] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719): tablet 7a4a910a8b9d4134adebb52a73dfae1d (table rebalance_test_table_14 [id=7455f5480e88494b83e1ac1d180ca4e6]) successfully deleted
I20240503 02:00:02.171828 6316 catalog_manager.cc:5484] T 3cf186a0901f420390c8fda54316c5af P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: config changed from index 6 to 7, NON_VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:02.173558 6850 raft_consensus.cc:2932] T 3cf186a0901f420390c8fda54316c5af P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } } }
I20240503 02:00:02.175724 10636 tablet_bootstrap.cc:654] T 88205f7c7ed4488e8fcfda3bfa0c4fca P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:02.186167 6582 consensus_queue.cc:237] T c3485701bf3b48f5ae2e1568afeed133 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } }
W20240503 02:00:02.187816 6523 consensus_peers.cc:485] T c3485701bf3b48f5ae2e1568afeed133 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c3485701bf3b48f5ae2e1568afeed133. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:02.188102 6850 raft_consensus.cc:1261] T c3485701bf3b48f5ae2e1568afeed133 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:02.188123 6719 raft_consensus.cc:1261] T c3485701bf3b48f5ae2e1568afeed133 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:02.188649 10345 consensus_queue.cc:1035] T c3485701bf3b48f5ae2e1568afeed133 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:02.188858 9484 consensus_queue.cc:1035] T c3485701bf3b48f5ae2e1568afeed133 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:02.189567 10636 tablet_bootstrap.cc:492] T 88205f7c7ed4488e8fcfda3bfa0c4fca P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:02.189684 10636 ts_tablet_manager.cc:1367] T 88205f7c7ed4488e8fcfda3bfa0c4fca P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.000s
I20240503 02:00:02.190150 10636 raft_consensus.cc:348] T 88205f7c7ed4488e8fcfda3bfa0c4fca P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:02.190295 10636 raft_consensus.cc:374] T 88205f7c7ed4488e8fcfda3bfa0c4fca P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:02.190336 10636 raft_consensus.cc:729] T 88205f7c7ed4488e8fcfda3bfa0c4fca P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:02.190456 10636 consensus_queue.cc:260] T 88205f7c7ed4488e8fcfda3bfa0c4fca P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:02.190733 10636 ts_tablet_manager.cc:1398] T 88205f7c7ed4488e8fcfda3bfa0c4fca P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:02.190781 10345 raft_consensus.cc:2932] T c3485701bf3b48f5ae2e1568afeed133 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } } }
I20240503 02:00:02.191720 6850 raft_consensus.cc:2932] T c3485701bf3b48f5ae2e1568afeed133 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } } }
I20240503 02:00:02.192936 6719 raft_consensus.cc:2932] T c3485701bf3b48f5ae2e1568afeed133 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } } }
I20240503 02:00:02.194011 6316 catalog_manager.cc:5484] T c3485701bf3b48f5ae2e1568afeed133 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: config changed from index -1 to 4, NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:02.206861 6458 consensus_queue.cc:237] T af00a839bb5e402791f87288493efef6 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } }
I20240503 02:00:02.208631 6719 raft_consensus.cc:1261] T af00a839bb5e402791f87288493efef6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:00:02.209038 6392 consensus_peers.cc:485] T af00a839bb5e402791f87288493efef6 P dec3e4a04d604f7eaaa3f077d081218a -> Peer 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157): Couldn't send request to peer 99ead02915954b4fab23b033ad6d546b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: af00a839bb5e402791f87288493efef6. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:02.209187 6582 raft_consensus.cc:1261] T af00a839bb5e402791f87288493efef6 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:02.209322 8633 consensus_queue.cc:1035] T af00a839bb5e402791f87288493efef6 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:02.209832 8633 consensus_queue.cc:1035] T af00a839bb5e402791f87288493efef6 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:02.211534 8633 raft_consensus.cc:2932] T af00a839bb5e402791f87288493efef6 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } } }
I20240503 02:00:02.212334 6582 raft_consensus.cc:2932] T af00a839bb5e402791f87288493efef6 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } } }
I20240503 02:00:02.212641 6719 raft_consensus.cc:2932] T af00a839bb5e402791f87288493efef6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } } }
I20240503 02:00:02.215493 6313 catalog_manager.cc:5484] T af00a839bb5e402791f87288493efef6 P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: config changed from index -1 to 4, NON_VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:02.226807 6582 consensus_queue.cc:237] T 1e445a3cc928429caf9e3c6193a63010 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } }
I20240503 02:00:02.228318 6458 raft_consensus.cc:1261] T 1e445a3cc928429caf9e3c6193a63010 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 02:00:02.228432 6525 consensus_peers.cc:485] T 1e445a3cc928429caf9e3c6193a63010 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211): Couldn't send request to peer 9942306855404cfd891adf227dd65eb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1e445a3cc928429caf9e3c6193a63010. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:02.228721 9178 raft_consensus.cc:1261] T 1e445a3cc928429caf9e3c6193a63010 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:02.229154 9484 consensus_queue.cc:1035] T 1e445a3cc928429caf9e3c6193a63010 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:02.229370 9973 consensus_queue.cc:1035] T 1e445a3cc928429caf9e3c6193a63010 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:02.231266 9484 raft_consensus.cc:2932] T 1e445a3cc928429caf9e3c6193a63010 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } } }
I20240503 02:00:02.231856 9178 raft_consensus.cc:2932] T 1e445a3cc928429caf9e3c6193a63010 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } } }
I20240503 02:00:02.233107 6458 raft_consensus.cc:2932] T 1e445a3cc928429caf9e3c6193a63010 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } } }
I20240503 02:00:02.235010 6315 catalog_manager.cc:5484] T 1e445a3cc928429caf9e3c6193a63010 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: config changed from index 6 to 7, NON_VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:02.237084 10670 ts_tablet_manager.cc:902] T 2fc834abc6a94c638d6202b03320a2d2 P 2a01ace80baf4057b6bf1995cf88468b: Initiating tablet copy from peer c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:02.238368 10257 raft_consensus.cc:482] T 0ee05e70ccbb496ca3cff08dc38112e8 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:02.238466 10257 raft_consensus.cc:504] T 0ee05e70ccbb496ca3cff08dc38112e8 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:02.238587 10670 tablet_copy_client.cc:308] T 2fc834abc6a94c638d6202b03320a2d2 P 2a01ace80baf4057b6bf1995cf88468b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:42655
I20240503 02:00:02.238902 10257 leader_election.cc:288] T 0ee05e70ccbb496ca3cff08dc38112e8 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:02.239003 6739 tablet_copy_service.cc:140] P c2ac2eb7cc264047b9ab32fe3a4fc025: Received BeginTabletCopySession request for tablet 2fc834abc6a94c638d6202b03320a2d2 from peer 2a01ace80baf4057b6bf1995cf88468b ({username='slave'} at 127.0.99.135:52621)
I20240503 02:00:02.239109 6739 tablet_copy_service.cc:161] P c2ac2eb7cc264047b9ab32fe3a4fc025: Beginning new tablet copy session on tablet 2fc834abc6a94c638d6202b03320a2d2 from peer 2a01ace80baf4057b6bf1995cf88468b at {username='slave'} at 127.0.99.135:52621: session id = 2a01ace80baf4057b6bf1995cf88468b-2fc834abc6a94c638d6202b03320a2d2
I20240503 02:00:02.239363 9039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ee05e70ccbb496ca3cff08dc38112e8" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:02.239481 9039 raft_consensus.cc:2454] T 0ee05e70ccbb496ca3cff08dc38112e8 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:02.239487 6719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ee05e70ccbb496ca3cff08dc38112e8" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 02:00:02.239607 6719 raft_consensus.cc:2454] T 0ee05e70ccbb496ca3cff08dc38112e8 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:02.239888 9113 leader_election.cc:302] T 0ee05e70ccbb496ca3cff08dc38112e8 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:02.240061 6739 tablet_copy_source_session.cc:205] T 2fc834abc6a94c638d6202b03320a2d2 P c2ac2eb7cc264047b9ab32fe3a4fc025: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:02.240257 10257 raft_consensus.cc:2781] T 0ee05e70ccbb496ca3cff08dc38112e8 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:02.240307 10257 raft_consensus.cc:482] T 0ee05e70ccbb496ca3cff08dc38112e8 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:02.240342 10257 raft_consensus.cc:3037] T 0ee05e70ccbb496ca3cff08dc38112e8 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:02.241361 10257 raft_consensus.cc:504] T 0ee05e70ccbb496ca3cff08dc38112e8 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:02.241641 10670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fc834abc6a94c638d6202b03320a2d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:02.241737 10257 leader_election.cc:288] T 0ee05e70ccbb496ca3cff08dc38112e8 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Requested vote from peers 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:02.242446 9039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ee05e70ccbb496ca3cff08dc38112e8" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:02.242566 9039 raft_consensus.cc:3037] T 0ee05e70ccbb496ca3cff08dc38112e8 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:02.243572 9039 raft_consensus.cc:2454] T 0ee05e70ccbb496ca3cff08dc38112e8 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:02.244110 6315 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:02.244241 6315 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:02.244836 9113 leader_election.cc:302] T 0ee05e70ccbb496ca3cff08dc38112e8 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:02.245663 6719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ee05e70ccbb496ca3cff08dc38112e8" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 02:00:02.245735 10670 tablet_copy_client.cc:783] T 2fc834abc6a94c638d6202b03320a2d2 P 2a01ace80baf4057b6bf1995cf88468b: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:02.245783 6719 raft_consensus.cc:3037] T 0ee05e70ccbb496ca3cff08dc38112e8 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:02.245992 10670 tablet_copy_client.cc:647] T 2fc834abc6a94c638d6202b03320a2d2 P 2a01ace80baf4057b6bf1995cf88468b: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:02.246568 6719 raft_consensus.cc:2454] T 0ee05e70ccbb496ca3cff08dc38112e8 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:02.247156 10257 raft_consensus.cc:2781] T 0ee05e70ccbb496ca3cff08dc38112e8 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:02.247347 10257 raft_consensus.cc:686] T 0ee05e70ccbb496ca3cff08dc38112e8 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Becoming Leader. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Running, Role: LEADER
I20240503 02:00:02.247481 10257 consensus_queue.cc:237] T 0ee05e70ccbb496ca3cff08dc38112e8 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:02.247628 6315 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet 68e0ec26972c425c8f5c1bd94ae5e0d7
I20240503 02:00:02.248220 8880 tablet_service.cc:1514] Processing DeleteTablet for tablet 68e0ec26972c425c8f5c1bd94ae5e0d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:02 UTC) from {username='slave'} at 127.0.0.1:40296
I20240503 02:00:02.248281 9285 tablet_service.cc:1514] Processing DeleteTablet for tablet 68e0ec26972c425c8f5c1bd94ae5e0d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:02 UTC) from {username='slave'} at 127.0.0.1:36004
I20240503 02:00:02.248867 10670 tablet_copy_client.cc:523] T 2fc834abc6a94c638d6202b03320a2d2 P 2a01ace80baf4057b6bf1995cf88468b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:02.249068 6829 tablet_service.cc:1514] Processing DeleteTablet for tablet 68e0ec26972c425c8f5c1bd94ae5e0d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:02 UTC) from {username='slave'} at 127.0.0.1:52812
I20240503 02:00:02.249255 9476 tablet_replica.cc:330] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P 492448570cc94ed1879feb2ec3b9051d: stopping tablet replica
I20240503 02:00:02.249385 9476 raft_consensus.cc:2229] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:02.249609 9476 raft_consensus.cc:2258] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:02.249665 10681 tablet_replica.cc:330] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P 9942306855404cfd891adf227dd65eb2: stopping tablet replica
I20240503 02:00:02.249645 6313 catalog_manager.cc:5484] T 0ee05e70ccbb496ca3cff08dc38112e8 P 2a01ace80baf4057b6bf1995cf88468b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135). New cstate: current_term: 1 leader_uuid: "2a01ace80baf4057b6bf1995cf88468b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:02.250020 10681 raft_consensus.cc:2229] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:02.250221 10681 raft_consensus.cc:2258] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:02.250502 9476 ts_tablet_manager.cc:1875] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P 492448570cc94ed1879feb2ec3b9051d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:02.251277 10681 ts_tablet_manager.cc:1875] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P 9942306855404cfd891adf227dd65eb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:02.252969 9476 ts_tablet_manager.cc:1888] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P 492448570cc94ed1879feb2ec3b9051d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:02.253051 9476 log.cc:1196] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P 492448570cc94ed1879feb2ec3b9051d: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/68e0ec26972c425c8f5c1bd94ae5e0d7
I20240503 02:00:02.253316 9476 ts_tablet_manager.cc:1909] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P 492448570cc94ed1879feb2ec3b9051d: Deleting consensus metadata
I20240503 02:00:02.253942 10681 ts_tablet_manager.cc:1888] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P 9942306855404cfd891adf227dd65eb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:02.254035 10681 log.cc:1196] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P 9942306855404cfd891adf227dd65eb2: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/68e0ec26972c425c8f5c1bd94ae5e0d7
I20240503 02:00:02.254151 6303 catalog_manager.cc:4830] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719): tablet 68e0ec26972c425c8f5c1bd94ae5e0d7 (table rebalancer_dynamic_table_new_name [id=281ac00372fe4c70b4cddfb02e3139b5]) successfully deleted
I20240503 02:00:02.254401 10681 ts_tablet_manager.cc:1909] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P 9942306855404cfd891adf227dd65eb2: Deleting consensus metadata
I20240503 02:00:02.254604 10670 tablet_bootstrap.cc:492] T 2fc834abc6a94c638d6202b03320a2d2 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:02.255427 6303 catalog_manager.cc:4830] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211): tablet 68e0ec26972c425c8f5c1bd94ae5e0d7 (table rebalancer_dynamic_table_new_name [id=281ac00372fe4c70b4cddfb02e3139b5]) successfully deleted
I20240503 02:00:02.257472 10682 tablet_replica.cc:330] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P e1a9251ce1cf4361b8b0ede9fbfc4023: stopping tablet replica
I20240503 02:00:02.262898 10682 raft_consensus.cc:2229] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:02.263106 10682 raft_consensus.cc:2258] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:02.263748 10682 ts_tablet_manager.cc:1875] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P e1a9251ce1cf4361b8b0ede9fbfc4023: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:02.264839 10685 ts_tablet_manager.cc:902] T 7df4c083550046e883e5e61d3d4e1572 P 9942306855404cfd891adf227dd65eb2: Initiating tablet copy from peer 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:02.265394 10685 tablet_copy_client.cc:308] T 7df4c083550046e883e5e61d3d4e1572 P 9942306855404cfd891adf227dd65eb2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43863
I20240503 02:00:02.266001 6609 tablet_copy_service.cc:140] P 80fd18ef4d604c86b909d5c008a46b4f: Received BeginTabletCopySession request for tablet 7df4c083550046e883e5e61d3d4e1572 from peer 9942306855404cfd891adf227dd65eb2 ({username='slave'} at 127.0.99.133:50723)
I20240503 02:00:02.266103 6609 tablet_copy_service.cc:161] P 80fd18ef4d604c86b909d5c008a46b4f: Beginning new tablet copy session on tablet 7df4c083550046e883e5e61d3d4e1572 from peer 9942306855404cfd891adf227dd65eb2 at {username='slave'} at 127.0.99.133:50723: session id = 9942306855404cfd891adf227dd65eb2-7df4c083550046e883e5e61d3d4e1572
I20240503 02:00:02.266309 10682 ts_tablet_manager.cc:1888] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:02.266433 10682 log.cc:1196] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P e1a9251ce1cf4361b8b0ede9fbfc4023: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/68e0ec26972c425c8f5c1bd94ae5e0d7
I20240503 02:00:02.266698 10682 ts_tablet_manager.cc:1909] T 68e0ec26972c425c8f5c1bd94ae5e0d7 P e1a9251ce1cf4361b8b0ede9fbfc4023: Deleting consensus metadata
I20240503 02:00:02.267045 6609 tablet_copy_source_session.cc:205] T 7df4c083550046e883e5e61d3d4e1572 P 80fd18ef4d604c86b909d5c008a46b4f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:02.267514 6301 catalog_manager.cc:4830] TS e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): tablet 68e0ec26972c425c8f5c1bd94ae5e0d7 (table rebalancer_dynamic_table_new_name [id=281ac00372fe4c70b4cddfb02e3139b5]) successfully deleted
I20240503 02:00:02.267675 6458 raft_consensus.cc:1261] T 8a76d3e2d4264687aa2972c179341b5e P dec3e4a04d604f7eaaa3f077d081218a [term 2 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:00:02.268114 10348 consensus_queue.cc:1035] T 8a76d3e2d4264687aa2972c179341b5e P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:02.268577 10685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7df4c083550046e883e5e61d3d4e1572. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:02.270836 10685 tablet_copy_client.cc:783] T 7df4c083550046e883e5e61d3d4e1572 P 9942306855404cfd891adf227dd65eb2: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:02.270990 10685 tablet_copy_client.cc:647] T 7df4c083550046e883e5e61d3d4e1572 P 9942306855404cfd891adf227dd65eb2: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:02.272125 10685 tablet_copy_client.cc:523] T 7df4c083550046e883e5e61d3d4e1572 P 9942306855404cfd891adf227dd65eb2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:02.273519 10685 tablet_bootstrap.cc:492] T 7df4c083550046e883e5e61d3d4e1572 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:02.273526 10679 raft_consensus.cc:482] T 88205f7c7ed4488e8fcfda3bfa0c4fca P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:02.273614 10679 raft_consensus.cc:504] T 88205f7c7ed4488e8fcfda3bfa0c4fca P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:02.274117 10679 leader_election.cc:288] T 88205f7c7ed4488e8fcfda3bfa0c4fca P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:02.274519 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88205f7c7ed4488e8fcfda3bfa0c4fca" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:02.274515 6582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88205f7c7ed4488e8fcfda3bfa0c4fca" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 02:00:02.274643 8916 raft_consensus.cc:2454] T 88205f7c7ed4488e8fcfda3bfa0c4fca P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:02.274653 6582 raft_consensus.cc:2454] T 88205f7c7ed4488e8fcfda3bfa0c4fca P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:02.274935 9114 leader_election.cc:302] T 88205f7c7ed4488e8fcfda3bfa0c4fca P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 80fd18ef4d604c86b909d5c008a46b4f; no voters:
I20240503 02:00:02.275177 10679 raft_consensus.cc:2781] T 88205f7c7ed4488e8fcfda3bfa0c4fca P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:02.275228 10679 raft_consensus.cc:482] T 88205f7c7ed4488e8fcfda3bfa0c4fca P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:02.275264 10679 raft_consensus.cc:3037] T 88205f7c7ed4488e8fcfda3bfa0c4fca P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:02.276538 10679 raft_consensus.cc:504] T 88205f7c7ed4488e8fcfda3bfa0c4fca P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:02.276908 10679 leader_election.cc:288] T 88205f7c7ed4488e8fcfda3bfa0c4fca P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Requested vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:02.277222 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88205f7c7ed4488e8fcfda3bfa0c4fca" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 02:00:02.277341 6589 raft_consensus.cc:3037] T 88205f7c7ed4488e8fcfda3bfa0c4fca P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:02.277410 10670 tablet_bootstrap.cc:492] T 2fc834abc6a94c638d6202b03320a2d2 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:02.277654 10670 tablet_bootstrap.cc:492] T 2fc834abc6a94c638d6202b03320a2d2 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap complete.
I20240503 02:00:02.277657 6582 consensus_queue.cc:237] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } }
I20240503 02:00:02.277819 10670 ts_tablet_manager.cc:1367] T 2fc834abc6a94c638d6202b03320a2d2 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.023s user 0.017s sys 0.005s
I20240503 02:00:02.278275 6589 raft_consensus.cc:2454] T 88205f7c7ed4488e8fcfda3bfa0c4fca P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:02.278270 10670 raft_consensus.cc:348] T 2fc834abc6a94c638d6202b03320a2d2 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } }
I20240503 02:00:02.278465 10670 raft_consensus.cc:729] T 2fc834abc6a94c638d6202b03320a2d2 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: LEARNER
I20240503 02:00:02.278601 10670 consensus_queue.cc:260] T 2fc834abc6a94c638d6202b03320a2d2 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } }
I20240503 02:00:02.278944 10670 ts_tablet_manager.cc:1398] T 2fc834abc6a94c638d6202b03320a2d2 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 02:00:02.279278 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88205f7c7ed4488e8fcfda3bfa0c4fca" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:02.279402 8916 raft_consensus.cc:3037] T 88205f7c7ed4488e8fcfda3bfa0c4fca P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:00:02.279776 6523 consensus_peers.cc:485] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 915028b3478341d588ed6031e657c259. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:02.280349 6739 tablet_copy_service.cc:342] P c2ac2eb7cc264047b9ab32fe3a4fc025: Request end of tablet copy session 2a01ace80baf4057b6bf1995cf88468b-2fc834abc6a94c638d6202b03320a2d2 received from {username='slave'} at 127.0.99.135:52621
I20240503 02:00:02.280473 6739 tablet_copy_service.cc:433] P c2ac2eb7cc264047b9ab32fe3a4fc025: ending tablet copy session 2a01ace80baf4057b6bf1995cf88468b-2fc834abc6a94c638d6202b03320a2d2 on tablet 2fc834abc6a94c638d6202b03320a2d2 with peer 2a01ace80baf4057b6bf1995cf88468b
I20240503 02:00:02.280452 8917 raft_consensus.cc:1261] T 915028b3478341d588ed6031e657c259 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:02.280532 6850 raft_consensus.cc:1261] T 915028b3478341d588ed6031e657c259 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:02.278970 9114 leader_election.cc:302] T 88205f7c7ed4488e8fcfda3bfa0c4fca P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 80fd18ef4d604c86b909d5c008a46b4f; no voters:
I20240503 02:00:02.280756 8916 raft_consensus.cc:2454] T 88205f7c7ed4488e8fcfda3bfa0c4fca P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:02.280931 10679 raft_consensus.cc:2781] T 88205f7c7ed4488e8fcfda3bfa0c4fca P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:02.281013 10679 raft_consensus.cc:686] T 88205f7c7ed4488e8fcfda3bfa0c4fca P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Becoming Leader. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Running, Role: LEADER
I20240503 02:00:02.281081 8833 consensus_queue.cc:1035] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:02.281131 10679 consensus_queue.cc:237] T 88205f7c7ed4488e8fcfda3bfa0c4fca P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:02.281497 10345 consensus_queue.cc:1035] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:02.282843 8833 raft_consensus.cc:2932] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } } }
I20240503 02:00:02.285642 6313 catalog_manager.cc:5484] T 88205f7c7ed4488e8fcfda3bfa0c4fca P 2a01ace80baf4057b6bf1995cf88468b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135). New cstate: current_term: 1 leader_uuid: "2a01ace80baf4057b6bf1995cf88468b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:02.285964 6315 catalog_manager.cc:5484] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: config changed from index 6 to 7, NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:02.287328 6849 raft_consensus.cc:2932] T 915028b3478341d588ed6031e657c259 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } } }
I20240503 02:00:02.288861 8916 raft_consensus.cc:2932] T 915028b3478341d588ed6031e657c259 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } } }
I20240503 02:00:02.299170 8916 consensus_queue.cc:237] T f28586e70bd9473fbdc6b9126b666fa9 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: true } }
I20240503 02:00:02.300709 9309 raft_consensus.cc:1261] T f28586e70bd9473fbdc6b9126b666fa9 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:02.301184 9178 raft_consensus.cc:1261] T f28586e70bd9473fbdc6b9126b666fa9 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:02.301539 9533 consensus_queue.cc:1035] T f28586e70bd9473fbdc6b9126b666fa9 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:02.301738 9758 consensus_queue.cc:1035] T f28586e70bd9473fbdc6b9126b666fa9 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:00:02.301967 8852 consensus_peers.cc:485] T f28586e70bd9473fbdc6b9126b666fa9 P 9942306855404cfd891adf227dd65eb2 -> Peer 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): Couldn't send request to peer 80fd18ef4d604c86b909d5c008a46b4f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f28586e70bd9473fbdc6b9126b666fa9. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:02.303601 9758 raft_consensus.cc:2932] T f28586e70bd9473fbdc6b9126b666fa9 P 9942306855404cfd891adf227dd65eb2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: true } } }
I20240503 02:00:02.303962 10685 tablet_bootstrap.cc:492] T 7df4c083550046e883e5e61d3d4e1572 P 9942306855404cfd891adf227dd65eb2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:02.304237 10685 tablet_bootstrap.cc:492] T 7df4c083550046e883e5e61d3d4e1572 P 9942306855404cfd891adf227dd65eb2: Bootstrap complete.
I20240503 02:00:02.304183 9309 raft_consensus.cc:2932] T f28586e70bd9473fbdc6b9126b666fa9 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: true } } }
I20240503 02:00:02.304440 10685 ts_tablet_manager.cc:1367] T 7df4c083550046e883e5e61d3d4e1572 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.031s user 0.022s sys 0.004s
I20240503 02:00:02.304913 10685 raft_consensus.cc:348] T 7df4c083550046e883e5e61d3d4e1572 P 9942306855404cfd891adf227dd65eb2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } }
I20240503 02:00:02.305068 10685 raft_consensus.cc:729] T 7df4c083550046e883e5e61d3d4e1572 P 9942306855404cfd891adf227dd65eb2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: LEARNER
I20240503 02:00:02.305179 10685 consensus_queue.cc:260] T 7df4c083550046e883e5e61d3d4e1572 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } }
I20240503 02:00:02.305480 10685 ts_tablet_manager.cc:1398] T 7df4c083550046e883e5e61d3d4e1572 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:02.306084 6609 tablet_copy_service.cc:342] P 80fd18ef4d604c86b909d5c008a46b4f: Request end of tablet copy session 9942306855404cfd891adf227dd65eb2-7df4c083550046e883e5e61d3d4e1572 received from {username='slave'} at 127.0.99.133:50723
I20240503 02:00:02.306181 6609 tablet_copy_service.cc:433] P 80fd18ef4d604c86b909d5c008a46b4f: ending tablet copy session 9942306855404cfd891adf227dd65eb2-7df4c083550046e883e5e61d3d4e1572 on tablet 7df4c083550046e883e5e61d3d4e1572 with peer 9942306855404cfd891adf227dd65eb2
I20240503 02:00:02.306164 9178 raft_consensus.cc:2932] T f28586e70bd9473fbdc6b9126b666fa9 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: true } } }
I20240503 02:00:02.306929 6315 catalog_manager.cc:5484] T f28586e70bd9473fbdc6b9126b666fa9 P e1a9251ce1cf4361b8b0ede9fbfc4023 reported cstate change: config changed from index -1 to 2, NON_VOTER 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "9942306855404cfd891adf227dd65eb2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: true } } }
I20240503 02:00:02.315333 6458 consensus_queue.cc:237] T ac1632cc5eb94554b1faa7eb715706ef P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } }
I20240503 02:00:02.316921 6719 raft_consensus.cc:1261] T ac1632cc5eb94554b1faa7eb715706ef P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:02.317126 6589 raft_consensus.cc:1261] T ac1632cc5eb94554b1faa7eb715706ef P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:02.317328 10382 consensus_queue.cc:1035] T ac1632cc5eb94554b1faa7eb715706ef P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:00:02.317530 6392 consensus_peers.cc:485] T ac1632cc5eb94554b1faa7eb715706ef P dec3e4a04d604f7eaaa3f077d081218a -> Peer 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157): Couldn't send request to peer 99ead02915954b4fab23b033ad6d546b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ac1632cc5eb94554b1faa7eb715706ef. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:02.317533 9955 consensus_queue.cc:1035] T ac1632cc5eb94554b1faa7eb715706ef P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:02.319618 9955 raft_consensus.cc:2932] T ac1632cc5eb94554b1faa7eb715706ef P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } } }
I20240503 02:00:02.320194 6719 raft_consensus.cc:2932] T ac1632cc5eb94554b1faa7eb715706ef P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } } }
I20240503 02:00:02.320725 6589 raft_consensus.cc:2932] T ac1632cc5eb94554b1faa7eb715706ef P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } } }
I20240503 02:00:02.322933 6313 catalog_manager.cc:5484] T ac1632cc5eb94554b1faa7eb715706ef P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: config changed from index -1 to 4, NON_VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:02.351608 6316 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:00:02.360123 8880 tablet_service.cc:1467] Processing CreateTablet for tablet f2716f6d9e554ec8a8b922a75e5d5d75 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=58ca374f38374792b2e80195b318352c]), partition=
I20240503 02:00:02.360307 6438 tablet_service.cc:1467] Processing CreateTablet for tablet f2716f6d9e554ec8a8b922a75e5d5d75 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=58ca374f38374792b2e80195b318352c]), partition=
I20240503 02:00:02.360637 9158 tablet_service.cc:1467] Processing CreateTablet for tablet f2716f6d9e554ec8a8b922a75e5d5d75 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=58ca374f38374792b2e80195b318352c]), partition=
I20240503 02:00:02.360795 8880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2716f6d9e554ec8a8b922a75e5d5d75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:02.360875 9158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2716f6d9e554ec8a8b922a75e5d5d75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:02.362607 10642 tablet_bootstrap.cc:492] T f2716f6d9e554ec8a8b922a75e5d5d75 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:02.362836 10636 tablet_bootstrap.cc:492] T f2716f6d9e554ec8a8b922a75e5d5d75 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:02.363370 6438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2716f6d9e554ec8a8b922a75e5d5d75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:02.363584 10642 tablet_bootstrap.cc:654] T f2716f6d9e554ec8a8b922a75e5d5d75 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:02.363657 10636 tablet_bootstrap.cc:654] T f2716f6d9e554ec8a8b922a75e5d5d75 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:02.365401 10636 tablet_bootstrap.cc:492] T f2716f6d9e554ec8a8b922a75e5d5d75 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:02.365494 10636 ts_tablet_manager.cc:1367] T f2716f6d9e554ec8a8b922a75e5d5d75 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:00:02.365968 10636 raft_consensus.cc:348] T f2716f6d9e554ec8a8b922a75e5d5d75 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:02.366083 10636 raft_consensus.cc:374] T f2716f6d9e554ec8a8b922a75e5d5d75 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:02.366117 10636 raft_consensus.cc:729] T f2716f6d9e554ec8a8b922a75e5d5d75 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:02.366272 10636 consensus_queue.cc:260] T f2716f6d9e554ec8a8b922a75e5d5d75 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:02.366473 10642 tablet_bootstrap.cc:492] T f2716f6d9e554ec8a8b922a75e5d5d75 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:02.366556 10642 ts_tablet_manager.cc:1367] T f2716f6d9e554ec8a8b922a75e5d5d75 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:02.366569 10636 ts_tablet_manager.cc:1398] T f2716f6d9e554ec8a8b922a75e5d5d75 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:02.366986 10642 raft_consensus.cc:348] T f2716f6d9e554ec8a8b922a75e5d5d75 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:02.367110 10642 raft_consensus.cc:374] T f2716f6d9e554ec8a8b922a75e5d5d75 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:02.367139 10642 raft_consensus.cc:729] T f2716f6d9e554ec8a8b922a75e5d5d75 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:02.367259 10642 consensus_queue.cc:260] T f2716f6d9e554ec8a8b922a75e5d5d75 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:02.367517 10642 ts_tablet_manager.cc:1398] T f2716f6d9e554ec8a8b922a75e5d5d75 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:02.372648 10739 tablet_bootstrap.cc:492] T f2716f6d9e554ec8a8b922a75e5d5d75 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:02.373607 10739 tablet_bootstrap.cc:654] T f2716f6d9e554ec8a8b922a75e5d5d75 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:02.377291 10739 tablet_bootstrap.cc:492] T f2716f6d9e554ec8a8b922a75e5d5d75 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:02.377404 10739 ts_tablet_manager.cc:1367] T f2716f6d9e554ec8a8b922a75e5d5d75 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:00:02.377827 10739 raft_consensus.cc:348] T f2716f6d9e554ec8a8b922a75e5d5d75 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:02.377939 10739 raft_consensus.cc:374] T f2716f6d9e554ec8a8b922a75e5d5d75 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:02.378613 10739 raft_consensus.cc:729] T f2716f6d9e554ec8a8b922a75e5d5d75 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:02.378960 10739 consensus_queue.cc:260] T f2716f6d9e554ec8a8b922a75e5d5d75 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:02.379299 10739 ts_tablet_manager.cc:1398] T f2716f6d9e554ec8a8b922a75e5d5d75 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:02.402995 9178 raft_consensus.cc:1261] T 8a76d3e2d4264687aa2972c179341b5e P 2a01ace80baf4057b6bf1995cf88468b [term 2 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:00:02.403486 10348 consensus_queue.cc:1035] T 8a76d3e2d4264687aa2972c179341b5e P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:02.409786 6849 raft_consensus.cc:1261] T 8a76d3e2d4264687aa2972c179341b5e P 492448570cc94ed1879feb2ec3b9051d [term 2 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:00:02.410578 10349 consensus_queue.cc:1035] T 8a76d3e2d4264687aa2972c179341b5e P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:02.412122 9309 raft_consensus.cc:1203] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:00:02.419517 10349 raft_consensus.cc:482] T cea1ea50fa434d31b42451f84f26bc76 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:02.419644 10349 raft_consensus.cc:504] T cea1ea50fa434d31b42451f84f26bc76 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:02.420082 10349 leader_election.cc:288] T cea1ea50fa434d31b42451f84f26bc76 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:02.421200 9039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cea1ea50fa434d31b42451f84f26bc76" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:02.421356 9039 raft_consensus.cc:2454] T cea1ea50fa434d31b42451f84f26bc76 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:02.421900 6654 leader_election.cc:302] T cea1ea50fa434d31b42451f84f26bc76 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99ead02915954b4fab23b033ad6d546b, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:02.422060 10349 raft_consensus.cc:2781] T cea1ea50fa434d31b42451f84f26bc76 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:02.422099 10349 raft_consensus.cc:482] T cea1ea50fa434d31b42451f84f26bc76 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:02.422132 10349 raft_consensus.cc:3037] T cea1ea50fa434d31b42451f84f26bc76 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:02.423169 10349 raft_consensus.cc:504] T cea1ea50fa434d31b42451f84f26bc76 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:02.423527 10349 leader_election.cc:288] T cea1ea50fa434d31b42451f84f26bc76 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Requested vote from peers e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:02.424183 9039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cea1ea50fa434d31b42451f84f26bc76" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:02.424310 9039 raft_consensus.cc:3037] T cea1ea50fa434d31b42451f84f26bc76 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:02.426019 9039 raft_consensus.cc:2454] T cea1ea50fa434d31b42451f84f26bc76 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:02.426290 6654 leader_election.cc:302] T cea1ea50fa434d31b42451f84f26bc76 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99ead02915954b4fab23b033ad6d546b, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:02.426455 10349 raft_consensus.cc:2781] T cea1ea50fa434d31b42451f84f26bc76 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:02.426539 10349 raft_consensus.cc:686] T cea1ea50fa434d31b42451f84f26bc76 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Becoming Leader. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Running, Role: LEADER
I20240503 02:00:02.426684 10349 consensus_queue.cc:237] T cea1ea50fa434d31b42451f84f26bc76 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:02.426851 9309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cea1ea50fa434d31b42451f84f26bc76" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" is_pre_election: true
I20240503 02:00:02.426976 9309 raft_consensus.cc:2454] T cea1ea50fa434d31b42451f84f26bc76 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:02.427165 9310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cea1ea50fa434d31b42451f84f26bc76" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023"
I20240503 02:00:02.427256 9310 raft_consensus.cc:3037] T cea1ea50fa434d31b42451f84f26bc76 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:02.428105 9310 raft_consensus.cc:2454] T cea1ea50fa434d31b42451f84f26bc76 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:02.428966 6315 catalog_manager.cc:5484] T cea1ea50fa434d31b42451f84f26bc76 P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:02.471714 10802 ts_tablet_manager.cc:902] T 925612b660ec471da5b3c548802ede6e P e1a9251ce1cf4361b8b0ede9fbfc4023: Initiating tablet copy from peer 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:02.472501 10802 tablet_copy_client.cc:308] T 925612b660ec471da5b3c548802ede6e P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43863
I20240503 02:00:02.473011 6609 tablet_copy_service.cc:140] P 80fd18ef4d604c86b909d5c008a46b4f: Received BeginTabletCopySession request for tablet 925612b660ec471da5b3c548802ede6e from peer e1a9251ce1cf4361b8b0ede9fbfc4023 ({username='slave'} at 127.0.99.136:60117)
I20240503 02:00:02.473147 6609 tablet_copy_service.cc:161] P 80fd18ef4d604c86b909d5c008a46b4f: Beginning new tablet copy session on tablet 925612b660ec471da5b3c548802ede6e from peer e1a9251ce1cf4361b8b0ede9fbfc4023 at {username='slave'} at 127.0.99.136:60117: session id = e1a9251ce1cf4361b8b0ede9fbfc4023-925612b660ec471da5b3c548802ede6e
I20240503 02:00:02.474016 6609 tablet_copy_source_session.cc:205] T 925612b660ec471da5b3c548802ede6e P 80fd18ef4d604c86b909d5c008a46b4f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:02.474929 10802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 925612b660ec471da5b3c548802ede6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:02.477126 10802 tablet_copy_client.cc:783] T 925612b660ec471da5b3c548802ede6e P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:02.477280 10802 tablet_copy_client.cc:647] T 925612b660ec471da5b3c548802ede6e P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:02.478482 10802 tablet_copy_client.cc:523] T 925612b660ec471da5b3c548802ede6e P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:02.478765 6313 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240503 02:00:02.478874 6313 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240503 02:00:02.479907 10802 tablet_bootstrap.cc:492] T 925612b660ec471da5b3c548802ede6e P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap starting.
I20240503 02:00:02.481566 6313 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet 7a4a910a8b9d4134adebb52a73dfae1d
I20240503 02:00:02.481930 6313 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet ceca04391cd446ae932b3485d958237a
I20240503 02:00:02.482151 6313 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet 70d0c699b33e44dcb7136f3d0456ce9c
I20240503 02:00:02.482234 6438 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a4a910a8b9d4134adebb52a73dfae1d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:02 UTC) from {username='slave'} at 127.0.0.1:51338
I20240503 02:00:02.482471 9474 tablet_replica.cc:330] T 7a4a910a8b9d4134adebb52a73dfae1d P dec3e4a04d604f7eaaa3f077d081218a: stopping tablet replica
I20240503 02:00:02.482625 9474 raft_consensus.cc:2229] T 7a4a910a8b9d4134adebb52a73dfae1d P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:02.482733 6569 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a4a910a8b9d4134adebb52a73dfae1d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:02 UTC) from {username='slave'} at 127.0.0.1:59846
I20240503 02:00:02.482802 6567 tablet_service.cc:1514] Processing DeleteTablet for tablet 70d0c699b33e44dcb7136f3d0456ce9c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:02 UTC) from {username='slave'} at 127.0.0.1:59846
I20240503 02:00:02.482829 9474 raft_consensus.cc:2258] T 7a4a910a8b9d4134adebb52a73dfae1d P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:02.482929 9473 tablet_replica.cc:330] T 7a4a910a8b9d4134adebb52a73dfae1d P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 02:00:02.483047 9473 raft_consensus.cc:2229] T 7a4a910a8b9d4134adebb52a73dfae1d P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:02.483250 9473 raft_consensus.cc:2258] T 7a4a910a8b9d4134adebb52a73dfae1d P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:02.483322 6829 tablet_service.cc:1514] Processing DeleteTablet for tablet ceca04391cd446ae932b3485d958237a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:02 UTC) from {username='slave'} at 127.0.0.1:52812
I20240503 02:00:02.483474 6831 tablet_service.cc:1514] Processing DeleteTablet for tablet 70d0c699b33e44dcb7136f3d0456ce9c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:02 UTC) from {username='slave'} at 127.0.0.1:52812
I20240503 02:00:02.483551 9474 ts_tablet_manager.cc:1875] T 7a4a910a8b9d4134adebb52a73dfae1d P dec3e4a04d604f7eaaa3f077d081218a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:02.483598 9476 tablet_replica.cc:330] T ceca04391cd446ae932b3485d958237a P 492448570cc94ed1879feb2ec3b9051d: stopping tablet replica
I20240503 02:00:02.483711 9476 raft_consensus.cc:2229] T ceca04391cd446ae932b3485d958237a P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:02.483879 9476 raft_consensus.cc:2258] T ceca04391cd446ae932b3485d958237a P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:02.483960 9473 ts_tablet_manager.cc:1875] T 7a4a910a8b9d4134adebb52a73dfae1d P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:02.484365 9476 ts_tablet_manager.cc:1875] T ceca04391cd446ae932b3485d958237a P 492448570cc94ed1879feb2ec3b9051d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:02.484480 6438 tablet_service.cc:1514] Processing DeleteTablet for tablet 70d0c699b33e44dcb7136f3d0456ce9c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:02 UTC) from {username='slave'} at 127.0.0.1:51338
I20240503 02:00:02.484557 9025 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a4a910a8b9d4134adebb52a73dfae1d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:02 UTC) from {username='slave'} at 127.0.0.1:52584
I20240503 02:00:02.486158 9474 ts_tablet_manager.cc:1888] T 7a4a910a8b9d4134adebb52a73dfae1d P dec3e4a04d604f7eaaa3f077d081218a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240503 02:00:02.486217 9474 log.cc:1196] T 7a4a910a8b9d4134adebb52a73dfae1d P dec3e4a04d604f7eaaa3f077d081218a: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/7a4a910a8b9d4134adebb52a73dfae1d
I20240503 02:00:02.486357 9473 ts_tablet_manager.cc:1888] T 7a4a910a8b9d4134adebb52a73dfae1d P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240503 02:00:02.486426 9473 log.cc:1196] T 7a4a910a8b9d4134adebb52a73dfae1d P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/7a4a910a8b9d4134adebb52a73dfae1d
I20240503 02:00:02.486430 9474 ts_tablet_manager.cc:1909] T 7a4a910a8b9d4134adebb52a73dfae1d P dec3e4a04d604f7eaaa3f077d081218a: Deleting consensus metadata
I20240503 02:00:02.486696 9473 ts_tablet_manager.cc:1909] T 7a4a910a8b9d4134adebb52a73dfae1d P 80fd18ef4d604c86b909d5c008a46b4f: Deleting consensus metadata
I20240503 02:00:02.486785 9476 ts_tablet_manager.cc:1888] T ceca04391cd446ae932b3485d958237a P 492448570cc94ed1879feb2ec3b9051d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240503 02:00:02.486850 9476 log.cc:1196] T ceca04391cd446ae932b3485d958237a P 492448570cc94ed1879feb2ec3b9051d: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/ceca04391cd446ae932b3485d958237a
I20240503 02:00:02.486989 9474 tablet_replica.cc:330] T 70d0c699b33e44dcb7136f3d0456ce9c P dec3e4a04d604f7eaaa3f077d081218a: stopping tablet replica
I20240503 02:00:02.487059 9476 ts_tablet_manager.cc:1909] T ceca04391cd446ae932b3485d958237a P 492448570cc94ed1879feb2ec3b9051d: Deleting consensus metadata
I20240503 02:00:02.487105 9474 raft_consensus.cc:2229] T 70d0c699b33e44dcb7136f3d0456ce9c P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:02.487145 6300 catalog_manager.cc:4830] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): tablet 7a4a910a8b9d4134adebb52a73dfae1d (table rebalance_test_table_14 [id=7455f5480e88494b83e1ac1d180ca4e6]) successfully deleted
I20240503 02:00:02.487282 9473 tablet_replica.cc:330] T 70d0c699b33e44dcb7136f3d0456ce9c P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 02:00:02.487308 9474 raft_consensus.cc:2258] T 70d0c699b33e44dcb7136f3d0456ce9c P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:02.487332 8880 tablet_service.cc:1514] Processing DeleteTablet for tablet ceca04391cd446ae932b3485d958237a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:02 UTC) from {username='slave'} at 127.0.0.1:40296
I20240503 02:00:02.487403 9473 raft_consensus.cc:2229] T 70d0c699b33e44dcb7136f3d0456ce9c P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:02.487509 10681 tablet_replica.cc:330] T ceca04391cd446ae932b3485d958237a P 9942306855404cfd891adf227dd65eb2: stopping tablet replica
I20240503 02:00:02.487599 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet 7a4a910a8b9d4134adebb52a73dfae1d (table rebalance_test_table_14 [id=7455f5480e88494b83e1ac1d180ca4e6]) successfully deleted
I20240503 02:00:02.487623 10681 raft_consensus.cc:2229] T ceca04391cd446ae932b3485d958237a P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:02.487749 10681 raft_consensus.cc:2258] T ceca04391cd446ae932b3485d958237a P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:02.487797 9474 ts_tablet_manager.cc:1875] T 70d0c699b33e44dcb7136f3d0456ce9c P dec3e4a04d604f7eaaa3f077d081218a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:02.488121 6303 catalog_manager.cc:4830] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719): tablet ceca04391cd446ae932b3485d958237a (table rebalance_test_table_14 [id=7455f5480e88494b83e1ac1d180ca4e6]) successfully deleted
I20240503 02:00:02.488233 10681 ts_tablet_manager.cc:1875] T ceca04391cd446ae932b3485d958237a P 9942306855404cfd891adf227dd65eb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:02.488322 9476 tablet_replica.cc:330] T 70d0c699b33e44dcb7136f3d0456ce9c P 492448570cc94ed1879feb2ec3b9051d: stopping tablet replica
I20240503 02:00:02.488481 9476 raft_consensus.cc:2229] T 70d0c699b33e44dcb7136f3d0456ce9c P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:02.488610 9476 raft_consensus.cc:2258] T 70d0c699b33e44dcb7136f3d0456ce9c P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:02.489073 9476 ts_tablet_manager.cc:1875] T 70d0c699b33e44dcb7136f3d0456ce9c P 492448570cc94ed1879feb2ec3b9051d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:02.490068 10601 tablet_replica.cc:330] T 7a4a910a8b9d4134adebb52a73dfae1d P 99ead02915954b4fab23b033ad6d546b: stopping tablet replica
I20240503 02:00:02.490193 10681 ts_tablet_manager.cc:1888] T ceca04391cd446ae932b3485d958237a P 9942306855404cfd891adf227dd65eb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240503 02:00:02.490201 10601 raft_consensus.cc:2229] T 7a4a910a8b9d4134adebb52a73dfae1d P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:02.490257 10681 log.cc:1196] T ceca04391cd446ae932b3485d958237a P 9942306855404cfd891adf227dd65eb2: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/ceca04391cd446ae932b3485d958237a
I20240503 02:00:02.490340 10601 raft_consensus.cc:2258] T 7a4a910a8b9d4134adebb52a73dfae1d P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:02.490504 10681 ts_tablet_manager.cc:1909] T ceca04391cd446ae932b3485d958237a P 9942306855404cfd891adf227dd65eb2: Deleting consensus metadata
I20240503 02:00:02.490705 6700 tablet_service.cc:1514] Processing DeleteTablet for tablet ceca04391cd446ae932b3485d958237a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:02 UTC) from {username='slave'} at 127.0.0.1:40336
I20240503 02:00:02.491003 10173 tablet_replica.cc:330] T ceca04391cd446ae932b3485d958237a P c2ac2eb7cc264047b9ab32fe3a4fc025: stopping tablet replica
I20240503 02:00:02.491154 10173 raft_consensus.cc:2229] T ceca04391cd446ae932b3485d958237a P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:02.491302 9476 ts_tablet_manager.cc:1888] T 70d0c699b33e44dcb7136f3d0456ce9c P 492448570cc94ed1879feb2ec3b9051d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:00:02.491369 9476 log.cc:1196] T 70d0c699b33e44dcb7136f3d0456ce9c P 492448570cc94ed1879feb2ec3b9051d: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/70d0c699b33e44dcb7136f3d0456ce9c
I20240503 02:00:02.491356 10173 raft_consensus.cc:2258] T ceca04391cd446ae932b3485d958237a P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:02.491595 9476 ts_tablet_manager.cc:1909] T 70d0c699b33e44dcb7136f3d0456ce9c P 492448570cc94ed1879feb2ec3b9051d: Deleting consensus metadata
I20240503 02:00:02.491889 10173 ts_tablet_manager.cc:1875] T ceca04391cd446ae932b3485d958237a P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:02.490177 9474 ts_tablet_manager.cc:1888] T 70d0c699b33e44dcb7136f3d0456ce9c P dec3e4a04d604f7eaaa3f077d081218a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:00:02.492492 9474 log.cc:1196] T 70d0c699b33e44dcb7136f3d0456ce9c P dec3e4a04d604f7eaaa3f077d081218a: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/70d0c699b33e44dcb7136f3d0456ce9c
I20240503 02:00:02.492480 6303 catalog_manager.cc:4830] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719): tablet 70d0c699b33e44dcb7136f3d0456ce9c (table rebalance_test_table_14 [id=7455f5480e88494b83e1ac1d180ca4e6]) successfully deleted
I20240503 02:00:02.492728 9474 ts_tablet_manager.cc:1909] T 70d0c699b33e44dcb7136f3d0456ce9c P dec3e4a04d604f7eaaa3f077d081218a: Deleting consensus metadata
I20240503 02:00:02.492784 10601 ts_tablet_manager.cc:1875] T 7a4a910a8b9d4134adebb52a73dfae1d P 99ead02915954b4fab23b033ad6d546b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:02.493249 10802 tablet_bootstrap.cc:492] T 925612b660ec471da5b3c548802ede6e P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:02.493439 10802 tablet_bootstrap.cc:492] T 925612b660ec471da5b3c548802ede6e P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap complete.
I20240503 02:00:02.493546 10802 ts_tablet_manager.cc:1367] T 925612b660ec471da5b3c548802ede6e P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20240503 02:00:02.493923 10802 raft_consensus.cc:348] T 925612b660ec471da5b3c548802ede6e P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } }
I20240503 02:00:02.494010 10802 raft_consensus.cc:729] T 925612b660ec471da5b3c548802ede6e P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e1a9251ce1cf4361b8b0ede9fbfc4023, State: Initialized, Role: LEARNER
I20240503 02:00:02.493992 10173 ts_tablet_manager.cc:1888] T ceca04391cd446ae932b3485d958237a P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240503 02:00:02.494053 10173 log.cc:1196] T ceca04391cd446ae932b3485d958237a P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/ceca04391cd446ae932b3485d958237a
I20240503 02:00:02.494081 10802 consensus_queue.cc:260] T 925612b660ec471da5b3c548802ede6e P e1a9251ce1cf4361b8b0ede9fbfc4023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } }
I20240503 02:00:02.494279 10173 ts_tablet_manager.cc:1909] T ceca04391cd446ae932b3485d958237a P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting consensus metadata
I20240503 02:00:02.494341 10802 ts_tablet_manager.cc:1398] T 925612b660ec471da5b3c548802ede6e P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:02.494884 10601 ts_tablet_manager.cc:1888] T 7a4a910a8b9d4134adebb52a73dfae1d P 99ead02915954b4fab23b033ad6d546b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240503 02:00:02.494953 10601 log.cc:1196] T 7a4a910a8b9d4134adebb52a73dfae1d P 99ead02915954b4fab23b033ad6d546b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/7a4a910a8b9d4134adebb52a73dfae1d
I20240503 02:00:02.495239 6609 tablet_copy_service.cc:342] P 80fd18ef4d604c86b909d5c008a46b4f: Request end of tablet copy session e1a9251ce1cf4361b8b0ede9fbfc4023-925612b660ec471da5b3c548802ede6e received from {username='slave'} at 127.0.99.136:60117
I20240503 02:00:02.495313 6609 tablet_copy_service.cc:433] P 80fd18ef4d604c86b909d5c008a46b4f: ending tablet copy session e1a9251ce1cf4361b8b0ede9fbfc4023-925612b660ec471da5b3c548802ede6e on tablet 925612b660ec471da5b3c548802ede6e with peer e1a9251ce1cf4361b8b0ede9fbfc4023
I20240503 02:00:02.496587 6300 catalog_manager.cc:4830] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): tablet 70d0c699b33e44dcb7136f3d0456ce9c (table rebalance_test_table_14 [id=7455f5480e88494b83e1ac1d180ca4e6]) successfully deleted
I20240503 02:00:02.496904 9473 raft_consensus.cc:2258] T 70d0c699b33e44dcb7136f3d0456ce9c P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:02.497030 6303 catalog_manager.cc:4830] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211): tablet ceca04391cd446ae932b3485d958237a (table rebalance_test_table_14 [id=7455f5480e88494b83e1ac1d180ca4e6]) successfully deleted
I20240503 02:00:02.497159 6303 catalog_manager.cc:4830] TS c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655): tablet ceca04391cd446ae932b3485d958237a (table rebalance_test_table_14 [id=7455f5480e88494b83e1ac1d180ca4e6]) successfully deleted
I20240503 02:00:02.497357 10601 ts_tablet_manager.cc:1909] T 7a4a910a8b9d4134adebb52a73dfae1d P 99ead02915954b4fab23b033ad6d546b: Deleting consensus metadata
I20240503 02:00:02.498057 6301 catalog_manager.cc:4830] TS 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157): tablet 7a4a910a8b9d4134adebb52a73dfae1d (table rebalance_test_table_14 [id=7455f5480e88494b83e1ac1d180ca4e6]) successfully deleted
I20240503 02:00:02.498328 9473 ts_tablet_manager.cc:1875] T 70d0c699b33e44dcb7136f3d0456ce9c P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:02.500032 9473 ts_tablet_manager.cc:1888] T 70d0c699b33e44dcb7136f3d0456ce9c P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:00:02.500098 9473 log.cc:1196] T 70d0c699b33e44dcb7136f3d0456ce9c P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/70d0c699b33e44dcb7136f3d0456ce9c
I20240503 02:00:02.500288 9473 ts_tablet_manager.cc:1909] T 70d0c699b33e44dcb7136f3d0456ce9c P 80fd18ef4d604c86b909d5c008a46b4f: Deleting consensus metadata
I20240503 02:00:02.501091 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet 70d0c699b33e44dcb7136f3d0456ce9c (table rebalance_test_table_14 [id=7455f5480e88494b83e1ac1d180ca4e6]) successfully deleted
I20240503 02:00:02.505795 6458 consensus_queue.cc:237] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } }
I20240503 02:00:02.507211 6849 raft_consensus.cc:1261] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:00:02.507352 6392 consensus_peers.cc:485] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a -> Peer 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157): Couldn't send request to peer 99ead02915954b4fab23b033ad6d546b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2d7244b0dc4d419cb5c3f3677acd70c7. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:02.507634 6719 raft_consensus.cc:1261] T 2d7244b0dc4d419cb5c3f3677acd70c7 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:02.507602 9721 consensus_queue.cc:1035] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:02.507959 9721 consensus_queue.cc:1035] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:02.509503 8679 raft_consensus.cc:2932] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } } }
I20240503 02:00:02.509969 6719 raft_consensus.cc:2932] T 2d7244b0dc4d419cb5c3f3677acd70c7 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } } }
I20240503 02:00:02.509954 6849 raft_consensus.cc:2932] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } } }
I20240503 02:00:02.512847 6315 catalog_manager.cc:5484] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 492448570cc94ed1879feb2ec3b9051d reported cstate change: config changed from index -1 to 4, NON_VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } } }
I20240503 02:00:02.516201 10826 ts_tablet_manager.cc:902] T 90ef3584830240acb0d8aad14ae851a6 P 99ead02915954b4fab23b033ad6d546b: Initiating tablet copy from peer 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:02.517074 10826 tablet_copy_client.cc:308] T 90ef3584830240acb0d8aad14ae851a6 P 99ead02915954b4fab23b033ad6d546b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:42719
I20240503 02:00:02.517653 6871 tablet_copy_service.cc:140] P 492448570cc94ed1879feb2ec3b9051d: Received BeginTabletCopySession request for tablet 90ef3584830240acb0d8aad14ae851a6 from peer 99ead02915954b4fab23b033ad6d546b ({username='slave'} at 127.0.99.134:60599)
I20240503 02:00:02.517771 6871 tablet_copy_service.cc:161] P 492448570cc94ed1879feb2ec3b9051d: Beginning new tablet copy session on tablet 90ef3584830240acb0d8aad14ae851a6 from peer 99ead02915954b4fab23b033ad6d546b at {username='slave'} at 127.0.99.134:60599: session id = 99ead02915954b4fab23b033ad6d546b-90ef3584830240acb0d8aad14ae851a6
I20240503 02:00:02.518746 6871 tablet_copy_source_session.cc:205] T 90ef3584830240acb0d8aad14ae851a6 P 492448570cc94ed1879feb2ec3b9051d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:02.519683 10826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90ef3584830240acb0d8aad14ae851a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:02.521931 10826 tablet_copy_client.cc:783] T 90ef3584830240acb0d8aad14ae851a6 P 99ead02915954b4fab23b033ad6d546b: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:02.521972 6589 consensus_queue.cc:237] T 4385d51d11574192883d01376e2d5620 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } }
I20240503 02:00:02.522109 10826 tablet_copy_client.cc:647] T 90ef3584830240acb0d8aad14ae851a6 P 99ead02915954b4fab23b033ad6d546b: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:02.523553 6458 raft_consensus.cc:1261] T 4385d51d11574192883d01376e2d5620 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:00:02.523695 6523 consensus_peers.cc:485] T 4385d51d11574192883d01376e2d5620 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4385d51d11574192883d01376e2d5620. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:02.523898 6849 raft_consensus.cc:1261] T 4385d51d11574192883d01376e2d5620 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:02.523952 10826 tablet_copy_client.cc:523] T 90ef3584830240acb0d8aad14ae851a6 P 99ead02915954b4fab23b033ad6d546b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:02.524247 10345 consensus_queue.cc:1035] T 4385d51d11574192883d01376e2d5620 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:02.524447 9973 consensus_queue.cc:1035] T 4385d51d11574192883d01376e2d5620 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:02.526392 8833 raft_consensus.cc:2932] T 4385d51d11574192883d01376e2d5620 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } } }
I20240503 02:00:02.526593 10826 tablet_bootstrap.cc:492] T 90ef3584830240acb0d8aad14ae851a6 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:02.526810 6849 raft_consensus.cc:2932] T 4385d51d11574192883d01376e2d5620 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } } }
I20240503 02:00:02.529096 10670 ts_tablet_manager.cc:902] T 4c13642db7fc40a5a934aa5e3c2b08fc P 2a01ace80baf4057b6bf1995cf88468b: Initiating tablet copy from peer c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:02.529635 10670 tablet_copy_client.cc:308] T 4c13642db7fc40a5a934aa5e3c2b08fc P 2a01ace80baf4057b6bf1995cf88468b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:42655
I20240503 02:00:02.530180 6739 tablet_copy_service.cc:140] P c2ac2eb7cc264047b9ab32fe3a4fc025: Received BeginTabletCopySession request for tablet 4c13642db7fc40a5a934aa5e3c2b08fc from peer 2a01ace80baf4057b6bf1995cf88468b ({username='slave'} at 127.0.99.135:52621)
I20240503 02:00:02.530300 6739 tablet_copy_service.cc:161] P c2ac2eb7cc264047b9ab32fe3a4fc025: Beginning new tablet copy session on tablet 4c13642db7fc40a5a934aa5e3c2b08fc from peer 2a01ace80baf4057b6bf1995cf88468b at {username='slave'} at 127.0.99.135:52621: session id = 2a01ace80baf4057b6bf1995cf88468b-4c13642db7fc40a5a934aa5e3c2b08fc
I20240503 02:00:02.530843 6458 raft_consensus.cc:2932] T 4385d51d11574192883d01376e2d5620 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } } }
I20240503 02:00:02.531288 6739 tablet_copy_source_session.cc:205] T 4c13642db7fc40a5a934aa5e3c2b08fc P c2ac2eb7cc264047b9ab32fe3a4fc025: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:02.532006 6313 catalog_manager.cc:5484] T 4385d51d11574192883d01376e2d5620 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: config changed from index -1 to 4, NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:02.533000 10670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c13642db7fc40a5a934aa5e3c2b08fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:02.535447 10670 tablet_copy_client.cc:783] T 4c13642db7fc40a5a934aa5e3c2b08fc P 2a01ace80baf4057b6bf1995cf88468b: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:02.535620 10670 tablet_copy_client.cc:647] T 4c13642db7fc40a5a934aa5e3c2b08fc P 2a01ace80baf4057b6bf1995cf88468b: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:02.538671 10670 tablet_copy_client.cc:523] T 4c13642db7fc40a5a934aa5e3c2b08fc P 2a01ace80baf4057b6bf1995cf88468b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:02.540748 10670 tablet_bootstrap.cc:492] T 4c13642db7fc40a5a934aa5e3c2b08fc P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:02.550665 10382 raft_consensus.cc:1050] T 0099654461ca41efa4323fc9f5adab4a P dec3e4a04d604f7eaaa3f077d081218a: attempting to promote NON_VOTER 80fd18ef4d604c86b909d5c008a46b4f to VOTER
I20240503 02:00:02.551112 10382 consensus_queue.cc:237] T 0099654461ca41efa4323fc9f5adab4a P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } }
I20240503 02:00:02.553023 8916 raft_consensus.cc:1261] T 0099654461ca41efa4323fc9f5adab4a P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:02.553611 8679 consensus_queue.cc:1035] T 0099654461ca41efa4323fc9f5adab4a P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:02.554466 6589 raft_consensus.cc:1261] T 0099654461ca41efa4323fc9f5adab4a P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEARNER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:02.555677 8679 consensus_queue.cc:1035] T 0099654461ca41efa4323fc9f5adab4a P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:02.556667 9178 raft_consensus.cc:1261] T 0099654461ca41efa4323fc9f5adab4a P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:02.557896 8679 consensus_queue.cc:1035] T 0099654461ca41efa4323fc9f5adab4a P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:02.559605 10826 tablet_bootstrap.cc:492] T 90ef3584830240acb0d8aad14ae851a6 P 99ead02915954b4fab23b033ad6d546b: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:02.560892 10826 tablet_bootstrap.cc:492] T 90ef3584830240acb0d8aad14ae851a6 P 99ead02915954b4fab23b033ad6d546b: Bootstrap complete.
I20240503 02:00:02.561151 10826 ts_tablet_manager.cc:1367] T 90ef3584830240acb0d8aad14ae851a6 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.035s user 0.029s sys 0.000s
I20240503 02:00:02.561668 10826 raft_consensus.cc:348] T 90ef3584830240acb0d8aad14ae851a6 P 99ead02915954b4fab23b033ad6d546b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } }
I20240503 02:00:02.561884 10826 raft_consensus.cc:729] T 90ef3584830240acb0d8aad14ae851a6 P 99ead02915954b4fab23b033ad6d546b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: LEARNER
I20240503 02:00:02.562026 10826 consensus_queue.cc:260] T 90ef3584830240acb0d8aad14ae851a6 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } }
I20240503 02:00:02.562415 10826 ts_tablet_manager.cc:1398] T 90ef3584830240acb0d8aad14ae851a6 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:02.563177 6871 tablet_copy_service.cc:342] P 492448570cc94ed1879feb2ec3b9051d: Request end of tablet copy session 99ead02915954b4fab23b033ad6d546b-90ef3584830240acb0d8aad14ae851a6 received from {username='slave'} at 127.0.99.134:60599
I20240503 02:00:02.563257 6871 tablet_copy_service.cc:433] P 492448570cc94ed1879feb2ec3b9051d: ending tablet copy session 99ead02915954b4fab23b033ad6d546b-90ef3584830240acb0d8aad14ae851a6 on tablet 90ef3584830240acb0d8aad14ae851a6 with peer 99ead02915954b4fab23b033ad6d546b
I20240503 02:00:02.564257 9955 raft_consensus.cc:2932] T 0099654461ca41efa4323fc9f5adab4a P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:02.567576 8916 raft_consensus.cc:2932] T 0099654461ca41efa4323fc9f5adab4a P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:02.569715 6582 raft_consensus.cc:2932] T 0099654461ca41efa4323fc9f5adab4a P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:02.571552 10670 tablet_bootstrap.cc:492] T 4c13642db7fc40a5a934aa5e3c2b08fc P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:02.571902 10670 tablet_bootstrap.cc:492] T 4c13642db7fc40a5a934aa5e3c2b08fc P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap complete.
I20240503 02:00:02.572121 10670 ts_tablet_manager.cc:1367] T 4c13642db7fc40a5a934aa5e3c2b08fc P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.031s user 0.021s sys 0.003s
I20240503 02:00:02.572736 6316 catalog_manager.cc:5484] T 0099654461ca41efa4323fc9f5adab4a P 9942306855404cfd891adf227dd65eb2 reported cstate change: config changed from index 2 to 3, 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:02.573287 10670 raft_consensus.cc:348] T 4c13642db7fc40a5a934aa5e3c2b08fc P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } }
I20240503 02:00:02.573482 10670 raft_consensus.cc:729] T 4c13642db7fc40a5a934aa5e3c2b08fc P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: LEARNER
I20240503 02:00:02.573647 10670 consensus_queue.cc:260] T 4c13642db7fc40a5a934aa5e3c2b08fc P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } }
I20240503 02:00:02.574694 10670 ts_tablet_manager.cc:1398] T 4c13642db7fc40a5a934aa5e3c2b08fc P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:02.575623 9178 raft_consensus.cc:2932] T 0099654461ca41efa4323fc9f5adab4a P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:02.576012 6739 tablet_copy_service.cc:342] P c2ac2eb7cc264047b9ab32fe3a4fc025: Request end of tablet copy session 2a01ace80baf4057b6bf1995cf88468b-4c13642db7fc40a5a934aa5e3c2b08fc received from {username='slave'} at 127.0.99.135:52621
I20240503 02:00:02.576102 6739 tablet_copy_service.cc:433] P c2ac2eb7cc264047b9ab32fe3a4fc025: ending tablet copy session 2a01ace80baf4057b6bf1995cf88468b-4c13642db7fc40a5a934aa5e3c2b08fc on tablet 4c13642db7fc40a5a934aa5e3c2b08fc with peer 2a01ace80baf4057b6bf1995cf88468b
I20240503 02:00:02.580472 6458 consensus_queue.cc:237] T 0099654461ca41efa4323fc9f5adab4a P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } }
I20240503 02:00:02.581820 9178 raft_consensus.cc:1261] T 0099654461ca41efa4323fc9f5adab4a P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:02.581840 6582 raft_consensus.cc:1261] T 0099654461ca41efa4323fc9f5adab4a P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:02.582240 10382 consensus_queue.cc:1035] T 0099654461ca41efa4323fc9f5adab4a P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:02.582443 9721 consensus_queue.cc:1035] T 0099654461ca41efa4323fc9f5adab4a P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:02.585183 8679 raft_consensus.cc:2932] T 0099654461ca41efa4323fc9f5adab4a P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:02.585561 9178 raft_consensus.cc:2932] T 0099654461ca41efa4323fc9f5adab4a P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:02.587884 6582 raft_consensus.cc:2932] T 0099654461ca41efa4323fc9f5adab4a P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:02.588129 6313 catalog_manager.cc:5484] T 0099654461ca41efa4323fc9f5adab4a P 2a01ace80baf4057b6bf1995cf88468b reported cstate change: config changed from index 3 to 4, VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:02.590866 8880 tablet_service.cc:1514] Processing DeleteTablet for tablet 0099654461ca41efa4323fc9f5adab4a with delete_type TABLET_DATA_TOMBSTONED (TS 9942306855404cfd891adf227dd65eb2 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40296
I20240503 02:00:02.591146 10681 tablet_replica.cc:330] T 0099654461ca41efa4323fc9f5adab4a P 9942306855404cfd891adf227dd65eb2: stopping tablet replica
I20240503 02:00:02.591284 10681 raft_consensus.cc:2229] T 0099654461ca41efa4323fc9f5adab4a P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:02.591423 10681 raft_consensus.cc:2258] T 0099654461ca41efa4323fc9f5adab4a P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:02.591600 6300 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0099654461ca41efa4323fc9f5adab4a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:02.593097 10681 ts_tablet_manager.cc:1875] T 0099654461ca41efa4323fc9f5adab4a P 9942306855404cfd891adf227dd65eb2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:02.595492 10681 ts_tablet_manager.cc:1888] T 0099654461ca41efa4323fc9f5adab4a P 9942306855404cfd891adf227dd65eb2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:00:02.595577 10681 log.cc:1196] T 0099654461ca41efa4323fc9f5adab4a P 9942306855404cfd891adf227dd65eb2: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/0099654461ca41efa4323fc9f5adab4a
I20240503 02:00:02.596474 6303 catalog_manager.cc:4830] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211): tablet 0099654461ca41efa4323fc9f5adab4a (table rebalancer_extra_table_2 [id=54f1891be3f54f19821aad5d1dfbd85d]) successfully deleted
I20240503 02:00:02.700140 10670 ts_tablet_manager.cc:902] T 3cf186a0901f420390c8fda54316c5af P 2a01ace80baf4057b6bf1995cf88468b: Initiating tablet copy from peer c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:02.700659 10670 tablet_copy_client.cc:308] T 3cf186a0901f420390c8fda54316c5af P 2a01ace80baf4057b6bf1995cf88468b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:42655
I20240503 02:00:02.701128 6739 tablet_copy_service.cc:140] P c2ac2eb7cc264047b9ab32fe3a4fc025: Received BeginTabletCopySession request for tablet 3cf186a0901f420390c8fda54316c5af from peer 2a01ace80baf4057b6bf1995cf88468b ({username='slave'} at 127.0.99.135:52621)
I20240503 02:00:02.701227 6739 tablet_copy_service.cc:161] P c2ac2eb7cc264047b9ab32fe3a4fc025: Beginning new tablet copy session on tablet 3cf186a0901f420390c8fda54316c5af from peer 2a01ace80baf4057b6bf1995cf88468b at {username='slave'} at 127.0.99.135:52621: session id = 2a01ace80baf4057b6bf1995cf88468b-3cf186a0901f420390c8fda54316c5af
I20240503 02:00:02.701959 6739 tablet_copy_source_session.cc:205] T 3cf186a0901f420390c8fda54316c5af P c2ac2eb7cc264047b9ab32fe3a4fc025: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:02.702785 10670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cf186a0901f420390c8fda54316c5af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:02.704596 10670 tablet_copy_client.cc:783] T 3cf186a0901f420390c8fda54316c5af P 2a01ace80baf4057b6bf1995cf88468b: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:02.704757 10670 tablet_copy_client.cc:647] T 3cf186a0901f420390c8fda54316c5af P 2a01ace80baf4057b6bf1995cf88468b: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:02.705884 10670 tablet_copy_client.cc:523] T 3cf186a0901f420390c8fda54316c5af P 2a01ace80baf4057b6bf1995cf88468b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:02.707264 10670 tablet_bootstrap.cc:492] T 3cf186a0901f420390c8fda54316c5af P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:02.708235 10685 ts_tablet_manager.cc:902] T 46ad540deb624a25b97d3b967fe1f6f4 P 9942306855404cfd891adf227dd65eb2: Initiating tablet copy from peer dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 02:00:02.708803 10685 tablet_copy_client.cc:308] T 46ad540deb624a25b97d3b967fe1f6f4 P 9942306855404cfd891adf227dd65eb2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:33093
I20240503 02:00:02.709332 6478 tablet_copy_service.cc:140] P dec3e4a04d604f7eaaa3f077d081218a: Received BeginTabletCopySession request for tablet 46ad540deb624a25b97d3b967fe1f6f4 from peer 9942306855404cfd891adf227dd65eb2 ({username='slave'} at 127.0.99.133:57637)
I20240503 02:00:02.709461 6478 tablet_copy_service.cc:161] P dec3e4a04d604f7eaaa3f077d081218a: Beginning new tablet copy session on tablet 46ad540deb624a25b97d3b967fe1f6f4 from peer 9942306855404cfd891adf227dd65eb2 at {username='slave'} at 127.0.99.133:57637: session id = 9942306855404cfd891adf227dd65eb2-46ad540deb624a25b97d3b967fe1f6f4
I20240503 02:00:02.710359 6478 tablet_copy_source_session.cc:205] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:02.711278 10685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46ad540deb624a25b97d3b967fe1f6f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:02.713187 10685 tablet_copy_client.cc:783] T 46ad540deb624a25b97d3b967fe1f6f4 P 9942306855404cfd891adf227dd65eb2: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:02.713347 10685 tablet_copy_client.cc:647] T 46ad540deb624a25b97d3b967fe1f6f4 P 9942306855404cfd891adf227dd65eb2: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:02.714447 10685 tablet_copy_client.cc:523] T 46ad540deb624a25b97d3b967fe1f6f4 P 9942306855404cfd891adf227dd65eb2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:02.715572 10685 tablet_bootstrap.cc:492] T 46ad540deb624a25b97d3b967fe1f6f4 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:02.718228 10826 ts_tablet_manager.cc:902] T af00a839bb5e402791f87288493efef6 P 99ead02915954b4fab23b033ad6d546b: Initiating tablet copy from peer dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 02:00:02.718708 10826 tablet_copy_client.cc:308] T af00a839bb5e402791f87288493efef6 P 99ead02915954b4fab23b033ad6d546b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:33093
I20240503 02:00:02.727419 8916 raft_consensus.cc:1203] T 7df4c083550046e883e5e61d3d4e1572 P 9942306855404cfd891adf227dd65eb2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:00:02.727795 6478 tablet_copy_service.cc:140] P dec3e4a04d604f7eaaa3f077d081218a: Received BeginTabletCopySession request for tablet af00a839bb5e402791f87288493efef6 from peer 99ead02915954b4fab23b033ad6d546b ({username='slave'} at 127.0.99.134:54509)
I20240503 02:00:02.727898 6478 tablet_copy_service.cc:161] P dec3e4a04d604f7eaaa3f077d081218a: Beginning new tablet copy session on tablet af00a839bb5e402791f87288493efef6 from peer 99ead02915954b4fab23b033ad6d546b at {username='slave'} at 127.0.99.134:54509: session id = 99ead02915954b4fab23b033ad6d546b-af00a839bb5e402791f87288493efef6
I20240503 02:00:02.728585 10956 ts_tablet_manager.cc:902] T f28586e70bd9473fbdc6b9126b666fa9 P 80fd18ef4d604c86b909d5c008a46b4f: Initiating tablet copy from peer 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:02.728868 6478 tablet_copy_source_session.cc:205] T af00a839bb5e402791f87288493efef6 P dec3e4a04d604f7eaaa3f077d081218a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:02.729100 10956 tablet_copy_client.cc:308] T f28586e70bd9473fbdc6b9126b666fa9 P 80fd18ef4d604c86b909d5c008a46b4f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:33211
I20240503 02:00:02.730783 8937 tablet_copy_service.cc:140] P 9942306855404cfd891adf227dd65eb2: Received BeginTabletCopySession request for tablet f28586e70bd9473fbdc6b9126b666fa9 from peer 80fd18ef4d604c86b909d5c008a46b4f ({username='slave'} at 127.0.99.130:36873)
I20240503 02:00:02.730922 8937 tablet_copy_service.cc:161] P 9942306855404cfd891adf227dd65eb2: Beginning new tablet copy session on tablet f28586e70bd9473fbdc6b9126b666fa9 from peer 80fd18ef4d604c86b909d5c008a46b4f at {username='slave'} at 127.0.99.130:36873: session id = 80fd18ef4d604c86b909d5c008a46b4f-f28586e70bd9473fbdc6b9126b666fa9
I20240503 02:00:02.730978 10826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af00a839bb5e402791f87288493efef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:02.732291 8937 tablet_copy_source_session.cc:205] T f28586e70bd9473fbdc6b9126b666fa9 P 9942306855404cfd891adf227dd65eb2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:02.733274 10826 tablet_copy_client.cc:783] T af00a839bb5e402791f87288493efef6 P 99ead02915954b4fab23b033ad6d546b: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:02.733436 10826 tablet_copy_client.cc:647] T af00a839bb5e402791f87288493efef6 P 99ead02915954b4fab23b033ad6d546b: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:02.734879 10826 tablet_copy_client.cc:523] T af00a839bb5e402791f87288493efef6 P 99ead02915954b4fab23b033ad6d546b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:02.735646 10956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f28586e70bd9473fbdc6b9126b666fa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:02.736220 10826 tablet_bootstrap.cc:492] T af00a839bb5e402791f87288493efef6 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:02.738659 10670 tablet_bootstrap.cc:492] T 3cf186a0901f420390c8fda54316c5af P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:02.738984 10670 tablet_bootstrap.cc:492] T 3cf186a0901f420390c8fda54316c5af P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap complete.
I20240503 02:00:02.739169 10670 ts_tablet_manager.cc:1367] T 3cf186a0901f420390c8fda54316c5af P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.032s user 0.024s sys 0.004s
I20240503 02:00:02.739640 10670 raft_consensus.cc:348] T 3cf186a0901f420390c8fda54316c5af P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } }
I20240503 02:00:02.739768 10670 raft_consensus.cc:729] T 3cf186a0901f420390c8fda54316c5af P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: LEARNER
I20240503 02:00:02.739861 10670 consensus_queue.cc:260] T 3cf186a0901f420390c8fda54316c5af P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } }
I20240503 02:00:02.740159 10670 ts_tablet_manager.cc:1398] T 3cf186a0901f420390c8fda54316c5af P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:02.740229 10679 consensus_queue.cc:1035] T 0ee05e70ccbb496ca3cff08dc38112e8 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:02.740923 6739 tablet_copy_service.cc:342] P c2ac2eb7cc264047b9ab32fe3a4fc025: Request end of tablet copy session 2a01ace80baf4057b6bf1995cf88468b-3cf186a0901f420390c8fda54316c5af received from {username='slave'} at 127.0.99.135:52621
I20240503 02:00:02.741010 6739 tablet_copy_service.cc:433] P c2ac2eb7cc264047b9ab32fe3a4fc025: ending tablet copy session 2a01ace80baf4057b6bf1995cf88468b-3cf186a0901f420390c8fda54316c5af on tablet 3cf186a0901f420390c8fda54316c5af with peer 2a01ace80baf4057b6bf1995cf88468b
I20240503 02:00:02.747274 10685 tablet_bootstrap.cc:492] T 46ad540deb624a25b97d3b967fe1f6f4 P 9942306855404cfd891adf227dd65eb2: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:02.747365 10956 tablet_copy_client.cc:783] T f28586e70bd9473fbdc6b9126b666fa9 P 80fd18ef4d604c86b909d5c008a46b4f: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:02.747543 10956 tablet_copy_client.cc:647] T f28586e70bd9473fbdc6b9126b666fa9 P 80fd18ef4d604c86b909d5c008a46b4f: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:02.747716 10685 tablet_bootstrap.cc:492] T 46ad540deb624a25b97d3b967fe1f6f4 P 9942306855404cfd891adf227dd65eb2: Bootstrap complete.
I20240503 02:00:02.747949 10685 ts_tablet_manager.cc:1367] T 46ad540deb624a25b97d3b967fe1f6f4 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.032s user 0.024s sys 0.004s
I20240503 02:00:02.748513 10685 raft_consensus.cc:348] T 46ad540deb624a25b97d3b967fe1f6f4 P 9942306855404cfd891adf227dd65eb2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } }
I20240503 02:00:02.748790 10685 raft_consensus.cc:729] T 46ad540deb624a25b97d3b967fe1f6f4 P 9942306855404cfd891adf227dd65eb2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: LEARNER
I20240503 02:00:02.748951 10685 consensus_queue.cc:260] T 46ad540deb624a25b97d3b967fe1f6f4 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } }
I20240503 02:00:02.749217 10956 tablet_copy_client.cc:523] T f28586e70bd9473fbdc6b9126b666fa9 P 80fd18ef4d604c86b909d5c008a46b4f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:02.749914 10685 ts_tablet_manager.cc:1398] T 46ad540deb624a25b97d3b967fe1f6f4 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:02.751559 10956 tablet_bootstrap.cc:492] T f28586e70bd9473fbdc6b9126b666fa9 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:02.752998 10257 consensus_queue.cc:1035] T 0ee05e70ccbb496ca3cff08dc38112e8 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:02.754657 6478 tablet_copy_service.cc:342] P dec3e4a04d604f7eaaa3f077d081218a: Request end of tablet copy session 9942306855404cfd891adf227dd65eb2-46ad540deb624a25b97d3b967fe1f6f4 received from {username='slave'} at 127.0.99.133:57637
I20240503 02:00:02.754750 6478 tablet_copy_service.cc:433] P dec3e4a04d604f7eaaa3f077d081218a: ending tablet copy session 9942306855404cfd891adf227dd65eb2-46ad540deb624a25b97d3b967fe1f6f4 on tablet 46ad540deb624a25b97d3b967fe1f6f4 with peer 9942306855404cfd891adf227dd65eb2
I20240503 02:00:02.756163 10956 tablet_bootstrap.cc:492] T f28586e70bd9473fbdc6b9126b666fa9 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:02.756431 10956 tablet_bootstrap.cc:492] T f28586e70bd9473fbdc6b9126b666fa9 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap complete.
I20240503 02:00:02.756616 10956 ts_tablet_manager.cc:1367] T f28586e70bd9473fbdc6b9126b666fa9 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.001s
I20240503 02:00:02.757521 10956 raft_consensus.cc:348] T f28586e70bd9473fbdc6b9126b666fa9 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: true } }
I20240503 02:00:02.757819 10956 raft_consensus.cc:729] T f28586e70bd9473fbdc6b9126b666fa9 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: LEARNER
I20240503 02:00:02.757926 10956 consensus_queue.cc:260] T f28586e70bd9473fbdc6b9126b666fa9 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: true } }
I20240503 02:00:02.758213 10956 ts_tablet_manager.cc:1398] T f28586e70bd9473fbdc6b9126b666fa9 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:00:02.758198 9721 raft_consensus.cc:482] T f2716f6d9e554ec8a8b922a75e5d5d75 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:02.758287 9721 raft_consensus.cc:504] T f2716f6d9e554ec8a8b922a75e5d5d75 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:02.758636 9721 leader_election.cc:288] T f2716f6d9e554ec8a8b922a75e5d5d75 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:02.759214 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2716f6d9e554ec8a8b922a75e5d5d75" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:02.759287 8937 tablet_copy_service.cc:342] P 9942306855404cfd891adf227dd65eb2: Request end of tablet copy session 80fd18ef4d604c86b909d5c008a46b4f-f28586e70bd9473fbdc6b9126b666fa9 received from {username='slave'} at 127.0.99.130:36873
I20240503 02:00:02.759348 8916 raft_consensus.cc:2454] T f2716f6d9e554ec8a8b922a75e5d5d75 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:02.759399 8937 tablet_copy_service.cc:433] P 9942306855404cfd891adf227dd65eb2: ending tablet copy session 80fd18ef4d604c86b909d5c008a46b4f-f28586e70bd9473fbdc6b9126b666fa9 on tablet f28586e70bd9473fbdc6b9126b666fa9 with peer 80fd18ef4d604c86b909d5c008a46b4f
I20240503 02:00:02.759460 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2716f6d9e554ec8a8b922a75e5d5d75" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:02.759594 9178 raft_consensus.cc:2454] T f2716f6d9e554ec8a8b922a75e5d5d75 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:02.759877 6391 leader_election.cc:302] T f2716f6d9e554ec8a8b922a75e5d5d75 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:02.760550 9721 raft_consensus.cc:2781] T f2716f6d9e554ec8a8b922a75e5d5d75 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:02.760622 9721 raft_consensus.cc:482] T f2716f6d9e554ec8a8b922a75e5d5d75 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:02.760660 9721 raft_consensus.cc:3037] T f2716f6d9e554ec8a8b922a75e5d5d75 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:02.761628 9721 raft_consensus.cc:504] T f2716f6d9e554ec8a8b922a75e5d5d75 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:02.762003 9721 leader_election.cc:288] T f2716f6d9e554ec8a8b922a75e5d5d75 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Requested vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:02.762701 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2716f6d9e554ec8a8b922a75e5d5d75" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:02.762830 8916 raft_consensus.cc:3037] T f2716f6d9e554ec8a8b922a75e5d5d75 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:02.763788 8916 raft_consensus.cc:2454] T f2716f6d9e554ec8a8b922a75e5d5d75 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:02.764125 6394 leader_election.cc:302] T f2716f6d9e554ec8a8b922a75e5d5d75 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9942306855404cfd891adf227dd65eb2, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:02.764302 9721 raft_consensus.cc:2781] T f2716f6d9e554ec8a8b922a75e5d5d75 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:02.764443 9721 raft_consensus.cc:686] T f2716f6d9e554ec8a8b922a75e5d5d75 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Becoming Leader. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Running, Role: LEADER
I20240503 02:00:02.764577 9721 consensus_queue.cc:237] T f2716f6d9e554ec8a8b922a75e5d5d75 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:02.767012 6315 catalog_manager.cc:5484] T f2716f6d9e554ec8a8b922a75e5d5d75 P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: term changed from 0 to 1, leader changed from <none> to dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129). New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:02.768505 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2716f6d9e554ec8a8b922a75e5d5d75" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:02.768657 9178 raft_consensus.cc:3037] T f2716f6d9e554ec8a8b922a75e5d5d75 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:02.769631 10802 ts_tablet_manager.cc:902] T c3485701bf3b48f5ae2e1568afeed133 P e1a9251ce1cf4361b8b0ede9fbfc4023: Initiating tablet copy from peer 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:02.769636 9178 raft_consensus.cc:2454] T f2716f6d9e554ec8a8b922a75e5d5d75 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:02.770074 10802 tablet_copy_client.cc:308] T c3485701bf3b48f5ae2e1568afeed133 P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43863
I20240503 02:00:02.770593 6609 tablet_copy_service.cc:140] P 80fd18ef4d604c86b909d5c008a46b4f: Received BeginTabletCopySession request for tablet c3485701bf3b48f5ae2e1568afeed133 from peer e1a9251ce1cf4361b8b0ede9fbfc4023 ({username='slave'} at 127.0.99.136:60117)
I20240503 02:00:02.770695 6609 tablet_copy_service.cc:161] P 80fd18ef4d604c86b909d5c008a46b4f: Beginning new tablet copy session on tablet c3485701bf3b48f5ae2e1568afeed133 from peer e1a9251ce1cf4361b8b0ede9fbfc4023 at {username='slave'} at 127.0.99.136:60117: session id = e1a9251ce1cf4361b8b0ede9fbfc4023-c3485701bf3b48f5ae2e1568afeed133
I20240503 02:00:02.771656 6609 tablet_copy_source_session.cc:205] T c3485701bf3b48f5ae2e1568afeed133 P 80fd18ef4d604c86b909d5c008a46b4f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:02.773417 10802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3485701bf3b48f5ae2e1568afeed133. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:02.776366 10802 tablet_copy_client.cc:783] T c3485701bf3b48f5ae2e1568afeed133 P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:02.776558 10802 tablet_copy_client.cc:647] T c3485701bf3b48f5ae2e1568afeed133 P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:02.777629 10802 tablet_copy_client.cc:523] T c3485701bf3b48f5ae2e1568afeed133 P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:02.779398 10802 tablet_bootstrap.cc:492] T c3485701bf3b48f5ae2e1568afeed133 P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap starting.
I20240503 02:00:02.784034 11000 ts_tablet_manager.cc:902] T ac1632cc5eb94554b1faa7eb715706ef P 99ead02915954b4fab23b033ad6d546b: Initiating tablet copy from peer dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 02:00:02.788251 10257 consensus_queue.cc:1035] T 88205f7c7ed4488e8fcfda3bfa0c4fca P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:02.792814 11000 tablet_copy_client.cc:308] T ac1632cc5eb94554b1faa7eb715706ef P 99ead02915954b4fab23b033ad6d546b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:33093
I20240503 02:00:02.794957 10685 ts_tablet_manager.cc:902] T 1e445a3cc928429caf9e3c6193a63010 P 9942306855404cfd891adf227dd65eb2: Initiating tablet copy from peer 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:02.795409 6478 tablet_copy_service.cc:140] P dec3e4a04d604f7eaaa3f077d081218a: Received BeginTabletCopySession request for tablet ac1632cc5eb94554b1faa7eb715706ef from peer 99ead02915954b4fab23b033ad6d546b ({username='slave'} at 127.0.99.134:54509)
I20240503 02:00:02.795521 6478 tablet_copy_service.cc:161] P dec3e4a04d604f7eaaa3f077d081218a: Beginning new tablet copy session on tablet ac1632cc5eb94554b1faa7eb715706ef from peer 99ead02915954b4fab23b033ad6d546b at {username='slave'} at 127.0.99.134:54509: session id = 99ead02915954b4fab23b033ad6d546b-ac1632cc5eb94554b1faa7eb715706ef
I20240503 02:00:02.795665 10685 tablet_copy_client.cc:308] T 1e445a3cc928429caf9e3c6193a63010 P 9942306855404cfd891adf227dd65eb2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43863
I20240503 02:00:02.796113 6609 tablet_copy_service.cc:140] P 80fd18ef4d604c86b909d5c008a46b4f: Received BeginTabletCopySession request for tablet 1e445a3cc928429caf9e3c6193a63010 from peer 9942306855404cfd891adf227dd65eb2 ({username='slave'} at 127.0.99.133:50723)
I20240503 02:00:02.796205 6609 tablet_copy_service.cc:161] P 80fd18ef4d604c86b909d5c008a46b4f: Beginning new tablet copy session on tablet 1e445a3cc928429caf9e3c6193a63010 from peer 9942306855404cfd891adf227dd65eb2 at {username='slave'} at 127.0.99.133:50723: session id = 9942306855404cfd891adf227dd65eb2-1e445a3cc928429caf9e3c6193a63010
I20240503 02:00:02.796519 6478 tablet_copy_source_session.cc:205] T ac1632cc5eb94554b1faa7eb715706ef P dec3e4a04d604f7eaaa3f077d081218a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:02.797122 6609 tablet_copy_source_session.cc:205] T 1e445a3cc928429caf9e3c6193a63010 P 80fd18ef4d604c86b909d5c008a46b4f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:02.797997 10685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e445a3cc928429caf9e3c6193a63010. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:02.798444 11000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac1632cc5eb94554b1faa7eb715706ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:02.800421 10685 tablet_copy_client.cc:783] T 1e445a3cc928429caf9e3c6193a63010 P 9942306855404cfd891adf227dd65eb2: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:02.800580 10685 tablet_copy_client.cc:647] T 1e445a3cc928429caf9e3c6193a63010 P 9942306855404cfd891adf227dd65eb2: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:02.800900 11000 tablet_copy_client.cc:783] T ac1632cc5eb94554b1faa7eb715706ef P 99ead02915954b4fab23b033ad6d546b: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:02.801100 11000 tablet_copy_client.cc:647] T ac1632cc5eb94554b1faa7eb715706ef P 99ead02915954b4fab23b033ad6d546b: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:02.801299 10257 consensus_queue.cc:1035] T 88205f7c7ed4488e8fcfda3bfa0c4fca P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:02.804778 9178 raft_consensus.cc:1203] T 2fc834abc6a94c638d6202b03320a2d2 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:00:02.813663 11000 tablet_copy_client.cc:523] T ac1632cc5eb94554b1faa7eb715706ef P 99ead02915954b4fab23b033ad6d546b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:02.815184 10685 tablet_copy_client.cc:523] T 1e445a3cc928429caf9e3c6193a63010 P 9942306855404cfd891adf227dd65eb2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:02.816910 10685 tablet_bootstrap.cc:492] T 1e445a3cc928429caf9e3c6193a63010 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:02.822278 10826 tablet_bootstrap.cc:492] T af00a839bb5e402791f87288493efef6 P 99ead02915954b4fab23b033ad6d546b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:02.822588 10826 tablet_bootstrap.cc:492] T af00a839bb5e402791f87288493efef6 P 99ead02915954b4fab23b033ad6d546b: Bootstrap complete.
I20240503 02:00:02.822789 10826 ts_tablet_manager.cc:1367] T af00a839bb5e402791f87288493efef6 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.087s user 0.027s sys 0.000s
I20240503 02:00:02.823269 10826 raft_consensus.cc:348] T af00a839bb5e402791f87288493efef6 P 99ead02915954b4fab23b033ad6d546b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } }
I20240503 02:00:02.823400 10826 raft_consensus.cc:729] T af00a839bb5e402791f87288493efef6 P 99ead02915954b4fab23b033ad6d546b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: LEARNER
I20240503 02:00:02.823500 10826 consensus_queue.cc:260] T af00a839bb5e402791f87288493efef6 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } }
I20240503 02:00:02.827106 11000 tablet_bootstrap.cc:492] T ac1632cc5eb94554b1faa7eb715706ef P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:02.845700 10826 ts_tablet_manager.cc:1398] T af00a839bb5e402791f87288493efef6 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.023s user 0.003s sys 0.000s
I20240503 02:00:02.846724 6478 tablet_copy_service.cc:342] P dec3e4a04d604f7eaaa3f077d081218a: Request end of tablet copy session 99ead02915954b4fab23b033ad6d546b-af00a839bb5e402791f87288493efef6 received from {username='slave'} at 127.0.99.134:54509
I20240503 02:00:02.846827 6478 tablet_copy_service.cc:433] P dec3e4a04d604f7eaaa3f077d081218a: ending tablet copy session 99ead02915954b4fab23b033ad6d546b-af00a839bb5e402791f87288493efef6 on tablet af00a839bb5e402791f87288493efef6 with peer 99ead02915954b4fab23b033ad6d546b
I20240503 02:00:02.847678 10685 tablet_bootstrap.cc:492] T 1e445a3cc928429caf9e3c6193a63010 P 9942306855404cfd891adf227dd65eb2: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:02.848065 10685 tablet_bootstrap.cc:492] T 1e445a3cc928429caf9e3c6193a63010 P 9942306855404cfd891adf227dd65eb2: Bootstrap complete.
I20240503 02:00:02.848248 10685 ts_tablet_manager.cc:1367] T 1e445a3cc928429caf9e3c6193a63010 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.031s user 0.023s sys 0.004s
I20240503 02:00:02.848752 10685 raft_consensus.cc:348] T 1e445a3cc928429caf9e3c6193a63010 P 9942306855404cfd891adf227dd65eb2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } }
I20240503 02:00:02.848878 10685 raft_consensus.cc:729] T 1e445a3cc928429caf9e3c6193a63010 P 9942306855404cfd891adf227dd65eb2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: LEARNER
I20240503 02:00:02.848980 10685 consensus_queue.cc:260] T 1e445a3cc928429caf9e3c6193a63010 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } }
I20240503 02:00:02.849244 10685 ts_tablet_manager.cc:1398] T 1e445a3cc928429caf9e3c6193a63010 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:02.851933 6609 tablet_copy_service.cc:342] P 80fd18ef4d604c86b909d5c008a46b4f: Request end of tablet copy session 9942306855404cfd891adf227dd65eb2-1e445a3cc928429caf9e3c6193a63010 received from {username='slave'} at 127.0.99.133:50723
I20240503 02:00:02.852032 6609 tablet_copy_service.cc:433] P 80fd18ef4d604c86b909d5c008a46b4f: ending tablet copy session 9942306855404cfd891adf227dd65eb2-1e445a3cc928429caf9e3c6193a63010 on tablet 1e445a3cc928429caf9e3c6193a63010 with peer 9942306855404cfd891adf227dd65eb2
I20240503 02:00:02.854789 10278 raft_consensus.cc:1050] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P 80fd18ef4d604c86b909d5c008a46b4f: attempting to promote NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 to VOTER
I20240503 02:00:02.855234 10278 consensus_queue.cc:237] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } }
I20240503 02:00:02.856801 10802 tablet_bootstrap.cc:492] T c3485701bf3b48f5ae2e1568afeed133 P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:02.857100 9310 raft_consensus.cc:1261] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 LEARNER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:02.857197 10802 tablet_bootstrap.cc:492] T c3485701bf3b48f5ae2e1568afeed133 P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap complete.
I20240503 02:00:02.857370 10802 ts_tablet_manager.cc:1367] T c3485701bf3b48f5ae2e1568afeed133 P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent bootstrapping tablet: real 0.078s user 0.025s sys 0.000s
I20240503 02:00:02.857529 10345 consensus_queue.cc:1035] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:02.857648 9984 consensus_queue.cc:1035] T cea1ea50fa434d31b42451f84f26bc76 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:02.857832 10802 raft_consensus.cc:348] T c3485701bf3b48f5ae2e1568afeed133 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } }
I20240503 02:00:02.857962 10802 raft_consensus.cc:729] T c3485701bf3b48f5ae2e1568afeed133 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e1a9251ce1cf4361b8b0ede9fbfc4023, State: Initialized, Role: LEARNER
I20240503 02:00:02.858079 10802 consensus_queue.cc:260] T c3485701bf3b48f5ae2e1568afeed133 P e1a9251ce1cf4361b8b0ede9fbfc4023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } }
I20240503 02:00:02.858368 10802 ts_tablet_manager.cc:1398] T c3485701bf3b48f5ae2e1568afeed133 P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:02.859789 6720 raft_consensus.cc:1261] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:02.860163 6458 raft_consensus.cc:1261] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:02.860911 9484 consensus_queue.cc:1035] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:02.861267 9484 consensus_queue.cc:1035] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:02.863350 6609 tablet_copy_service.cc:342] P 80fd18ef4d604c86b909d5c008a46b4f: Request end of tablet copy session e1a9251ce1cf4361b8b0ede9fbfc4023-c3485701bf3b48f5ae2e1568afeed133 received from {username='slave'} at 127.0.99.136:60117
I20240503 02:00:02.863452 6609 tablet_copy_service.cc:433] P 80fd18ef4d604c86b909d5c008a46b4f: ending tablet copy session e1a9251ce1cf4361b8b0ede9fbfc4023-c3485701bf3b48f5ae2e1568afeed133 on tablet c3485701bf3b48f5ae2e1568afeed133 with peer e1a9251ce1cf4361b8b0ede9fbfc4023
I20240503 02:00:02.863890 9484 raft_consensus.cc:2932] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:02.864344 10802 ts_tablet_manager.cc:902] T 915028b3478341d588ed6031e657c259 P e1a9251ce1cf4361b8b0ede9fbfc4023: Initiating tablet copy from peer 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:02.864748 10802 tablet_copy_client.cc:308] T 915028b3478341d588ed6031e657c259 P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43863
I20240503 02:00:02.866145 6609 tablet_copy_service.cc:140] P 80fd18ef4d604c86b909d5c008a46b4f: Received BeginTabletCopySession request for tablet 915028b3478341d588ed6031e657c259 from peer e1a9251ce1cf4361b8b0ede9fbfc4023 ({username='slave'} at 127.0.99.136:60117)
I20240503 02:00:02.866250 6609 tablet_copy_service.cc:161] P 80fd18ef4d604c86b909d5c008a46b4f: Beginning new tablet copy session on tablet 915028b3478341d588ed6031e657c259 from peer e1a9251ce1cf4361b8b0ede9fbfc4023 at {username='slave'} at 127.0.99.136:60117: session id = e1a9251ce1cf4361b8b0ede9fbfc4023-915028b3478341d588ed6031e657c259
I20240503 02:00:02.866928 6458 raft_consensus.cc:2932] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:02.867111 6720 raft_consensus.cc:2932] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:02.867277 6609 tablet_copy_source_session.cc:205] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:02.868147 9310 raft_consensus.cc:2932] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:02.870419 10802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 915028b3478341d588ed6031e657c259. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:02.870501 6315 catalog_manager.cc:5484] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: config changed from index 4 to 5, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:02.875149 10802 tablet_copy_client.cc:783] T 915028b3478341d588ed6031e657c259 P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:02.875332 10802 tablet_copy_client.cc:647] T 915028b3478341d588ed6031e657c259 P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:02.878127 6582 consensus_queue.cc:237] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } }
I20240503 02:00:02.880518 6458 raft_consensus.cc:1261] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:02.881570 9310 raft_consensus.cc:1261] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:02.881515 10345 consensus_queue.cc:1035] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:02.881717 9711 consensus_queue.cc:1035] T cea1ea50fa434d31b42451f84f26bc76 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:02.882783 9484 consensus_queue.cc:1035] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:02.884698 10345 raft_consensus.cc:2932] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:02.885532 10802 tablet_copy_client.cc:523] T 915028b3478341d588ed6031e657c259 P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:02.886852 9310 raft_consensus.cc:2932] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:02.887090 10802 tablet_bootstrap.cc:492] T 915028b3478341d588ed6031e657c259 P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap starting.
I20240503 02:00:02.888442 6302 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9f7ade1e2af24e86b6e54e2b5769e4c0 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:02.888985 6458 raft_consensus.cc:2932] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:02.891212 6315 catalog_manager.cc:5484] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: config changed from index 5 to 6, VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:02.894035 6700 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f7ade1e2af24e86b6e54e2b5769e4c0 with delete_type TABLET_DATA_TOMBSTONED (TS c2ac2eb7cc264047b9ab32fe3a4fc025 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:40336
I20240503 02:00:02.894290 10173 tablet_replica.cc:330] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P c2ac2eb7cc264047b9ab32fe3a4fc025: stopping tablet replica
I20240503 02:00:02.894435 10173 raft_consensus.cc:2229] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:02.894613 10173 raft_consensus.cc:2258] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:02.895310 10173 ts_tablet_manager.cc:1875] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:02.897732 10173 ts_tablet_manager.cc:1888] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:00:02.897817 10173 log.cc:1196] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/9f7ade1e2af24e86b6e54e2b5769e4c0
I20240503 02:00:02.898408 11000 tablet_bootstrap.cc:492] T ac1632cc5eb94554b1faa7eb715706ef P 99ead02915954b4fab23b033ad6d546b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:02.899144 6303 catalog_manager.cc:4830] TS c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655): tablet 9f7ade1e2af24e86b6e54e2b5769e4c0 (table rebalance_test_table_18 [id=d732c32bc9004748904a109f59db08fd]) successfully deleted
I20240503 02:00:02.900287 11000 tablet_bootstrap.cc:492] T ac1632cc5eb94554b1faa7eb715706ef P 99ead02915954b4fab23b033ad6d546b: Bootstrap complete.
I20240503 02:00:02.900557 11000 ts_tablet_manager.cc:1367] T ac1632cc5eb94554b1faa7eb715706ef P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.074s user 0.026s sys 0.003s
I20240503 02:00:02.901121 11000 raft_consensus.cc:348] T ac1632cc5eb94554b1faa7eb715706ef P 99ead02915954b4fab23b033ad6d546b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } }
I20240503 02:00:02.901327 11000 raft_consensus.cc:729] T ac1632cc5eb94554b1faa7eb715706ef P 99ead02915954b4fab23b033ad6d546b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: LEARNER
I20240503 02:00:02.901486 11000 consensus_queue.cc:260] T ac1632cc5eb94554b1faa7eb715706ef P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } }
I20240503 02:00:02.901866 11000 ts_tablet_manager.cc:1398] T ac1632cc5eb94554b1faa7eb715706ef P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:02.902714 6478 tablet_copy_service.cc:342] P dec3e4a04d604f7eaaa3f077d081218a: Request end of tablet copy session 99ead02915954b4fab23b033ad6d546b-ac1632cc5eb94554b1faa7eb715706ef received from {username='slave'} at 127.0.99.134:54509
I20240503 02:00:02.902793 6478 tablet_copy_service.cc:433] P dec3e4a04d604f7eaaa3f077d081218a: ending tablet copy session 99ead02915954b4fab23b033ad6d546b-ac1632cc5eb94554b1faa7eb715706ef on tablet ac1632cc5eb94554b1faa7eb715706ef with peer 99ead02915954b4fab23b033ad6d546b
I20240503 02:00:02.906271 9158 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:00:02.916721 6567 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:00:02.916989 9025 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:02.921203 6831 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:02.924609 8880 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:02.925608 6700 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:00:02.928242 6312 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:02.932550 6438 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 02:00:02.933207 9285 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:02.943790 6312 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_extra_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:02.951013 9158 tablet_service.cc:1467] Processing CreateTablet for tablet 01d0ff19ac53453888570a51948235e8 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=e43866993a7646be8fc624a6f2150d34]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:02.951342 9158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01d0ff19ac53453888570a51948235e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:02.951522 9285 tablet_service.cc:1467] Processing CreateTablet for tablet 9f6bcf448d424d3e8bb18bf12cfb64c5 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=e43866993a7646be8fc624a6f2150d34]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:02.951526 9286 tablet_service.cc:1467] Processing CreateTablet for tablet 1c721b33983b4c47afd8faa8f155ca84 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=e43866993a7646be8fc624a6f2150d34]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:02.951854 9286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c721b33983b4c47afd8faa8f155ca84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:02.952076 9285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f6bcf448d424d3e8bb18bf12cfb64c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:02.953025 8897 tablet_service.cc:1467] Processing CreateTablet for tablet 1c721b33983b4c47afd8faa8f155ca84 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=e43866993a7646be8fc624a6f2150d34]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:02.953291 8897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c721b33983b4c47afd8faa8f155ca84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:02.953347 8880 tablet_service.cc:1467] Processing CreateTablet for tablet 9f6bcf448d424d3e8bb18bf12cfb64c5 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=e43866993a7646be8fc624a6f2150d34]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:02.953557 8880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f6bcf448d424d3e8bb18bf12cfb64c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:02.953934 6831 tablet_service.cc:1467] Processing CreateTablet for tablet 01d0ff19ac53453888570a51948235e8 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=e43866993a7646be8fc624a6f2150d34]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:02.954185 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01d0ff19ac53453888570a51948235e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:02.955271 11045 tablet_bootstrap.cc:492] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:02.956506 6829 tablet_service.cc:1467] Processing CreateTablet for tablet 1c721b33983b4c47afd8faa8f155ca84 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=e43866993a7646be8fc624a6f2150d34]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:02.956763 6829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c721b33983b4c47afd8faa8f155ca84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:02.957859 11045 tablet_bootstrap.cc:654] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:02.959178 6567 tablet_service.cc:1467] Processing CreateTablet for tablet 01d0ff19ac53453888570a51948235e8 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=e43866993a7646be8fc624a6f2150d34]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:02.959177 9025 tablet_service.cc:1467] Processing CreateTablet for tablet 9f6bcf448d424d3e8bb18bf12cfb64c5 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=e43866993a7646be8fc624a6f2150d34]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:02.959436 6567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01d0ff19ac53453888570a51948235e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:02.959455 9025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f6bcf448d424d3e8bb18bf12cfb64c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:02.977200 11043 tablet_bootstrap.cc:492] T 01d0ff19ac53453888570a51948235e8 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:02.977376 11046 tablet_bootstrap.cc:492] T 01d0ff19ac53453888570a51948235e8 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:02.979368 11048 tablet_bootstrap.cc:492] T 01d0ff19ac53453888570a51948235e8 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:02.980111 11043 tablet_bootstrap.cc:654] T 01d0ff19ac53453888570a51948235e8 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:02.980257 10802 tablet_bootstrap.cc:492] T 915028b3478341d588ed6031e657c259 P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:02.980892 11046 tablet_bootstrap.cc:654] T 01d0ff19ac53453888570a51948235e8 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:02.981608 11048 tablet_bootstrap.cc:654] T 01d0ff19ac53453888570a51948235e8 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:02.982169 10802 tablet_bootstrap.cc:492] T 915028b3478341d588ed6031e657c259 P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap complete.
I20240503 02:00:02.982559 10802 ts_tablet_manager.cc:1367] T 915028b3478341d588ed6031e657c259 P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent bootstrapping tablet: real 0.096s user 0.029s sys 0.000s
I20240503 02:00:02.983091 10802 raft_consensus.cc:348] T 915028b3478341d588ed6031e657c259 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } }
I20240503 02:00:02.983942 10802 raft_consensus.cc:729] T 915028b3478341d588ed6031e657c259 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e1a9251ce1cf4361b8b0ede9fbfc4023, State: Initialized, Role: LEARNER
I20240503 02:00:02.984126 10802 consensus_queue.cc:260] T 915028b3478341d588ed6031e657c259 P e1a9251ce1cf4361b8b0ede9fbfc4023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } }
I20240503 02:00:02.984211 6316 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240503 02:00:02.984339 6316 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240503 02:00:02.984608 10802 ts_tablet_manager.cc:1398] T 915028b3478341d588ed6031e657c259 P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:02.986264 6609 tablet_copy_service.cc:342] P 80fd18ef4d604c86b909d5c008a46b4f: Request end of tablet copy session e1a9251ce1cf4361b8b0ede9fbfc4023-915028b3478341d588ed6031e657c259 received from {username='slave'} at 127.0.99.136:60117
I20240503 02:00:02.986343 6609 tablet_copy_service.cc:433] P 80fd18ef4d604c86b909d5c008a46b4f: ending tablet copy session e1a9251ce1cf4361b8b0ede9fbfc4023-915028b3478341d588ed6031e657c259 on tablet 915028b3478341d588ed6031e657c259 with peer e1a9251ce1cf4361b8b0ede9fbfc4023
I20240503 02:00:02.987440 6316 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet 9b5c2152b64449de9a4a0990a15ee3f8
I20240503 02:00:02.987761 6316 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet 0b659e9d57db4ebf93c9143938970090
I20240503 02:00:02.987943 6567 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b5c2152b64449de9a4a0990a15ee3f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:02 UTC) from {username='slave'} at 127.0.0.1:59846
I20240503 02:00:02.987995 6316 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet c85a74a168964629a6c2b527f0e02632
I20240503 02:00:02.988214 6700 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b659e9d57db4ebf93c9143938970090 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:02 UTC) from {username='slave'} at 127.0.0.1:40336
I20240503 02:00:02.988416 10173 tablet_replica.cc:330] T 0b659e9d57db4ebf93c9143938970090 P c2ac2eb7cc264047b9ab32fe3a4fc025: stopping tablet replica
I20240503 02:00:02.988468 9473 tablet_replica.cc:330] T 9b5c2152b64449de9a4a0990a15ee3f8 P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 02:00:02.988577 9473 raft_consensus.cc:2229] T 9b5c2152b64449de9a4a0990a15ee3f8 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:02.988575 10173 raft_consensus.cc:2229] T 0b659e9d57db4ebf93c9143938970090 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:02.988750 10173 raft_consensus.cc:2258] T 0b659e9d57db4ebf93c9143938970090 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:02.988780 9473 raft_consensus.cc:2258] T 9b5c2152b64449de9a4a0990a15ee3f8 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:02.989287 9473 ts_tablet_manager.cc:1875] T 9b5c2152b64449de9a4a0990a15ee3f8 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:02.989287 10173 ts_tablet_manager.cc:1875] T 0b659e9d57db4ebf93c9143938970090 P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:02.989297 9286 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b659e9d57db4ebf93c9143938970090 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:02 UTC) from {username='slave'} at 127.0.0.1:36004
I20240503 02:00:02.989413 6438 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b5c2152b64449de9a4a0990a15ee3f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:02 UTC) from {username='slave'} at 127.0.0.1:51338
I20240503 02:00:02.989576 9474 tablet_replica.cc:330] T 9b5c2152b64449de9a4a0990a15ee3f8 P dec3e4a04d604f7eaaa3f077d081218a: stopping tablet replica
I20240503 02:00:02.989655 6436 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b659e9d57db4ebf93c9143938970090 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:02 UTC) from {username='slave'} at 127.0.0.1:51338
I20240503 02:00:02.989727 9474 raft_consensus.cc:2229] T 9b5c2152b64449de9a4a0990a15ee3f8 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:02.989858 9474 raft_consensus.cc:2258] T 9b5c2152b64449de9a4a0990a15ee3f8 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:02.990334 9474 ts_tablet_manager.cc:1875] T 9b5c2152b64449de9a4a0990a15ee3f8 P dec3e4a04d604f7eaaa3f077d081218a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:02.991537 6829 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b5c2152b64449de9a4a0990a15ee3f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:02 UTC) from {username='slave'} at 127.0.0.1:52812
I20240503 02:00:02.991706 6831 tablet_service.cc:1514] Processing DeleteTablet for tablet c85a74a168964629a6c2b527f0e02632 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:02 UTC) from {username='slave'} at 127.0.0.1:52812
I20240503 02:00:02.991827 9476 tablet_replica.cc:330] T 9b5c2152b64449de9a4a0990a15ee3f8 P 492448570cc94ed1879feb2ec3b9051d: stopping tablet replica
I20240503 02:00:02.991935 9476 raft_consensus.cc:2229] T 9b5c2152b64449de9a4a0990a15ee3f8 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:02.992069 9476 raft_consensus.cc:2258] T 9b5c2152b64449de9a4a0990a15ee3f8 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:02.992625 9476 ts_tablet_manager.cc:1875] T 9b5c2152b64449de9a4a0990a15ee3f8 P 492448570cc94ed1879feb2ec3b9051d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:02.993014 9474 ts_tablet_manager.cc:1888] T 9b5c2152b64449de9a4a0990a15ee3f8 P dec3e4a04d604f7eaaa3f077d081218a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:00:02.993165 9474 log.cc:1196] T 9b5c2152b64449de9a4a0990a15ee3f8 P dec3e4a04d604f7eaaa3f077d081218a: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/9b5c2152b64449de9a4a0990a15ee3f8
I20240503 02:00:02.993427 9474 ts_tablet_manager.cc:1909] T 9b5c2152b64449de9a4a0990a15ee3f8 P dec3e4a04d604f7eaaa3f077d081218a: Deleting consensus metadata
I20240503 02:00:02.993470 11043 tablet_bootstrap.cc:492] T 01d0ff19ac53453888570a51948235e8 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:02.993571 11043 ts_tablet_manager.cc:1367] T 01d0ff19ac53453888570a51948235e8 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240503 02:00:02.994231 9474 tablet_replica.cc:330] T 0b659e9d57db4ebf93c9143938970090 P dec3e4a04d604f7eaaa3f077d081218a: stopping tablet replica
I20240503 02:00:02.994192 11043 raft_consensus.cc:348] T 01d0ff19ac53453888570a51948235e8 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:02.994298 11043 raft_consensus.cc:374] T 01d0ff19ac53453888570a51948235e8 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:02.994328 11043 raft_consensus.cc:729] T 01d0ff19ac53453888570a51948235e8 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:02.994347 9474 raft_consensus.cc:2229] T 0b659e9d57db4ebf93c9143938970090 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:02.994525 9474 raft_consensus.cc:2258] T 0b659e9d57db4ebf93c9143938970090 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:02.994479 11043 consensus_queue.cc:260] T 01d0ff19ac53453888570a51948235e8 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:02.994863 9476 ts_tablet_manager.cc:1888] T 9b5c2152b64449de9a4a0990a15ee3f8 P 492448570cc94ed1879feb2ec3b9051d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:00:02.994928 9476 log.cc:1196] T 9b5c2152b64449de9a4a0990a15ee3f8 P 492448570cc94ed1879feb2ec3b9051d: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/9b5c2152b64449de9a4a0990a15ee3f8
I20240503 02:00:02.994984 9474 ts_tablet_manager.cc:1875] T 0b659e9d57db4ebf93c9143938970090 P dec3e4a04d604f7eaaa3f077d081218a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:02.995167 9476 ts_tablet_manager.cc:1909] T 9b5c2152b64449de9a4a0990a15ee3f8 P 492448570cc94ed1879feb2ec3b9051d: Deleting consensus metadata
I20240503 02:00:02.995556 11043 ts_tablet_manager.cc:1398] T 01d0ff19ac53453888570a51948235e8 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:02.995833 6700 tablet_service.cc:1514] Processing DeleteTablet for tablet c85a74a168964629a6c2b527f0e02632 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:02 UTC) from {username='slave'} at 127.0.0.1:40336
I20240503 02:00:02.996547 6567 tablet_service.cc:1514] Processing DeleteTablet for tablet c85a74a168964629a6c2b527f0e02632 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:02 UTC) from {username='slave'} at 127.0.0.1:59846
I20240503 02:00:02.996593 6300 catalog_manager.cc:4830] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): tablet 9b5c2152b64449de9a4a0990a15ee3f8 (table rebalance_test_table_16 [id=eecab7e70bfa4621832bb99bd9ef066c]) successfully deleted
I20240503 02:00:02.996598 11046 tablet_bootstrap.cc:492] T 01d0ff19ac53453888570a51948235e8 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:02.996747 11046 ts_tablet_manager.cc:1367] T 01d0ff19ac53453888570a51948235e8 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.000s
I20240503 02:00:02.997699 9473 ts_tablet_manager.cc:1888] T 9b5c2152b64449de9a4a0990a15ee3f8 P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:00:02.997774 9473 log.cc:1196] T 9b5c2152b64449de9a4a0990a15ee3f8 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/9b5c2152b64449de9a4a0990a15ee3f8
I20240503 02:00:02.998029 9473 ts_tablet_manager.cc:1909] T 9b5c2152b64449de9a4a0990a15ee3f8 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting consensus metadata
I20240503 02:00:02.998471 9474 ts_tablet_manager.cc:1888] T 0b659e9d57db4ebf93c9143938970090 P dec3e4a04d604f7eaaa3f077d081218a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240503 02:00:02.998534 9474 log.cc:1196] T 0b659e9d57db4ebf93c9143938970090 P dec3e4a04d604f7eaaa3f077d081218a: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/0b659e9d57db4ebf93c9143938970090
I20240503 02:00:02.997221 11046 raft_consensus.cc:348] T 01d0ff19ac53453888570a51948235e8 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:02.998585 11046 raft_consensus.cc:374] T 01d0ff19ac53453888570a51948235e8 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:02.998621 11046 raft_consensus.cc:729] T 01d0ff19ac53453888570a51948235e8 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:02.998637 9473 tablet_replica.cc:330] T c85a74a168964629a6c2b527f0e02632 P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 02:00:02.998752 9473 raft_consensus.cc:2229] T c85a74a168964629a6c2b527f0e02632 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:02.998757 11046 consensus_queue.cc:260] T 01d0ff19ac53453888570a51948235e8 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:02.998816 9474 ts_tablet_manager.cc:1909] T 0b659e9d57db4ebf93c9143938970090 P dec3e4a04d604f7eaaa3f077d081218a: Deleting consensus metadata
I20240503 02:00:02.998880 9473 raft_consensus.cc:2258] T c85a74a168964629a6c2b527f0e02632 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:02.999370 9473 ts_tablet_manager.cc:1875] T c85a74a168964629a6c2b527f0e02632 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:02.999819 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet 9b5c2152b64449de9a4a0990a15ee3f8 (table rebalance_test_table_16 [id=eecab7e70bfa4621832bb99bd9ef066c]) successfully deleted
I20240503 02:00:03.000139 10173 ts_tablet_manager.cc:1888] T 0b659e9d57db4ebf93c9143938970090 P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240503 02:00:03.000522 10173 log.cc:1196] T 0b659e9d57db4ebf93c9143938970090 P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/0b659e9d57db4ebf93c9143938970090
I20240503 02:00:03.000806 10173 ts_tablet_manager.cc:1909] T 0b659e9d57db4ebf93c9143938970090 P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting consensus metadata
I20240503 02:00:03.001376 10173 tablet_replica.cc:330] T c85a74a168964629a6c2b527f0e02632 P c2ac2eb7cc264047b9ab32fe3a4fc025: stopping tablet replica
I20240503 02:00:03.001502 10173 raft_consensus.cc:2229] T c85a74a168964629a6c2b527f0e02632 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:03.001698 10173 raft_consensus.cc:2258] T c85a74a168964629a6c2b527f0e02632 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:03.002020 6303 catalog_manager.cc:4830] TS c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655): tablet 0b659e9d57db4ebf93c9143938970090 (table rebalance_test_table_16 [id=eecab7e70bfa4621832bb99bd9ef066c]) successfully deleted
I20240503 02:00:03.002609 9476 tablet_replica.cc:330] T c85a74a168964629a6c2b527f0e02632 P 492448570cc94ed1879feb2ec3b9051d: stopping tablet replica
I20240503 02:00:03.002664 6303 catalog_manager.cc:4830] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719): tablet 9b5c2152b64449de9a4a0990a15ee3f8 (table rebalance_test_table_16 [id=eecab7e70bfa4621832bb99bd9ef066c]) successfully deleted
I20240503 02:00:03.002835 9476 raft_consensus.cc:2229] T c85a74a168964629a6c2b527f0e02632 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:03.003005 9476 raft_consensus.cc:2258] T c85a74a168964629a6c2b527f0e02632 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:03.003564 9476 ts_tablet_manager.cc:1875] T c85a74a168964629a6c2b527f0e02632 P 492448570cc94ed1879feb2ec3b9051d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:03.003654 9178 raft_consensus.cc:1203] T 4c13642db7fc40a5a934aa5e3c2b08fc P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:00:03.004227 10173 ts_tablet_manager.cc:1875] T c85a74a168964629a6c2b527f0e02632 P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:03.005317 6300 catalog_manager.cc:4830] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): tablet 0b659e9d57db4ebf93c9143938970090 (table rebalance_test_table_16 [id=eecab7e70bfa4621832bb99bd9ef066c]) successfully deleted
I20240503 02:00:03.005692 11046 ts_tablet_manager.cc:1398] T 01d0ff19ac53453888570a51948235e8 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20240503 02:00:03.006762 9476 ts_tablet_manager.cc:1888] T c85a74a168964629a6c2b527f0e02632 P 492448570cc94ed1879feb2ec3b9051d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:00:03.006757 11046 tablet_bootstrap.cc:492] T 1c721b33983b4c47afd8faa8f155ca84 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:03.008071 11046 tablet_bootstrap.cc:654] T 1c721b33983b4c47afd8faa8f155ca84 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:03.006855 9476 log.cc:1196] T c85a74a168964629a6c2b527f0e02632 P 492448570cc94ed1879feb2ec3b9051d: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/c85a74a168964629a6c2b527f0e02632
I20240503 02:00:03.008970 9476 ts_tablet_manager.cc:1909] T c85a74a168964629a6c2b527f0e02632 P 492448570cc94ed1879feb2ec3b9051d: Deleting consensus metadata
I20240503 02:00:03.009821 6303 catalog_manager.cc:4830] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719): tablet c85a74a168964629a6c2b527f0e02632 (table rebalance_test_table_16 [id=eecab7e70bfa4621832bb99bd9ef066c]) successfully deleted
I20240503 02:00:03.010178 10173 ts_tablet_manager.cc:1888] T c85a74a168964629a6c2b527f0e02632 P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:00:03.010819 11046 tablet_bootstrap.cc:492] T 1c721b33983b4c47afd8faa8f155ca84 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:03.010922 11046 ts_tablet_manager.cc:1367] T 1c721b33983b4c47afd8faa8f155ca84 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:03.011402 11046 raft_consensus.cc:348] T 1c721b33983b4c47afd8faa8f155ca84 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:03.011530 11046 raft_consensus.cc:374] T 1c721b33983b4c47afd8faa8f155ca84 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:03.011569 11046 raft_consensus.cc:729] T 1c721b33983b4c47afd8faa8f155ca84 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:03.011724 11046 consensus_queue.cc:260] T 1c721b33983b4c47afd8faa8f155ca84 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:03.011799 11053 tablet_replica.cc:330] T 0b659e9d57db4ebf93c9143938970090 P e1a9251ce1cf4361b8b0ede9fbfc4023: stopping tablet replica
I20240503 02:00:03.011948 11053 raft_consensus.cc:2229] T 0b659e9d57db4ebf93c9143938970090 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:03.012048 9473 ts_tablet_manager.cc:1888] T c85a74a168964629a6c2b527f0e02632 P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:00:03.012096 11053 raft_consensus.cc:2258] T 0b659e9d57db4ebf93c9143938970090 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:03.012184 9473 log.cc:1196] T c85a74a168964629a6c2b527f0e02632 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/c85a74a168964629a6c2b527f0e02632
I20240503 02:00:03.012323 10173 log.cc:1196] T c85a74a168964629a6c2b527f0e02632 P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/c85a74a168964629a6c2b527f0e02632
I20240503 02:00:03.012646 11053 ts_tablet_manager.cc:1875] T 0b659e9d57db4ebf93c9143938970090 P e1a9251ce1cf4361b8b0ede9fbfc4023: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:03.012682 9473 ts_tablet_manager.cc:1909] T c85a74a168964629a6c2b527f0e02632 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting consensus metadata
I20240503 02:00:03.012845 11046 ts_tablet_manager.cc:1398] T 1c721b33983b4c47afd8faa8f155ca84 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:03.013334 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet c85a74a168964629a6c2b527f0e02632 (table rebalance_test_table_16 [id=eecab7e70bfa4621832bb99bd9ef066c]) successfully deleted
I20240503 02:00:03.013819 10173 ts_tablet_manager.cc:1909] T c85a74a168964629a6c2b527f0e02632 P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting consensus metadata
I20240503 02:00:03.014451 6303 catalog_manager.cc:4830] TS c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655): tablet c85a74a168964629a6c2b527f0e02632 (table rebalance_test_table_16 [id=eecab7e70bfa4621832bb99bd9ef066c]) successfully deleted
I20240503 02:00:03.015282 11048 tablet_bootstrap.cc:492] T 01d0ff19ac53453888570a51948235e8 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:03.015383 11048 ts_tablet_manager.cc:1367] T 01d0ff19ac53453888570a51948235e8 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.000s
I20240503 02:00:03.016160 11048 raft_consensus.cc:348] T 01d0ff19ac53453888570a51948235e8 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:03.016258 11048 raft_consensus.cc:374] T 01d0ff19ac53453888570a51948235e8 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:03.016393 11048 raft_consensus.cc:729] T 01d0ff19ac53453888570a51948235e8 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:03.016566 11048 consensus_queue.cc:260] T 01d0ff19ac53453888570a51948235e8 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:03.016965 11048 ts_tablet_manager.cc:1398] T 01d0ff19ac53453888570a51948235e8 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:00:03.020555 11053 ts_tablet_manager.cc:1888] T 0b659e9d57db4ebf93c9143938970090 P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240503 02:00:03.020648 11053 log.cc:1196] T 0b659e9d57db4ebf93c9143938970090 P e1a9251ce1cf4361b8b0ede9fbfc4023: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/0b659e9d57db4ebf93c9143938970090
I20240503 02:00:03.020947 11053 ts_tablet_manager.cc:1909] T 0b659e9d57db4ebf93c9143938970090 P e1a9251ce1cf4361b8b0ede9fbfc4023: Deleting consensus metadata
I20240503 02:00:03.021709 6301 catalog_manager.cc:4830] TS e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): tablet 0b659e9d57db4ebf93c9143938970090 (table rebalance_test_table_16 [id=eecab7e70bfa4621832bb99bd9ef066c]) successfully deleted
I20240503 02:00:03.031133 8917 raft_consensus.cc:1261] T f2716f6d9e554ec8a8b922a75e5d5d75 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:03.032657 9955 consensus_queue.cc:1035] T f2716f6d9e554ec8a8b922a75e5d5d75 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:03.032999 9178 raft_consensus.cc:1261] T f2716f6d9e554ec8a8b922a75e5d5d75 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:03.033416 9955 consensus_queue.cc:1035] T f2716f6d9e554ec8a8b922a75e5d5d75 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:03.035825 6720 consensus_queue.cc:237] T 8a76d3e2d4264687aa2972c179341b5e P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } }
I20240503 02:00:03.037190 11049 tablet_bootstrap.cc:492] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:03.037396 9179 raft_consensus.cc:1261] T 8a76d3e2d4264687aa2972c179341b5e P 2a01ace80baf4057b6bf1995cf88468b [term 2 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:00:03.037544 6312 catalog_manager.cc:5576] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079) does not have the latest schema for tablet f2716f6d9e554ec8a8b922a75e5d5d75 (table rebalancer_dynamic_table [id=58ca374f38374792b2e80195b318352c]). Expected version 1 got 0
I20240503 02:00:03.037878 10347 consensus_queue.cc:1035] T 8a76d3e2d4264687aa2972c179341b5e P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:03.038290 11049 tablet_bootstrap.cc:654] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:03.040210 6850 raft_consensus.cc:1261] T 8a76d3e2d4264687aa2972c179341b5e P 492448570cc94ed1879feb2ec3b9051d [term 2 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:00:03.041070 9711 raft_consensus.cc:2932] T 8a76d3e2d4264687aa2972c179341b5e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:03.041721 9179 raft_consensus.cc:2932] T 8a76d3e2d4264687aa2972c179341b5e P 2a01ace80baf4057b6bf1995cf88468b [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:03.044169 6312 catalog_manager.cc:5484] T 8a76d3e2d4264687aa2972c179341b5e P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: config changed from index 5 to 7, VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:03.044445 10268 consensus_queue.cc:1035] T 8a76d3e2d4264687aa2972c179341b5e P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:03.045171 11044 tablet_bootstrap.cc:492] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap starting.
I20240503 02:00:03.046227 11044 tablet_bootstrap.cc:654] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P e1a9251ce1cf4361b8b0ede9fbfc4023: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:03.046412 9039 raft_consensus.cc:1203] T 90ef3584830240acb0d8aad14ae851a6 P 99ead02915954b4fab23b033ad6d546b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 02:00:03.047369 6849 raft_consensus.cc:2932] T 8a76d3e2d4264687aa2972c179341b5e P 492448570cc94ed1879feb2ec3b9051d [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:03.047554 11045 tablet_bootstrap.cc:492] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:03.047657 11045 ts_tablet_manager.cc:1367] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.092s user 0.000s sys 0.002s
I20240503 02:00:03.048112 11045 raft_consensus.cc:348] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:03.048213 11045 raft_consensus.cc:374] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:03.048241 11045 raft_consensus.cc:729] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:03.048357 11045 consensus_queue.cc:260] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:03.048691 11045 ts_tablet_manager.cc:1398] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:03.049858 11045 tablet_bootstrap.cc:492] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:03.050801 6436 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a76d3e2d4264687aa2972c179341b5e with delete_type TABLET_DATA_TOMBSTONED (TS dec3e4a04d604f7eaaa3f077d081218a not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:51338
I20240503 02:00:03.050925 11045 tablet_bootstrap.cc:654] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:03.051162 6316 catalog_manager.cc:5576] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211) does not have the latest schema for tablet f2716f6d9e554ec8a8b922a75e5d5d75 (table rebalancer_dynamic_table [id=58ca374f38374792b2e80195b318352c]). Expected version 1 got 0
I20240503 02:00:03.051810 9474 tablet_replica.cc:330] T 8a76d3e2d4264687aa2972c179341b5e P dec3e4a04d604f7eaaa3f077d081218a: stopping tablet replica
I20240503 02:00:03.051954 9474 raft_consensus.cc:2229] T 8a76d3e2d4264687aa2972c179341b5e P dec3e4a04d604f7eaaa3f077d081218a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:03.051992 6303 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8a76d3e2d4264687aa2972c179341b5e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:03.052058 9474 raft_consensus.cc:2258] T 8a76d3e2d4264687aa2972c179341b5e P dec3e4a04d604f7eaaa3f077d081218a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:03.053138 11045 tablet_bootstrap.cc:492] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:03.053233 11045 ts_tablet_manager.cc:1367] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20240503 02:00:03.053568 9474 ts_tablet_manager.cc:1875] T 8a76d3e2d4264687aa2972c179341b5e P dec3e4a04d604f7eaaa3f077d081218a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:03.053570 9310 raft_consensus.cc:1203] T 925612b660ec471da5b3c548802ede6e P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:00:03.053700 11045 raft_consensus.cc:348] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:03.053812 11045 raft_consensus.cc:374] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:03.053843 11045 raft_consensus.cc:729] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:03.053951 11045 consensus_queue.cc:260] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:03.054230 11045 ts_tablet_manager.cc:1398] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:03.055039 6311 catalog_manager.cc:5576] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093) does not have the latest schema for tablet f2716f6d9e554ec8a8b922a75e5d5d75 (table rebalancer_dynamic_table [id=58ca374f38374792b2e80195b318352c]). Expected version 1 got 0
I20240503 02:00:03.056098 9474 ts_tablet_manager.cc:1888] T 8a76d3e2d4264687aa2972c179341b5e P dec3e4a04d604f7eaaa3f077d081218a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 02:00:03.056160 9474 log.cc:1196] T 8a76d3e2d4264687aa2972c179341b5e P dec3e4a04d604f7eaaa3f077d081218a: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/8a76d3e2d4264687aa2972c179341b5e
I20240503 02:00:03.057545 6300 catalog_manager.cc:4830] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): tablet 8a76d3e2d4264687aa2972c179341b5e (table rebalance_test_table_9 [id=c65135643094446abc8eba15be23cd99]) successfully deleted
I20240503 02:00:03.058547 6316 catalog_manager.cc:5576] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211) does not have the latest schema for tablet f2716f6d9e554ec8a8b922a75e5d5d75 (table rebalancer_dynamic_table [id=58ca374f38374792b2e80195b318352c]). Expected version 1 got 0
I20240503 02:00:03.059898 11062 tablet.cc:1722] T f2716f6d9e554ec8a8b922a75e5d5d75 P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:03.060824 11061 tablet.cc:1722] T f2716f6d9e554ec8a8b922a75e5d5d75 P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:03.061669 11061 tablet.cc:1620] T f2716f6d9e554ec8a8b922a75e5d5d75 P dec3e4a04d604f7eaaa3f077d081218a: MemRowSet was empty: no flush needed.
I20240503 02:00:03.062498 11062 tablet.cc:1620] T f2716f6d9e554ec8a8b922a75e5d5d75 P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
I20240503 02:00:03.062765 11044 tablet_bootstrap.cc:492] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P e1a9251ce1cf4361b8b0ede9fbfc4023: No bootstrap required, opened a new log
I20240503 02:00:03.062872 11044 ts_tablet_manager.cc:1367] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20240503 02:00:03.063356 11044 raft_consensus.cc:348] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:03.063462 11044 raft_consensus.cc:374] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:03.063493 11044 raft_consensus.cc:729] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1a9251ce1cf4361b8b0ede9fbfc4023, State: Initialized, Role: FOLLOWER
I20240503 02:00:03.063625 11044 consensus_queue.cc:260] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P e1a9251ce1cf4361b8b0ede9fbfc4023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:03.063746 10763 mvcc.cc:205] Tried to move back new op lower bound from 7023417766006300672 to 7023417764925472768. Current Snapshot: MvccSnapshot[applied={T|T < 7023417766006300672}]
I20240503 02:00:03.063903 11044 ts_tablet_manager.cc:1398] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:03.064064 11044 tablet_bootstrap.cc:492] T 1c721b33983b4c47afd8faa8f155ca84 P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap starting.
I20240503 02:00:03.065651 11044 tablet_bootstrap.cc:654] T 1c721b33983b4c47afd8faa8f155ca84 P e1a9251ce1cf4361b8b0ede9fbfc4023: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:03.065711 6307 catalog_manager.cc:5576] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079) does not have the latest schema for tablet f2716f6d9e554ec8a8b922a75e5d5d75 (table rebalancer_dynamic_table [id=58ca374f38374792b2e80195b318352c]). Expected version 1 got 0
I20240503 02:00:03.069558 6316 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=58ca374f38374792b2e80195b318352c] alter complete (version 1)
I20240503 02:00:03.071736 11049 tablet_bootstrap.cc:492] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:03.071846 11049 ts_tablet_manager.cc:1367] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.035s user 0.002s sys 0.000s
I20240503 02:00:03.071715 11064 tablet.cc:1722] T f2716f6d9e554ec8a8b922a75e5d5d75 P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:03.072368 11049 raft_consensus.cc:348] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:03.072501 11049 raft_consensus.cc:374] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:03.072528 11049 raft_consensus.cc:729] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:03.072659 11049 consensus_queue.cc:260] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:03.072933 11049 ts_tablet_manager.cc:1398] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:03.073215 11064 tablet.cc:1620] T f2716f6d9e554ec8a8b922a75e5d5d75 P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
I20240503 02:00:03.075431 11044 tablet_bootstrap.cc:492] T 1c721b33983b4c47afd8faa8f155ca84 P e1a9251ce1cf4361b8b0ede9fbfc4023: No bootstrap required, opened a new log
I20240503 02:00:03.075563 11044 ts_tablet_manager.cc:1367] T 1c721b33983b4c47afd8faa8f155ca84 P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20240503 02:00:03.076051 11044 raft_consensus.cc:348] T 1c721b33983b4c47afd8faa8f155ca84 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:03.076161 11044 raft_consensus.cc:374] T 1c721b33983b4c47afd8faa8f155ca84 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:03.076192 11044 raft_consensus.cc:729] T 1c721b33983b4c47afd8faa8f155ca84 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1a9251ce1cf4361b8b0ede9fbfc4023, State: Initialized, Role: FOLLOWER
I20240503 02:00:03.076313 11044 consensus_queue.cc:260] T 1c721b33983b4c47afd8faa8f155ca84 P e1a9251ce1cf4361b8b0ede9fbfc4023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:03.076633 11044 ts_tablet_manager.cc:1398] T 1c721b33983b4c47afd8faa8f155ca84 P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:03.085381 9758 raft_consensus.cc:482] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:03.085503 9758 raft_consensus.cc:504] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:03.086026 9758 leader_election.cc:288] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:03.086855 9310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c721b33983b4c47afd8faa8f155ca84" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" is_pre_election: true
I20240503 02:00:03.087011 9310 raft_consensus.cc:2454] T 1c721b33983b4c47afd8faa8f155ca84 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 0.
I20240503 02:00:03.087347 8851 leader_election.cc:302] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9942306855404cfd891adf227dd65eb2, e1a9251ce1cf4361b8b0ede9fbfc4023; no voters:
I20240503 02:00:03.087814 9758 raft_consensus.cc:2781] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:03.087872 9758 raft_consensus.cc:482] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:03.087906 9758 raft_consensus.cc:3037] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:03.088824 9758 raft_consensus.cc:504] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:03.089013 6582 consensus_queue.cc:237] T cb873018448549ccbe3d14b784e47d26 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { promote: true } }
I20240503 02:00:03.089133 9758 leader_election.cc:288] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 election: Requested vote from peers e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:03.089497 9310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c721b33983b4c47afd8faa8f155ca84" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023"
I20240503 02:00:03.089623 9310 raft_consensus.cc:3037] T 1c721b33983b4c47afd8faa8f155ca84 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:03.091547 9310 raft_consensus.cc:2454] T 1c721b33983b4c47afd8faa8f155ca84 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 1.
I20240503 02:00:03.092015 8851 leader_election.cc:302] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9942306855404cfd891adf227dd65eb2, e1a9251ce1cf4361b8b0ede9fbfc4023; no voters:
I20240503 02:00:03.092561 8917 raft_consensus.cc:1261] T cb873018448549ccbe3d14b784e47d26 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:03.092679 9039 raft_consensus.cc:1261] T cb873018448549ccbe3d14b784e47d26 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:00:03.092693 6525 consensus_peers.cc:485] T cb873018448549ccbe3d14b784e47d26 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719): Couldn't send request to peer 492448570cc94ed1879feb2ec3b9051d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cb873018448549ccbe3d14b784e47d26. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:03.092900 9973 consensus_queue.cc:1035] T cb873018448549ccbe3d14b784e47d26 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:03.093149 9758 raft_consensus.cc:2781] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:03.093201 9484 consensus_queue.cc:1035] T cb873018448549ccbe3d14b784e47d26 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:03.093329 9758 raft_consensus.cc:686] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2 [term 1 LEADER]: Becoming Leader. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Running, Role: LEADER
I20240503 02:00:03.093461 9758 consensus_queue.cc:237] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:03.094413 6850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c721b33983b4c47afd8faa8f155ca84" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 02:00:03.094540 6850 raft_consensus.cc:3037] T 1c721b33983b4c47afd8faa8f155ca84 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:03.095579 6850 raft_consensus.cc:2454] T 1c721b33983b4c47afd8faa8f155ca84 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 1.
I20240503 02:00:03.095862 9973 raft_consensus.cc:2932] T cb873018448549ccbe3d14b784e47d26 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { promote: true } } }
I20240503 02:00:03.096194 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c721b33983b4c47afd8faa8f155ca84" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 02:00:03.096330 6849 raft_consensus.cc:2362] T 1c721b33983b4c47afd8faa8f155ca84 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 1. Re-sending same reply.
I20240503 02:00:03.097064 6311 catalog_manager.cc:5484] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9942306855404cfd891adf227dd65eb2 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "9942306855404cfd891adf227dd65eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:03.097204 8917 raft_consensus.cc:2932] T cb873018448549ccbe3d14b784e47d26 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { promote: true } } }
I20240503 02:00:03.098462 9039 raft_consensus.cc:2932] T cb873018448549ccbe3d14b784e47d26 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { promote: true } } }
I20240503 02:00:03.100232 6316 catalog_manager.cc:5484] T cb873018448549ccbe3d14b784e47d26 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: config changed from index -1 to 2, NON_VOTER 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:03.107110 11000 ts_tablet_manager.cc:902] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 99ead02915954b4fab23b033ad6d546b: Initiating tablet copy from peer dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 02:00:03.107573 11000 tablet_copy_client.cc:308] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 99ead02915954b4fab23b033ad6d546b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:33093
I20240503 02:00:03.108040 6478 tablet_copy_service.cc:140] P dec3e4a04d604f7eaaa3f077d081218a: Received BeginTabletCopySession request for tablet 2d7244b0dc4d419cb5c3f3677acd70c7 from peer 99ead02915954b4fab23b033ad6d546b ({username='slave'} at 127.0.99.134:54509)
I20240503 02:00:03.108145 6478 tablet_copy_service.cc:161] P dec3e4a04d604f7eaaa3f077d081218a: Beginning new tablet copy session on tablet 2d7244b0dc4d419cb5c3f3677acd70c7 from peer 99ead02915954b4fab23b033ad6d546b at {username='slave'} at 127.0.99.134:54509: session id = 99ead02915954b4fab23b033ad6d546b-2d7244b0dc4d419cb5c3f3677acd70c7
I20240503 02:00:03.109190 6478 tablet_copy_source_session.cc:205] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:03.110217 11000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d7244b0dc4d419cb5c3f3677acd70c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:03.110996 9178 consensus_queue.cc:237] T f5ed01740019467d8c1d55cc2fa18ca3 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: true } }
I20240503 02:00:03.112643 8917 raft_consensus.cc:1261] T f5ed01740019467d8c1d55cc2fa18ca3 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Refusing update from remote peer 2a01ace80baf4057b6bf1995cf88468b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:00:03.112833 9114 consensus_peers.cc:485] T f5ed01740019467d8c1d55cc2fa18ca3 P 2a01ace80baf4057b6bf1995cf88468b -> Peer 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): Couldn't send request to peer 80fd18ef4d604c86b909d5c008a46b4f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f5ed01740019467d8c1d55cc2fa18ca3. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:03.112859 9039 raft_consensus.cc:1261] T f5ed01740019467d8c1d55cc2fa18ca3 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer 2a01ace80baf4057b6bf1995cf88468b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:03.113072 11087 consensus_queue.cc:1035] T f5ed01740019467d8c1d55cc2fa18ca3 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:03.113333 10257 consensus_queue.cc:1035] T f5ed01740019467d8c1d55cc2fa18ca3 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:03.113384 11000 tablet_copy_client.cc:783] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 99ead02915954b4fab23b033ad6d546b: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:03.113543 11000 tablet_copy_client.cc:647] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 99ead02915954b4fab23b033ad6d546b: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:03.114755 11000 tablet_copy_client.cc:523] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 99ead02915954b4fab23b033ad6d546b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:03.115360 10257 raft_consensus.cc:2932] T f5ed01740019467d8c1d55cc2fa18ca3 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: true } } }
I20240503 02:00:03.115784 9039 raft_consensus.cc:2932] T f5ed01740019467d8c1d55cc2fa18ca3 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: true } } }
I20240503 02:00:03.115927 8917 raft_consensus.cc:2932] T f5ed01740019467d8c1d55cc2fa18ca3 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: true } } }
I20240503 02:00:03.116575 11000 tablet_bootstrap.cc:492] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:03.119174 6307 catalog_manager.cc:5484] T f5ed01740019467d8c1d55cc2fa18ca3 P 9942306855404cfd891adf227dd65eb2 reported cstate change: config changed from index -1 to 2, NON_VOTER 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "2a01ace80baf4057b6bf1995cf88468b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: true } } }
I20240503 02:00:03.126796 8833 raft_consensus.cc:1050] T 7df4c083550046e883e5e61d3d4e1572 P 80fd18ef4d604c86b909d5c008a46b4f: attempting to promote NON_VOTER 9942306855404cfd891adf227dd65eb2 to VOTER
I20240503 02:00:03.127264 8833 consensus_queue.cc:237] T 7df4c083550046e883e5e61d3d4e1572 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } }
I20240503 02:00:03.128600 9484 raft_consensus.cc:482] T 01d0ff19ac53453888570a51948235e8 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:03.128701 9484 raft_consensus.cc:504] T 01d0ff19ac53453888570a51948235e8 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:03.129058 9484 leader_election.cc:288] T 01d0ff19ac53453888570a51948235e8 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:03.129513 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01d0ff19ac53453888570a51948235e8" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:03.129662 9178 raft_consensus.cc:2454] T 01d0ff19ac53453888570a51948235e8 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:03.129954 6522 leader_election.cc:302] T 01d0ff19ac53453888570a51948235e8 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 80fd18ef4d604c86b909d5c008a46b4f; no voters:
I20240503 02:00:03.130112 9484 raft_consensus.cc:2781] T 01d0ff19ac53453888570a51948235e8 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:03.130168 9484 raft_consensus.cc:482] T 01d0ff19ac53453888570a51948235e8 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:03.130204 9484 raft_consensus.cc:3037] T 01d0ff19ac53453888570a51948235e8 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:03.130684 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01d0ff19ac53453888570a51948235e8" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 02:00:03.130847 6849 raft_consensus.cc:2454] T 01d0ff19ac53453888570a51948235e8 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:03.131410 9484 raft_consensus.cc:504] T 01d0ff19ac53453888570a51948235e8 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:03.131772 9484 leader_election.cc:288] T 01d0ff19ac53453888570a51948235e8 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Requested vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:03.132520 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01d0ff19ac53453888570a51948235e8" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:03.132557 6720 raft_consensus.cc:1261] T 7df4c083550046e883e5e61d3d4e1572 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:03.132655 9178 raft_consensus.cc:3037] T 01d0ff19ac53453888570a51948235e8 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:03.133103 10345 consensus_queue.cc:1035] T 7df4c083550046e883e5e61d3d4e1572 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:03.133205 8917 raft_consensus.cc:1261] T 7df4c083550046e883e5e61d3d4e1572 P 9942306855404cfd891adf227dd65eb2 [term 1 LEARNER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:03.133639 10345 consensus_queue.cc:1035] T 7df4c083550046e883e5e61d3d4e1572 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:03.133790 9178 raft_consensus.cc:2454] T 01d0ff19ac53453888570a51948235e8 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:03.134394 6458 raft_consensus.cc:1261] T 7df4c083550046e883e5e61d3d4e1572 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:03.134764 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01d0ff19ac53453888570a51948235e8" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 02:00:03.134896 6849 raft_consensus.cc:3037] T 01d0ff19ac53453888570a51948235e8 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:03.135856 6849 raft_consensus.cc:2454] T 01d0ff19ac53453888570a51948235e8 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:03.136217 6525 leader_election.cc:302] T 01d0ff19ac53453888570a51948235e8 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 80fd18ef4d604c86b909d5c008a46b4f; no voters:
I20240503 02:00:03.136432 10345 raft_consensus.cc:2781] T 01d0ff19ac53453888570a51948235e8 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:03.136523 10345 raft_consensus.cc:686] T 01d0ff19ac53453888570a51948235e8 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Becoming Leader. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Running, Role: LEADER
I20240503 02:00:03.136627 9973 consensus_queue.cc:1035] T 7df4c083550046e883e5e61d3d4e1572 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:03.136667 10345 consensus_queue.cc:237] T 01d0ff19ac53453888570a51948235e8 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:03.138011 10802 ts_tablet_manager.cc:902] T 4385d51d11574192883d01376e2d5620 P e1a9251ce1cf4361b8b0ede9fbfc4023: Initiating tablet copy from peer 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:03.138494 10802 tablet_copy_client.cc:308] T 4385d51d11574192883d01376e2d5620 P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43863
I20240503 02:00:03.139130 6609 tablet_copy_service.cc:140] P 80fd18ef4d604c86b909d5c008a46b4f: Received BeginTabletCopySession request for tablet 4385d51d11574192883d01376e2d5620 from peer e1a9251ce1cf4361b8b0ede9fbfc4023 ({username='slave'} at 127.0.99.136:60117)
I20240503 02:00:03.139223 6609 tablet_copy_service.cc:161] P 80fd18ef4d604c86b909d5c008a46b4f: Beginning new tablet copy session on tablet 4385d51d11574192883d01376e2d5620 from peer e1a9251ce1cf4361b8b0ede9fbfc4023 at {username='slave'} at 127.0.99.136:60117: session id = e1a9251ce1cf4361b8b0ede9fbfc4023-4385d51d11574192883d01376e2d5620
I20240503 02:00:03.139994 6311 catalog_manager.cc:5484] T 01d0ff19ac53453888570a51948235e8 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:03.140131 6609 tablet_copy_source_session.cc:205] T 4385d51d11574192883d01376e2d5620 P 80fd18ef4d604c86b909d5c008a46b4f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:03.140482 11000 tablet_bootstrap.cc:492] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 99ead02915954b4fab23b033ad6d546b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:03.140790 11000 tablet_bootstrap.cc:492] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 99ead02915954b4fab23b033ad6d546b: Bootstrap complete.
I20240503 02:00:03.141044 11000 ts_tablet_manager.cc:1367] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.025s user 0.020s sys 0.004s
I20240503 02:00:03.141152 10802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4385d51d11574192883d01376e2d5620. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:03.141549 11000 raft_consensus.cc:348] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 99ead02915954b4fab23b033ad6d546b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } }
I20240503 02:00:03.141659 11000 raft_consensus.cc:729] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 99ead02915954b4fab23b033ad6d546b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: LEARNER
I20240503 02:00:03.141745 11000 consensus_queue.cc:260] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } }
I20240503 02:00:03.141999 11000 ts_tablet_manager.cc:1398] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:03.142776 10278 raft_consensus.cc:2932] T 7df4c083550046e883e5e61d3d4e1572 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:03.143273 6458 raft_consensus.cc:2932] T 7df4c083550046e883e5e61d3d4e1572 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:03.144701 6478 tablet_copy_service.cc:342] P dec3e4a04d604f7eaaa3f077d081218a: Request end of tablet copy session 99ead02915954b4fab23b033ad6d546b-2d7244b0dc4d419cb5c3f3677acd70c7 received from {username='slave'} at 127.0.99.134:54509
I20240503 02:00:03.144775 6478 tablet_copy_service.cc:433] P dec3e4a04d604f7eaaa3f077d081218a: ending tablet copy session 99ead02915954b4fab23b033ad6d546b-2d7244b0dc4d419cb5c3f3677acd70c7 on tablet 2d7244b0dc4d419cb5c3f3677acd70c7 with peer 99ead02915954b4fab23b033ad6d546b
I20240503 02:00:03.145309 8917 raft_consensus.cc:2932] T 7df4c083550046e883e5e61d3d4e1572 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:03.145824 6720 raft_consensus.cc:2932] T 7df4c083550046e883e5e61d3d4e1572 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:03.148705 6307 catalog_manager.cc:5484] T 7df4c083550046e883e5e61d3d4e1572 P 9942306855404cfd891adf227dd65eb2 reported cstate change: config changed from index 4 to 5, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:03.149211 10802 tablet_copy_client.cc:783] T 4385d51d11574192883d01376e2d5620 P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:03.149421 10802 tablet_copy_client.cc:647] T 4385d51d11574192883d01376e2d5620 P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:03.151021 6458 consensus_queue.cc:237] T 3b189fba0d6c461694dda002a3c5eb2f P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } }
I20240503 02:00:03.151162 10802 tablet_copy_client.cc:523] T 4385d51d11574192883d01376e2d5620 P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:03.152869 10802 tablet_bootstrap.cc:492] T 4385d51d11574192883d01376e2d5620 P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap starting.
I20240503 02:00:03.152969 6582 consensus_queue.cc:237] T 7df4c083550046e883e5e61d3d4e1572 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } }
I20240503 02:00:03.153196 6849 raft_consensus.cc:1261] T 3b189fba0d6c461694dda002a3c5eb2f P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:03.154282 8916 raft_consensus.cc:1261] T 7df4c083550046e883e5e61d3d4e1572 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:03.154621 9721 consensus_queue.cc:1035] T 3b189fba0d6c461694dda002a3c5eb2f P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:03.156641 6720 raft_consensus.cc:1261] T 3b189fba0d6c461694dda002a3c5eb2f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:00:03.156702 6394 consensus_peers.cc:485] T 3b189fba0d6c461694dda002a3c5eb2f P dec3e4a04d604f7eaaa3f077d081218a -> Peer 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211): Couldn't send request to peer 9942306855404cfd891adf227dd65eb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3b189fba0d6c461694dda002a3c5eb2f. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:03.157032 6458 raft_consensus.cc:1261] T 7df4c083550046e883e5e61d3d4e1572 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:03.157757 9484 consensus_queue.cc:1035] T 7df4c083550046e883e5e61d3d4e1572 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:03.158373 9721 consensus_queue.cc:1035] T 3b189fba0d6c461694dda002a3c5eb2f P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:03.160583 9484 consensus_queue.cc:1035] T 7df4c083550046e883e5e61d3d4e1572 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:03.160697 10381 raft_consensus.cc:2932] T 3b189fba0d6c461694dda002a3c5eb2f P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } } }
I20240503 02:00:03.164486 6311 catalog_manager.cc:5484] T 3b189fba0d6c461694dda002a3c5eb2f P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: config changed from index -1 to 4, NON_VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:03.167699 6720 raft_consensus.cc:2932] T 3b189fba0d6c461694dda002a3c5eb2f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } } }
I20240503 02:00:03.171169 6849 raft_consensus.cc:2932] T 3b189fba0d6c461694dda002a3c5eb2f P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } } }
I20240503 02:00:03.171862 9484 raft_consensus.cc:2932] T 7df4c083550046e883e5e61d3d4e1572 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:03.174731 6458 raft_consensus.cc:2932] T 7df4c083550046e883e5e61d3d4e1572 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:03.175441 6311 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:03.177604 6316 catalog_manager.cc:5484] T 7df4c083550046e883e5e61d3d4e1572 P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: config changed from index 5 to 6, VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:03.178876 6302 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7df4c083550046e883e5e61d3d4e1572 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:03.180192 10802 tablet_bootstrap.cc:492] T 4385d51d11574192883d01376e2d5620 P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:03.180091 8917 raft_consensus.cc:2932] T 7df4c083550046e883e5e61d3d4e1572 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:03.180555 10802 tablet_bootstrap.cc:492] T 4385d51d11574192883d01376e2d5620 P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap complete.
I20240503 02:00:03.180773 10802 ts_tablet_manager.cc:1367] T 4385d51d11574192883d01376e2d5620 P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent bootstrapping tablet: real 0.028s user 0.024s sys 0.001s
I20240503 02:00:03.181298 10802 raft_consensus.cc:348] T 4385d51d11574192883d01376e2d5620 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } }
I20240503 02:00:03.181455 10802 raft_consensus.cc:729] T 4385d51d11574192883d01376e2d5620 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e1a9251ce1cf4361b8b0ede9fbfc4023, State: Initialized, Role: LEARNER
I20240503 02:00:03.181594 10802 consensus_queue.cc:260] T 4385d51d11574192883d01376e2d5620 P e1a9251ce1cf4361b8b0ede9fbfc4023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } }
I20240503 02:00:03.182546 10802 ts_tablet_manager.cc:1398] T 4385d51d11574192883d01376e2d5620 P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:00:03.182574 9039 raft_consensus.cc:1203] T af00a839bb5e402791f87288493efef6 P 99ead02915954b4fab23b033ad6d546b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:00:03.183529 6700 tablet_service.cc:1514] Processing DeleteTablet for tablet 7df4c083550046e883e5e61d3d4e1572 with delete_type TABLET_DATA_TOMBSTONED (TS c2ac2eb7cc264047b9ab32fe3a4fc025 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:40336
I20240503 02:00:03.183794 10173 tablet_replica.cc:330] T 7df4c083550046e883e5e61d3d4e1572 P c2ac2eb7cc264047b9ab32fe3a4fc025: stopping tablet replica
I20240503 02:00:03.183923 10173 raft_consensus.cc:2229] T 7df4c083550046e883e5e61d3d4e1572 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:03.184144 10173 raft_consensus.cc:2258] T 7df4c083550046e883e5e61d3d4e1572 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:03.184113 11061 tablet.cc:1722] T f2716f6d9e554ec8a8b922a75e5d5d75 P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:03.184465 11061 tablet.cc:1620] T f2716f6d9e554ec8a8b922a75e5d5d75 P dec3e4a04d604f7eaaa3f077d081218a: MemRowSet was empty: no flush needed.
I20240503 02:00:03.185240 6609 tablet_copy_service.cc:342] P 80fd18ef4d604c86b909d5c008a46b4f: Request end of tablet copy session e1a9251ce1cf4361b8b0ede9fbfc4023-4385d51d11574192883d01376e2d5620 received from {username='slave'} at 127.0.99.136:60117
I20240503 02:00:03.185344 6609 tablet_copy_service.cc:433] P 80fd18ef4d604c86b909d5c008a46b4f: ending tablet copy session e1a9251ce1cf4361b8b0ede9fbfc4023-4385d51d11574192883d01376e2d5620 on tablet 4385d51d11574192883d01376e2d5620 with peer e1a9251ce1cf4361b8b0ede9fbfc4023
I20240503 02:00:03.185874 11062 tablet.cc:1722] T f2716f6d9e554ec8a8b922a75e5d5d75 P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:03.186125 10173 ts_tablet_manager.cc:1875] T 7df4c083550046e883e5e61d3d4e1572 P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:03.186229 11062 tablet.cc:1620] T f2716f6d9e554ec8a8b922a75e5d5d75 P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
I20240503 02:00:03.188557 10173 ts_tablet_manager.cc:1888] T 7df4c083550046e883e5e61d3d4e1572 P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:00:03.189893 10173 log.cc:1196] T 7df4c083550046e883e5e61d3d4e1572 P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/7df4c083550046e883e5e61d3d4e1572
I20240503 02:00:03.190603 8916 raft_consensus.cc:1203] T 1e445a3cc928429caf9e3c6193a63010 P 9942306855404cfd891adf227dd65eb2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 02:00:03.193166 11064 tablet.cc:1722] T f2716f6d9e554ec8a8b922a75e5d5d75 P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:03.193521 11064 tablet.cc:1620] T f2716f6d9e554ec8a8b922a75e5d5d75 P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
I20240503 02:00:03.196574 6582 raft_consensus.cc:1203] T f28586e70bd9473fbdc6b9126b666fa9 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:00:03.197935 6303 catalog_manager.cc:4830] TS c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655): tablet 7df4c083550046e883e5e61d3d4e1572 (table rebalance_test_table_15 [id=4e740b0167ed439291b74d77653ec386]) successfully deleted
I20240503 02:00:03.202513 6315 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=58ca374f38374792b2e80195b318352c] alter complete (version 2)
I20240503 02:00:03.223260 9178 raft_consensus.cc:1203] T 3cf186a0901f420390c8fda54316c5af P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 02:00:03.234899 9310 raft_consensus.cc:1203] T c3485701bf3b48f5ae2e1568afeed133 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:00:03.261096 8916 raft_consensus.cc:1203] T 46ad540deb624a25b97d3b967fe1f6f4 P 9942306855404cfd891adf227dd65eb2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 02:00:03.309226 6312 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:03.315157 11061 tablet.cc:1722] T f2716f6d9e554ec8a8b922a75e5d5d75 P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:03.315901 11064 tablet.cc:1722] T f2716f6d9e554ec8a8b922a75e5d5d75 P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:03.317616 11062 tablet.cc:1722] T f2716f6d9e554ec8a8b922a75e5d5d75 P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:03.321516 6316 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=58ca374f38374792b2e80195b318352c] alter complete (version 3)
I20240503 02:00:03.328997 9758 raft_consensus.cc:482] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:03.329130 9758 raft_consensus.cc:504] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:03.329557 9758 leader_election.cc:288] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:03.330106 9039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f6bcf448d424d3e8bb18bf12cfb64c5" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:03.330089 9310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f6bcf448d424d3e8bb18bf12cfb64c5" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" is_pre_election: true
I20240503 02:00:03.330233 9039 raft_consensus.cc:2454] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 0.
I20240503 02:00:03.330233 9310 raft_consensus.cc:2454] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 0.
I20240503 02:00:03.330581 8851 leader_election.cc:302] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9942306855404cfd891adf227dd65eb2, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:03.330775 9758 raft_consensus.cc:2781] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:03.330849 9758 raft_consensus.cc:482] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:03.330899 9758 raft_consensus.cc:3037] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:03.331725 9758 raft_consensus.cc:504] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:03.332043 9758 leader_election.cc:288] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 election: Requested vote from peers e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:03.332461 9310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f6bcf448d424d3e8bb18bf12cfb64c5" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023"
I20240503 02:00:03.332530 9039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f6bcf448d424d3e8bb18bf12cfb64c5" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:03.332607 9310 raft_consensus.cc:3037] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:03.332649 9039 raft_consensus.cc:3037] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:03.333586 9310 raft_consensus.cc:2454] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 1.
I20240503 02:00:03.333588 9039 raft_consensus.cc:2454] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 1.
I20240503 02:00:03.334318 8851 leader_election.cc:302] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9942306855404cfd891adf227dd65eb2, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:03.334607 9758 raft_consensus.cc:2781] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:03.334764 9758 raft_consensus.cc:686] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P 9942306855404cfd891adf227dd65eb2 [term 1 LEADER]: Becoming Leader. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Running, Role: LEADER
I20240503 02:00:03.334935 9758 consensus_queue.cc:237] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:03.337040 6307 catalog_manager.cc:5484] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P 9942306855404cfd891adf227dd65eb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9942306855404cfd891adf227dd65eb2 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "9942306855404cfd891adf227dd65eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:03.349102 6582 consensus_queue.cc:237] T d0d5dd42db3e4883af7fd16d8813b550 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } }
I20240503 02:00:03.350883 6720 raft_consensus.cc:1261] T d0d5dd42db3e4883af7fd16d8813b550 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:00:03.350998 6523 consensus_peers.cc:485] T d0d5dd42db3e4883af7fd16d8813b550 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d0d5dd42db3e4883af7fd16d8813b550. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:03.351004 6849 raft_consensus.cc:1261] T d0d5dd42db3e4883af7fd16d8813b550 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:03.351395 8833 consensus_queue.cc:1035] T d0d5dd42db3e4883af7fd16d8813b550 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:03.351794 10345 consensus_queue.cc:1035] T d0d5dd42db3e4883af7fd16d8813b550 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:03.353621 8833 raft_consensus.cc:2932] T d0d5dd42db3e4883af7fd16d8813b550 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } } }
I20240503 02:00:03.354251 6849 raft_consensus.cc:2932] T d0d5dd42db3e4883af7fd16d8813b550 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } } }
I20240503 02:00:03.354692 6720 raft_consensus.cc:2932] T d0d5dd42db3e4883af7fd16d8813b550 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } } }
I20240503 02:00:03.358642 6307 catalog_manager.cc:5484] T d0d5dd42db3e4883af7fd16d8813b550 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: config changed from index -1 to 4, NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:03.360453 10268 raft_consensus.cc:1050] T 2fc834abc6a94c638d6202b03320a2d2 P c2ac2eb7cc264047b9ab32fe3a4fc025: attempting to promote NON_VOTER 2a01ace80baf4057b6bf1995cf88468b to VOTER
I20240503 02:00:03.360898 10268 consensus_queue.cc:237] T 2fc834abc6a94c638d6202b03320a2d2 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } }
I20240503 02:00:03.362493 6849 raft_consensus.cc:1261] T 2fc834abc6a94c638d6202b03320a2d2 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:03.363211 10347 consensus_queue.cc:1035] T 2fc834abc6a94c638d6202b03320a2d2 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:03.363761 6458 raft_consensus.cc:1261] T 2fc834abc6a94c638d6202b03320a2d2 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:03.364917 9178 raft_consensus.cc:1261] T 2fc834abc6a94c638d6202b03320a2d2 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEARNER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:03.365227 10347 consensus_queue.cc:1035] T 2fc834abc6a94c638d6202b03320a2d2 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:03.365511 10349 consensus_queue.cc:1035] T 2fc834abc6a94c638d6202b03320a2d2 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:03.368631 9711 raft_consensus.cc:2932] T 2fc834abc6a94c638d6202b03320a2d2 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:03.368822 6720 consensus_queue.cc:237] T 485d1d89a8b94505969d663be92124cd P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } }
I20240503 02:00:03.370484 6850 raft_consensus.cc:2932] T 2fc834abc6a94c638d6202b03320a2d2 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:03.371898 6456 raft_consensus.cc:1261] T 485d1d89a8b94505969d663be92124cd P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:03.372875 6316 catalog_manager.cc:5484] T 2fc834abc6a94c638d6202b03320a2d2 P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: config changed from index 4 to 5, 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:03.374550 9039 raft_consensus.cc:1261] T 485d1d89a8b94505969d663be92124cd P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:03.375161 10349 consensus_queue.cc:1035] T 485d1d89a8b94505969d663be92124cd P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:03.376649 9178 raft_consensus.cc:2932] T 2fc834abc6a94c638d6202b03320a2d2 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:03.377082 6720 consensus_queue.cc:237] T 2fc834abc6a94c638d6202b03320a2d2 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } }
I20240503 02:00:03.378283 6850 raft_consensus.cc:1261] T 2fc834abc6a94c638d6202b03320a2d2 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:03.378321 9984 raft_consensus.cc:2932] T 485d1d89a8b94505969d663be92124cd P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } } }
I20240503 02:00:03.378777 9039 raft_consensus.cc:2932] T 485d1d89a8b94505969d663be92124cd P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } } }
I20240503 02:00:03.379685 9984 consensus_queue.cc:1035] T 2fc834abc6a94c638d6202b03320a2d2 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:03.381403 6316 catalog_manager.cc:5484] T 485d1d89a8b94505969d663be92124cd P 99ead02915954b4fab23b033ad6d546b reported cstate change: config changed from index 6 to 7, NON_VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } } }
I20240503 02:00:03.382037 10347 consensus_queue.cc:1035] T 485d1d89a8b94505969d663be92124cd P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 02:00:03.382663 6656 consensus_peers.cc:485] T 485d1d89a8b94505969d663be92124cd P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211): Couldn't send request to peer 9942306855404cfd891adf227dd65eb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 485d1d89a8b94505969d663be92124cd. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:03.383464 6458 raft_consensus.cc:2932] T 485d1d89a8b94505969d663be92124cd P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } } }
I20240503 02:00:03.387563 9984 raft_consensus.cc:2932] T 2fc834abc6a94c638d6202b03320a2d2 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:03.388958 6303 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2fc834abc6a94c638d6202b03320a2d2 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:03.388854 6850 raft_consensus.cc:2932] T 2fc834abc6a94c638d6202b03320a2d2 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:03.389622 9178 raft_consensus.cc:1261] T 2fc834abc6a94c638d6202b03320a2d2 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:03.390080 6316 catalog_manager.cc:5484] T 2fc834abc6a94c638d6202b03320a2d2 P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: config changed from index 5 to 6, VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:03.390604 10268 consensus_queue.cc:1035] T 2fc834abc6a94c638d6202b03320a2d2 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:03.391580 9178 raft_consensus.cc:2932] T 2fc834abc6a94c638d6202b03320a2d2 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:03.393399 9039 raft_consensus.cc:1203] T ac1632cc5eb94554b1faa7eb715706ef P 99ead02915954b4fab23b033ad6d546b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:00:03.393541 6436 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fc834abc6a94c638d6202b03320a2d2 with delete_type TABLET_DATA_TOMBSTONED (TS dec3e4a04d604f7eaaa3f077d081218a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51338
I20240503 02:00:03.393792 9474 tablet_replica.cc:330] T 2fc834abc6a94c638d6202b03320a2d2 P dec3e4a04d604f7eaaa3f077d081218a: stopping tablet replica
I20240503 02:00:03.393927 9474 raft_consensus.cc:2229] T 2fc834abc6a94c638d6202b03320a2d2 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:03.394068 9474 pending_rounds.cc:62] T 2fc834abc6a94c638d6202b03320a2d2 P dec3e4a04d604f7eaaa3f077d081218a: Trying to abort 1 pending ops.
I20240503 02:00:03.394094 9474 pending_rounds.cc:68] T 2fc834abc6a94c638d6202b03320a2d2 P dec3e4a04d604f7eaaa3f077d081218a: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023417767365242880 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "2fc834abc6a94c638d6202b03320a2d2" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } } }
I20240503 02:00:03.394199 9474 raft_consensus.cc:2894] T 2fc834abc6a94c638d6202b03320a2d2 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240503 02:00:03.394240 9474 raft_consensus.cc:2866] T 2fc834abc6a94c638d6202b03320a2d2 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 02:00:03.394276 9474 raft_consensus.cc:2258] T 2fc834abc6a94c638d6202b03320a2d2 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:03.394869 9474 ts_tablet_manager.cc:1875] T 2fc834abc6a94c638d6202b03320a2d2 P dec3e4a04d604f7eaaa3f077d081218a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:03.397128 9474 ts_tablet_manager.cc:1888] T 2fc834abc6a94c638d6202b03320a2d2 P dec3e4a04d604f7eaaa3f077d081218a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:00:03.397212 9474 log.cc:1196] T 2fc834abc6a94c638d6202b03320a2d2 P dec3e4a04d604f7eaaa3f077d081218a: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/2fc834abc6a94c638d6202b03320a2d2
I20240503 02:00:03.397720 6300 catalog_manager.cc:4830] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): tablet 2fc834abc6a94c638d6202b03320a2d2 (table rebalance_test_table_7 [id=e0e5e479de824fb18516cfe56d5892c6]) successfully deleted
I20240503 02:00:03.423585 6316 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:03.423759 6316 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:03.426126 6316 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet f2716f6d9e554ec8a8b922a75e5d5d75
I20240503 02:00:03.426803 8880 tablet_service.cc:1514] Processing DeleteTablet for tablet f2716f6d9e554ec8a8b922a75e5d5d75 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:03 UTC) from {username='slave'} at 127.0.0.1:40296
I20240503 02:00:03.426987 9158 tablet_service.cc:1514] Processing DeleteTablet for tablet f2716f6d9e554ec8a8b922a75e5d5d75 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:03 UTC) from {username='slave'} at 127.0.0.1:51940
I20240503 02:00:03.427222 6436 tablet_service.cc:1514] Processing DeleteTablet for tablet f2716f6d9e554ec8a8b922a75e5d5d75 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:03 UTC) from {username='slave'} at 127.0.0.1:51338
I20240503 02:00:03.427439 9474 tablet_replica.cc:330] T f2716f6d9e554ec8a8b922a75e5d5d75 P dec3e4a04d604f7eaaa3f077d081218a: stopping tablet replica
I20240503 02:00:03.427590 9474 raft_consensus.cc:2229] T f2716f6d9e554ec8a8b922a75e5d5d75 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:03.427924 11251 tablet_replica.cc:330] T f2716f6d9e554ec8a8b922a75e5d5d75 P 9942306855404cfd891adf227dd65eb2: stopping tablet replica
I20240503 02:00:03.427986 9474 raft_consensus.cc:2258] T f2716f6d9e554ec8a8b922a75e5d5d75 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:03.428097 11251 raft_consensus.cc:2229] T f2716f6d9e554ec8a8b922a75e5d5d75 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:03.428251 11251 raft_consensus.cc:2258] T f2716f6d9e554ec8a8b922a75e5d5d75 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:03.428736 9474 ts_tablet_manager.cc:1875] T f2716f6d9e554ec8a8b922a75e5d5d75 P dec3e4a04d604f7eaaa3f077d081218a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:03.428776 11251 ts_tablet_manager.cc:1875] T f2716f6d9e554ec8a8b922a75e5d5d75 P 9942306855404cfd891adf227dd65eb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:03.428779 11252 tablet_replica.cc:330] T f2716f6d9e554ec8a8b922a75e5d5d75 P 2a01ace80baf4057b6bf1995cf88468b: stopping tablet replica
I20240503 02:00:03.429108 11252 raft_consensus.cc:2229] T f2716f6d9e554ec8a8b922a75e5d5d75 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:03.429265 11252 raft_consensus.cc:2258] T f2716f6d9e554ec8a8b922a75e5d5d75 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:03.430809 11252 ts_tablet_manager.cc:1875] T f2716f6d9e554ec8a8b922a75e5d5d75 P 2a01ace80baf4057b6bf1995cf88468b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:03.431013 9474 ts_tablet_manager.cc:1888] T f2716f6d9e554ec8a8b922a75e5d5d75 P dec3e4a04d604f7eaaa3f077d081218a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:03.431013 11251 ts_tablet_manager.cc:1888] T f2716f6d9e554ec8a8b922a75e5d5d75 P 9942306855404cfd891adf227dd65eb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:03.431092 11251 log.cc:1196] T f2716f6d9e554ec8a8b922a75e5d5d75 P 9942306855404cfd891adf227dd65eb2: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/f2716f6d9e554ec8a8b922a75e5d5d75
I20240503 02:00:03.431092 9474 log.cc:1196] T f2716f6d9e554ec8a8b922a75e5d5d75 P dec3e4a04d604f7eaaa3f077d081218a: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/f2716f6d9e554ec8a8b922a75e5d5d75
I20240503 02:00:03.431337 9474 ts_tablet_manager.cc:1909] T f2716f6d9e554ec8a8b922a75e5d5d75 P dec3e4a04d604f7eaaa3f077d081218a: Deleting consensus metadata
I20240503 02:00:03.431337 11251 ts_tablet_manager.cc:1909] T f2716f6d9e554ec8a8b922a75e5d5d75 P 9942306855404cfd891adf227dd65eb2: Deleting consensus metadata
I20240503 02:00:03.432801 6303 catalog_manager.cc:4830] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211): tablet f2716f6d9e554ec8a8b922a75e5d5d75 (table rebalancer_dynamic_table_new_name [id=58ca374f38374792b2e80195b318352c]) successfully deleted
I20240503 02:00:03.432977 11252 ts_tablet_manager.cc:1888] T f2716f6d9e554ec8a8b922a75e5d5d75 P 2a01ace80baf4057b6bf1995cf88468b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:03.433068 11252 log.cc:1196] T f2716f6d9e554ec8a8b922a75e5d5d75 P 2a01ace80baf4057b6bf1995cf88468b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/f2716f6d9e554ec8a8b922a75e5d5d75
I20240503 02:00:03.433297 11252 ts_tablet_manager.cc:1909] T f2716f6d9e554ec8a8b922a75e5d5d75 P 2a01ace80baf4057b6bf1995cf88468b: Deleting consensus metadata
I20240503 02:00:03.433533 9310 raft_consensus.cc:1203] T 915028b3478341d588ed6031e657c259 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 02:00:03.433564 6300 catalog_manager.cc:4830] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): tablet f2716f6d9e554ec8a8b922a75e5d5d75 (table rebalancer_dynamic_table_new_name [id=58ca374f38374792b2e80195b318352c]) successfully deleted
I20240503 02:00:03.436488 6300 catalog_manager.cc:4830] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079): tablet f2716f6d9e554ec8a8b922a75e5d5d75 (table rebalancer_dynamic_table_new_name [id=58ca374f38374792b2e80195b318352c]) successfully deleted
I20240503 02:00:03.451618 9973 raft_consensus.cc:1050] T 925612b660ec471da5b3c548802ede6e P 80fd18ef4d604c86b909d5c008a46b4f: attempting to promote NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 to VOTER
I20240503 02:00:03.452085 9973 consensus_queue.cc:237] T 925612b660ec471da5b3c548802ede6e P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } }
I20240503 02:00:03.453531 9178 raft_consensus.cc:1261] T 925612b660ec471da5b3c548802ede6e P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:03.453558 8916 raft_consensus.cc:1261] T 925612b660ec471da5b3c548802ede6e P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:03.453567 9310 raft_consensus.cc:1261] T 925612b660ec471da5b3c548802ede6e P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 LEARNER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:03.453975 10278 consensus_queue.cc:1035] T 925612b660ec471da5b3c548802ede6e P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:03.454339 10278 consensus_queue.cc:1035] T 925612b660ec471da5b3c548802ede6e P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:03.454803 10278 consensus_queue.cc:1035] T 925612b660ec471da5b3c548802ede6e P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:03.456080 9973 raft_consensus.cc:2932] T 925612b660ec471da5b3c548802ede6e P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:03.456964 9178 raft_consensus.cc:2932] T 925612b660ec471da5b3c548802ede6e P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:03.457932 9310 raft_consensus.cc:2932] T 925612b660ec471da5b3c548802ede6e P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:03.458590 8916 raft_consensus.cc:2932] T 925612b660ec471da5b3c548802ede6e P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:03.460143 6316 catalog_manager.cc:5484] T 925612b660ec471da5b3c548802ede6e P 2a01ace80baf4057b6bf1995cf88468b reported cstate change: config changed from index 2 to 3, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:03.465193 6582 consensus_queue.cc:237] T 925612b660ec471da5b3c548802ede6e P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } }
I20240503 02:00:03.466527 9178 raft_consensus.cc:1261] T 925612b660ec471da5b3c548802ede6e P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:03.466666 9310 raft_consensus.cc:1261] T 925612b660ec471da5b3c548802ede6e P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:03.467113 9973 consensus_queue.cc:1035] T 925612b660ec471da5b3c548802ede6e P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:03.467499 9973 consensus_queue.cc:1035] T 925612b660ec471da5b3c548802ede6e P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:03.469599 9973 raft_consensus.cc:2932] T 925612b660ec471da5b3c548802ede6e P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:03.470263 9310 raft_consensus.cc:2932] T 925612b660ec471da5b3c548802ede6e P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:03.471665 9178 raft_consensus.cc:2932] T 925612b660ec471da5b3c548802ede6e P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:03.472577 6302 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 925612b660ec471da5b3c548802ede6e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:03.473426 6307 catalog_manager.cc:5484] T 925612b660ec471da5b3c548802ede6e P e1a9251ce1cf4361b8b0ede9fbfc4023 reported cstate change: config changed from index 3 to 4, VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:03.476743 8880 tablet_service.cc:1514] Processing DeleteTablet for tablet 925612b660ec471da5b3c548802ede6e with delete_type TABLET_DATA_TOMBSTONED (TS 9942306855404cfd891adf227dd65eb2 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40296
I20240503 02:00:03.477068 11251 tablet_replica.cc:330] T 925612b660ec471da5b3c548802ede6e P 9942306855404cfd891adf227dd65eb2: stopping tablet replica
I20240503 02:00:03.477228 11251 raft_consensus.cc:2229] T 925612b660ec471da5b3c548802ede6e P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:03.477389 11251 raft_consensus.cc:2258] T 925612b660ec471da5b3c548802ede6e P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:03.478013 11251 ts_tablet_manager.cc:1875] T 925612b660ec471da5b3c548802ede6e P 9942306855404cfd891adf227dd65eb2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:03.480466 11251 ts_tablet_manager.cc:1888] T 925612b660ec471da5b3c548802ede6e P 9942306855404cfd891adf227dd65eb2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:00:03.480587 11251 log.cc:1196] T 925612b660ec471da5b3c548802ede6e P 9942306855404cfd891adf227dd65eb2: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/925612b660ec471da5b3c548802ede6e
I20240503 02:00:03.481117 6303 catalog_manager.cc:4830] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211): tablet 925612b660ec471da5b3c548802ede6e (table rebalancer_extra_table_6 [id=0392495bb8a743099943f2bba5f15608]) successfully deleted
I20240503 02:00:03.481766 10268 raft_consensus.cc:1050] T 4c13642db7fc40a5a934aa5e3c2b08fc P c2ac2eb7cc264047b9ab32fe3a4fc025: attempting to promote NON_VOTER 2a01ace80baf4057b6bf1995cf88468b to VOTER
I20240503 02:00:03.482204 10268 consensus_queue.cc:237] T 4c13642db7fc40a5a934aa5e3c2b08fc P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } }
I20240503 02:00:03.483719 6582 raft_consensus.cc:1261] T 4c13642db7fc40a5a934aa5e3c2b08fc P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:03.483956 6850 raft_consensus.cc:1261] T 4c13642db7fc40a5a934aa5e3c2b08fc P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:03.484102 9984 consensus_queue.cc:1035] T 4c13642db7fc40a5a934aa5e3c2b08fc P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:03.484297 9178 raft_consensus.cc:1261] T 4c13642db7fc40a5a934aa5e3c2b08fc P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEARNER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:03.484591 9984 consensus_queue.cc:1035] T 4c13642db7fc40a5a934aa5e3c2b08fc P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:03.484758 10349 consensus_queue.cc:1035] T 4c13642db7fc40a5a934aa5e3c2b08fc P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:03.486704 9711 raft_consensus.cc:2932] T 4c13642db7fc40a5a934aa5e3c2b08fc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:03.487138 9178 raft_consensus.cc:2932] T 4c13642db7fc40a5a934aa5e3c2b08fc P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:03.489161 6582 raft_consensus.cc:2932] T 4c13642db7fc40a5a934aa5e3c2b08fc P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:03.489480 6316 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240503 02:00:03.489625 6316 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240503 02:00:03.491412 6850 raft_consensus.cc:2932] T 4c13642db7fc40a5a934aa5e3c2b08fc P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:03.494266 6316 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 4 replicas of tablet af00a839bb5e402791f87288493efef6
I20240503 02:00:03.494638 6316 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 4 replicas of tablet 4c13642db7fc40a5a934aa5e3c2b08fc
I20240503 02:00:03.495026 9158 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c13642db7fc40a5a934aa5e3c2b08fc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:03 UTC) from {username='slave'} at 127.0.0.1:51940
I20240503 02:00:03.494999 6436 tablet_service.cc:1514] Processing DeleteTablet for tablet af00a839bb5e402791f87288493efef6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:03 UTC) from {username='slave'} at 127.0.0.1:51338
I20240503 02:00:03.495211 11252 tablet_replica.cc:330] T 4c13642db7fc40a5a934aa5e3c2b08fc P 2a01ace80baf4057b6bf1995cf88468b: stopping tablet replica
I20240503 02:00:03.495204 6700 tablet_service.cc:1514] Processing DeleteTablet for tablet af00a839bb5e402791f87288493efef6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:03 UTC) from {username='slave'} at 127.0.0.1:40336
I20240503 02:00:03.495350 9025 tablet_service.cc:1514] Processing DeleteTablet for tablet af00a839bb5e402791f87288493efef6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:03 UTC) from {username='slave'} at 127.0.0.1:52584
I20240503 02:00:03.495385 11252 raft_consensus.cc:2229] T 4c13642db7fc40a5a934aa5e3c2b08fc P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:03.495533 11252 raft_consensus.cc:2258] T 4c13642db7fc40a5a934aa5e3c2b08fc P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:03.495627 6567 tablet_service.cc:1514] Processing DeleteTablet for tablet af00a839bb5e402791f87288493efef6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:03 UTC) from {username='slave'} at 127.0.0.1:59846
I20240503 02:00:03.495673 6569 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c13642db7fc40a5a934aa5e3c2b08fc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:03 UTC) from {username='slave'} at 127.0.0.1:59846
I20240503 02:00:03.495730 9474 tablet_replica.cc:330] T af00a839bb5e402791f87288493efef6 P dec3e4a04d604f7eaaa3f077d081218a: stopping tablet replica
I20240503 02:00:03.495803 6831 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c13642db7fc40a5a934aa5e3c2b08fc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:03 UTC) from {username='slave'} at 127.0.0.1:52812
I20240503 02:00:03.495880 9474 raft_consensus.cc:2229] T af00a839bb5e402791f87288493efef6 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:03.495962 9473 tablet_replica.cc:330] T af00a839bb5e402791f87288493efef6 P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 02:00:03.496088 9473 raft_consensus.cc:2229] T af00a839bb5e402791f87288493efef6 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:03.496172 9474 raft_consensus.cc:2258] T af00a839bb5e402791f87288493efef6 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:03.496250 9473 raft_consensus.cc:2258] T af00a839bb5e402791f87288493efef6 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:03.496392 6316 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet 9f7ade1e2af24e86b6e54e2b5769e4c0
I20240503 02:00:03.496803 9473 ts_tablet_manager.cc:1875] T af00a839bb5e402791f87288493efef6 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:03.496799 9474 ts_tablet_manager.cc:1875] T af00a839bb5e402791f87288493efef6 P dec3e4a04d604f7eaaa3f077d081218a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:03.498957 9476 tablet_replica.cc:330] T 4c13642db7fc40a5a934aa5e3c2b08fc P 492448570cc94ed1879feb2ec3b9051d: stopping tablet replica
I20240503 02:00:03.499150 9476 raft_consensus.cc:2229] T 4c13642db7fc40a5a934aa5e3c2b08fc P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:03.499214 9473 ts_tablet_manager.cc:1888] T af00a839bb5e402791f87288493efef6 P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:03.499228 9474 ts_tablet_manager.cc:1888] T af00a839bb5e402791f87288493efef6 P dec3e4a04d604f7eaaa3f077d081218a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:03.499287 9473 log.cc:1196] T af00a839bb5e402791f87288493efef6 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/af00a839bb5e402791f87288493efef6
I20240503 02:00:03.499338 9476 raft_consensus.cc:2258] T 4c13642db7fc40a5a934aa5e3c2b08fc P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:03.499384 9474 log.cc:1196] T af00a839bb5e402791f87288493efef6 P dec3e4a04d604f7eaaa3f077d081218a: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/af00a839bb5e402791f87288493efef6
I20240503 02:00:03.499573 9473 ts_tablet_manager.cc:1909] T af00a839bb5e402791f87288493efef6 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting consensus metadata
I20240503 02:00:03.499665 9474 ts_tablet_manager.cc:1909] T af00a839bb5e402791f87288493efef6 P dec3e4a04d604f7eaaa3f077d081218a: Deleting consensus metadata
I20240503 02:00:03.499768 6436 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f7ade1e2af24e86b6e54e2b5769e4c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:03 UTC) from {username='slave'} at 127.0.0.1:51338
I20240503 02:00:03.499783 6700 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c13642db7fc40a5a934aa5e3c2b08fc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:03 UTC) from {username='slave'} at 127.0.0.1:40336
I20240503 02:00:03.499927 6567 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f7ade1e2af24e86b6e54e2b5769e4c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:03 UTC) from {username='slave'} at 127.0.0.1:59846
I20240503 02:00:03.500242 11296 tablet_replica.cc:330] T af00a839bb5e402791f87288493efef6 P 99ead02915954b4fab23b033ad6d546b: stopping tablet replica
I20240503 02:00:03.500236 9158 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c13642db7fc40a5a934aa5e3c2b08fc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:03 UTC) from {username='slave'} at 127.0.0.1:51940
I20240503 02:00:03.500294 9474 tablet_replica.cc:330] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P dec3e4a04d604f7eaaa3f077d081218a: stopping tablet replica
I20240503 02:00:03.500409 11296 raft_consensus.cc:2229] T af00a839bb5e402791f87288493efef6 P 99ead02915954b4fab23b033ad6d546b [term 1 LEARNER]: Raft consensus shutting down.
I20240503 02:00:03.500480 10173 tablet_replica.cc:330] T af00a839bb5e402791f87288493efef6 P c2ac2eb7cc264047b9ab32fe3a4fc025: stopping tablet replica
I20240503 02:00:03.500519 11296 raft_consensus.cc:2258] T af00a839bb5e402791f87288493efef6 P 99ead02915954b4fab23b033ad6d546b [term 1 LEARNER]: Raft consensus is shut down!
I20240503 02:00:03.500543 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet af00a839bb5e402791f87288493efef6 (table rebalance_test_table_18 [id=d732c32bc9004748904a109f59db08fd]) successfully deleted
I20240503 02:00:03.500604 10173 raft_consensus.cc:2229] T af00a839bb5e402791f87288493efef6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:03.500741 9473 tablet_replica.cc:330] T 4c13642db7fc40a5a934aa5e3c2b08fc P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 02:00:03.500747 10173 raft_consensus.cc:2258] T af00a839bb5e402791f87288493efef6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:03.500854 9473 raft_consensus.cc:2229] T 4c13642db7fc40a5a934aa5e3c2b08fc P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:03.500918 6300 catalog_manager.cc:4830] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): tablet af00a839bb5e402791f87288493efef6 (table rebalance_test_table_18 [id=d732c32bc9004748904a109f59db08fd]) successfully deleted
I20240503 02:00:03.501011 9473 raft_consensus.cc:2258] T 4c13642db7fc40a5a934aa5e3c2b08fc P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:03.501106 11296 ts_tablet_manager.cc:1875] T af00a839bb5e402791f87288493efef6 P 99ead02915954b4fab23b033ad6d546b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:03.501184 10173 ts_tablet_manager.cc:1875] T af00a839bb5e402791f87288493efef6 P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:03.501597 9473 ts_tablet_manager.cc:1875] T 4c13642db7fc40a5a934aa5e3c2b08fc P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:03.501792 9758 consensus_queue.cc:1035] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:03.503356 11296 ts_tablet_manager.cc:1888] T af00a839bb5e402791f87288493efef6 P 99ead02915954b4fab23b033ad6d546b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:03.503356 10173 ts_tablet_manager.cc:1888] T af00a839bb5e402791f87288493efef6 P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:03.503423 10173 log.cc:1196] T af00a839bb5e402791f87288493efef6 P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/af00a839bb5e402791f87288493efef6
I20240503 02:00:03.503423 11296 log.cc:1196] T af00a839bb5e402791f87288493efef6 P 99ead02915954b4fab23b033ad6d546b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/af00a839bb5e402791f87288493efef6
I20240503 02:00:03.503428 9476 ts_tablet_manager.cc:1875] T 4c13642db7fc40a5a934aa5e3c2b08fc P 492448570cc94ed1879feb2ec3b9051d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:03.503665 10173 ts_tablet_manager.cc:1909] T af00a839bb5e402791f87288493efef6 P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting consensus metadata
I20240503 02:00:03.503978 9286 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f7ade1e2af24e86b6e54e2b5769e4c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:03 UTC) from {username='slave'} at 127.0.0.1:36004
I20240503 02:00:03.504510 11053 tablet_replica.cc:330] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P e1a9251ce1cf4361b8b0ede9fbfc4023: stopping tablet replica
I20240503 02:00:03.504554 9473 ts_tablet_manager.cc:1888] T 4c13642db7fc40a5a934aa5e3c2b08fc P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240503 02:00:03.504623 9473 log.cc:1196] T 4c13642db7fc40a5a934aa5e3c2b08fc P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/4c13642db7fc40a5a934aa5e3c2b08fc
I20240503 02:00:03.504643 11053 raft_consensus.cc:2229] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:03.503670 11296 ts_tablet_manager.cc:1909] T af00a839bb5e402791f87288493efef6 P 99ead02915954b4fab23b033ad6d546b: Deleting consensus metadata
I20240503 02:00:03.504787 11053 raft_consensus.cc:2258] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:03.505102 9473 ts_tablet_manager.cc:1909] T 4c13642db7fc40a5a934aa5e3c2b08fc P 80fd18ef4d604c86b909d5c008a46b4f: Deleting consensus metadata
I20240503 02:00:03.505371 6301 catalog_manager.cc:4830] TS 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157): tablet af00a839bb5e402791f87288493efef6 (table rebalance_test_table_18 [id=d732c32bc9004748904a109f59db08fd]) successfully deleted
I20240503 02:00:03.505563 9476 ts_tablet_manager.cc:1888] T 4c13642db7fc40a5a934aa5e3c2b08fc P 492448570cc94ed1879feb2ec3b9051d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240503 02:00:03.505625 9476 log.cc:1196] T 4c13642db7fc40a5a934aa5e3c2b08fc P 492448570cc94ed1879feb2ec3b9051d: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/4c13642db7fc40a5a934aa5e3c2b08fc
I20240503 02:00:03.505641 11053 ts_tablet_manager.cc:1875] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P e1a9251ce1cf4361b8b0ede9fbfc4023: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:03.505844 9476 ts_tablet_manager.cc:1909] T 4c13642db7fc40a5a934aa5e3c2b08fc P 492448570cc94ed1879feb2ec3b9051d: Deleting consensus metadata
I20240503 02:00:03.505906 9473 tablet_replica.cc:330] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 02:00:03.506034 9473 raft_consensus.cc:2229] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:03.506034 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet 4c13642db7fc40a5a934aa5e3c2b08fc (table rebalance_test_table_18 [id=d732c32bc9004748904a109f59db08fd]) successfully deleted
I20240503 02:00:03.506253 9473 raft_consensus.cc:2258] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:03.506551 10173 tablet_replica.cc:330] T 4c13642db7fc40a5a934aa5e3c2b08fc P c2ac2eb7cc264047b9ab32fe3a4fc025: stopping tablet replica
I20240503 02:00:03.506613 6303 catalog_manager.cc:4830] TS c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655): tablet af00a839bb5e402791f87288493efef6 (table rebalance_test_table_18 [id=d732c32bc9004748904a109f59db08fd]) successfully deleted
I20240503 02:00:03.506673 10173 raft_consensus.cc:2229] T 4c13642db7fc40a5a934aa5e3c2b08fc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:03.506844 10173 raft_consensus.cc:2258] T 4c13642db7fc40a5a934aa5e3c2b08fc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:03.506906 9473 ts_tablet_manager.cc:1875] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:03.507431 6303 catalog_manager.cc:4830] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719): tablet 4c13642db7fc40a5a934aa5e3c2b08fc (table rebalance_test_table_18 [id=d732c32bc9004748904a109f59db08fd]) successfully deleted
I20240503 02:00:03.507566 11053 ts_tablet_manager.cc:1888] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240503 02:00:03.507625 11053 log.cc:1196] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P e1a9251ce1cf4361b8b0ede9fbfc4023: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/9f7ade1e2af24e86b6e54e2b5769e4c0
I20240503 02:00:03.507876 11053 ts_tablet_manager.cc:1909] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P e1a9251ce1cf4361b8b0ede9fbfc4023: Deleting consensus metadata
I20240503 02:00:03.509137 9473 ts_tablet_manager.cc:1888] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240503 02:00:03.509240 9473 log.cc:1196] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/9f7ade1e2af24e86b6e54e2b5769e4c0
I20240503 02:00:03.509790 9473 ts_tablet_manager.cc:1909] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting consensus metadata
I20240503 02:00:03.509963 11073 consensus_queue.cc:1035] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:03.510691 9474 raft_consensus.cc:2229] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:03.511235 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet 9f7ade1e2af24e86b6e54e2b5769e4c0 (table rebalance_test_table_18 [id=d732c32bc9004748904a109f59db08fd]) successfully deleted
I20240503 02:00:03.511433 9474 raft_consensus.cc:2258] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:03.511490 6301 catalog_manager.cc:4830] TS e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): tablet 9f7ade1e2af24e86b6e54e2b5769e4c0 (table rebalance_test_table_18 [id=d732c32bc9004748904a109f59db08fd]) successfully deleted
I20240503 02:00:03.512419 9474 ts_tablet_manager.cc:1875] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P dec3e4a04d604f7eaaa3f077d081218a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:03.516212 9474 ts_tablet_manager.cc:1888] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P dec3e4a04d604f7eaaa3f077d081218a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240503 02:00:03.516412 9474 log.cc:1196] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P dec3e4a04d604f7eaaa3f077d081218a: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/9f7ade1e2af24e86b6e54e2b5769e4c0
I20240503 02:00:03.516745 9474 ts_tablet_manager.cc:1909] T 9f7ade1e2af24e86b6e54e2b5769e4c0 P dec3e4a04d604f7eaaa3f077d081218a: Deleting consensus metadata
I20240503 02:00:03.517874 6300 catalog_manager.cc:4830] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): tablet 9f7ade1e2af24e86b6e54e2b5769e4c0 (table rebalance_test_table_18 [id=d732c32bc9004748904a109f59db08fd]) successfully deleted
I20240503 02:00:03.518505 10173 ts_tablet_manager.cc:1875] T 4c13642db7fc40a5a934aa5e3c2b08fc P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:03.518723 11252 ts_tablet_manager.cc:1875] T 4c13642db7fc40a5a934aa5e3c2b08fc P 2a01ace80baf4057b6bf1995cf88468b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:03.520512 11252 ts_tablet_manager.cc:1888] T 4c13642db7fc40a5a934aa5e3c2b08fc P 2a01ace80baf4057b6bf1995cf88468b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240503 02:00:03.520563 10173 ts_tablet_manager.cc:1888] T 4c13642db7fc40a5a934aa5e3c2b08fc P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240503 02:00:03.520597 11252 log.cc:1196] T 4c13642db7fc40a5a934aa5e3c2b08fc P 2a01ace80baf4057b6bf1995cf88468b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/4c13642db7fc40a5a934aa5e3c2b08fc
I20240503 02:00:03.520638 10173 log.cc:1196] T 4c13642db7fc40a5a934aa5e3c2b08fc P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/4c13642db7fc40a5a934aa5e3c2b08fc
I20240503 02:00:03.520833 10173 ts_tablet_manager.cc:1909] T 4c13642db7fc40a5a934aa5e3c2b08fc P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting consensus metadata
I20240503 02:00:03.520838 11252 ts_tablet_manager.cc:1909] T 4c13642db7fc40a5a934aa5e3c2b08fc P 2a01ace80baf4057b6bf1995cf88468b: Deleting consensus metadata
I20240503 02:00:03.521728 6300 catalog_manager.cc:4830] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079): tablet 4c13642db7fc40a5a934aa5e3c2b08fc (table rebalance_test_table_18 [id=d732c32bc9004748904a109f59db08fd]) successfully deleted
I20240503 02:00:03.521852 6303 catalog_manager.cc:4830] TS c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655): tablet 4c13642db7fc40a5a934aa5e3c2b08fc (table rebalance_test_table_18 [id=d732c32bc9004748904a109f59db08fd]) successfully deleted
W20240503 02:00:03.521927 6300 catalog_manager.cc:4797] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079): delete failed for tablet 4c13642db7fc40a5a934aa5e3c2b08fc because the tablet was not found. No further retry: Not found: Tablet not found: 4c13642db7fc40a5a934aa5e3c2b08fc
I20240503 02:00:03.527487 9310 raft_consensus.cc:1203] T 4385d51d11574192883d01376e2d5620 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:00:03.528095 6315 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:00:03.533522 9286 tablet_service.cc:1467] Processing CreateTablet for tablet b1b5646b12744ac9a57c871622c35dea (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9471894d04b248e9879771dd5d8e4bae]), partition=
I20240503 02:00:03.533749 9158 tablet_service.cc:1467] Processing CreateTablet for tablet b1b5646b12744ac9a57c871622c35dea (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9471894d04b248e9879771dd5d8e4bae]), partition=
I20240503 02:00:03.533798 9286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1b5646b12744ac9a57c871622c35dea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:03.533975 9158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1b5646b12744ac9a57c871622c35dea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:03.535683 11044 tablet_bootstrap.cc:492] T b1b5646b12744ac9a57c871622c35dea P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap starting.
I20240503 02:00:03.536095 8880 tablet_service.cc:1467] Processing CreateTablet for tablet b1b5646b12744ac9a57c871622c35dea (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9471894d04b248e9879771dd5d8e4bae]), partition=
I20240503 02:00:03.536329 8880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1b5646b12744ac9a57c871622c35dea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:03.536638 11044 tablet_bootstrap.cc:654] T b1b5646b12744ac9a57c871622c35dea P e1a9251ce1cf4361b8b0ede9fbfc4023: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:03.538404 11045 tablet_bootstrap.cc:492] T b1b5646b12744ac9a57c871622c35dea P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:03.538657 11044 tablet_bootstrap.cc:492] T b1b5646b12744ac9a57c871622c35dea P e1a9251ce1cf4361b8b0ede9fbfc4023: No bootstrap required, opened a new log
I20240503 02:00:03.538733 11044 ts_tablet_manager.cc:1367] T b1b5646b12744ac9a57c871622c35dea P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:03.535815 11316 tablet_bootstrap.cc:492] T b1b5646b12744ac9a57c871622c35dea P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:03.539216 11044 raft_consensus.cc:348] T b1b5646b12744ac9a57c871622c35dea P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:03.539290 11045 tablet_bootstrap.cc:654] T b1b5646b12744ac9a57c871622c35dea P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:03.539324 11044 raft_consensus.cc:374] T b1b5646b12744ac9a57c871622c35dea P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:03.539353 11044 raft_consensus.cc:729] T b1b5646b12744ac9a57c871622c35dea P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1a9251ce1cf4361b8b0ede9fbfc4023, State: Initialized, Role: FOLLOWER
I20240503 02:00:03.539489 11044 consensus_queue.cc:260] T b1b5646b12744ac9a57c871622c35dea P e1a9251ce1cf4361b8b0ede9fbfc4023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:03.539758 11044 ts_tablet_manager.cc:1398] T b1b5646b12744ac9a57c871622c35dea P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:03.540037 11316 tablet_bootstrap.cc:654] T b1b5646b12744ac9a57c871622c35dea P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:03.540539 11045 tablet_bootstrap.cc:492] T b1b5646b12744ac9a57c871622c35dea P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:03.540628 11045 ts_tablet_manager.cc:1367] T b1b5646b12744ac9a57c871622c35dea P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:00:03.541010 11045 raft_consensus.cc:348] T b1b5646b12744ac9a57c871622c35dea P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:03.541100 11045 raft_consensus.cc:374] T b1b5646b12744ac9a57c871622c35dea P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:03.541127 11045 raft_consensus.cc:729] T b1b5646b12744ac9a57c871622c35dea P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:03.541250 11045 consensus_queue.cc:260] T b1b5646b12744ac9a57c871622c35dea P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:03.541527 11045 ts_tablet_manager.cc:1398] T b1b5646b12744ac9a57c871622c35dea P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:03.543195 11316 tablet_bootstrap.cc:492] T b1b5646b12744ac9a57c871622c35dea P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:03.543355 11316 ts_tablet_manager.cc:1367] T b1b5646b12744ac9a57c871622c35dea P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20240503 02:00:03.545987 11316 raft_consensus.cc:348] T b1b5646b12744ac9a57c871622c35dea P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:03.546478 11316 raft_consensus.cc:374] T b1b5646b12744ac9a57c871622c35dea P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:03.546572 11316 raft_consensus.cc:729] T b1b5646b12744ac9a57c871622c35dea P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:03.546767 11316 consensus_queue.cc:260] T b1b5646b12744ac9a57c871622c35dea P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:03.547066 11316 ts_tablet_manager.cc:1398] T b1b5646b12744ac9a57c871622c35dea P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240503 02:00:03.570695 11340 ts_tablet_manager.cc:902] T cb873018448549ccbe3d14b784e47d26 P 492448570cc94ed1879feb2ec3b9051d: Initiating tablet copy from peer 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:03.571475 11340 tablet_copy_client.cc:308] T cb873018448549ccbe3d14b784e47d26 P 492448570cc94ed1879feb2ec3b9051d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43863
I20240503 02:00:03.572147 6609 tablet_copy_service.cc:140] P 80fd18ef4d604c86b909d5c008a46b4f: Received BeginTabletCopySession request for tablet cb873018448549ccbe3d14b784e47d26 from peer 492448570cc94ed1879feb2ec3b9051d ({username='slave'} at 127.0.99.132:33379)
I20240503 02:00:03.572273 6609 tablet_copy_service.cc:161] P 80fd18ef4d604c86b909d5c008a46b4f: Beginning new tablet copy session on tablet cb873018448549ccbe3d14b784e47d26 from peer 492448570cc94ed1879feb2ec3b9051d at {username='slave'} at 127.0.99.132:33379: session id = 492448570cc94ed1879feb2ec3b9051d-cb873018448549ccbe3d14b784e47d26
I20240503 02:00:03.573305 6609 tablet_copy_source_session.cc:205] T cb873018448549ccbe3d14b784e47d26 P 80fd18ef4d604c86b909d5c008a46b4f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:03.575201 11340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb873018448549ccbe3d14b784e47d26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:03.578826 11340 tablet_copy_client.cc:783] T cb873018448549ccbe3d14b784e47d26 P 492448570cc94ed1879feb2ec3b9051d: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:03.579157 11340 tablet_copy_client.cc:647] T cb873018448549ccbe3d14b784e47d26 P 492448570cc94ed1879feb2ec3b9051d: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:03.580237 11350 ts_tablet_manager.cc:902] T 3b189fba0d6c461694dda002a3c5eb2f P 9942306855404cfd891adf227dd65eb2: Initiating tablet copy from peer dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 02:00:03.580785 11350 tablet_copy_client.cc:308] T 3b189fba0d6c461694dda002a3c5eb2f P 9942306855404cfd891adf227dd65eb2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:33093
I20240503 02:00:03.580927 11340 tablet_copy_client.cc:523] T cb873018448549ccbe3d14b784e47d26 P 492448570cc94ed1879feb2ec3b9051d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:03.581255 6478 tablet_copy_service.cc:140] P dec3e4a04d604f7eaaa3f077d081218a: Received BeginTabletCopySession request for tablet 3b189fba0d6c461694dda002a3c5eb2f from peer 9942306855404cfd891adf227dd65eb2 ({username='slave'} at 127.0.99.133:57637)
I20240503 02:00:03.581354 6478 tablet_copy_service.cc:161] P dec3e4a04d604f7eaaa3f077d081218a: Beginning new tablet copy session on tablet 3b189fba0d6c461694dda002a3c5eb2f from peer 9942306855404cfd891adf227dd65eb2 at {username='slave'} at 127.0.99.133:57637: session id = 9942306855404cfd891adf227dd65eb2-3b189fba0d6c461694dda002a3c5eb2f
I20240503 02:00:03.582295 6478 tablet_copy_source_session.cc:205] T 3b189fba0d6c461694dda002a3c5eb2f P dec3e4a04d604f7eaaa3f077d081218a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:03.582746 11340 tablet_bootstrap.cc:492] T cb873018448549ccbe3d14b784e47d26 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:03.583189 11350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b189fba0d6c461694dda002a3c5eb2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:03.585623 11350 tablet_copy_client.cc:783] T 3b189fba0d6c461694dda002a3c5eb2f P 9942306855404cfd891adf227dd65eb2: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:03.585827 8916 consensus_queue.cc:237] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: true } }
I20240503 02:00:03.585928 11350 tablet_copy_client.cc:647] T 3b189fba0d6c461694dda002a3c5eb2f P 9942306855404cfd891adf227dd65eb2: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:03.587608 9039 raft_consensus.cc:1261] T 96c395771fd44056abacceb6c1acdb9f P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:03.587749 11340 tablet_bootstrap.cc:492] T cb873018448549ccbe3d14b784e47d26 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:03.587813 11350 tablet_copy_client.cc:523] T 3b189fba0d6c461694dda002a3c5eb2f P 9942306855404cfd891adf227dd65eb2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:03.588030 11340 tablet_bootstrap.cc:492] T cb873018448549ccbe3d14b784e47d26 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap complete.
W20240503 02:00:03.588145 8850 consensus_peers.cc:485] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2 -> Peer dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): Couldn't send request to peer dec3e4a04d604f7eaaa3f077d081218a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 96c395771fd44056abacceb6c1acdb9f. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:03.588158 9178 raft_consensus.cc:1261] T 96c395771fd44056abacceb6c1acdb9f P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:03.588204 11340 ts_tablet_manager.cc:1367] T cb873018448549ccbe3d14b784e47d26 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240503 02:00:03.588744 11340 raft_consensus.cc:348] T cb873018448549ccbe3d14b784e47d26 P 492448570cc94ed1879feb2ec3b9051d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { promote: true } }
I20240503 02:00:03.588804 11357 consensus_queue.cc:1035] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:03.588858 11340 raft_consensus.cc:729] T cb873018448549ccbe3d14b784e47d26 P 492448570cc94ed1879feb2ec3b9051d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: LEARNER
I20240503 02:00:03.588960 11340 consensus_queue.cc:260] T cb873018448549ccbe3d14b784e47d26 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { promote: true } }
I20240503 02:00:03.589236 11340 ts_tablet_manager.cc:1398] T cb873018448549ccbe3d14b784e47d26 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 02:00:03.589253 11350 tablet_bootstrap.cc:492] T 3b189fba0d6c461694dda002a3c5eb2f P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:03.590056 6609 tablet_copy_service.cc:342] P 80fd18ef4d604c86b909d5c008a46b4f: Request end of tablet copy session 492448570cc94ed1879feb2ec3b9051d-cb873018448549ccbe3d14b784e47d26 received from {username='slave'} at 127.0.99.132:33379
I20240503 02:00:03.590148 6609 tablet_copy_service.cc:433] P 80fd18ef4d604c86b909d5c008a46b4f: ending tablet copy session 492448570cc94ed1879feb2ec3b9051d-cb873018448549ccbe3d14b784e47d26 on tablet cb873018448549ccbe3d14b784e47d26 with peer 492448570cc94ed1879feb2ec3b9051d
I20240503 02:00:03.591502 9758 raft_consensus.cc:2932] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: true } } }
I20240503 02:00:03.593045 9758 consensus_queue.cc:1035] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:03.594399 9039 raft_consensus.cc:2932] T 96c395771fd44056abacceb6c1acdb9f P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: true } } }
I20240503 02:00:03.595705 9178 raft_consensus.cc:2932] T 96c395771fd44056abacceb6c1acdb9f P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: true } } }
I20240503 02:00:03.596915 6315 catalog_manager.cc:5484] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2 reported cstate change: config changed from index -1 to 2, NON_VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "9942306855404cfd891adf227dd65eb2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:03.603639 11357 raft_consensus.cc:1050] T f28586e70bd9473fbdc6b9126b666fa9 P 9942306855404cfd891adf227dd65eb2: attempting to promote NON_VOTER 80fd18ef4d604c86b909d5c008a46b4f to VOTER
I20240503 02:00:03.604141 11357 consensus_queue.cc:237] T f28586e70bd9473fbdc6b9126b666fa9 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } }
I20240503 02:00:03.605716 9178 raft_consensus.cc:1261] T f28586e70bd9473fbdc6b9126b666fa9 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:03.605945 6582 raft_consensus.cc:1261] T f28586e70bd9473fbdc6b9126b666fa9 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEARNER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:03.606143 9758 consensus_queue.cc:1035] T f28586e70bd9473fbdc6b9126b666fa9 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:03.606333 9310 raft_consensus.cc:1261] T f28586e70bd9473fbdc6b9126b666fa9 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:03.606844 11359 raft_consensus.cc:482] T b1b5646b12744ac9a57c871622c35dea P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:03.606899 11357 consensus_queue.cc:1035] T f28586e70bd9473fbdc6b9126b666fa9 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:03.606937 11359 raft_consensus.cc:504] T b1b5646b12744ac9a57c871622c35dea P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:03.607213 11357 consensus_queue.cc:1035] T f28586e70bd9473fbdc6b9126b666fa9 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:03.607748 9309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1b5646b12744ac9a57c871622c35dea" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" is_pre_election: true
I20240503 02:00:03.607860 9309 raft_consensus.cc:2454] T b1b5646b12744ac9a57c871622c35dea P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 0.
I20240503 02:00:03.608906 8851 leader_election.cc:302] T b1b5646b12744ac9a57c871622c35dea P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9942306855404cfd891adf227dd65eb2, e1a9251ce1cf4361b8b0ede9fbfc4023; no voters:
I20240503 02:00:03.609056 11357 raft_consensus.cc:2781] T b1b5646b12744ac9a57c871622c35dea P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:03.609230 11357 raft_consensus.cc:482] T b1b5646b12744ac9a57c871622c35dea P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:03.609323 11357 raft_consensus.cc:3037] T b1b5646b12744ac9a57c871622c35dea P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:03.609297 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1b5646b12744ac9a57c871622c35dea" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:03.609444 9179 raft_consensus.cc:2454] T b1b5646b12744ac9a57c871622c35dea P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 0.
I20240503 02:00:03.607256 11359 leader_election.cc:288] T b1b5646b12744ac9a57c871622c35dea P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:03.611460 11357 raft_consensus.cc:504] T b1b5646b12744ac9a57c871622c35dea P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:03.611557 11359 raft_consensus.cc:2932] T f28586e70bd9473fbdc6b9126b666fa9 P 9942306855404cfd891adf227dd65eb2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:03.611837 11357 leader_election.cc:288] T b1b5646b12744ac9a57c871622c35dea P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 election: Requested vote from peers e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:03.612087 9310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1b5646b12744ac9a57c871622c35dea" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023"
I20240503 02:00:03.612247 9310 raft_consensus.cc:3037] T b1b5646b12744ac9a57c871622c35dea P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:03.613006 9310 raft_consensus.cc:2454] T b1b5646b12744ac9a57c871622c35dea P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 1.
I20240503 02:00:03.613679 9178 raft_consensus.cc:2932] T f28586e70bd9473fbdc6b9126b666fa9 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:03.614655 6315 catalog_manager.cc:5484] T f28586e70bd9473fbdc6b9126b666fa9 P 9942306855404cfd891adf227dd65eb2 reported cstate change: config changed from index 2 to 3, 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9942306855404cfd891adf227dd65eb2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:03.615900 8851 leader_election.cc:302] T b1b5646b12744ac9a57c871622c35dea P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9942306855404cfd891adf227dd65eb2, e1a9251ce1cf4361b8b0ede9fbfc4023; no voters:
I20240503 02:00:03.617976 9758 raft_consensus.cc:2781] T b1b5646b12744ac9a57c871622c35dea P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:03.618088 9758 raft_consensus.cc:686] T b1b5646b12744ac9a57c871622c35dea P 9942306855404cfd891adf227dd65eb2 [term 1 LEADER]: Becoming Leader. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Running, Role: LEADER
I20240503 02:00:03.618225 9758 consensus_queue.cc:237] T b1b5646b12744ac9a57c871622c35dea P 9942306855404cfd891adf227dd65eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:03.619995 9310 raft_consensus.cc:2932] T f28586e70bd9473fbdc6b9126b666fa9 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:03.622254 6312 catalog_manager.cc:5484] T b1b5646b12744ac9a57c871622c35dea P 9942306855404cfd891adf227dd65eb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9942306855404cfd891adf227dd65eb2 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "9942306855404cfd891adf227dd65eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:03.622773 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1b5646b12744ac9a57c871622c35dea" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:03.622898 9178 raft_consensus.cc:3037] T b1b5646b12744ac9a57c871622c35dea P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:03.623456 11350 tablet_bootstrap.cc:492] T 3b189fba0d6c461694dda002a3c5eb2f P 9942306855404cfd891adf227dd65eb2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:03.623919 11350 tablet_bootstrap.cc:492] T 3b189fba0d6c461694dda002a3c5eb2f P 9942306855404cfd891adf227dd65eb2: Bootstrap complete.
I20240503 02:00:03.623916 9178 raft_consensus.cc:2454] T b1b5646b12744ac9a57c871622c35dea P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 1.
I20240503 02:00:03.624167 11350 ts_tablet_manager.cc:1367] T 3b189fba0d6c461694dda002a3c5eb2f P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.035s user 0.026s sys 0.000s
I20240503 02:00:03.624701 11350 raft_consensus.cc:348] T 3b189fba0d6c461694dda002a3c5eb2f P 9942306855404cfd891adf227dd65eb2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } }
I20240503 02:00:03.624904 11350 raft_consensus.cc:729] T 3b189fba0d6c461694dda002a3c5eb2f P 9942306855404cfd891adf227dd65eb2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: LEARNER
I20240503 02:00:03.625090 11350 consensus_queue.cc:260] T 3b189fba0d6c461694dda002a3c5eb2f P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } }
I20240503 02:00:03.625836 11350 ts_tablet_manager.cc:1398] T 3b189fba0d6c461694dda002a3c5eb2f P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:03.625929 6582 raft_consensus.cc:2932] T f28586e70bd9473fbdc6b9126b666fa9 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:03.626477 11058 raft_consensus.cc:1050] T 90ef3584830240acb0d8aad14ae851a6 P 492448570cc94ed1879feb2ec3b9051d: attempting to promote NON_VOTER 99ead02915954b4fab23b033ad6d546b to VOTER
I20240503 02:00:03.626796 6478 tablet_copy_service.cc:342] P dec3e4a04d604f7eaaa3f077d081218a: Request end of tablet copy session 9942306855404cfd891adf227dd65eb2-3b189fba0d6c461694dda002a3c5eb2f received from {username='slave'} at 127.0.99.133:57637
I20240503 02:00:03.626875 6478 tablet_copy_service.cc:433] P dec3e4a04d604f7eaaa3f077d081218a: ending tablet copy session 9942306855404cfd891adf227dd65eb2-3b189fba0d6c461694dda002a3c5eb2f on tablet 3b189fba0d6c461694dda002a3c5eb2f with peer 9942306855404cfd891adf227dd65eb2
I20240503 02:00:03.626863 11058 consensus_queue.cc:237] T 90ef3584830240acb0d8aad14ae851a6 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } }
I20240503 02:00:03.628481 6719 raft_consensus.cc:1261] T 90ef3584830240acb0d8aad14ae851a6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:03.628535 9178 raft_consensus.cc:1261] T 90ef3584830240acb0d8aad14ae851a6 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:03.628983 11377 consensus_queue.cc:1035] T 90ef3584830240acb0d8aad14ae851a6 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:03.629489 9039 raft_consensus.cc:1261] T 90ef3584830240acb0d8aad14ae851a6 P 99ead02915954b4fab23b033ad6d546b [term 1 LEARNER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:03.629851 11377 consensus_queue.cc:1035] T 90ef3584830240acb0d8aad14ae851a6 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:03.630230 11377 consensus_queue.cc:1035] T 90ef3584830240acb0d8aad14ae851a6 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:03.631613 8916 consensus_queue.cc:237] T f28586e70bd9473fbdc6b9126b666fa9 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } }
I20240503 02:00:03.632925 6582 raft_consensus.cc:1261] T f28586e70bd9473fbdc6b9126b666fa9 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:03.633281 11373 raft_consensus.cc:2932] T 90ef3584830240acb0d8aad14ae851a6 P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 99ead02915954b4fab23b033ad6d546b (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:03.633590 9758 consensus_queue.cc:1035] T f28586e70bd9473fbdc6b9126b666fa9 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:03.633795 9310 raft_consensus.cc:1261] T f28586e70bd9473fbdc6b9126b666fa9 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:03.633759 6719 raft_consensus.cc:2932] T 90ef3584830240acb0d8aad14ae851a6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 99ead02915954b4fab23b033ad6d546b (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:03.634171 9758 consensus_queue.cc:1035] T f28586e70bd9473fbdc6b9126b666fa9 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:03.635243 9758 raft_consensus.cc:2932] T f28586e70bd9473fbdc6b9126b666fa9 P 9942306855404cfd891adf227dd65eb2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:03.635722 6582 raft_consensus.cc:2932] T f28586e70bd9473fbdc6b9126b666fa9 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:03.636693 6303 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f28586e70bd9473fbdc6b9126b666fa9 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:03.636662 9039 raft_consensus.cc:2932] T 90ef3584830240acb0d8aad14ae851a6 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 99ead02915954b4fab23b033ad6d546b (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:03.638512 6315 catalog_manager.cc:5484] T 90ef3584830240acb0d8aad14ae851a6 P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: config changed from index 7 to 8, 99ead02915954b4fab23b033ad6d546b (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:03.638844 6316 catalog_manager.cc:5484] T f28586e70bd9473fbdc6b9126b666fa9 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: config changed from index 3 to 4, VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "9942306855404cfd891adf227dd65eb2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:03.642020 9158 tablet_service.cc:1514] Processing DeleteTablet for tablet f28586e70bd9473fbdc6b9126b666fa9 with delete_type TABLET_DATA_TOMBSTONED (TS 2a01ace80baf4057b6bf1995cf88468b not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:51940
I20240503 02:00:03.642294 11252 tablet_replica.cc:330] T f28586e70bd9473fbdc6b9126b666fa9 P 2a01ace80baf4057b6bf1995cf88468b: stopping tablet replica
I20240503 02:00:03.642436 11252 raft_consensus.cc:2229] T f28586e70bd9473fbdc6b9126b666fa9 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:03.642566 11252 raft_consensus.cc:2258] T f28586e70bd9473fbdc6b9126b666fa9 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:03.644784 9310 raft_consensus.cc:2932] T f28586e70bd9473fbdc6b9126b666fa9 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:03.645198 8833 raft_consensus.cc:1050] T c3485701bf3b48f5ae2e1568afeed133 P 80fd18ef4d604c86b909d5c008a46b4f: attempting to promote NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 to VOTER
I20240503 02:00:03.645596 8833 consensus_queue.cc:237] T c3485701bf3b48f5ae2e1568afeed133 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } }
I20240503 02:00:03.646997 9310 raft_consensus.cc:1261] T c3485701bf3b48f5ae2e1568afeed133 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 LEARNER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:03.647051 11252 ts_tablet_manager.cc:1875] T f28586e70bd9473fbdc6b9126b666fa9 P 2a01ace80baf4057b6bf1995cf88468b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:03.647446 10278 consensus_queue.cc:1035] T c3485701bf3b48f5ae2e1568afeed133 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:03.649389 11252 ts_tablet_manager.cc:1888] T f28586e70bd9473fbdc6b9126b666fa9 P 2a01ace80baf4057b6bf1995cf88468b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:00:03.649470 11252 log.cc:1196] T f28586e70bd9473fbdc6b9126b666fa9 P 2a01ace80baf4057b6bf1995cf88468b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/f28586e70bd9473fbdc6b9126b666fa9
I20240503 02:00:03.650269 6300 catalog_manager.cc:4830] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079): tablet f28586e70bd9473fbdc6b9126b666fa9 (table rebalancer_extra_table_4 [id=c1969b96b61041869ddd81db8c84bf44]) successfully deleted
I20240503 02:00:03.650524 6719 raft_consensus.cc:1261] T c3485701bf3b48f5ae2e1568afeed133 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:03.650660 6850 raft_consensus.cc:1261] T c3485701bf3b48f5ae2e1568afeed133 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:03.650981 10278 consensus_queue.cc:1035] T c3485701bf3b48f5ae2e1568afeed133 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:03.651189 9484 consensus_queue.cc:1035] T c3485701bf3b48f5ae2e1568afeed133 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:03.651742 6850 consensus_queue.cc:237] T 90ef3584830240acb0d8aad14ae851a6 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } }
I20240503 02:00:03.653050 9178 raft_consensus.cc:1261] T 90ef3584830240acb0d8aad14ae851a6 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:03.653033 9484 raft_consensus.cc:2932] T c3485701bf3b48f5ae2e1568afeed133 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:03.653692 11058 consensus_queue.cc:1035] T 90ef3584830240acb0d8aad14ae851a6 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:03.654445 9178 raft_consensus.cc:2932] T 90ef3584830240acb0d8aad14ae851a6 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 99ead02915954b4fab23b033ad6d546b (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:03.655047 6849 raft_consensus.cc:2932] T c3485701bf3b48f5ae2e1568afeed133 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:03.655982 9310 raft_consensus.cc:2932] T c3485701bf3b48f5ae2e1568afeed133 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:03.658044 9039 raft_consensus.cc:1261] T 90ef3584830240acb0d8aad14ae851a6 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:03.657979 6312 catalog_manager.cc:5484] T c3485701bf3b48f5ae2e1568afeed133 P 492448570cc94ed1879feb2ec3b9051d reported cstate change: config changed from index 4 to 5, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:03.658754 11058 consensus_queue.cc:1035] T 90ef3584830240acb0d8aad14ae851a6 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:03.661638 11376 raft_consensus.cc:2932] T 90ef3584830240acb0d8aad14ae851a6 P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:03.661942 9179 raft_consensus.cc:2932] T 90ef3584830240acb0d8aad14ae851a6 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:03.663916 6303 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 90ef3584830240acb0d8aad14ae851a6 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:03.664609 6311 catalog_manager.cc:5484] T 90ef3584830240acb0d8aad14ae851a6 P 2a01ace80baf4057b6bf1995cf88468b reported cstate change: config changed from index 8 to 9, VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:03.664747 9039 raft_consensus.cc:2932] T 90ef3584830240acb0d8aad14ae851a6 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:03.667199 6700 tablet_service.cc:1514] Processing DeleteTablet for tablet 90ef3584830240acb0d8aad14ae851a6 with delete_type TABLET_DATA_TOMBSTONED (TS c2ac2eb7cc264047b9ab32fe3a4fc025 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:40336
I20240503 02:00:03.667450 10173 tablet_replica.cc:330] T 90ef3584830240acb0d8aad14ae851a6 P c2ac2eb7cc264047b9ab32fe3a4fc025: stopping tablet replica
I20240503 02:00:03.667793 10173 raft_consensus.cc:2229] T 90ef3584830240acb0d8aad14ae851a6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:03.668035 10173 raft_consensus.cc:2258] T 90ef3584830240acb0d8aad14ae851a6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:03.669029 10173 ts_tablet_manager.cc:1875] T 90ef3584830240acb0d8aad14ae851a6 P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:03.668970 6719 raft_consensus.cc:2932] T c3485701bf3b48f5ae2e1568afeed133 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:03.671485 10173 ts_tablet_manager.cc:1888] T 90ef3584830240acb0d8aad14ae851a6 P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 02:00:03.671572 10173 log.cc:1196] T 90ef3584830240acb0d8aad14ae851a6 P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/90ef3584830240acb0d8aad14ae851a6
I20240503 02:00:03.674000 6303 catalog_manager.cc:4830] TS c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655): tablet 90ef3584830240acb0d8aad14ae851a6 (table rebalance_test_table_5 [id=9b77b848294749e09f0854d24da5d9f5]) successfully deleted
I20240503 02:00:03.675083 11396 ts_tablet_manager.cc:902] T f5ed01740019467d8c1d55cc2fa18ca3 P 80fd18ef4d604c86b909d5c008a46b4f: Initiating tablet copy from peer 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:03.676041 11396 tablet_copy_client.cc:308] T f5ed01740019467d8c1d55cc2fa18ca3 P 80fd18ef4d604c86b909d5c008a46b4f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:34079
I20240503 02:00:03.676753 9199 tablet_copy_service.cc:140] P 2a01ace80baf4057b6bf1995cf88468b: Received BeginTabletCopySession request for tablet f5ed01740019467d8c1d55cc2fa18ca3 from peer 80fd18ef4d604c86b909d5c008a46b4f ({username='slave'} at 127.0.99.130:60621)
I20240503 02:00:03.676894 9199 tablet_copy_service.cc:161] P 2a01ace80baf4057b6bf1995cf88468b: Beginning new tablet copy session on tablet f5ed01740019467d8c1d55cc2fa18ca3 from peer 80fd18ef4d604c86b909d5c008a46b4f at {username='slave'} at 127.0.99.130:60621: session id = 80fd18ef4d604c86b909d5c008a46b4f-f5ed01740019467d8c1d55cc2fa18ca3
I20240503 02:00:03.677629 8833 consensus_queue.cc:1035] T 01d0ff19ac53453888570a51948235e8 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:03.678105 9199 tablet_copy_source_session.cc:205] T f5ed01740019467d8c1d55cc2fa18ca3 P 2a01ace80baf4057b6bf1995cf88468b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:03.679664 11396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5ed01740019467d8c1d55cc2fa18ca3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:03.681871 9484 consensus_queue.cc:1035] T 01d0ff19ac53453888570a51948235e8 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:03.682073 11396 tablet_copy_client.cc:783] T f5ed01740019467d8c1d55cc2fa18ca3 P 80fd18ef4d604c86b909d5c008a46b4f: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:03.682221 11396 tablet_copy_client.cc:647] T f5ed01740019467d8c1d55cc2fa18ca3 P 80fd18ef4d604c86b909d5c008a46b4f: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:03.685359 11396 tablet_copy_client.cc:523] T f5ed01740019467d8c1d55cc2fa18ca3 P 80fd18ef4d604c86b909d5c008a46b4f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:03.686873 11396 tablet_bootstrap.cc:492] T f5ed01740019467d8c1d55cc2fa18ca3 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:03.690140 11396 tablet_bootstrap.cc:492] T f5ed01740019467d8c1d55cc2fa18ca3 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:03.690366 11396 tablet_bootstrap.cc:492] T f5ed01740019467d8c1d55cc2fa18ca3 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap complete.
I20240503 02:00:03.690536 11396 ts_tablet_manager.cc:1367] T f5ed01740019467d8c1d55cc2fa18ca3 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:00:03.690948 11396 raft_consensus.cc:348] T f5ed01740019467d8c1d55cc2fa18ca3 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: true } }
I20240503 02:00:03.691058 11396 raft_consensus.cc:729] T f5ed01740019467d8c1d55cc2fa18ca3 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: LEARNER
I20240503 02:00:03.691152 11396 consensus_queue.cc:260] T f5ed01740019467d8c1d55cc2fa18ca3 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: true } }
I20240503 02:00:03.691399 11396 ts_tablet_manager.cc:1398] T f5ed01740019467d8c1d55cc2fa18ca3 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:03.692409 9199 tablet_copy_service.cc:342] P 2a01ace80baf4057b6bf1995cf88468b: Request end of tablet copy session 80fd18ef4d604c86b909d5c008a46b4f-f5ed01740019467d8c1d55cc2fa18ca3 received from {username='slave'} at 127.0.99.130:60621
I20240503 02:00:03.692605 9199 tablet_copy_service.cc:433] P 2a01ace80baf4057b6bf1995cf88468b: ending tablet copy session 80fd18ef4d604c86b909d5c008a46b4f-f5ed01740019467d8c1d55cc2fa18ca3 on tablet f5ed01740019467d8c1d55cc2fa18ca3 with peer 80fd18ef4d604c86b909d5c008a46b4f
I20240503 02:00:03.709439 9039 raft_consensus.cc:1203] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 99ead02915954b4fab23b033ad6d546b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:00:03.715145 9984 raft_consensus.cc:1050] T 3cf186a0901f420390c8fda54316c5af P c2ac2eb7cc264047b9ab32fe3a4fc025: attempting to promote NON_VOTER 2a01ace80baf4057b6bf1995cf88468b to VOTER
I20240503 02:00:03.715626 9984 consensus_queue.cc:237] T 3cf186a0901f420390c8fda54316c5af P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } }
I20240503 02:00:03.717185 9039 raft_consensus.cc:1261] T 3cf186a0901f420390c8fda54316c5af P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:03.717309 9179 raft_consensus.cc:1261] T 3cf186a0901f420390c8fda54316c5af P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEARNER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:03.717857 10268 consensus_queue.cc:1035] T 3cf186a0901f420390c8fda54316c5af P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:03.717896 6850 raft_consensus.cc:1261] T 3cf186a0901f420390c8fda54316c5af P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:03.718634 10268 consensus_queue.cc:1035] T 3cf186a0901f420390c8fda54316c5af P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:03.719512 10268 consensus_queue.cc:1035] T 3cf186a0901f420390c8fda54316c5af P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:03.721261 10347 raft_consensus.cc:2932] T 3cf186a0901f420390c8fda54316c5af P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:03.721827 6850 raft_consensus.cc:2932] T 3cf186a0901f420390c8fda54316c5af P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:03.724623 6309 catalog_manager.cc:5484] T 3cf186a0901f420390c8fda54316c5af P 492448570cc94ed1879feb2ec3b9051d reported cstate change: config changed from index 7 to 8, 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:03.726204 9179 raft_consensus.cc:2932] T 3cf186a0901f420390c8fda54316c5af P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:03.730211 9039 raft_consensus.cc:2932] T 3cf186a0901f420390c8fda54316c5af P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:03.731676 6719 consensus_queue.cc:237] T 3cf186a0901f420390c8fda54316c5af P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } }
I20240503 02:00:03.733278 9179 raft_consensus.cc:1261] T 3cf186a0901f420390c8fda54316c5af P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:03.734481 6314 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:03.734601 9984 consensus_queue.cc:1035] T 3cf186a0901f420390c8fda54316c5af P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:03.735719 9039 raft_consensus.cc:1261] T 3cf186a0901f420390c8fda54316c5af P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:03.736142 9984 consensus_queue.cc:1035] T 3cf186a0901f420390c8fda54316c5af P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:03.736573 9984 raft_consensus.cc:2932] T 3cf186a0901f420390c8fda54316c5af P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:03.738353 6303 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3cf186a0901f420390c8fda54316c5af with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:03.739030 9178 raft_consensus.cc:1261] T b1b5646b12744ac9a57c871622c35dea P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:03.739183 9310 raft_consensus.cc:1261] T b1b5646b12744ac9a57c871622c35dea P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:03.739722 11357 consensus_queue.cc:1035] T b1b5646b12744ac9a57c871622c35dea P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:03.739992 11357 consensus_queue.cc:1035] T b1b5646b12744ac9a57c871622c35dea P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:03.739919 6307 catalog_manager.cc:5484] T 3cf186a0901f420390c8fda54316c5af P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: config changed from index 8 to 9, VOTER 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:03.740906 9179 raft_consensus.cc:2932] T 3cf186a0901f420390c8fda54316c5af P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:03.743196 6831 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cf186a0901f420390c8fda54316c5af with delete_type TABLET_DATA_TOMBSTONED (TS 492448570cc94ed1879feb2ec3b9051d not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:52812
I20240503 02:00:03.743448 9476 tablet_replica.cc:330] T 3cf186a0901f420390c8fda54316c5af P 492448570cc94ed1879feb2ec3b9051d: stopping tablet replica
I20240503 02:00:03.743602 9476 raft_consensus.cc:2229] T 3cf186a0901f420390c8fda54316c5af P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:03.743738 6314 catalog_manager.cc:5576] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079) does not have the latest schema for tablet b1b5646b12744ac9a57c871622c35dea (table rebalancer_dynamic_table [id=9471894d04b248e9879771dd5d8e4bae]). Expected version 1 got 0
I20240503 02:00:03.743844 9476 raft_consensus.cc:2258] T 3cf186a0901f420390c8fda54316c5af P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:03.744808 9039 raft_consensus.cc:2932] T 3cf186a0901f420390c8fda54316c5af P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:03.745779 6312 catalog_manager.cc:5576] TS e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311) does not have the latest schema for tablet b1b5646b12744ac9a57c871622c35dea (table rebalancer_dynamic_table [id=9471894d04b248e9879771dd5d8e4bae]). Expected version 1 got 0
I20240503 02:00:03.746676 9476 ts_tablet_manager.cc:1875] T 3cf186a0901f420390c8fda54316c5af P 492448570cc94ed1879feb2ec3b9051d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:03.749225 9476 ts_tablet_manager.cc:1888] T 3cf186a0901f420390c8fda54316c5af P 492448570cc94ed1879feb2ec3b9051d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 02:00:03.749332 9476 log.cc:1196] T 3cf186a0901f420390c8fda54316c5af P 492448570cc94ed1879feb2ec3b9051d: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/3cf186a0901f420390c8fda54316c5af
I20240503 02:00:03.749944 6303 catalog_manager.cc:4830] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719): tablet 3cf186a0901f420390c8fda54316c5af (table rebalance_test_table_11 [id=805c97ce87c74e019deaa2acee6a5329]) successfully deleted
I20240503 02:00:03.753545 11062 tablet.cc:1722] T b1b5646b12744ac9a57c871622c35dea P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:03.753887 11062 tablet.cc:1620] T b1b5646b12744ac9a57c871622c35dea P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
I20240503 02:00:03.755455 11437 tablet.cc:1722] T b1b5646b12744ac9a57c871622c35dea P e1a9251ce1cf4361b8b0ede9fbfc4023: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:03.756498 11437 tablet.cc:1620] T b1b5646b12744ac9a57c871622c35dea P e1a9251ce1cf4361b8b0ede9fbfc4023: MemRowSet was empty: no flush needed.
I20240503 02:00:03.756981 11064 tablet.cc:1722] T b1b5646b12744ac9a57c871622c35dea P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:03.757268 11064 tablet.cc:1620] T b1b5646b12744ac9a57c871622c35dea P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
I20240503 02:00:03.757544 6314 catalog_manager.cc:5576] TS e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311) does not have the latest schema for tablet b1b5646b12744ac9a57c871622c35dea (table rebalancer_dynamic_table [id=9471894d04b248e9879771dd5d8e4bae]). Expected version 1 got 0
I20240503 02:00:03.759236 6309 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=9471894d04b248e9879771dd5d8e4bae] alter complete (version 1)
I20240503 02:00:03.769011 9158 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:00:03.778082 6436 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:00:03.782141 9286 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:03.783267 9484 raft_consensus.cc:1050] T 1e445a3cc928429caf9e3c6193a63010 P 80fd18ef4d604c86b909d5c008a46b4f: attempting to promote NON_VOTER 9942306855404cfd891adf227dd65eb2 to VOTER
I20240503 02:00:03.783736 9484 consensus_queue.cc:237] T 1e445a3cc928429caf9e3c6193a63010 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } }
I20240503 02:00:03.783833 6567 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 02:00:03.786695 9179 raft_consensus.cc:1261] T 1e445a3cc928429caf9e3c6193a63010 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:03.786844 8916 raft_consensus.cc:1261] T 1e445a3cc928429caf9e3c6193a63010 P 9942306855404cfd891adf227dd65eb2 [term 1 LEARNER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:03.787834 10278 consensus_queue.cc:1035] T 1e445a3cc928429caf9e3c6193a63010 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:03.788019 8833 consensus_queue.cc:1035] T 1e445a3cc928429caf9e3c6193a63010 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:03.789881 6831 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:03.794404 6458 raft_consensus.cc:1261] T 1e445a3cc928429caf9e3c6193a63010 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:03.794867 10278 consensus_queue.cc:1035] T 1e445a3cc928429caf9e3c6193a63010 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:03.795789 8880 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:00:03.799561 6700 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:00:03.803911 9955 raft_consensus.cc:1050] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a: attempting to promote NON_VOTER 9942306855404cfd891adf227dd65eb2 to VOTER
I20240503 02:00:03.804360 9955 consensus_queue.cc:237] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } }
I20240503 02:00:03.806399 9310 raft_consensus.cc:1261] T 46ad540deb624a25b97d3b967fe1f6f4 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:03.806828 10381 consensus_queue.cc:1035] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:03.808473 6850 raft_consensus.cc:1261] T 46ad540deb624a25b97d3b967fe1f6f4 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:03.809057 10381 consensus_queue.cc:1035] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:03.813154 11359 consensus_queue.cc:1035] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:03.816785 8917 raft_consensus.cc:1261] T 46ad540deb624a25b97d3b967fe1f6f4 P 9942306855404cfd891adf227dd65eb2 [term 1 LEARNER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:03.817461 10381 consensus_queue.cc:1035] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:03.820174 11452 ts_tablet_manager.cc:902] T d0d5dd42db3e4883af7fd16d8813b550 P e1a9251ce1cf4361b8b0ede9fbfc4023: Initiating tablet copy from peer 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:03.822125 11452 tablet_copy_client.cc:308] T d0d5dd42db3e4883af7fd16d8813b550 P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43863
I20240503 02:00:03.822649 6609 tablet_copy_service.cc:140] P 80fd18ef4d604c86b909d5c008a46b4f: Received BeginTabletCopySession request for tablet d0d5dd42db3e4883af7fd16d8813b550 from peer e1a9251ce1cf4361b8b0ede9fbfc4023 ({username='slave'} at 127.0.99.136:60117)
I20240503 02:00:03.822757 6609 tablet_copy_service.cc:161] P 80fd18ef4d604c86b909d5c008a46b4f: Beginning new tablet copy session on tablet d0d5dd42db3e4883af7fd16d8813b550 from peer e1a9251ce1cf4361b8b0ede9fbfc4023 at {username='slave'} at 127.0.99.136:60117: session id = e1a9251ce1cf4361b8b0ede9fbfc4023-d0d5dd42db3e4883af7fd16d8813b550
I20240503 02:00:03.823480 6609 tablet_copy_source_session.cc:205] T d0d5dd42db3e4883af7fd16d8813b550 P 80fd18ef4d604c86b909d5c008a46b4f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:03.824548 11452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0d5dd42db3e4883af7fd16d8813b550. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:03.826783 11452 tablet_copy_client.cc:783] T d0d5dd42db3e4883af7fd16d8813b550 P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:03.826918 11452 tablet_copy_client.cc:647] T d0d5dd42db3e4883af7fd16d8813b550 P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:03.830574 11073 consensus_queue.cc:1035] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:03.839622 11452 tablet_copy_client.cc:523] T d0d5dd42db3e4883af7fd16d8813b550 P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:03.841173 11452 tablet_bootstrap.cc:492] T d0d5dd42db3e4883af7fd16d8813b550 P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap starting.
I20240503 02:00:03.842471 11447 raft_consensus.cc:2932] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:03.844663 10278 raft_consensus.cc:2932] T 1e445a3cc928429caf9e3c6193a63010 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:03.845813 8917 raft_consensus.cc:2932] T 1e445a3cc928429caf9e3c6193a63010 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:03.848026 9179 raft_consensus.cc:2932] T 1e445a3cc928429caf9e3c6193a63010 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:03.848222 6458 raft_consensus.cc:2932] T 1e445a3cc928429caf9e3c6193a63010 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:03.848333 6307 catalog_manager.cc:5484] T 1e445a3cc928429caf9e3c6193a63010 P 9942306855404cfd891adf227dd65eb2 reported cstate change: config changed from index 7 to 8, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:03.849254 6309 catalog_manager.cc:5484] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: config changed from index 7 to 8, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:03.849952 6850 raft_consensus.cc:2932] T 46ad540deb624a25b97d3b967fe1f6f4 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:03.851604 9310 raft_consensus.cc:2932] T 46ad540deb624a25b97d3b967fe1f6f4 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:03.853466 8917 raft_consensus.cc:2932] T 46ad540deb624a25b97d3b967fe1f6f4 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:03.858214 9025 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:03.859205 6582 consensus_queue.cc:237] T 1e445a3cc928429caf9e3c6193a63010 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } }
I20240503 02:00:03.863054 8917 raft_consensus.cc:1261] T 1e445a3cc928429caf9e3c6193a63010 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:03.863497 9484 consensus_queue.cc:1035] T 1e445a3cc928429caf9e3c6193a63010 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:03.863726 9179 raft_consensus.cc:1261] T 1e445a3cc928429caf9e3c6193a63010 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:03.864409 9484 consensus_queue.cc:1035] T 1e445a3cc928429caf9e3c6193a63010 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:03.865607 10345 raft_consensus.cc:2932] T 1e445a3cc928429caf9e3c6193a63010 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:03.866036 8916 raft_consensus.cc:2932] T 1e445a3cc928429caf9e3c6193a63010 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:03.866943 9179 raft_consensus.cc:2932] T 1e445a3cc928429caf9e3c6193a63010 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:03.868259 6302 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1e445a3cc928429caf9e3c6193a63010 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:03.869067 6314 catalog_manager.cc:5484] T 1e445a3cc928429caf9e3c6193a63010 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: config changed from index 8 to 9, VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:03.871394 6309 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:03.871758 9955 raft_consensus.cc:1050] T ac1632cc5eb94554b1faa7eb715706ef P dec3e4a04d604f7eaaa3f077d081218a: attempting to promote NON_VOTER 99ead02915954b4fab23b033ad6d546b to VOTER
I20240503 02:00:03.872221 9955 consensus_queue.cc:237] T ac1632cc5eb94554b1faa7eb715706ef P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } }
I20240503 02:00:03.873515 9048 raft_consensus.cc:1261] T ac1632cc5eb94554b1faa7eb715706ef P 99ead02915954b4fab23b033ad6d546b [term 1 LEARNER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:03.873515 6582 raft_consensus.cc:1261] T ac1632cc5eb94554b1faa7eb715706ef P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:03.873965 10381 consensus_queue.cc:1035] T ac1632cc5eb94554b1faa7eb715706ef P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:03.874233 9721 consensus_queue.cc:1035] T ac1632cc5eb94554b1faa7eb715706ef P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:03.874347 6436 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e445a3cc928429caf9e3c6193a63010 with delete_type TABLET_DATA_TOMBSTONED (TS dec3e4a04d604f7eaaa3f077d081218a not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:51338
I20240503 02:00:03.874595 9474 tablet_replica.cc:330] T 1e445a3cc928429caf9e3c6193a63010 P dec3e4a04d604f7eaaa3f077d081218a: stopping tablet replica
I20240503 02:00:03.874737 9474 raft_consensus.cc:2229] T 1e445a3cc928429caf9e3c6193a63010 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:03.874923 9474 raft_consensus.cc:2258] T 1e445a3cc928429caf9e3c6193a63010 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:03.875803 9474 ts_tablet_manager.cc:1875] T 1e445a3cc928429caf9e3c6193a63010 P dec3e4a04d604f7eaaa3f077d081218a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:03.876490 6312 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_extra_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:03.878182 9474 ts_tablet_manager.cc:1888] T 1e445a3cc928429caf9e3c6193a63010 P dec3e4a04d604f7eaaa3f077d081218a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 02:00:03.878253 9474 log.cc:1196] T 1e445a3cc928429caf9e3c6193a63010 P dec3e4a04d604f7eaaa3f077d081218a: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/1e445a3cc928429caf9e3c6193a63010
I20240503 02:00:03.879719 6300 catalog_manager.cc:4830] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): tablet 1e445a3cc928429caf9e3c6193a63010 (table rebalance_test_table_19 [id=250f0720cafa4425b45a0360ebe04a03]) successfully deleted
I20240503 02:00:03.884837 11062 tablet.cc:1722] T b1b5646b12744ac9a57c871622c35dea P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:03.885208 11062 tablet.cc:1620] T b1b5646b12744ac9a57c871622c35dea P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
I20240503 02:00:03.885761 6436 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e445a3cc928429caf9e3c6193a63010 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 8 (current committed config index is 9)) from {username='slave'} at 127.0.0.1:51338
I20240503 02:00:03.887341 11437 tablet.cc:1722] T b1b5646b12744ac9a57c871622c35dea P e1a9251ce1cf4361b8b0ede9fbfc4023: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:03.887671 11437 tablet.cc:1620] T b1b5646b12744ac9a57c871622c35dea P e1a9251ce1cf4361b8b0ede9fbfc4023: MemRowSet was empty: no flush needed.
I20240503 02:00:03.889994 6719 raft_consensus.cc:1261] T ac1632cc5eb94554b1faa7eb715706ef P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:03.890669 11447 consensus_queue.cc:1035] T ac1632cc5eb94554b1faa7eb715706ef P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:03.891247 11064 tablet.cc:1722] T b1b5646b12744ac9a57c871622c35dea P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:03.891579 11064 tablet.cc:1620] T b1b5646b12744ac9a57c871622c35dea P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
I20240503 02:00:03.895216 6300 catalog_manager.cc:4830] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): tablet 1e445a3cc928429caf9e3c6193a63010 (table rebalance_test_table_19 [id=250f0720cafa4425b45a0360ebe04a03]) successfully deleted
I20240503 02:00:03.896553 9025 tablet_service.cc:1467] Processing CreateTablet for tablet e52373099af3428497d176e3fcd8ef27 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=2d54fb96d5ab4898ac53ffc16334115d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:03.896881 9025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e52373099af3428497d176e3fcd8ef27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:03.897454 9158 tablet_service.cc:1467] Processing CreateTablet for tablet e52373099af3428497d176e3fcd8ef27 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=2d54fb96d5ab4898ac53ffc16334115d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:03.897506 9159 tablet_service.cc:1467] Processing CreateTablet for tablet bd8b9532bd744ecdbe8da66a1225724b (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=2d54fb96d5ab4898ac53ffc16334115d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:03.897727 9158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e52373099af3428497d176e3fcd8ef27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:03.897750 6436 tablet_service.cc:1467] Processing CreateTablet for tablet bd8b9532bd744ecdbe8da66a1225724b (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=2d54fb96d5ab4898ac53ffc16334115d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:03.897835 6438 tablet_service.cc:1467] Processing CreateTablet for tablet 485833ae7cdf41ba9695f6502e1d83f7 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=2d54fb96d5ab4898ac53ffc16334115d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:03.897976 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd8b9532bd744ecdbe8da66a1225724b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:03.898140 6438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 485833ae7cdf41ba9695f6502e1d83f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:03.899854 6831 tablet_service.cc:1467] Processing CreateTablet for tablet 485833ae7cdf41ba9695f6502e1d83f7 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=2d54fb96d5ab4898ac53ffc16334115d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:03.900133 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 485833ae7cdf41ba9695f6502e1d83f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:03.900631 9025 tablet_service.cc:1467] Processing CreateTablet for tablet bd8b9532bd744ecdbe8da66a1225724b (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=2d54fb96d5ab4898ac53ffc16334115d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:03.900866 9025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd8b9532bd744ecdbe8da66a1225724b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:03.901211 8880 tablet_service.cc:1467] Processing CreateTablet for tablet 485833ae7cdf41ba9695f6502e1d83f7 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=2d54fb96d5ab4898ac53ffc16334115d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:03.901471 8880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 485833ae7cdf41ba9695f6502e1d83f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:03.902642 6700 tablet_service.cc:1467] Processing CreateTablet for tablet e52373099af3428497d176e3fcd8ef27 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=2d54fb96d5ab4898ac53ffc16334115d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:03.902887 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e52373099af3428497d176e3fcd8ef27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:03.903535 11045 tablet_bootstrap.cc:492] T 485833ae7cdf41ba9695f6502e1d83f7 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:03.904436 11045 tablet_bootstrap.cc:654] T 485833ae7cdf41ba9695f6502e1d83f7 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:03.905987 11316 tablet_bootstrap.cc:492] T e52373099af3428497d176e3fcd8ef27 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:03.906100 9159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd8b9532bd744ecdbe8da66a1225724b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:03.908057 11316 tablet_bootstrap.cc:654] T e52373099af3428497d176e3fcd8ef27 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:03.911675 11452 tablet_bootstrap.cc:492] T d0d5dd42db3e4883af7fd16d8813b550 P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:03.911417 6312 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=9471894d04b248e9879771dd5d8e4bae] alter complete (version 2)
I20240503 02:00:03.912209 11452 tablet_bootstrap.cc:492] T d0d5dd42db3e4883af7fd16d8813b550 P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap complete.
I20240503 02:00:03.912508 11452 ts_tablet_manager.cc:1367] T d0d5dd42db3e4883af7fd16d8813b550 P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent bootstrapping tablet: real 0.071s user 0.026s sys 0.000s
I20240503 02:00:03.913057 11452 raft_consensus.cc:348] T d0d5dd42db3e4883af7fd16d8813b550 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } }
I20240503 02:00:03.913285 11452 raft_consensus.cc:729] T d0d5dd42db3e4883af7fd16d8813b550 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e1a9251ce1cf4361b8b0ede9fbfc4023, State: Initialized, Role: LEARNER
I20240503 02:00:03.913304 11465 tablet_bootstrap.cc:492] T e52373099af3428497d176e3fcd8ef27 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:03.913406 11452 consensus_queue.cc:260] T d0d5dd42db3e4883af7fd16d8813b550 P e1a9251ce1cf4361b8b0ede9fbfc4023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } }
I20240503 02:00:03.913784 11452 ts_tablet_manager.cc:1398] T d0d5dd42db3e4883af7fd16d8813b550 P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:03.914261 11465 tablet_bootstrap.cc:654] T e52373099af3428497d176e3fcd8ef27 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:03.914305 6609 tablet_copy_service.cc:342] P 80fd18ef4d604c86b909d5c008a46b4f: Request end of tablet copy session e1a9251ce1cf4361b8b0ede9fbfc4023-d0d5dd42db3e4883af7fd16d8813b550 received from {username='slave'} at 127.0.99.136:60117
I20240503 02:00:03.914395 6609 tablet_copy_service.cc:433] P 80fd18ef4d604c86b909d5c008a46b4f: ending tablet copy session e1a9251ce1cf4361b8b0ede9fbfc4023-d0d5dd42db3e4883af7fd16d8813b550 on tablet d0d5dd42db3e4883af7fd16d8813b550 with peer e1a9251ce1cf4361b8b0ede9fbfc4023
I20240503 02:00:03.915071 11045 tablet_bootstrap.cc:492] T 485833ae7cdf41ba9695f6502e1d83f7 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:03.915102 11316 tablet_bootstrap.cc:492] T e52373099af3428497d176e3fcd8ef27 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:03.915171 11045 ts_tablet_manager.cc:1367] T 485833ae7cdf41ba9695f6502e1d83f7 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.001s
I20240503 02:00:03.915200 11316 ts_tablet_manager.cc:1367] T e52373099af3428497d176e3fcd8ef27 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20240503 02:00:03.915607 11045 raft_consensus.cc:348] T 485833ae7cdf41ba9695f6502e1d83f7 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:03.915724 11045 raft_consensus.cc:374] T 485833ae7cdf41ba9695f6502e1d83f7 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:03.915668 11316 raft_consensus.cc:348] T e52373099af3428497d176e3fcd8ef27 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:03.915755 11045 raft_consensus.cc:729] T 485833ae7cdf41ba9695f6502e1d83f7 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:03.915769 11316 raft_consensus.cc:374] T e52373099af3428497d176e3fcd8ef27 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:03.915799 11316 raft_consensus.cc:729] T e52373099af3428497d176e3fcd8ef27 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:03.915869 11045 consensus_queue.cc:260] T 485833ae7cdf41ba9695f6502e1d83f7 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:03.915913 11316 consensus_queue.cc:260] T e52373099af3428497d176e3fcd8ef27 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:03.916129 11045 ts_tablet_manager.cc:1398] T 485833ae7cdf41ba9695f6502e1d83f7 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:03.916188 11316 ts_tablet_manager.cc:1398] T e52373099af3428497d176e3fcd8ef27 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:03.916354 11316 tablet_bootstrap.cc:492] T bd8b9532bd744ecdbe8da66a1225724b P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:03.916754 11467 tablet_bootstrap.cc:492] T 485833ae7cdf41ba9695f6502e1d83f7 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:03.917407 11316 tablet_bootstrap.cc:654] T bd8b9532bd744ecdbe8da66a1225724b P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:03.917768 11469 tablet_bootstrap.cc:492] T e52373099af3428497d176e3fcd8ef27 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:03.920528 11316 tablet_bootstrap.cc:492] T bd8b9532bd744ecdbe8da66a1225724b P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:03.920625 11316 ts_tablet_manager.cc:1367] T bd8b9532bd744ecdbe8da66a1225724b P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:03.920966 11467 tablet_bootstrap.cc:654] T 485833ae7cdf41ba9695f6502e1d83f7 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:03.921097 11316 raft_consensus.cc:348] T bd8b9532bd744ecdbe8da66a1225724b P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:03.921204 11316 raft_consensus.cc:374] T bd8b9532bd744ecdbe8da66a1225724b P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:03.921233 11316 raft_consensus.cc:729] T bd8b9532bd744ecdbe8da66a1225724b P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:03.921380 11469 tablet_bootstrap.cc:654] T e52373099af3428497d176e3fcd8ef27 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:03.921348 11316 consensus_queue.cc:260] T bd8b9532bd744ecdbe8da66a1225724b P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:03.921614 11316 ts_tablet_manager.cc:1398] T bd8b9532bd744ecdbe8da66a1225724b P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:03.922099 11466 tablet_bootstrap.cc:492] T 485833ae7cdf41ba9695f6502e1d83f7 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:03.922137 11467 tablet_bootstrap.cc:492] T 485833ae7cdf41ba9695f6502e1d83f7 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:03.922212 11467 ts_tablet_manager.cc:1367] T 485833ae7cdf41ba9695f6502e1d83f7 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:00:03.923184 11466 tablet_bootstrap.cc:654] T 485833ae7cdf41ba9695f6502e1d83f7 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:03.923249 9955 raft_consensus.cc:2932] T ac1632cc5eb94554b1faa7eb715706ef P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 99ead02915954b4fab23b033ad6d546b (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:03.925383 11467 raft_consensus.cc:348] T 485833ae7cdf41ba9695f6502e1d83f7 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:03.925514 11467 raft_consensus.cc:374] T 485833ae7cdf41ba9695f6502e1d83f7 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:03.925550 11467 raft_consensus.cc:729] T 485833ae7cdf41ba9695f6502e1d83f7 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:03.925571 6719 raft_consensus.cc:2932] T ac1632cc5eb94554b1faa7eb715706ef P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 99ead02915954b4fab23b033ad6d546b (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:03.925663 11467 consensus_queue.cc:260] T 485833ae7cdf41ba9695f6502e1d83f7 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:03.926251 6582 raft_consensus.cc:2932] T ac1632cc5eb94554b1faa7eb715706ef P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 99ead02915954b4fab23b033ad6d546b (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:03.926504 6309 catalog_manager.cc:5484] T ac1632cc5eb94554b1faa7eb715706ef P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: config changed from index 4 to 5, 99ead02915954b4fab23b033ad6d546b (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:03.927002 11467 ts_tablet_manager.cc:1398] T 485833ae7cdf41ba9695f6502e1d83f7 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240503 02:00:03.929366 11465 tablet_bootstrap.cc:492] T e52373099af3428497d176e3fcd8ef27 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:03.929567 11465 ts_tablet_manager.cc:1367] T e52373099af3428497d176e3fcd8ef27 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240503 02:00:03.929684 11350 ts_tablet_manager.cc:902] T 485d1d89a8b94505969d663be92124cd P 9942306855404cfd891adf227dd65eb2: Initiating tablet copy from peer c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:03.930192 11350 tablet_copy_client.cc:308] T 485d1d89a8b94505969d663be92124cd P 9942306855404cfd891adf227dd65eb2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:42655
I20240503 02:00:03.930619 6739 tablet_copy_service.cc:140] P c2ac2eb7cc264047b9ab32fe3a4fc025: Received BeginTabletCopySession request for tablet 485d1d89a8b94505969d663be92124cd from peer 9942306855404cfd891adf227dd65eb2 ({username='slave'} at 127.0.99.133:46077)
I20240503 02:00:03.930711 6739 tablet_copy_service.cc:161] P c2ac2eb7cc264047b9ab32fe3a4fc025: Beginning new tablet copy session on tablet 485d1d89a8b94505969d663be92124cd from peer 9942306855404cfd891adf227dd65eb2 at {username='slave'} at 127.0.99.133:46077: session id = 9942306855404cfd891adf227dd65eb2-485d1d89a8b94505969d663be92124cd
I20240503 02:00:03.931581 6739 tablet_copy_source_session.cc:205] T 485d1d89a8b94505969d663be92124cd P c2ac2eb7cc264047b9ab32fe3a4fc025: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:03.932862 11350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 485d1d89a8b94505969d663be92124cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:03.933153 11465 raft_consensus.cc:348] T e52373099af3428497d176e3fcd8ef27 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:03.933290 11465 raft_consensus.cc:374] T e52373099af3428497d176e3fcd8ef27 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:03.933351 11465 raft_consensus.cc:729] T e52373099af3428497d176e3fcd8ef27 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:03.933502 11465 consensus_queue.cc:260] T e52373099af3428497d176e3fcd8ef27 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:03.935011 11350 tablet_copy_client.cc:783] T 485d1d89a8b94505969d663be92124cd P 9942306855404cfd891adf227dd65eb2: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:03.935166 11350 tablet_copy_client.cc:647] T 485d1d89a8b94505969d663be92124cd P 9942306855404cfd891adf227dd65eb2: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:03.934437 11465 ts_tablet_manager.cc:1398] T e52373099af3428497d176e3fcd8ef27 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:00:03.936048 11465 tablet_bootstrap.cc:492] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:03.935951 9048 raft_consensus.cc:2932] T ac1632cc5eb94554b1faa7eb715706ef P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 99ead02915954b4fab23b033ad6d546b (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:03.937120 11465 tablet_bootstrap.cc:654] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:03.938441 11350 tablet_copy_client.cc:523] T 485d1d89a8b94505969d663be92124cd P 9942306855404cfd891adf227dd65eb2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:03.939802 11350 tablet_bootstrap.cc:492] T 485d1d89a8b94505969d663be92124cd P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:03.939772 6458 consensus_queue.cc:237] T ac1632cc5eb94554b1faa7eb715706ef P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } }
I20240503 02:00:03.941188 6589 raft_consensus.cc:1261] T ac1632cc5eb94554b1faa7eb715706ef P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:03.941653 10381 consensus_queue.cc:1035] T ac1632cc5eb94554b1faa7eb715706ef P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:03.942395 11469 tablet_bootstrap.cc:492] T e52373099af3428497d176e3fcd8ef27 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:03.942449 9048 raft_consensus.cc:1261] T ac1632cc5eb94554b1faa7eb715706ef P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:03.942515 11469 ts_tablet_manager.cc:1367] T e52373099af3428497d176e3fcd8ef27 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.000s
I20240503 02:00:03.942845 10381 consensus_queue.cc:1035] T ac1632cc5eb94554b1faa7eb715706ef P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:03.942991 11469 raft_consensus.cc:348] T e52373099af3428497d176e3fcd8ef27 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:03.943094 11469 raft_consensus.cc:374] T e52373099af3428497d176e3fcd8ef27 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:03.943120 11469 raft_consensus.cc:729] T e52373099af3428497d176e3fcd8ef27 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:03.943243 11469 consensus_queue.cc:260] T e52373099af3428497d176e3fcd8ef27 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:03.943538 11469 ts_tablet_manager.cc:1398] T e52373099af3428497d176e3fcd8ef27 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:03.944341 11466 tablet_bootstrap.cc:492] T 485833ae7cdf41ba9695f6502e1d83f7 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:03.944523 11466 ts_tablet_manager.cc:1367] T 485833ae7cdf41ba9695f6502e1d83f7 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.000s
I20240503 02:00:03.944959 11466 raft_consensus.cc:348] T 485833ae7cdf41ba9695f6502e1d83f7 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:03.945057 11466 raft_consensus.cc:374] T 485833ae7cdf41ba9695f6502e1d83f7 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:03.945081 11466 raft_consensus.cc:729] T 485833ae7cdf41ba9695f6502e1d83f7 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:03.945188 11466 consensus_queue.cc:260] T 485833ae7cdf41ba9695f6502e1d83f7 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:03.945428 11466 ts_tablet_manager.cc:1398] T 485833ae7cdf41ba9695f6502e1d83f7 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:03.945609 11466 tablet_bootstrap.cc:492] T bd8b9532bd744ecdbe8da66a1225724b P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:03.946503 11466 tablet_bootstrap.cc:654] T bd8b9532bd744ecdbe8da66a1225724b P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:03.946717 10381 raft_consensus.cc:2932] T ac1632cc5eb94554b1faa7eb715706ef P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:03.948436 6300 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ac1632cc5eb94554b1faa7eb715706ef with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:03.948992 11466 tablet_bootstrap.cc:492] T bd8b9532bd744ecdbe8da66a1225724b P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:03.949008 6589 raft_consensus.cc:2932] T ac1632cc5eb94554b1faa7eb715706ef P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:03.949167 11466 ts_tablet_manager.cc:1367] T bd8b9532bd744ecdbe8da66a1225724b P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 02:00:03.950277 9039 raft_consensus.cc:2932] T ac1632cc5eb94554b1faa7eb715706ef P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:03.950961 11466 raft_consensus.cc:348] T bd8b9532bd744ecdbe8da66a1225724b P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:03.951139 11466 raft_consensus.cc:374] T bd8b9532bd744ecdbe8da66a1225724b P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:03.951211 11466 raft_consensus.cc:729] T bd8b9532bd744ecdbe8da66a1225724b P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:03.951407 11466 consensus_queue.cc:260] T bd8b9532bd744ecdbe8da66a1225724b P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:03.951444 6307 catalog_manager.cc:5484] T ac1632cc5eb94554b1faa7eb715706ef P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: config changed from index 5 to 6, VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:03.952587 11465 tablet_bootstrap.cc:492] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:03.952685 11465 ts_tablet_manager.cc:1367] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240503 02:00:03.953133 11465 raft_consensus.cc:348] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:03.953224 11465 raft_consensus.cc:374] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:03.953244 11465 raft_consensus.cc:729] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:03.953351 11465 consensus_queue.cc:260] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:03.953580 11466 ts_tablet_manager.cc:1398] T bd8b9532bd744ecdbe8da66a1225724b P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240503 02:00:03.953636 11465 ts_tablet_manager.cc:1398] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:03.954295 6700 tablet_service.cc:1514] Processing DeleteTablet for tablet ac1632cc5eb94554b1faa7eb715706ef with delete_type TABLET_DATA_TOMBSTONED (TS c2ac2eb7cc264047b9ab32fe3a4fc025 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:40336
I20240503 02:00:03.954531 10173 tablet_replica.cc:330] T ac1632cc5eb94554b1faa7eb715706ef P c2ac2eb7cc264047b9ab32fe3a4fc025: stopping tablet replica
I20240503 02:00:03.954670 10173 raft_consensus.cc:2229] T ac1632cc5eb94554b1faa7eb715706ef P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:03.954834 10173 raft_consensus.cc:2258] T ac1632cc5eb94554b1faa7eb715706ef P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:03.956440 10173 ts_tablet_manager.cc:1875] T ac1632cc5eb94554b1faa7eb715706ef P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:03.958786 10173 ts_tablet_manager.cc:1888] T ac1632cc5eb94554b1faa7eb715706ef P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:00:03.958846 10173 log.cc:1196] T ac1632cc5eb94554b1faa7eb715706ef P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/ac1632cc5eb94554b1faa7eb715706ef
I20240503 02:00:03.959353 6303 catalog_manager.cc:4830] TS c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655): tablet ac1632cc5eb94554b1faa7eb715706ef (table rebalance_test_table_9 [id=c65135643094446abc8eba15be23cd99]) successfully deleted
I20240503 02:00:03.964956 6589 consensus_queue.cc:237] T 7df4c083550046e883e5e61d3d4e1572 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } }
I20240503 02:00:03.966589 6458 raft_consensus.cc:1261] T 7df4c083550046e883e5e61d3d4e1572 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 02:00:03.966675 6522 consensus_peers.cc:485] T 7df4c083550046e883e5e61d3d4e1572 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079): Couldn't send request to peer 2a01ace80baf4057b6bf1995cf88468b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7df4c083550046e883e5e61d3d4e1572. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:03.967005 8916 raft_consensus.cc:1261] T 7df4c083550046e883e5e61d3d4e1572 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:03.967064 9973 raft_consensus.cc:1050] T 4385d51d11574192883d01376e2d5620 P 80fd18ef4d604c86b909d5c008a46b4f: attempting to promote NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 to VOTER
I20240503 02:00:03.967458 9973 consensus_queue.cc:237] T 4385d51d11574192883d01376e2d5620 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } }
I20240503 02:00:03.968217 8833 consensus_queue.cc:1035] T 7df4c083550046e883e5e61d3d4e1572 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:03.968894 6850 raft_consensus.cc:1261] T 4385d51d11574192883d01376e2d5620 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:03.969416 10278 consensus_queue.cc:1035] T 7df4c083550046e883e5e61d3d4e1572 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:03.969537 8833 consensus_queue.cc:1035] T 4385d51d11574192883d01376e2d5620 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:03.971274 9310 raft_consensus.cc:1261] T 4385d51d11574192883d01376e2d5620 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 LEARNER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:03.971698 9973 consensus_queue.cc:1035] T 4385d51d11574192883d01376e2d5620 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:03.972072 6458 raft_consensus.cc:1261] T 4385d51d11574192883d01376e2d5620 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:03.972704 8833 raft_consensus.cc:2932] T 7df4c083550046e883e5e61d3d4e1572 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } } }
I20240503 02:00:03.974515 11490 consensus_queue.cc:1035] T 4385d51d11574192883d01376e2d5620 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:03.975455 6456 raft_consensus.cc:2932] T 7df4c083550046e883e5e61d3d4e1572 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } } }
I20240503 02:00:03.977465 8916 raft_consensus.cc:2932] T 7df4c083550046e883e5e61d3d4e1572 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } } }
I20240503 02:00:03.978404 6309 catalog_manager.cc:5484] T 7df4c083550046e883e5e61d3d4e1572 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: config changed from index 6 to 7, NON_VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:03.979194 6850 raft_consensus.cc:1203] T cb873018448549ccbe3d14b784e47d26 P 492448570cc94ed1879feb2ec3b9051d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:00:03.981999 11350 tablet_bootstrap.cc:492] T 485d1d89a8b94505969d663be92124cd P 9942306855404cfd891adf227dd65eb2: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:03.982285 11350 tablet_bootstrap.cc:492] T 485d1d89a8b94505969d663be92124cd P 9942306855404cfd891adf227dd65eb2: Bootstrap complete.
I20240503 02:00:03.982471 11350 ts_tablet_manager.cc:1367] T 485d1d89a8b94505969d663be92124cd P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.043s user 0.025s sys 0.000s
I20240503 02:00:03.982914 11350 raft_consensus.cc:348] T 485d1d89a8b94505969d663be92124cd P 9942306855404cfd891adf227dd65eb2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } }
I20240503 02:00:03.983033 11350 raft_consensus.cc:729] T 485d1d89a8b94505969d663be92124cd P 9942306855404cfd891adf227dd65eb2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: LEARNER
I20240503 02:00:03.983122 11350 consensus_queue.cc:260] T 485d1d89a8b94505969d663be92124cd P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } }
I20240503 02:00:03.983893 11350 ts_tablet_manager.cc:1398] T 485d1d89a8b94505969d663be92124cd P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 02:00:03.984647 6739 tablet_copy_service.cc:342] P c2ac2eb7cc264047b9ab32fe3a4fc025: Request end of tablet copy session 9942306855404cfd891adf227dd65eb2-485d1d89a8b94505969d663be92124cd received from {username='slave'} at 127.0.99.133:46077
I20240503 02:00:03.984726 6739 tablet_copy_service.cc:433] P c2ac2eb7cc264047b9ab32fe3a4fc025: ending tablet copy session 9942306855404cfd891adf227dd65eb2-485d1d89a8b94505969d663be92124cd on tablet 485d1d89a8b94505969d663be92124cd with peer 9942306855404cfd891adf227dd65eb2
I20240503 02:00:03.986773 9484 raft_consensus.cc:2932] T 4385d51d11574192883d01376e2d5620 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:03.988734 6850 raft_consensus.cc:2932] T 4385d51d11574192883d01376e2d5620 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:03.988734 9310 raft_consensus.cc:2932] T 4385d51d11574192883d01376e2d5620 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:03.991852 6458 raft_consensus.cc:2932] T 4385d51d11574192883d01376e2d5620 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:03.993724 6312 catalog_manager.cc:5484] T 4385d51d11574192883d01376e2d5620 P 492448570cc94ed1879feb2ec3b9051d reported cstate change: config changed from index 4 to 5, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:03.998577 6589 consensus_queue.cc:237] T 4385d51d11574192883d01376e2d5620 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } }
I20240503 02:00:03.999974 6849 raft_consensus.cc:1261] T 4385d51d11574192883d01376e2d5620 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:04.000265 9310 raft_consensus.cc:1261] T 4385d51d11574192883d01376e2d5620 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:04.001492 11491 consensus_queue.cc:1035] T 4385d51d11574192883d01376e2d5620 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:04.001870 11491 consensus_queue.cc:1035] T 4385d51d11574192883d01376e2d5620 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:04.002990 9179 consensus_queue.cc:237] T 51b82ef8e4cc474387f6902f2e526ac0 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: true } }
I20240503 02:00:04.003304 8833 raft_consensus.cc:2932] T 4385d51d11574192883d01376e2d5620 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:04.003659 9310 raft_consensus.cc:2932] T 4385d51d11574192883d01376e2d5620 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:04.004822 9039 raft_consensus.cc:1261] T 51b82ef8e4cc474387f6902f2e526ac0 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer 2a01ace80baf4057b6bf1995cf88468b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:04.005160 6302 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4385d51d11574192883d01376e2d5620 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:04.006393 10679 consensus_queue.cc:1035] T 51b82ef8e4cc474387f6902f2e526ac0 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:00:04.006887 9115 consensus_peers.cc:485] T 51b82ef8e4cc474387f6902f2e526ac0 P 2a01ace80baf4057b6bf1995cf88468b -> Peer c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655): Couldn't send request to peer c2ac2eb7cc264047b9ab32fe3a4fc025. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 51b82ef8e4cc474387f6902f2e526ac0. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:04.006812 6312 catalog_manager.cc:5484] T 4385d51d11574192883d01376e2d5620 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: config changed from index 5 to 6, VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:04.009740 6849 raft_consensus.cc:2932] T 4385d51d11574192883d01376e2d5620 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:04.010066 9310 raft_consensus.cc:1261] T 51b82ef8e4cc474387f6902f2e526ac0 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Refusing update from remote peer 2a01ace80baf4057b6bf1995cf88468b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:04.010093 11501 raft_consensus.cc:2932] T 51b82ef8e4cc474387f6902f2e526ac0 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: true } } }
I20240503 02:00:04.011590 6436 tablet_service.cc:1514] Processing DeleteTablet for tablet 4385d51d11574192883d01376e2d5620 with delete_type TABLET_DATA_TOMBSTONED (TS dec3e4a04d604f7eaaa3f077d081218a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51338
I20240503 02:00:04.012109 9474 tablet_replica.cc:330] T 4385d51d11574192883d01376e2d5620 P dec3e4a04d604f7eaaa3f077d081218a: stopping tablet replica
I20240503 02:00:04.012275 9474 raft_consensus.cc:2229] T 4385d51d11574192883d01376e2d5620 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:04.012777 9474 raft_consensus.cc:2258] T 4385d51d11574192883d01376e2d5620 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:04.014092 6314 catalog_manager.cc:5484] T 51b82ef8e4cc474387f6902f2e526ac0 P 2a01ace80baf4057b6bf1995cf88468b reported cstate change: config changed from index -1 to 2, NON_VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "2a01ace80baf4057b6bf1995cf88468b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:04.014317 10679 consensus_queue.cc:1035] T 51b82ef8e4cc474387f6902f2e526ac0 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:04.014221 9039 raft_consensus.cc:2932] T 51b82ef8e4cc474387f6902f2e526ac0 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: true } } }
I20240503 02:00:04.014775 6309 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:04.015806 9310 raft_consensus.cc:2932] T 51b82ef8e4cc474387f6902f2e526ac0 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: true } } }
I20240503 02:00:04.018174 9474 ts_tablet_manager.cc:1875] T 4385d51d11574192883d01376e2d5620 P dec3e4a04d604f7eaaa3f077d081218a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:04.020197 9474 ts_tablet_manager.cc:1888] T 4385d51d11574192883d01376e2d5620 P dec3e4a04d604f7eaaa3f077d081218a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:00:04.020260 9474 log.cc:1196] T 4385d51d11574192883d01376e2d5620 P dec3e4a04d604f7eaaa3f077d081218a: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/4385d51d11574192883d01376e2d5620
I20240503 02:00:04.021013 6300 catalog_manager.cc:4830] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): tablet 4385d51d11574192883d01376e2d5620 (table rebalance_test_table_17 [id=5ccf0adaf2de4b79bbf38c053646e60e]) successfully deleted
I20240503 02:00:04.024070 10473 raft_consensus.cc:482] T e52373099af3428497d176e3fcd8ef27 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:04.024189 10473 raft_consensus.cc:504] T e52373099af3428497d176e3fcd8ef27 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:04.024789 10473 leader_election.cc:288] T e52373099af3428497d176e3fcd8ef27 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:04.026640 6719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e52373099af3428497d176e3fcd8ef27" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 02:00:04.026788 6719 raft_consensus.cc:2454] T e52373099af3428497d176e3fcd8ef27 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:04.028914 11062 tablet.cc:1722] T b1b5646b12744ac9a57c871622c35dea P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:04.029301 11064 tablet.cc:1722] T b1b5646b12744ac9a57c871622c35dea P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:04.029551 8984 leader_election.cc:302] T e52373099af3428497d176e3fcd8ef27 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99ead02915954b4fab23b033ad6d546b, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:04.029650 9484 raft_consensus.cc:1050] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f: attempting to promote NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 to VOTER
I20240503 02:00:04.030071 9484 consensus_queue.cc:237] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } }
I20240503 02:00:04.030191 10473 raft_consensus.cc:2781] T e52373099af3428497d176e3fcd8ef27 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:04.030251 10473 raft_consensus.cc:482] T e52373099af3428497d176e3fcd8ef27 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:04.030283 10473 raft_consensus.cc:3037] T e52373099af3428497d176e3fcd8ef27 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:04.031260 10473 raft_consensus.cc:504] T e52373099af3428497d176e3fcd8ef27 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:04.031643 10473 leader_election.cc:288] T e52373099af3428497d176e3fcd8ef27 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Requested vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:04.032317 6719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e52373099af3428497d176e3fcd8ef27" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 02:00:04.032469 6719 raft_consensus.cc:3037] T e52373099af3428497d176e3fcd8ef27 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:04.032708 6849 raft_consensus.cc:1261] T 915028b3478341d588ed6031e657c259 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:04.033030 8916 raft_consensus.cc:1261] T 915028b3478341d588ed6031e657c259 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:04.033480 6719 raft_consensus.cc:2454] T e52373099af3428497d176e3fcd8ef27 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:04.033445 11492 consensus_queue.cc:1035] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:04.033875 9309 raft_consensus.cc:1261] T 915028b3478341d588ed6031e657c259 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 LEARNER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:04.034708 8984 leader_election.cc:302] T e52373099af3428497d176e3fcd8ef27 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99ead02915954b4fab23b033ad6d546b, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:04.035243 10473 raft_consensus.cc:2781] T e52373099af3428497d176e3fcd8ef27 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:04.035450 10473 raft_consensus.cc:686] T e52373099af3428497d176e3fcd8ef27 P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Becoming Leader. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Running, Role: LEADER
I20240503 02:00:04.035598 10473 consensus_queue.cc:237] T e52373099af3428497d176e3fcd8ef27 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:04.036000 6312 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=9471894d04b248e9879771dd5d8e4bae] alter complete (version 3)
I20240503 02:00:04.036181 11492 consensus_queue.cc:1035] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:04.036585 11492 consensus_queue.cc:1035] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:04.036931 10473 raft_consensus.cc:482] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:04.037045 10473 raft_consensus.cc:504] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:04.037992 6309 catalog_manager.cc:5484] T e52373099af3428497d176e3fcd8ef27 P 99ead02915954b4fab23b033ad6d546b reported cstate change: term changed from 0 to 1, leader changed from <none> to 99ead02915954b4fab23b033ad6d546b (127.0.99.134). New cstate: current_term: 1 leader_uuid: "99ead02915954b4fab23b033ad6d546b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:04.038064 11437 tablet.cc:1722] T b1b5646b12744ac9a57c871622c35dea P e1a9251ce1cf4361b8b0ede9fbfc4023: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:04.039901 6849 consensus_queue.cc:237] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } }
I20240503 02:00:04.040709 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8b9532bd744ecdbe8da66a1225724b" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 02:00:04.040855 6458 raft_consensus.cc:2454] T bd8b9532bd744ecdbe8da66a1225724b P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:04.041301 8981 leader_election.cc:302] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99ead02915954b4fab23b033ad6d546b, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:04.041766 6589 raft_consensus.cc:1261] T 1fa274c98fc147f29344ed537e2ccf17 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:04.042174 9310 raft_consensus.cc:1261] T 1fa274c98fc147f29344ed537e2ccf17 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:04.042241 10473 leader_election.cc:288] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 02:00:04.042220 11058 consensus_queue.cc:1035] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:04.042333 10473 raft_consensus.cc:2781] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:04.042378 10473 raft_consensus.cc:482] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:04.042419 10473 raft_consensus.cc:3037] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:04.042527 11377 consensus_queue.cc:1035] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 02:00:04.043386 6784 consensus_peers.cc:485] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d -> Peer 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079): Couldn't send request to peer 2a01ace80baf4057b6bf1995cf88468b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1fa274c98fc147f29344ed537e2ccf17. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:04.043466 10473 raft_consensus.cc:504] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:04.044639 10473 leader_election.cc:288] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Requested vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 02:00:04.044961 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8b9532bd744ecdbe8da66a1225724b" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 02:00:04.045100 6458 raft_consensus.cc:3037] T bd8b9532bd744ecdbe8da66a1225724b P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:04.046202 6458 raft_consensus.cc:2454] T bd8b9532bd744ecdbe8da66a1225724b P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:04.046715 11377 raft_consensus.cc:2932] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } } }
I20240503 02:00:04.047614 9310 raft_consensus.cc:2932] T 1fa274c98fc147f29344ed537e2ccf17 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } } }
I20240503 02:00:04.048749 8981 leader_election.cc:302] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99ead02915954b4fab23b033ad6d546b, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:04.049685 10473 raft_consensus.cc:2781] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:04.049777 10473 raft_consensus.cc:686] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Becoming Leader. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Running, Role: LEADER
I20240503 02:00:04.049680 6582 raft_consensus.cc:2932] T 1fa274c98fc147f29344ed537e2ccf17 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } } }
I20240503 02:00:04.049904 10473 consensus_queue.cc:237] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:04.052625 6309 catalog_manager.cc:5484] T 1fa274c98fc147f29344ed537e2ccf17 P e1a9251ce1cf4361b8b0ede9fbfc4023 reported cstate change: config changed from index 6 to 7, NON_VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } } }
I20240503 02:00:04.052796 6312 catalog_manager.cc:5484] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b reported cstate change: term changed from 0 to 1, leader changed from <none> to 99ead02915954b4fab23b033ad6d546b (127.0.99.134). New cstate: current_term: 1 leader_uuid: "99ead02915954b4fab23b033ad6d546b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:04.059343 11490 raft_consensus.cc:2932] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:04.060330 8916 raft_consensus.cc:2932] T 915028b3478341d588ed6031e657c259 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:04.061401 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e52373099af3428497d176e3fcd8ef27" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:04.061486 9177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8b9532bd744ecdbe8da66a1225724b" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:04.061535 9178 raft_consensus.cc:3037] T e52373099af3428497d176e3fcd8ef27 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:04.061571 9177 raft_consensus.cc:2454] T bd8b9532bd744ecdbe8da66a1225724b P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:04.062507 9178 raft_consensus.cc:2454] T e52373099af3428497d176e3fcd8ef27 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:04.062753 9176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8b9532bd744ecdbe8da66a1225724b" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:04.062865 9176 raft_consensus.cc:3037] T bd8b9532bd744ecdbe8da66a1225724b P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:04.063061 6849 raft_consensus.cc:2932] T 915028b3478341d588ed6031e657c259 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:04.063588 6312 catalog_manager.cc:5484] T 915028b3478341d588ed6031e657c259 P 9942306855404cfd891adf227dd65eb2 reported cstate change: config changed from index 7 to 8, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:04.063920 9176 raft_consensus.cc:2454] T bd8b9532bd744ecdbe8da66a1225724b P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:04.064473 9309 raft_consensus.cc:2932] T 915028b3478341d588ed6031e657c259 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:04.066870 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e52373099af3428497d176e3fcd8ef27" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:04.067041 9179 raft_consensus.cc:2362] T e52373099af3428497d176e3fcd8ef27 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1. Re-sending same reply.
I20240503 02:00:04.067979 6582 consensus_queue.cc:237] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } }
I20240503 02:00:04.069276 9309 raft_consensus.cc:1261] T 915028b3478341d588ed6031e657c259 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:04.069518 8916 raft_consensus.cc:1261] T 915028b3478341d588ed6031e657c259 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:04.069921 9973 consensus_queue.cc:1035] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:04.070303 9973 consensus_queue.cc:1035] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:04.072356 11491 raft_consensus.cc:2932] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:04.072780 8916 raft_consensus.cc:2932] T 915028b3478341d588ed6031e657c259 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:04.073552 9309 raft_consensus.cc:2932] T 915028b3478341d588ed6031e657c259 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:04.073966 6302 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 915028b3478341d588ed6031e657c259 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:04.075722 6312 catalog_manager.cc:5484] T 915028b3478341d588ed6031e657c259 P 9942306855404cfd891adf227dd65eb2 reported cstate change: config changed from index 8 to 9, VOTER 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:04.078209 6831 tablet_service.cc:1514] Processing DeleteTablet for tablet 915028b3478341d588ed6031e657c259 with delete_type TABLET_DATA_TOMBSTONED (TS 492448570cc94ed1879feb2ec3b9051d not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:52812
I20240503 02:00:04.078584 9476 tablet_replica.cc:330] T 915028b3478341d588ed6031e657c259 P 492448570cc94ed1879feb2ec3b9051d: stopping tablet replica
I20240503 02:00:04.078727 9476 raft_consensus.cc:2229] T 915028b3478341d588ed6031e657c259 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:04.078933 9476 raft_consensus.cc:2258] T 915028b3478341d588ed6031e657c259 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:04.080708 9476 ts_tablet_manager.cc:1875] T 915028b3478341d588ed6031e657c259 P 492448570cc94ed1879feb2ec3b9051d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:04.082974 9476 ts_tablet_manager.cc:1888] T 915028b3478341d588ed6031e657c259 P 492448570cc94ed1879feb2ec3b9051d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 02:00:04.083070 9476 log.cc:1196] T 915028b3478341d588ed6031e657c259 P 492448570cc94ed1879feb2ec3b9051d: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/915028b3478341d588ed6031e657c259
I20240503 02:00:04.083603 6303 catalog_manager.cc:4830] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719): tablet 915028b3478341d588ed6031e657c259 (table rebalance_test_table_3 [id=72f29547da0b4a95974018c2782cb359]) successfully deleted
I20240503 02:00:04.121852 11554 ts_tablet_manager.cc:902] T 96c395771fd44056abacceb6c1acdb9f P dec3e4a04d604f7eaaa3f077d081218a: Initiating tablet copy from peer 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:04.122776 11554 tablet_copy_client.cc:308] T 96c395771fd44056abacceb6c1acdb9f P dec3e4a04d604f7eaaa3f077d081218a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:33211
I20240503 02:00:04.123492 8937 tablet_copy_service.cc:140] P 9942306855404cfd891adf227dd65eb2: Received BeginTabletCopySession request for tablet 96c395771fd44056abacceb6c1acdb9f from peer dec3e4a04d604f7eaaa3f077d081218a ({username='slave'} at 127.0.99.129:38483)
I20240503 02:00:04.123623 8937 tablet_copy_service.cc:161] P 9942306855404cfd891adf227dd65eb2: Beginning new tablet copy session on tablet 96c395771fd44056abacceb6c1acdb9f from peer dec3e4a04d604f7eaaa3f077d081218a at {username='slave'} at 127.0.99.129:38483: session id = dec3e4a04d604f7eaaa3f077d081218a-96c395771fd44056abacceb6c1acdb9f
I20240503 02:00:04.124680 8937 tablet_copy_source_session.cc:205] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:04.125883 11554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96c395771fd44056abacceb6c1acdb9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:04.128065 11554 tablet_copy_client.cc:783] T 96c395771fd44056abacceb6c1acdb9f P dec3e4a04d604f7eaaa3f077d081218a: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:04.128288 11554 tablet_copy_client.cc:647] T 96c395771fd44056abacceb6c1acdb9f P dec3e4a04d604f7eaaa3f077d081218a: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:04.130077 11554 tablet_copy_client.cc:523] T 96c395771fd44056abacceb6c1acdb9f P dec3e4a04d604f7eaaa3f077d081218a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:04.131894 11554 tablet_bootstrap.cc:492] T 96c395771fd44056abacceb6c1acdb9f P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:04.136655 11554 tablet_bootstrap.cc:492] T 96c395771fd44056abacceb6c1acdb9f P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:04.136916 11554 tablet_bootstrap.cc:492] T 96c395771fd44056abacceb6c1acdb9f P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap complete.
I20240503 02:00:04.137111 11554 ts_tablet_manager.cc:1367] T 96c395771fd44056abacceb6c1acdb9f P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20240503 02:00:04.137809 11554 raft_consensus.cc:348] T 96c395771fd44056abacceb6c1acdb9f P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: true } }
I20240503 02:00:04.137969 11554 raft_consensus.cc:729] T 96c395771fd44056abacceb6c1acdb9f P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: LEARNER
I20240503 02:00:04.138094 11554 consensus_queue.cc:260] T 96c395771fd44056abacceb6c1acdb9f P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: true } }
I20240503 02:00:04.138551 11554 ts_tablet_manager.cc:1398] T 96c395771fd44056abacceb6c1acdb9f P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 02:00:04.139200 6582 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c3485701bf3b48f5ae2e1568afeed133"
dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49756
I20240503 02:00:04.139391 6582 raft_consensus.cc:595] T c3485701bf3b48f5ae2e1568afeed133 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Received request to transfer leadership
I20240503 02:00:04.139693 8937 tablet_copy_service.cc:342] P 9942306855404cfd891adf227dd65eb2: Request end of tablet copy session dec3e4a04d604f7eaaa3f077d081218a-96c395771fd44056abacceb6c1acdb9f received from {username='slave'} at 127.0.99.129:38483
I20240503 02:00:04.139781 8937 tablet_copy_service.cc:433] P 9942306855404cfd891adf227dd65eb2: ending tablet copy session dec3e4a04d604f7eaaa3f077d081218a-96c395771fd44056abacceb6c1acdb9f on tablet 96c395771fd44056abacceb6c1acdb9f with peer dec3e4a04d604f7eaaa3f077d081218a
I20240503 02:00:04.144906 6314 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:04.145071 6314 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:04.147330 6314 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet b1b5646b12744ac9a57c871622c35dea
I20240503 02:00:04.148226 9286 tablet_service.cc:1514] Processing DeleteTablet for tablet b1b5646b12744ac9a57c871622c35dea with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:04 UTC) from {username='slave'} at 127.0.0.1:36004
I20240503 02:00:04.148237 9159 tablet_service.cc:1514] Processing DeleteTablet for tablet b1b5646b12744ac9a57c871622c35dea with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:04 UTC) from {username='slave'} at 127.0.0.1:51940
I20240503 02:00:04.148485 11252 tablet_replica.cc:330] T b1b5646b12744ac9a57c871622c35dea P 2a01ace80baf4057b6bf1995cf88468b: stopping tablet replica
I20240503 02:00:04.148631 11252 raft_consensus.cc:2229] T b1b5646b12744ac9a57c871622c35dea P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:04.148759 11252 raft_consensus.cc:2258] T b1b5646b12744ac9a57c871622c35dea P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:04.149356 11252 ts_tablet_manager.cc:1875] T b1b5646b12744ac9a57c871622c35dea P 2a01ace80baf4057b6bf1995cf88468b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:04.150005 11575 tablet_replica.cc:330] T b1b5646b12744ac9a57c871622c35dea P e1a9251ce1cf4361b8b0ede9fbfc4023: stopping tablet replica
I20240503 02:00:04.150198 11575 raft_consensus.cc:2229] T b1b5646b12744ac9a57c871622c35dea P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:04.150388 11575 raft_consensus.cc:2258] T b1b5646b12744ac9a57c871622c35dea P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:04.150796 8880 tablet_service.cc:1514] Processing DeleteTablet for tablet b1b5646b12744ac9a57c871622c35dea with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:04 UTC) from {username='slave'} at 127.0.0.1:40296
I20240503 02:00:04.150944 6582 raft_consensus.cc:1203] T f5ed01740019467d8c1d55cc2fa18ca3 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:00:04.152042 11252 ts_tablet_manager.cc:1888] T b1b5646b12744ac9a57c871622c35dea P 2a01ace80baf4057b6bf1995cf88468b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:04.152151 11252 log.cc:1196] T b1b5646b12744ac9a57c871622c35dea P 2a01ace80baf4057b6bf1995cf88468b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/b1b5646b12744ac9a57c871622c35dea
I20240503 02:00:04.152186 11576 tablet_replica.cc:330] T b1b5646b12744ac9a57c871622c35dea P 9942306855404cfd891adf227dd65eb2: stopping tablet replica
I20240503 02:00:04.152424 11576 raft_consensus.cc:2229] T b1b5646b12744ac9a57c871622c35dea P 9942306855404cfd891adf227dd65eb2 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:04.152853 11575 ts_tablet_manager.cc:1875] T b1b5646b12744ac9a57c871622c35dea P e1a9251ce1cf4361b8b0ede9fbfc4023: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:04.153200 11576 raft_consensus.cc:2258] T b1b5646b12744ac9a57c871622c35dea P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:04.154393 11252 ts_tablet_manager.cc:1909] T b1b5646b12744ac9a57c871622c35dea P 2a01ace80baf4057b6bf1995cf88468b: Deleting consensus metadata
I20240503 02:00:04.155263 6300 catalog_manager.cc:4830] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079): tablet b1b5646b12744ac9a57c871622c35dea (table rebalancer_dynamic_table_new_name [id=9471894d04b248e9879771dd5d8e4bae]) successfully deleted
I20240503 02:00:04.159307 11575 ts_tablet_manager.cc:1888] T b1b5646b12744ac9a57c871622c35dea P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:04.159525 11575 log.cc:1196] T b1b5646b12744ac9a57c871622c35dea P e1a9251ce1cf4361b8b0ede9fbfc4023: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/b1b5646b12744ac9a57c871622c35dea
I20240503 02:00:04.159839 11575 ts_tablet_manager.cc:1909] T b1b5646b12744ac9a57c871622c35dea P e1a9251ce1cf4361b8b0ede9fbfc4023: Deleting consensus metadata
I20240503 02:00:04.160619 6301 catalog_manager.cc:4830] TS e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): tablet b1b5646b12744ac9a57c871622c35dea (table rebalancer_dynamic_table_new_name [id=9471894d04b248e9879771dd5d8e4bae]) successfully deleted
I20240503 02:00:04.164865 11576 ts_tablet_manager.cc:1875] T b1b5646b12744ac9a57c871622c35dea P 9942306855404cfd891adf227dd65eb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:04.167892 11576 ts_tablet_manager.cc:1888] T b1b5646b12744ac9a57c871622c35dea P 9942306855404cfd891adf227dd65eb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:04.168037 11576 log.cc:1196] T b1b5646b12744ac9a57c871622c35dea P 9942306855404cfd891adf227dd65eb2: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/b1b5646b12744ac9a57c871622c35dea
I20240503 02:00:04.168303 11576 ts_tablet_manager.cc:1909] T b1b5646b12744ac9a57c871622c35dea P 9942306855404cfd891adf227dd65eb2: Deleting consensus metadata
I20240503 02:00:04.169118 6303 catalog_manager.cc:4830] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211): tablet b1b5646b12744ac9a57c871622c35dea (table rebalancer_dynamic_table_new_name [id=9471894d04b248e9879771dd5d8e4bae]) successfully deleted
I20240503 02:00:04.176789 6458 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "46ad540deb624a25b97d3b967fe1f6f4"
dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58942
I20240503 02:00:04.176976 6458 raft_consensus.cc:595] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Received request to transfer leadership
I20240503 02:00:04.188778 10381 raft_consensus.cc:1050] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a: attempting to promote NON_VOTER 99ead02915954b4fab23b033ad6d546b to VOTER
I20240503 02:00:04.189431 10381 consensus_queue.cc:237] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } }
I20240503 02:00:04.191499 9039 raft_consensus.cc:1261] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 99ead02915954b4fab23b033ad6d546b [term 1 LEARNER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:04.191694 6849 raft_consensus.cc:1261] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:04.191810 6719 raft_consensus.cc:1261] T 2d7244b0dc4d419cb5c3f3677acd70c7 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:04.191965 9955 consensus_queue.cc:1035] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:04.192402 9721 consensus_queue.cc:1035] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:04.192907 9721 consensus_queue.cc:1035] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:04.195118 9955 raft_consensus.cc:2932] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 99ead02915954b4fab23b033ad6d546b (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:04.197494 9039 raft_consensus.cc:2932] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 99ead02915954b4fab23b033ad6d546b (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:04.199858 6719 raft_consensus.cc:2932] T 2d7244b0dc4d419cb5c3f3677acd70c7 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 99ead02915954b4fab23b033ad6d546b (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:04.200599 8916 raft_consensus.cc:1203] T 3b189fba0d6c461694dda002a3c5eb2f P 9942306855404cfd891adf227dd65eb2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:00:04.200937 6311 catalog_manager.cc:5484] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: config changed from index 4 to 5, 99ead02915954b4fab23b033ad6d546b (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:04.204897 6458 consensus_queue.cc:237] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } }
I20240503 02:00:04.206319 9039 raft_consensus.cc:1261] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:04.206666 6719 raft_consensus.cc:1261] T 2d7244b0dc4d419cb5c3f3677acd70c7 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:04.207185 11447 consensus_queue.cc:1035] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:04.207374 9721 consensus_queue.cc:1035] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:04.209362 9721 raft_consensus.cc:2932] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:04.210095 6719 raft_consensus.cc:2932] T 2d7244b0dc4d419cb5c3f3677acd70c7 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:04.210536 9039 raft_consensus.cc:2932] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:04.211709 6300 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2d7244b0dc4d419cb5c3f3677acd70c7 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:04.213120 6312 catalog_manager.cc:5484] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: config changed from index 5 to 6, VOTER 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:04.215145 9955 raft_consensus.cc:482] T 485833ae7cdf41ba9695f6502e1d83f7 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:04.215255 9955 raft_consensus.cc:504] T 485833ae7cdf41ba9695f6502e1d83f7 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:04.215670 9955 leader_election.cc:288] T 485833ae7cdf41ba9695f6502e1d83f7 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:04.216061 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "485833ae7cdf41ba9695f6502e1d83f7" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:04.216100 6850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "485833ae7cdf41ba9695f6502e1d83f7" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 02:00:04.216189 8916 raft_consensus.cc:2454] T 485833ae7cdf41ba9695f6502e1d83f7 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:04.216230 6850 raft_consensus.cc:2454] T 485833ae7cdf41ba9695f6502e1d83f7 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:04.216506 6394 leader_election.cc:302] T 485833ae7cdf41ba9695f6502e1d83f7 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9942306855404cfd891adf227dd65eb2, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:04.217104 6831 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d7244b0dc4d419cb5c3f3677acd70c7 with delete_type TABLET_DATA_TOMBSTONED (TS 492448570cc94ed1879feb2ec3b9051d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52812
I20240503 02:00:04.217308 9476 tablet_replica.cc:330] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 492448570cc94ed1879feb2ec3b9051d: stopping tablet replica
I20240503 02:00:04.217314 9955 raft_consensus.cc:2781] T 485833ae7cdf41ba9695f6502e1d83f7 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:04.217360 9955 raft_consensus.cc:482] T 485833ae7cdf41ba9695f6502e1d83f7 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:04.217384 9955 raft_consensus.cc:3037] T 485833ae7cdf41ba9695f6502e1d83f7 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:04.217443 9476 raft_consensus.cc:2229] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:04.217590 9476 pending_rounds.cc:62] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 492448570cc94ed1879feb2ec3b9051d: Trying to abort 1 pending ops.
I20240503 02:00:04.217615 9476 pending_rounds.cc:68] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 492448570cc94ed1879feb2ec3b9051d: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023417770757607424 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "2d7244b0dc4d419cb5c3f3677acd70c7" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } } }
I20240503 02:00:04.217748 9476 raft_consensus.cc:2894] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240503 02:00:04.217811 9476 raft_consensus.cc:2866] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 02:00:04.217850 9476 raft_consensus.cc:2258] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:04.218472 9955 raft_consensus.cc:504] T 485833ae7cdf41ba9695f6502e1d83f7 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:04.218839 9955 leader_election.cc:288] T 485833ae7cdf41ba9695f6502e1d83f7 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Requested vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:04.219473 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "485833ae7cdf41ba9695f6502e1d83f7" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:04.219606 8916 raft_consensus.cc:3037] T 485833ae7cdf41ba9695f6502e1d83f7 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:04.219945 6850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "485833ae7cdf41ba9695f6502e1d83f7" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 02:00:04.220074 6850 raft_consensus.cc:3037] T 485833ae7cdf41ba9695f6502e1d83f7 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:00:04.220317 11604 consensus_queue.cc:1159] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "492448570cc94ed1879feb2ec3b9051d" status { last_received { term: 1 index: 5 } last_committed_idx: 4 last_received_current_leader { term: 1 index: 5 } }
I20240503 02:00:04.220667 8916 raft_consensus.cc:2454] T 485833ae7cdf41ba9695f6502e1d83f7 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:04.221123 6394 leader_election.cc:302] T 485833ae7cdf41ba9695f6502e1d83f7 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9942306855404cfd891adf227dd65eb2, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:04.221279 9955 raft_consensus.cc:2781] T 485833ae7cdf41ba9695f6502e1d83f7 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:04.221366 9955 raft_consensus.cc:686] T 485833ae7cdf41ba9695f6502e1d83f7 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Becoming Leader. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Running, Role: LEADER
I20240503 02:00:04.221578 9955 consensus_queue.cc:237] T 485833ae7cdf41ba9695f6502e1d83f7 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:04.220973 6850 raft_consensus.cc:2454] T 485833ae7cdf41ba9695f6502e1d83f7 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:04.223064 9476 ts_tablet_manager.cc:1875] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 492448570cc94ed1879feb2ec3b9051d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:04.225317 9476 ts_tablet_manager.cc:1888] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 492448570cc94ed1879feb2ec3b9051d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:00:04.225415 9476 log.cc:1196] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 492448570cc94ed1879feb2ec3b9051d: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/2d7244b0dc4d419cb5c3f3677acd70c7
I20240503 02:00:04.225453 6311 catalog_manager.cc:5484] T 485833ae7cdf41ba9695f6502e1d83f7 P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: term changed from 0 to 1, leader changed from <none> to dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129). New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:04.226181 6303 catalog_manager.cc:4830] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719): tablet 2d7244b0dc4d419cb5c3f3677acd70c7 (table rebalance_test_table_17 [id=5ccf0adaf2de4b79bbf38c053646e60e]) successfully deleted
I20240503 02:00:04.251276 6312 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:00:04.256996 8880 tablet_service.cc:1467] Processing CreateTablet for tablet 478399c0161f439a9f7fd4b53d4506cb (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e49bcea404d2492ea04582ccfa57537b]), partition=
I20240503 02:00:04.257149 9159 tablet_service.cc:1467] Processing CreateTablet for tablet 478399c0161f439a9f7fd4b53d4506cb (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e49bcea404d2492ea04582ccfa57537b]), partition=
I20240503 02:00:04.257191 9286 tablet_service.cc:1467] Processing CreateTablet for tablet 478399c0161f439a9f7fd4b53d4506cb (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e49bcea404d2492ea04582ccfa57537b]), partition=
I20240503 02:00:04.257299 8880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 478399c0161f439a9f7fd4b53d4506cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:04.257355 9159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 478399c0161f439a9f7fd4b53d4506cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:04.257433 9286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 478399c0161f439a9f7fd4b53d4506cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:04.259527 11045 tablet_bootstrap.cc:492] T 478399c0161f439a9f7fd4b53d4506cb P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:04.259527 11316 tablet_bootstrap.cc:492] T 478399c0161f439a9f7fd4b53d4506cb P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:04.259843 11637 tablet_bootstrap.cc:492] T 478399c0161f439a9f7fd4b53d4506cb P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap starting.
I20240503 02:00:04.260620 11045 tablet_bootstrap.cc:654] T 478399c0161f439a9f7fd4b53d4506cb P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:04.260620 11316 tablet_bootstrap.cc:654] T 478399c0161f439a9f7fd4b53d4506cb P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:04.260775 11637 tablet_bootstrap.cc:654] T 478399c0161f439a9f7fd4b53d4506cb P e1a9251ce1cf4361b8b0ede9fbfc4023: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:04.261790 11637 tablet_bootstrap.cc:492] T 478399c0161f439a9f7fd4b53d4506cb P e1a9251ce1cf4361b8b0ede9fbfc4023: No bootstrap required, opened a new log
I20240503 02:00:04.261888 11637 ts_tablet_manager.cc:1367] T 478399c0161f439a9f7fd4b53d4506cb P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:04.261931 11316 tablet_bootstrap.cc:492] T 478399c0161f439a9f7fd4b53d4506cb P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:04.262004 11316 ts_tablet_manager.cc:1367] T 478399c0161f439a9f7fd4b53d4506cb P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:04.262022 11045 tablet_bootstrap.cc:492] T 478399c0161f439a9f7fd4b53d4506cb P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:04.262109 11045 ts_tablet_manager.cc:1367] T 478399c0161f439a9f7fd4b53d4506cb P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:04.262359 11637 raft_consensus.cc:348] T 478399c0161f439a9f7fd4b53d4506cb P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:04.262391 11316 raft_consensus.cc:348] T 478399c0161f439a9f7fd4b53d4506cb P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:04.262471 11637 raft_consensus.cc:374] T 478399c0161f439a9f7fd4b53d4506cb P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:04.262473 11316 raft_consensus.cc:374] T 478399c0161f439a9f7fd4b53d4506cb P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:04.262501 11637 raft_consensus.cc:729] T 478399c0161f439a9f7fd4b53d4506cb P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1a9251ce1cf4361b8b0ede9fbfc4023, State: Initialized, Role: FOLLOWER
I20240503 02:00:04.262513 11316 raft_consensus.cc:729] T 478399c0161f439a9f7fd4b53d4506cb P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:04.262564 11045 raft_consensus.cc:348] T 478399c0161f439a9f7fd4b53d4506cb P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:04.262624 11316 consensus_queue.cc:260] T 478399c0161f439a9f7fd4b53d4506cb P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:04.262624 11637 consensus_queue.cc:260] T 478399c0161f439a9f7fd4b53d4506cb P e1a9251ce1cf4361b8b0ede9fbfc4023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:04.262671 11045 raft_consensus.cc:374] T 478399c0161f439a9f7fd4b53d4506cb P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:04.262702 11045 raft_consensus.cc:729] T 478399c0161f439a9f7fd4b53d4506cb P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:04.262815 11045 consensus_queue.cc:260] T 478399c0161f439a9f7fd4b53d4506cb P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:04.262897 11316 ts_tablet_manager.cc:1398] T 478399c0161f439a9f7fd4b53d4506cb P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:04.262897 11637 ts_tablet_manager.cc:1398] T 478399c0161f439a9f7fd4b53d4506cb P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:04.263092 11045 ts_tablet_manager.cc:1398] T 478399c0161f439a9f7fd4b53d4506cb P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 02:00:04.270238 9955 raft_consensus.cc:979] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a: : Instructing follower 492448570cc94ed1879feb2ec3b9051d to start an election
I20240503 02:00:04.270352 9955 raft_consensus.cc:1067] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Signalling peer 492448570cc94ed1879feb2ec3b9051d to start an election
I20240503 02:00:04.270828 6849 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "46ad540deb624a25b97d3b967fe1f6f4"
dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
from {username='slave'} at 127.0.99.129:50625
I20240503 02:00:04.271020 6849 raft_consensus.cc:482] T 46ad540deb624a25b97d3b967fe1f6f4 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:04.271093 6849 raft_consensus.cc:3037] T 46ad540deb624a25b97d3b967fe1f6f4 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:04.272131 6849 raft_consensus.cc:504] T 46ad540deb624a25b97d3b967fe1f6f4 P 492448570cc94ed1879feb2ec3b9051d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } }
I20240503 02:00:04.272646 6849 leader_election.cc:288] T 46ad540deb624a25b97d3b967fe1f6f4 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 2 election: Requested vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:04.273041 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46ad540deb624a25b97d3b967fe1f6f4" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:04.273161 8916 raft_consensus.cc:3037] T 46ad540deb624a25b97d3b967fe1f6f4 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:04.273699 9309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46ad540deb624a25b97d3b967fe1f6f4" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023"
I20240503 02:00:04.273823 9309 raft_consensus.cc:3037] T 46ad540deb624a25b97d3b967fe1f6f4 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:04.274173 8916 raft_consensus.cc:2454] T 46ad540deb624a25b97d3b967fe1f6f4 P 9942306855404cfd891adf227dd65eb2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 2.
I20240503 02:00:04.274356 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46ad540deb624a25b97d3b967fe1f6f4" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 02:00:04.274477 6458 raft_consensus.cc:3032] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:00:04.274513 6458 raft_consensus.cc:729] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Running, Role: LEADER
I20240503 02:00:04.274657 6458 consensus_queue.cc:260] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } }
I20240503 02:00:04.274817 6458 raft_consensus.cc:3037] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:04.274803 9309 raft_consensus.cc:2454] T 46ad540deb624a25b97d3b967fe1f6f4 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 2.
I20240503 02:00:04.275745 6458 raft_consensus.cc:2454] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 2.
I20240503 02:00:04.276131 6785 leader_election.cc:302] T 46ad540deb624a25b97d3b967fe1f6f4 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 9942306855404cfd891adf227dd65eb2, e1a9251ce1cf4361b8b0ede9fbfc4023; no voters:
I20240503 02:00:04.276324 11376 raft_consensus.cc:2781] T 46ad540deb624a25b97d3b967fe1f6f4 P 492448570cc94ed1879feb2ec3b9051d [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:04.276427 11376 raft_consensus.cc:686] T 46ad540deb624a25b97d3b967fe1f6f4 P 492448570cc94ed1879feb2ec3b9051d [term 2 LEADER]: Becoming Leader. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Running, Role: LEADER
I20240503 02:00:04.276580 11376 consensus_queue.cc:237] T 46ad540deb624a25b97d3b967fe1f6f4 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } }
I20240503 02:00:04.278489 6312 catalog_manager.cc:5484] T 46ad540deb624a25b97d3b967fe1f6f4 P 492448570cc94ed1879feb2ec3b9051d reported cstate change: term changed from 1 to 2, leader changed from dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) to 492448570cc94ed1879feb2ec3b9051d (127.0.99.132). New cstate: current_term: 2 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:04.283094 6720 consensus_queue.cc:237] T 3dc213593e4048c99ccbe10161ff8e7f P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } }
I20240503 02:00:04.284654 6849 raft_consensus.cc:1261] T 3dc213593e4048c99ccbe10161ff8e7f P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:04.284838 6582 raft_consensus.cc:1261] T 3dc213593e4048c99ccbe10161ff8e7f P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:00:04.285197 6654 consensus_peers.cc:485] T 3dc213593e4048c99ccbe10161ff8e7f P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157): Couldn't send request to peer 99ead02915954b4fab23b033ad6d546b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3dc213593e4048c99ccbe10161ff8e7f. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:04.285316 9984 consensus_queue.cc:1035] T 3dc213593e4048c99ccbe10161ff8e7f P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:04.285818 9984 consensus_queue.cc:1035] T 3dc213593e4048c99ccbe10161ff8e7f P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:04.287303 9984 raft_consensus.cc:2932] T 3dc213593e4048c99ccbe10161ff8e7f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } } }
I20240503 02:00:04.287834 6849 raft_consensus.cc:2932] T 3dc213593e4048c99ccbe10161ff8e7f P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } } }
I20240503 02:00:04.288933 6582 raft_consensus.cc:2932] T 3dc213593e4048c99ccbe10161ff8e7f P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } } }
I20240503 02:00:04.290230 6312 catalog_manager.cc:5484] T 3dc213593e4048c99ccbe10161ff8e7f P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: config changed from index -1 to 4, NON_VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:04.300820 9179 consensus_queue.cc:237] T 0ee05e70ccbb496ca3cff08dc38112e8 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: true } }
W20240503 02:00:04.302508 9112 consensus_peers.cc:485] T 0ee05e70ccbb496ca3cff08dc38112e8 P 2a01ace80baf4057b6bf1995cf88468b -> Peer dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): Couldn't send request to peer dec3e4a04d604f7eaaa3f077d081218a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0ee05e70ccbb496ca3cff08dc38112e8. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:04.302474 6720 raft_consensus.cc:1261] T 0ee05e70ccbb496ca3cff08dc38112e8 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer 2a01ace80baf4057b6bf1995cf88468b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:04.302603 9039 raft_consensus.cc:1261] T 0ee05e70ccbb496ca3cff08dc38112e8 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer 2a01ace80baf4057b6bf1995cf88468b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:04.303010 11502 consensus_queue.cc:1035] T 0ee05e70ccbb496ca3cff08dc38112e8 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:04.303221 10679 consensus_queue.cc:1035] T 0ee05e70ccbb496ca3cff08dc38112e8 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:04.305239 10679 raft_consensus.cc:2932] T 0ee05e70ccbb496ca3cff08dc38112e8 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: true } } }
I20240503 02:00:04.305666 9039 raft_consensus.cc:2932] T 0ee05e70ccbb496ca3cff08dc38112e8 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: true } } }
I20240503 02:00:04.305666 6720 raft_consensus.cc:2932] T 0ee05e70ccbb496ca3cff08dc38112e8 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: true } } }
I20240503 02:00:04.307878 6309 catalog_manager.cc:5484] T 0ee05e70ccbb496ca3cff08dc38112e8 P 2a01ace80baf4057b6bf1995cf88468b reported cstate change: config changed from index -1 to 2, NON_VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "2a01ace80baf4057b6bf1995cf88468b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:04.316910 6720 consensus_queue.cc:237] T d897c21e630d4770832bb10f43778e82 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } }
I20240503 02:00:04.318434 6849 raft_consensus.cc:1261] T d897c21e630d4770832bb10f43778e82 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:04.318475 6458 raft_consensus.cc:1261] T d897c21e630d4770832bb10f43778e82 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:00:04.318527 6654 consensus_peers.cc:485] T d897c21e630d4770832bb10f43778e82 P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d897c21e630d4770832bb10f43778e82. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:04.318850 10268 consensus_queue.cc:1035] T d897c21e630d4770832bb10f43778e82 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:04.319056 9711 consensus_queue.cc:1035] T d897c21e630d4770832bb10f43778e82 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:04.320487 9309 raft_consensus.cc:1203] T d0d5dd42db3e4883af7fd16d8813b550 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:00:04.321192 10349 raft_consensus.cc:2932] T d897c21e630d4770832bb10f43778e82 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } } }
I20240503 02:00:04.321671 6458 raft_consensus.cc:2932] T d897c21e630d4770832bb10f43778e82 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } } }
I20240503 02:00:04.322852 6849 raft_consensus.cc:2932] T d897c21e630d4770832bb10f43778e82 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } } }
I20240503 02:00:04.325946 6309 catalog_manager.cc:5484] T d897c21e630d4770832bb10f43778e82 P 492448570cc94ed1879feb2ec3b9051d reported cstate change: config changed from index -1 to 4, NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } } }
I20240503 02:00:04.335052 6582 consensus_queue.cc:237] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } }
I20240503 02:00:04.336555 6849 raft_consensus.cc:1261] T 3416b6107e1848f299ac49ccd88ae7c8 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:04.336575 9309 raft_consensus.cc:1261] T 3416b6107e1848f299ac49ccd88ae7c8 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 02:00:04.336828 6525 consensus_peers.cc:485] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211): Couldn't send request to peer 9942306855404cfd891adf227dd65eb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3416b6107e1848f299ac49ccd88ae7c8. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:04.336972 11492 consensus_queue.cc:1035] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:04.337163 9973 consensus_queue.cc:1035] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:04.339111 9484 raft_consensus.cc:2932] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } } }
I20240503 02:00:04.339619 9309 raft_consensus.cc:2932] T 3416b6107e1848f299ac49ccd88ae7c8 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } } }
I20240503 02:00:04.339869 6849 raft_consensus.cc:2932] T 3416b6107e1848f299ac49ccd88ae7c8 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } } }
I20240503 02:00:04.342849 6312 catalog_manager.cc:5484] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: config changed from index 6 to 7, NON_VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:04.350502 6720 consensus_queue.cc:237] T 2e967e377984468d90799041c29882bd P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } }
I20240503 02:00:04.351994 6582 raft_consensus.cc:1261] T 2e967e377984468d90799041c29882bd P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:04.351972 6458 raft_consensus.cc:1261] T 2e967e377984468d90799041c29882bd P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:00:04.352332 6654 consensus_peers.cc:485] T 2e967e377984468d90799041c29882bd P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157): Couldn't send request to peer 99ead02915954b4fab23b033ad6d546b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2e967e377984468d90799041c29882bd. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:04.352450 10349 consensus_queue.cc:1035] T 2e967e377984468d90799041c29882bd P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:04.352623 9984 consensus_queue.cc:1035] T 2e967e377984468d90799041c29882bd P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:04.354329 9711 raft_consensus.cc:2932] T 2e967e377984468d90799041c29882bd P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } } }
I20240503 02:00:04.354801 6582 raft_consensus.cc:2932] T 2e967e377984468d90799041c29882bd P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } } }
I20240503 02:00:04.357105 6309 catalog_manager.cc:5484] T 2e967e377984468d90799041c29882bd P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: config changed from index -1 to 4, NON_VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:04.357169 6458 raft_consensus.cc:2932] T 2e967e377984468d90799041c29882bd P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } } }
I20240503 02:00:04.360654 11501 raft_consensus.cc:482] T 478399c0161f439a9f7fd4b53d4506cb P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:04.360782 11501 raft_consensus.cc:504] T 478399c0161f439a9f7fd4b53d4506cb P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:04.361166 11501 leader_election.cc:288] T 478399c0161f439a9f7fd4b53d4506cb P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:04.362425 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "478399c0161f439a9f7fd4b53d4506cb" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:04.362562 8916 raft_consensus.cc:2454] T 478399c0161f439a9f7fd4b53d4506cb P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:04.362658 9309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "478399c0161f439a9f7fd4b53d4506cb" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" is_pre_election: true
I20240503 02:00:04.362788 9309 raft_consensus.cc:2454] T 478399c0161f439a9f7fd4b53d4506cb P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:04.362831 9115 leader_election.cc:302] T 478399c0161f439a9f7fd4b53d4506cb P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:04.363485 11501 raft_consensus.cc:2781] T 478399c0161f439a9f7fd4b53d4506cb P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:04.363561 11501 raft_consensus.cc:482] T 478399c0161f439a9f7fd4b53d4506cb P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:04.363600 11501 raft_consensus.cc:3037] T 478399c0161f439a9f7fd4b53d4506cb P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:04.364439 11501 raft_consensus.cc:504] T 478399c0161f439a9f7fd4b53d4506cb P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:04.364849 11501 leader_election.cc:288] T 478399c0161f439a9f7fd4b53d4506cb P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Requested vote from peers e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:04.365274 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "478399c0161f439a9f7fd4b53d4506cb" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:04.365401 8916 raft_consensus.cc:3037] T 478399c0161f439a9f7fd4b53d4506cb P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:04.365458 9309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "478399c0161f439a9f7fd4b53d4506cb" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023"
I20240503 02:00:04.365572 9309 raft_consensus.cc:3037] T 478399c0161f439a9f7fd4b53d4506cb P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:04.366464 9309 raft_consensus.cc:2454] T 478399c0161f439a9f7fd4b53d4506cb P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:04.366482 8916 raft_consensus.cc:2454] T 478399c0161f439a9f7fd4b53d4506cb P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:04.366773 9113 leader_election.cc:302] T 478399c0161f439a9f7fd4b53d4506cb P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, e1a9251ce1cf4361b8b0ede9fbfc4023; no voters:
I20240503 02:00:04.367015 11501 raft_consensus.cc:2781] T 478399c0161f439a9f7fd4b53d4506cb P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:04.367089 11501 raft_consensus.cc:686] T 478399c0161f439a9f7fd4b53d4506cb P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Becoming Leader. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Running, Role: LEADER
I20240503 02:00:04.367231 11501 consensus_queue.cc:237] T 478399c0161f439a9f7fd4b53d4506cb P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:04.369210 6309 catalog_manager.cc:5484] T 478399c0161f439a9f7fd4b53d4506cb P 2a01ace80baf4057b6bf1995cf88468b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135). New cstate: current_term: 1 leader_uuid: "2a01ace80baf4057b6bf1995cf88468b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:04.386794 8916 consensus_queue.cc:237] T 067f54011a37432d99157884e70fc601 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { promote: true } }
I20240503 02:00:04.388270 9309 raft_consensus.cc:1261] T 067f54011a37432d99157884e70fc601 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:00:04.388463 8853 consensus_peers.cc:485] T 067f54011a37432d99157884e70fc601 P 9942306855404cfd891adf227dd65eb2 -> Peer 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719): Couldn't send request to peer 492448570cc94ed1879feb2ec3b9051d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 067f54011a37432d99157884e70fc601. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:04.388530 9179 raft_consensus.cc:1261] T 067f54011a37432d99157884e70fc601 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:04.388783 11073 consensus_queue.cc:1035] T 067f54011a37432d99157884e70fc601 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:04.388957 11357 consensus_queue.cc:1035] T 067f54011a37432d99157884e70fc601 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:04.391218 11357 raft_consensus.cc:2932] T 067f54011a37432d99157884e70fc601 P 9942306855404cfd891adf227dd65eb2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { promote: true } } }
I20240503 02:00:04.392768 9179 raft_consensus.cc:2932] T 067f54011a37432d99157884e70fc601 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { promote: true } } }
I20240503 02:00:04.394232 6309 catalog_manager.cc:5484] T 067f54011a37432d99157884e70fc601 P 9942306855404cfd891adf227dd65eb2 reported cstate change: config changed from index -1 to 2, NON_VOTER 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "9942306855404cfd891adf227dd65eb2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:04.407493 9310 raft_consensus.cc:2932] T 067f54011a37432d99157884e70fc601 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { promote: true } } }
I20240503 02:00:04.407835 8916 consensus_queue.cc:237] T 9c8a53980b00421d8c2bd3ae73f0c8cc P 9942306855404cfd891adf227dd65eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: true } }
W20240503 02:00:04.409837 8853 consensus_peers.cc:485] T 9c8a53980b00421d8c2bd3ae73f0c8cc P 9942306855404cfd891adf227dd65eb2 -> Peer c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655): Couldn't send request to peer c2ac2eb7cc264047b9ab32fe3a4fc025. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9c8a53980b00421d8c2bd3ae73f0c8cc. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:04.410506 6458 raft_consensus.cc:1261] T 9c8a53980b00421d8c2bd3ae73f0c8cc P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:04.410836 9039 raft_consensus.cc:1261] T 9c8a53980b00421d8c2bd3ae73f0c8cc P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:04.410974 11073 consensus_queue.cc:1035] T 9c8a53980b00421d8c2bd3ae73f0c8cc P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:04.411206 9758 consensus_queue.cc:1035] T 9c8a53980b00421d8c2bd3ae73f0c8cc P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:04.413542 11357 raft_consensus.cc:2932] T 9c8a53980b00421d8c2bd3ae73f0c8cc P 9942306855404cfd891adf227dd65eb2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: true } } }
I20240503 02:00:04.416046 6458 raft_consensus.cc:2932] T 9c8a53980b00421d8c2bd3ae73f0c8cc P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: true } } }
I20240503 02:00:04.417640 11515 consensus_queue.cc:1035] T e52373099af3428497d176e3fcd8ef27 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:04.418879 6314 catalog_manager.cc:5484] T 9c8a53980b00421d8c2bd3ae73f0c8cc P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: config changed from index -1 to 2, NON_VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "9942306855404cfd891adf227dd65eb2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: true } } }
I20240503 02:00:04.420125 9039 raft_consensus.cc:2932] T 9c8a53980b00421d8c2bd3ae73f0c8cc P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: true } } }
I20240503 02:00:04.421967 11515 consensus_queue.cc:1035] T e52373099af3428497d176e3fcd8ef27 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:04.434546 11515 consensus_queue.cc:1035] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:04.438493 11514 consensus_queue.cc:1035] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:04.470645 11740 ts_tablet_manager.cc:902] T 7df4c083550046e883e5e61d3d4e1572 P 2a01ace80baf4057b6bf1995cf88468b: Initiating tablet copy from peer 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:04.471361 11740 tablet_copy_client.cc:308] T 7df4c083550046e883e5e61d3d4e1572 P 2a01ace80baf4057b6bf1995cf88468b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43863
I20240503 02:00:04.471843 6609 tablet_copy_service.cc:140] P 80fd18ef4d604c86b909d5c008a46b4f: Received BeginTabletCopySession request for tablet 7df4c083550046e883e5e61d3d4e1572 from peer 2a01ace80baf4057b6bf1995cf88468b ({username='slave'} at 127.0.99.135:40117)
I20240503 02:00:04.471957 6609 tablet_copy_service.cc:161] P 80fd18ef4d604c86b909d5c008a46b4f: Beginning new tablet copy session on tablet 7df4c083550046e883e5e61d3d4e1572 from peer 2a01ace80baf4057b6bf1995cf88468b at {username='slave'} at 127.0.99.135:40117: session id = 2a01ace80baf4057b6bf1995cf88468b-7df4c083550046e883e5e61d3d4e1572
I20240503 02:00:04.472998 6609 tablet_copy_source_session.cc:205] T 7df4c083550046e883e5e61d3d4e1572 P 80fd18ef4d604c86b909d5c008a46b4f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:04.473886 11740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7df4c083550046e883e5e61d3d4e1572. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:04.476128 11740 tablet_copy_client.cc:783] T 7df4c083550046e883e5e61d3d4e1572 P 2a01ace80baf4057b6bf1995cf88468b: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:04.476284 11740 tablet_copy_client.cc:647] T 7df4c083550046e883e5e61d3d4e1572 P 2a01ace80baf4057b6bf1995cf88468b: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:04.477535 11740 tablet_copy_client.cc:523] T 7df4c083550046e883e5e61d3d4e1572 P 2a01ace80baf4057b6bf1995cf88468b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:04.478951 11740 tablet_bootstrap.cc:492] T 7df4c083550046e883e5e61d3d4e1572 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:04.491303 6309 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:04.495687 8916 raft_consensus.cc:1261] T 478399c0161f439a9f7fd4b53d4506cb P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Refusing update from remote peer 2a01ace80baf4057b6bf1995cf88468b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:04.495796 9310 raft_consensus.cc:1261] T 478399c0161f439a9f7fd4b53d4506cb P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Refusing update from remote peer 2a01ace80baf4057b6bf1995cf88468b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:04.496187 11502 consensus_queue.cc:1035] T 478399c0161f439a9f7fd4b53d4506cb P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:04.496973 11502 consensus_queue.cc:1035] T 478399c0161f439a9f7fd4b53d4506cb P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:04.498898 6314 catalog_manager.cc:5576] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211) does not have the latest schema for tablet 478399c0161f439a9f7fd4b53d4506cb (table rebalancer_dynamic_table [id=e49bcea404d2492ea04582ccfa57537b]). Expected version 1 got 0
I20240503 02:00:04.499625 6309 catalog_manager.cc:5576] TS e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311) does not have the latest schema for tablet 478399c0161f439a9f7fd4b53d4506cb (table rebalancer_dynamic_table [id=e49bcea404d2492ea04582ccfa57537b]). Expected version 1 got 0
I20240503 02:00:04.502007 11064 tablet.cc:1722] T 478399c0161f439a9f7fd4b53d4506cb P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:04.502357 11064 tablet.cc:1620] T 478399c0161f439a9f7fd4b53d4506cb P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
I20240503 02:00:04.502806 11062 tablet.cc:1722] T 478399c0161f439a9f7fd4b53d4506cb P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:04.503103 11062 tablet.cc:1620] T 478399c0161f439a9f7fd4b53d4506cb P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
I20240503 02:00:04.505244 11740 tablet_bootstrap.cc:492] T 7df4c083550046e883e5e61d3d4e1572 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:04.505509 11740 tablet_bootstrap.cc:492] T 7df4c083550046e883e5e61d3d4e1572 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap complete.
I20240503 02:00:04.505638 8917 raft_consensus.cc:1203] T 485d1d89a8b94505969d663be92124cd P 9942306855404cfd891adf227dd65eb2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 02:00:04.505673 11740 ts_tablet_manager.cc:1367] T 7df4c083550046e883e5e61d3d4e1572 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.027s user 0.021s sys 0.005s
I20240503 02:00:04.506096 11740 raft_consensus.cc:348] T 7df4c083550046e883e5e61d3d4e1572 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } }
I20240503 02:00:04.506220 11740 raft_consensus.cc:729] T 7df4c083550046e883e5e61d3d4e1572 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: LEARNER
I20240503 02:00:04.506314 11740 consensus_queue.cc:260] T 7df4c083550046e883e5e61d3d4e1572 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } }
I20240503 02:00:04.506624 11740 ts_tablet_manager.cc:1398] T 7df4c083550046e883e5e61d3d4e1572 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:04.507162 6609 tablet_copy_service.cc:342] P 80fd18ef4d604c86b909d5c008a46b4f: Request end of tablet copy session 2a01ace80baf4057b6bf1995cf88468b-7df4c083550046e883e5e61d3d4e1572 received from {username='slave'} at 127.0.99.135:40117
I20240503 02:00:04.507254 6609 tablet_copy_service.cc:433] P 80fd18ef4d604c86b909d5c008a46b4f: ending tablet copy session 2a01ace80baf4057b6bf1995cf88468b-7df4c083550046e883e5e61d3d4e1572 on tablet 7df4c083550046e883e5e61d3d4e1572 with peer 2a01ace80baf4057b6bf1995cf88468b
I20240503 02:00:04.507738 11437 tablet.cc:1722] T 478399c0161f439a9f7fd4b53d4506cb P e1a9251ce1cf4361b8b0ede9fbfc4023: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:04.508064 11437 tablet.cc:1620] T 478399c0161f439a9f7fd4b53d4506cb P e1a9251ce1cf4361b8b0ede9fbfc4023: MemRowSet was empty: no flush needed.
I20240503 02:00:04.509541 6314 catalog_manager.cc:5576] TS e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311) does not have the latest schema for tablet 478399c0161f439a9f7fd4b53d4506cb (table rebalancer_dynamic_table [id=e49bcea404d2492ea04582ccfa57537b]). Expected version 1 got 0
I20240503 02:00:04.509701 6309 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=e49bcea404d2492ea04582ccfa57537b] alter complete (version 1)
I20240503 02:00:04.531471 6582 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c3485701bf3b48f5ae2e1568afeed133"
dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49872
I20240503 02:00:04.531626 6582 raft_consensus.cc:595] T c3485701bf3b48f5ae2e1568afeed133 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Received request to transfer leadership
I20240503 02:00:04.534840 11492 raft_consensus.cc:1050] T cb873018448549ccbe3d14b784e47d26 P 80fd18ef4d604c86b909d5c008a46b4f: attempting to promote NON_VOTER 492448570cc94ed1879feb2ec3b9051d to VOTER
I20240503 02:00:04.535300 11492 consensus_queue.cc:237] T cb873018448549ccbe3d14b784e47d26 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { promote: false } }
I20240503 02:00:04.536698 8917 raft_consensus.cc:1261] T cb873018448549ccbe3d14b784e47d26 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:04.537230 6849 raft_consensus.cc:1261] T cb873018448549ccbe3d14b784e47d26 P 492448570cc94ed1879feb2ec3b9051d [term 1 LEARNER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:04.537425 9973 consensus_queue.cc:1035] T cb873018448549ccbe3d14b784e47d26 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:04.537603 10345 consensus_queue.cc:1035] T cb873018448549ccbe3d14b784e47d26 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:04.539113 11795 ts_tablet_manager.cc:902] T 51b82ef8e4cc474387f6902f2e526ac0 P c2ac2eb7cc264047b9ab32fe3a4fc025: Initiating tablet copy from peer 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:04.539175 9039 raft_consensus.cc:1261] T cb873018448549ccbe3d14b784e47d26 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:04.539613 9973 consensus_queue.cc:1035] T cb873018448549ccbe3d14b784e47d26 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:04.540802 11795 tablet_copy_client.cc:308] T 51b82ef8e4cc474387f6902f2e526ac0 P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:34079
I20240503 02:00:04.541896 10278 raft_consensus.cc:2932] T cb873018448549ccbe3d14b784e47d26 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { promote: false } } }
I20240503 02:00:04.542201 9199 tablet_copy_service.cc:140] P 2a01ace80baf4057b6bf1995cf88468b: Received BeginTabletCopySession request for tablet 51b82ef8e4cc474387f6902f2e526ac0 from peer c2ac2eb7cc264047b9ab32fe3a4fc025 ({username='slave'} at 127.0.99.131:49829)
I20240503 02:00:04.542322 9199 tablet_copy_service.cc:161] P 2a01ace80baf4057b6bf1995cf88468b: Beginning new tablet copy session on tablet 51b82ef8e4cc474387f6902f2e526ac0 from peer c2ac2eb7cc264047b9ab32fe3a4fc025 at {username='slave'} at 127.0.99.131:49829: session id = c2ac2eb7cc264047b9ab32fe3a4fc025-51b82ef8e4cc474387f6902f2e526ac0
I20240503 02:00:04.542460 6849 raft_consensus.cc:2932] T cb873018448549ccbe3d14b784e47d26 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { promote: false } } }
I20240503 02:00:04.543236 9199 tablet_copy_source_session.cc:205] T 51b82ef8e4cc474387f6902f2e526ac0 P 2a01ace80baf4057b6bf1995cf88468b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:04.543859 8917 raft_consensus.cc:2932] T cb873018448549ccbe3d14b784e47d26 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { promote: false } } }
I20240503 02:00:04.544725 9039 raft_consensus.cc:2932] T cb873018448549ccbe3d14b784e47d26 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { promote: false } } }
I20240503 02:00:04.544991 6309 catalog_manager.cc:5484] T cb873018448549ccbe3d14b784e47d26 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: config changed from index 2 to 3, 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:04.545643 11795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51b82ef8e4cc474387f6902f2e526ac0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:04.548502 11795 tablet_copy_client.cc:783] T 51b82ef8e4cc474387f6902f2e526ac0 P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:04.548741 11795 tablet_copy_client.cc:647] T 51b82ef8e4cc474387f6902f2e526ac0 P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:04.550158 11795 tablet_copy_client.cc:523] T 51b82ef8e4cc474387f6902f2e526ac0 P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:04.551937 11795 tablet_bootstrap.cc:492] T 51b82ef8e4cc474387f6902f2e526ac0 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:04.566490 11795 tablet_bootstrap.cc:492] T 51b82ef8e4cc474387f6902f2e526ac0 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:04.566799 11795 tablet_bootstrap.cc:492] T 51b82ef8e4cc474387f6902f2e526ac0 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap complete.
I20240503 02:00:04.566992 11795 ts_tablet_manager.cc:1367] T 51b82ef8e4cc474387f6902f2e526ac0 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.000s
I20240503 02:00:04.567476 11795 raft_consensus.cc:348] T 51b82ef8e4cc474387f6902f2e526ac0 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: true } }
I20240503 02:00:04.567603 11795 raft_consensus.cc:729] T 51b82ef8e4cc474387f6902f2e526ac0 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: LEARNER
I20240503 02:00:04.567700 11795 consensus_queue.cc:260] T 51b82ef8e4cc474387f6902f2e526ac0 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: true } }
I20240503 02:00:04.567967 11795 ts_tablet_manager.cc:1398] T 51b82ef8e4cc474387f6902f2e526ac0 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:04.568590 9199 tablet_copy_service.cc:342] P 2a01ace80baf4057b6bf1995cf88468b: Request end of tablet copy session c2ac2eb7cc264047b9ab32fe3a4fc025-51b82ef8e4cc474387f6902f2e526ac0 received from {username='slave'} at 127.0.99.131:49829
I20240503 02:00:04.568676 9199 tablet_copy_service.cc:433] P 2a01ace80baf4057b6bf1995cf88468b: ending tablet copy session c2ac2eb7cc264047b9ab32fe3a4fc025-51b82ef8e4cc474387f6902f2e526ac0 on tablet 51b82ef8e4cc474387f6902f2e526ac0 with peer c2ac2eb7cc264047b9ab32fe3a4fc025
I20240503 02:00:04.576314 6458 raft_consensus.cc:1203] T 96c395771fd44056abacceb6c1acdb9f P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:00:04.584887 11740 ts_tablet_manager.cc:902] T 1fa274c98fc147f29344ed537e2ccf17 P 2a01ace80baf4057b6bf1995cf88468b: Initiating tablet copy from peer 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:04.585372 11740 tablet_copy_client.cc:308] T 1fa274c98fc147f29344ed537e2ccf17 P 2a01ace80baf4057b6bf1995cf88468b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:42719
I20240503 02:00:04.585878 6871 tablet_copy_service.cc:140] P 492448570cc94ed1879feb2ec3b9051d: Received BeginTabletCopySession request for tablet 1fa274c98fc147f29344ed537e2ccf17 from peer 2a01ace80baf4057b6bf1995cf88468b ({username='slave'} at 127.0.99.135:38375)
I20240503 02:00:04.586004 6871 tablet_copy_service.cc:161] P 492448570cc94ed1879feb2ec3b9051d: Beginning new tablet copy session on tablet 1fa274c98fc147f29344ed537e2ccf17 from peer 2a01ace80baf4057b6bf1995cf88468b at {username='slave'} at 127.0.99.135:38375: session id = 2a01ace80baf4057b6bf1995cf88468b-1fa274c98fc147f29344ed537e2ccf17
I20240503 02:00:04.586830 6871 tablet_copy_source_session.cc:205] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:04.587741 11740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fa274c98fc147f29344ed537e2ccf17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:04.589955 11740 tablet_copy_client.cc:783] T 1fa274c98fc147f29344ed537e2ccf17 P 2a01ace80baf4057b6bf1995cf88468b: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:04.590114 11740 tablet_copy_client.cc:647] T 1fa274c98fc147f29344ed537e2ccf17 P 2a01ace80baf4057b6bf1995cf88468b: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:04.591336 11740 tablet_copy_client.cc:523] T 1fa274c98fc147f29344ed537e2ccf17 P 2a01ace80baf4057b6bf1995cf88468b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:04.592438 11740 tablet_bootstrap.cc:492] T 1fa274c98fc147f29344ed537e2ccf17 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:04.608399 9484 raft_consensus.cc:979] T c3485701bf3b48f5ae2e1568afeed133 P 80fd18ef4d604c86b909d5c008a46b4f: : Instructing follower 492448570cc94ed1879feb2ec3b9051d to start an election
I20240503 02:00:04.608527 9484 raft_consensus.cc:1067] T c3485701bf3b48f5ae2e1568afeed133 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Signalling peer 492448570cc94ed1879feb2ec3b9051d to start an election
I20240503 02:00:04.609120 6849 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c3485701bf3b48f5ae2e1568afeed133"
dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
from {username='slave'} at 127.0.99.130:55559
I20240503 02:00:04.609279 6849 raft_consensus.cc:482] T c3485701bf3b48f5ae2e1568afeed133 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:04.609342 6849 raft_consensus.cc:3037] T c3485701bf3b48f5ae2e1568afeed133 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:04.610199 6849 raft_consensus.cc:504] T c3485701bf3b48f5ae2e1568afeed133 P 492448570cc94ed1879feb2ec3b9051d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } }
I20240503 02:00:04.610412 11740 tablet_bootstrap.cc:492] T 1fa274c98fc147f29344ed537e2ccf17 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:04.610616 6849 leader_election.cc:288] T c3485701bf3b48f5ae2e1568afeed133 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 2 election: Requested vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311)
I20240503 02:00:04.610661 11740 tablet_bootstrap.cc:492] T 1fa274c98fc147f29344ed537e2ccf17 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap complete.
I20240503 02:00:04.610836 11740 ts_tablet_manager.cc:1367] T 1fa274c98fc147f29344ed537e2ccf17 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.018s user 0.017s sys 0.000s
I20240503 02:00:04.611090 9310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3485701bf3b48f5ae2e1568afeed133" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023"
I20240503 02:00:04.611091 6720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3485701bf3b48f5ae2e1568afeed133" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 02:00:04.611217 9310 raft_consensus.cc:3037] T c3485701bf3b48f5ae2e1568afeed133 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:04.611218 6720 raft_consensus.cc:3037] T c3485701bf3b48f5ae2e1568afeed133 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:04.611343 11740 raft_consensus.cc:348] T 1fa274c98fc147f29344ed537e2ccf17 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } }
I20240503 02:00:04.611449 11740 raft_consensus.cc:729] T 1fa274c98fc147f29344ed537e2ccf17 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: LEARNER
I20240503 02:00:04.611541 11740 consensus_queue.cc:260] T 1fa274c98fc147f29344ed537e2ccf17 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } }
I20240503 02:00:04.611790 11740 ts_tablet_manager.cc:1398] T 1fa274c98fc147f29344ed537e2ccf17 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:04.611827 6582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3485701bf3b48f5ae2e1568afeed133" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 02:00:04.611969 6582 raft_consensus.cc:3032] T c3485701bf3b48f5ae2e1568afeed133 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:00:04.612005 6582 raft_consensus.cc:729] T c3485701bf3b48f5ae2e1568afeed133 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Running, Role: LEADER
I20240503 02:00:04.612215 9310 raft_consensus.cc:2454] T c3485701bf3b48f5ae2e1568afeed133 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 2.
I20240503 02:00:04.612212 6582 consensus_queue.cc:260] T c3485701bf3b48f5ae2e1568afeed133 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } }
I20240503 02:00:04.612349 6720 raft_consensus.cc:2454] T c3485701bf3b48f5ae2e1568afeed133 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 2.
I20240503 02:00:04.612435 6582 raft_consensus.cc:3037] T c3485701bf3b48f5ae2e1568afeed133 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:04.613097 9310 raft_consensus.cc:1226] T c3485701bf3b48f5ae2e1568afeed133 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 2 FOLLOWER]: Rejecting Update request from peer 80fd18ef4d604c86b909d5c008a46b4f for earlier term 1. Current term is 2. Ops: []
I20240503 02:00:04.613265 6309 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:04.613454 6582 raft_consensus.cc:2454] T c3485701bf3b48f5ae2e1568afeed133 P 80fd18ef4d604c86b909d5c008a46b4f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 2.
I20240503 02:00:04.613596 6787 leader_election.cc:302] T c3485701bf3b48f5ae2e1568afeed133 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, c2ac2eb7cc264047b9ab32fe3a4fc025, e1a9251ce1cf4361b8b0ede9fbfc4023; no voters:
I20240503 02:00:04.613778 11376 raft_consensus.cc:2781] T c3485701bf3b48f5ae2e1568afeed133 P 492448570cc94ed1879feb2ec3b9051d [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:04.613855 11376 raft_consensus.cc:686] T c3485701bf3b48f5ae2e1568afeed133 P 492448570cc94ed1879feb2ec3b9051d [term 2 LEADER]: Becoming Leader. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Running, Role: LEADER
I20240503 02:00:04.613983 11376 consensus_queue.cc:237] T c3485701bf3b48f5ae2e1568afeed133 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } }
I20240503 02:00:04.614809 6871 tablet_copy_service.cc:342] P 492448570cc94ed1879feb2ec3b9051d: Request end of tablet copy session 2a01ace80baf4057b6bf1995cf88468b-1fa274c98fc147f29344ed537e2ccf17 received from {username='slave'} at 127.0.99.135:38375
I20240503 02:00:04.614877 6871 tablet_copy_service.cc:433] P 492448570cc94ed1879feb2ec3b9051d: ending tablet copy session 2a01ace80baf4057b6bf1995cf88468b-1fa274c98fc147f29344ed537e2ccf17 on tablet 1fa274c98fc147f29344ed537e2ccf17 with peer 2a01ace80baf4057b6bf1995cf88468b
I20240503 02:00:04.617997 6309 catalog_manager.cc:5484] T c3485701bf3b48f5ae2e1568afeed133 P 492448570cc94ed1879feb2ec3b9051d reported cstate change: term changed from 1 to 2, leader changed from 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) to 492448570cc94ed1879feb2ec3b9051d (127.0.99.132). New cstate: current_term: 2 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:04.619048 11064 tablet.cc:1722] T 478399c0161f439a9f7fd4b53d4506cb P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:04.619415 11064 tablet.cc:1620] T 478399c0161f439a9f7fd4b53d4506cb P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
I20240503 02:00:04.619823 11062 tablet.cc:1722] T 478399c0161f439a9f7fd4b53d4506cb P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:04.620119 11062 tablet.cc:1620] T 478399c0161f439a9f7fd4b53d4506cb P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
I20240503 02:00:04.622416 11437 tablet.cc:1722] T 478399c0161f439a9f7fd4b53d4506cb P e1a9251ce1cf4361b8b0ede9fbfc4023: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:04.622745 11437 tablet.cc:1620] T 478399c0161f439a9f7fd4b53d4506cb P e1a9251ce1cf4361b8b0ede9fbfc4023: MemRowSet was empty: no flush needed.
I20240503 02:00:04.624980 6309 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=e49bcea404d2492ea04582ccfa57537b] alter complete (version 2)
I20240503 02:00:04.665347 6458 raft_consensus.cc:1261] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a [term 2 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:00:04.665810 11828 ts_tablet_manager.cc:902] T 3dc213593e4048c99ccbe10161ff8e7f P 99ead02915954b4fab23b033ad6d546b: Initiating tablet copy from peer c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:04.665824 11376 consensus_queue.cc:1035] T 46ad540deb624a25b97d3b967fe1f6f4 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:04.666577 11828 tablet_copy_client.cc:308] T 3dc213593e4048c99ccbe10161ff8e7f P 99ead02915954b4fab23b033ad6d546b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:42655
I20240503 02:00:04.667424 6739 tablet_copy_service.cc:140] P c2ac2eb7cc264047b9ab32fe3a4fc025: Received BeginTabletCopySession request for tablet 3dc213593e4048c99ccbe10161ff8e7f from peer 99ead02915954b4fab23b033ad6d546b ({username='slave'} at 127.0.99.134:48173)
I20240503 02:00:04.667537 6739 tablet_copy_service.cc:161] P c2ac2eb7cc264047b9ab32fe3a4fc025: Beginning new tablet copy session on tablet 3dc213593e4048c99ccbe10161ff8e7f from peer 99ead02915954b4fab23b033ad6d546b at {username='slave'} at 127.0.99.134:48173: session id = 99ead02915954b4fab23b033ad6d546b-3dc213593e4048c99ccbe10161ff8e7f
I20240503 02:00:04.668367 6739 tablet_copy_source_session.cc:205] T 3dc213593e4048c99ccbe10161ff8e7f P c2ac2eb7cc264047b9ab32fe3a4fc025: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:04.669417 11828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dc213593e4048c99ccbe10161ff8e7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:04.671626 11828 tablet_copy_client.cc:783] T 3dc213593e4048c99ccbe10161ff8e7f P 99ead02915954b4fab23b033ad6d546b: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:04.671850 11828 tablet_copy_client.cc:647] T 3dc213593e4048c99ccbe10161ff8e7f P 99ead02915954b4fab23b033ad6d546b: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:04.673133 11828 tablet_copy_client.cc:523] T 3dc213593e4048c99ccbe10161ff8e7f P 99ead02915954b4fab23b033ad6d546b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:04.674674 11828 tablet_bootstrap.cc:492] T 3dc213593e4048c99ccbe10161ff8e7f P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:04.693730 11828 tablet_bootstrap.cc:492] T 3dc213593e4048c99ccbe10161ff8e7f P 99ead02915954b4fab23b033ad6d546b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:04.694031 11828 tablet_bootstrap.cc:492] T 3dc213593e4048c99ccbe10161ff8e7f P 99ead02915954b4fab23b033ad6d546b: Bootstrap complete.
I20240503 02:00:04.694231 11828 ts_tablet_manager.cc:1367] T 3dc213593e4048c99ccbe10161ff8e7f P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.020s user 0.019s sys 0.000s
I20240503 02:00:04.694698 11828 raft_consensus.cc:348] T 3dc213593e4048c99ccbe10161ff8e7f P 99ead02915954b4fab23b033ad6d546b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } }
I20240503 02:00:04.694813 11828 raft_consensus.cc:729] T 3dc213593e4048c99ccbe10161ff8e7f P 99ead02915954b4fab23b033ad6d546b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: LEARNER
I20240503 02:00:04.694900 11828 consensus_queue.cc:260] T 3dc213593e4048c99ccbe10161ff8e7f P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } }
I20240503 02:00:04.695223 11828 ts_tablet_manager.cc:1398] T 3dc213593e4048c99ccbe10161ff8e7f P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:04.695896 6739 tablet_copy_service.cc:342] P c2ac2eb7cc264047b9ab32fe3a4fc025: Request end of tablet copy session 99ead02915954b4fab23b033ad6d546b-3dc213593e4048c99ccbe10161ff8e7f received from {username='slave'} at 127.0.99.134:48173
I20240503 02:00:04.695993 6739 tablet_copy_service.cc:433] P c2ac2eb7cc264047b9ab32fe3a4fc025: ending tablet copy session 99ead02915954b4fab23b033ad6d546b-3dc213593e4048c99ccbe10161ff8e7f on tablet 3dc213593e4048c99ccbe10161ff8e7f with peer 99ead02915954b4fab23b033ad6d546b
I20240503 02:00:04.722208 11834 ts_tablet_manager.cc:902] T d897c21e630d4770832bb10f43778e82 P e1a9251ce1cf4361b8b0ede9fbfc4023: Initiating tablet copy from peer c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:04.722762 11834 tablet_copy_client.cc:308] T d897c21e630d4770832bb10f43778e82 P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:42655
I20240503 02:00:04.725093 11447 raft_consensus.cc:1050] T 3b189fba0d6c461694dda002a3c5eb2f P dec3e4a04d604f7eaaa3f077d081218a: attempting to promote NON_VOTER 9942306855404cfd891adf227dd65eb2 to VOTER
I20240503 02:00:04.725493 11447 consensus_queue.cc:237] T 3b189fba0d6c461694dda002a3c5eb2f P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } }
I20240503 02:00:04.727253 6849 raft_consensus.cc:1261] T 3b189fba0d6c461694dda002a3c5eb2f P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:04.727339 8917 raft_consensus.cc:1261] T 3b189fba0d6c461694dda002a3c5eb2f P 9942306855404cfd891adf227dd65eb2 [term 1 LEARNER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:04.727427 6720 raft_consensus.cc:1261] T 3b189fba0d6c461694dda002a3c5eb2f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:04.727567 6739 tablet_copy_service.cc:140] P c2ac2eb7cc264047b9ab32fe3a4fc025: Received BeginTabletCopySession request for tablet d897c21e630d4770832bb10f43778e82 from peer e1a9251ce1cf4361b8b0ede9fbfc4023 ({username='slave'} at 127.0.99.136:40237)
I20240503 02:00:04.727666 6739 tablet_copy_service.cc:161] P c2ac2eb7cc264047b9ab32fe3a4fc025: Beginning new tablet copy session on tablet d897c21e630d4770832bb10f43778e82 from peer e1a9251ce1cf4361b8b0ede9fbfc4023 at {username='slave'} at 127.0.99.136:40237: session id = e1a9251ce1cf4361b8b0ede9fbfc4023-d897c21e630d4770832bb10f43778e82
I20240503 02:00:04.727989 9955 consensus_queue.cc:1035] T 3b189fba0d6c461694dda002a3c5eb2f P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:04.728078 6314 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:04.728741 9955 consensus_queue.cc:1035] T 3b189fba0d6c461694dda002a3c5eb2f P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:04.729126 6739 tablet_copy_source_session.cc:205] T d897c21e630d4770832bb10f43778e82 P c2ac2eb7cc264047b9ab32fe3a4fc025: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:04.730216 9955 consensus_queue.cc:1035] T 3b189fba0d6c461694dda002a3c5eb2f P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:04.730533 11834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d897c21e630d4770832bb10f43778e82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:04.732998 11834 tablet_copy_client.cc:783] T d897c21e630d4770832bb10f43778e82 P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:04.733158 11834 tablet_copy_client.cc:647] T d897c21e630d4770832bb10f43778e82 P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:04.733242 9955 raft_consensus.cc:2932] T 3b189fba0d6c461694dda002a3c5eb2f P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:04.734609 11834 tablet_copy_client.cc:523] T d897c21e630d4770832bb10f43778e82 P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:04.735107 6720 raft_consensus.cc:2932] T 3b189fba0d6c461694dda002a3c5eb2f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:04.735589 11064 tablet.cc:1722] T 478399c0161f439a9f7fd4b53d4506cb P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:04.736271 11502 raft_consensus.cc:1050] T f5ed01740019467d8c1d55cc2fa18ca3 P 2a01ace80baf4057b6bf1995cf88468b: attempting to promote NON_VOTER 80fd18ef4d604c86b909d5c008a46b4f to VOTER
I20240503 02:00:04.736796 11502 consensus_queue.cc:237] T f5ed01740019467d8c1d55cc2fa18ca3 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } }
I20240503 02:00:04.737169 11834 tablet_bootstrap.cc:492] T d897c21e630d4770832bb10f43778e82 P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap starting.
I20240503 02:00:04.738461 6582 raft_consensus.cc:1261] T f5ed01740019467d8c1d55cc2fa18ca3 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEARNER]: Refusing update from remote peer 2a01ace80baf4057b6bf1995cf88468b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:04.738551 11437 tablet.cc:1722] T 478399c0161f439a9f7fd4b53d4506cb P e1a9251ce1cf4361b8b0ede9fbfc4023: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:04.738898 11501 consensus_queue.cc:1035] T f5ed01740019467d8c1d55cc2fa18ca3 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:04.739060 6314 catalog_manager.cc:5484] T 3b189fba0d6c461694dda002a3c5eb2f P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: config changed from index 4 to 5, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:04.740605 9039 raft_consensus.cc:1261] T f5ed01740019467d8c1d55cc2fa18ca3 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer 2a01ace80baf4057b6bf1995cf88468b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:04.742581 10679 consensus_queue.cc:1035] T f5ed01740019467d8c1d55cc2fa18ca3 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:04.743307 8916 raft_consensus.cc:1261] T f5ed01740019467d8c1d55cc2fa18ca3 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Refusing update from remote peer 2a01ace80baf4057b6bf1995cf88468b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:04.744433 11504 consensus_queue.cc:1035] T f5ed01740019467d8c1d55cc2fa18ca3 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:04.745150 11062 tablet.cc:1722] T 478399c0161f439a9f7fd4b53d4506cb P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:04.745723 6312 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=e49bcea404d2492ea04582ccfa57537b] alter complete (version 3)
I20240503 02:00:04.747571 11502 raft_consensus.cc:2932] T f5ed01740019467d8c1d55cc2fa18ca3 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:04.748430 9039 raft_consensus.cc:2932] T f5ed01740019467d8c1d55cc2fa18ca3 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:04.750659 6849 raft_consensus.cc:2932] T 3b189fba0d6c461694dda002a3c5eb2f P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:04.751322 6582 raft_consensus.cc:2932] T f5ed01740019467d8c1d55cc2fa18ca3 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:04.751437 8917 raft_consensus.cc:2932] T f5ed01740019467d8c1d55cc2fa18ca3 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:04.754555 6309 catalog_manager.cc:5484] T f5ed01740019467d8c1d55cc2fa18ca3 P 2a01ace80baf4057b6bf1995cf88468b reported cstate change: config changed from index 2 to 3, 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2a01ace80baf4057b6bf1995cf88468b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:04.756086 6458 consensus_queue.cc:237] T 3b189fba0d6c461694dda002a3c5eb2f P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } }
I20240503 02:00:04.756983 8917 raft_consensus.cc:2932] T 3b189fba0d6c461694dda002a3c5eb2f P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:04.758592 6720 raft_consensus.cc:1261] T 3b189fba0d6c461694dda002a3c5eb2f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:04.759181 11841 consensus_queue.cc:1035] T 3b189fba0d6c461694dda002a3c5eb2f P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:04.759451 8916 raft_consensus.cc:1261] T 3b189fba0d6c461694dda002a3c5eb2f P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:04.760208 11842 consensus_queue.cc:1035] T 3b189fba0d6c461694dda002a3c5eb2f P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:04.761788 9179 consensus_queue.cc:237] T f5ed01740019467d8c1d55cc2fa18ca3 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } }
I20240503 02:00:04.763509 6582 raft_consensus.cc:1261] T f5ed01740019467d8c1d55cc2fa18ca3 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer 2a01ace80baf4057b6bf1995cf88468b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:04.763952 11504 consensus_queue.cc:1035] T f5ed01740019467d8c1d55cc2fa18ca3 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:04.764631 9955 raft_consensus.cc:2932] T 3b189fba0d6c461694dda002a3c5eb2f P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:04.765339 8917 raft_consensus.cc:1261] T f5ed01740019467d8c1d55cc2fa18ca3 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Refusing update from remote peer 2a01ace80baf4057b6bf1995cf88468b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:04.765341 6720 raft_consensus.cc:2932] T 3b189fba0d6c461694dda002a3c5eb2f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:04.765771 11504 consensus_queue.cc:1035] T f5ed01740019467d8c1d55cc2fa18ca3 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:04.768311 6314 catalog_manager.cc:5484] T 3b189fba0d6c461694dda002a3c5eb2f P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: config changed from index 5 to 6, VOTER 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:04.769604 6300 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3b189fba0d6c461694dda002a3c5eb2f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:04.772326 6831 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b189fba0d6c461694dda002a3c5eb2f with delete_type TABLET_DATA_TOMBSTONED (TS 492448570cc94ed1879feb2ec3b9051d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52812
I20240503 02:00:04.772922 11853 tablet_replica.cc:330] T 3b189fba0d6c461694dda002a3c5eb2f P 492448570cc94ed1879feb2ec3b9051d: stopping tablet replica
I20240503 02:00:04.774317 11853 raft_consensus.cc:2229] T 3b189fba0d6c461694dda002a3c5eb2f P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:04.774571 11853 raft_consensus.cc:2258] T 3b189fba0d6c461694dda002a3c5eb2f P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:04.775373 11853 ts_tablet_manager.cc:1875] T 3b189fba0d6c461694dda002a3c5eb2f P 492448570cc94ed1879feb2ec3b9051d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:04.778321 11853 ts_tablet_manager.cc:1888] T 3b189fba0d6c461694dda002a3c5eb2f P 492448570cc94ed1879feb2ec3b9051d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:00:04.778278 8916 raft_consensus.cc:2932] T 3b189fba0d6c461694dda002a3c5eb2f P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:04.778719 11853 log.cc:1196] T 3b189fba0d6c461694dda002a3c5eb2f P 492448570cc94ed1879feb2ec3b9051d: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/3b189fba0d6c461694dda002a3c5eb2f
I20240503 02:00:04.780119 6303 catalog_manager.cc:4830] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719): tablet 3b189fba0d6c461694dda002a3c5eb2f (table rebalance_test_table_17 [id=5ccf0adaf2de4b79bbf38c053646e60e]) successfully deleted
I20240503 02:00:04.780928 11841 consensus_queue.cc:1035] T 485833ae7cdf41ba9695f6502e1d83f7 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:04.782856 11501 raft_consensus.cc:2932] T f5ed01740019467d8c1d55cc2fa18ca3 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:04.783363 8917 raft_consensus.cc:2932] T f5ed01740019467d8c1d55cc2fa18ca3 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:04.785903 6582 raft_consensus.cc:2932] T f5ed01740019467d8c1d55cc2fa18ca3 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:04.786487 6312 catalog_manager.cc:5484] T f5ed01740019467d8c1d55cc2fa18ca3 P 9942306855404cfd891adf227dd65eb2 reported cstate change: config changed from index 3 to 4, VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "2a01ace80baf4057b6bf1995cf88468b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:04.787113 6314 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_extra_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:04.788661 6300 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f5ed01740019467d8c1d55cc2fa18ca3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:04.789367 9025 tablet_service.cc:1514] Processing DeleteTablet for tablet f5ed01740019467d8c1d55cc2fa18ca3 with delete_type TABLET_DATA_TOMBSTONED (TS 99ead02915954b4fab23b033ad6d546b not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52584
I20240503 02:00:04.789842 11859 tablet_replica.cc:330] T f5ed01740019467d8c1d55cc2fa18ca3 P 99ead02915954b4fab23b033ad6d546b: stopping tablet replica
I20240503 02:00:04.790045 11859 raft_consensus.cc:2229] T f5ed01740019467d8c1d55cc2fa18ca3 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:04.790257 11859 raft_consensus.cc:2258] T f5ed01740019467d8c1d55cc2fa18ca3 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:04.793879 11859 ts_tablet_manager.cc:1875] T f5ed01740019467d8c1d55cc2fa18ca3 P 99ead02915954b4fab23b033ad6d546b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:04.795742 8880 tablet_service.cc:1467] Processing CreateTablet for tablet a3175dfc79d641a3bd70f2df5c579bdc (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=f789477e4c27430e8d7c180dee243ee8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:04.796046 8880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3175dfc79d641a3bd70f2df5c579bdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:04.796496 6831 tablet_service.cc:1467] Processing CreateTablet for tablet a3175dfc79d641a3bd70f2df5c579bdc (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=f789477e4c27430e8d7c180dee243ee8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:04.796763 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3175dfc79d641a3bd70f2df5c579bdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:04.797276 11834 tablet_bootstrap.cc:492] T d897c21e630d4770832bb10f43778e82 P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:04.797523 11834 tablet_bootstrap.cc:492] T d897c21e630d4770832bb10f43778e82 P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap complete.
I20240503 02:00:04.797685 11834 ts_tablet_manager.cc:1367] T d897c21e630d4770832bb10f43778e82 P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent bootstrapping tablet: real 0.061s user 0.025s sys 0.000s
I20240503 02:00:04.798184 11834 raft_consensus.cc:348] T d897c21e630d4770832bb10f43778e82 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } }
I20240503 02:00:04.798310 11834 raft_consensus.cc:729] T d897c21e630d4770832bb10f43778e82 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e1a9251ce1cf4361b8b0ede9fbfc4023, State: Initialized, Role: LEARNER
I20240503 02:00:04.798523 11834 consensus_queue.cc:260] T d897c21e630d4770832bb10f43778e82 P e1a9251ce1cf4361b8b0ede9fbfc4023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } }
I20240503 02:00:04.799484 6436 tablet_service.cc:1467] Processing CreateTablet for tablet 1a7ee79312ea4bad816264375ca5bb2d (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=f789477e4c27430e8d7c180dee243ee8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:04.799775 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a7ee79312ea4bad816264375ca5bb2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:04.800161 11834 ts_tablet_manager.cc:1398] T d897c21e630d4770832bb10f43778e82 P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:04.801352 6739 tablet_copy_service.cc:342] P c2ac2eb7cc264047b9ab32fe3a4fc025: Request end of tablet copy session e1a9251ce1cf4361b8b0ede9fbfc4023-d897c21e630d4770832bb10f43778e82 received from {username='slave'} at 127.0.99.136:40237
I20240503 02:00:04.801435 6739 tablet_copy_service.cc:433] P c2ac2eb7cc264047b9ab32fe3a4fc025: ending tablet copy session e1a9251ce1cf4361b8b0ede9fbfc4023-d897c21e630d4770832bb10f43778e82 on tablet d897c21e630d4770832bb10f43778e82 with peer e1a9251ce1cf4361b8b0ede9fbfc4023
I20240503 02:00:04.802443 11862 tablet_bootstrap.cc:492] T 1a7ee79312ea4bad816264375ca5bb2d P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:04.802892 6700 tablet_service.cc:1467] Processing CreateTablet for tablet ba17d4d3e0754b2798eb95484d37229a (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=f789477e4c27430e8d7c180dee243ee8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:04.803154 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba17d4d3e0754b2798eb95484d37229a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:04.803416 11862 tablet_bootstrap.cc:654] T 1a7ee79312ea4bad816264375ca5bb2d P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:04.804136 11859 ts_tablet_manager.cc:1888] T f5ed01740019467d8c1d55cc2fa18ca3 P 99ead02915954b4fab23b033ad6d546b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:00:04.804212 11859 log.cc:1196] T f5ed01740019467d8c1d55cc2fa18ca3 P 99ead02915954b4fab23b033ad6d546b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/f5ed01740019467d8c1d55cc2fa18ca3
I20240503 02:00:04.805446 11861 tablet_bootstrap.cc:492] T a3175dfc79d641a3bd70f2df5c579bdc P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:04.806089 6301 catalog_manager.cc:4830] TS 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157): tablet f5ed01740019467d8c1d55cc2fa18ca3 (table rebalancer_extra_table_0 [id=296df7c9486144058745436d9b814a78]) successfully deleted
I20240503 02:00:04.806195 9159 tablet_service.cc:1467] Processing CreateTablet for tablet a3175dfc79d641a3bd70f2df5c579bdc (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=f789477e4c27430e8d7c180dee243ee8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:04.806493 9159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3175dfc79d641a3bd70f2df5c579bdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:04.809176 9286 tablet_service.cc:1467] Processing CreateTablet for tablet 1a7ee79312ea4bad816264375ca5bb2d (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=f789477e4c27430e8d7c180dee243ee8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:04.809219 9025 tablet_service.cc:1467] Processing CreateTablet for tablet 1a7ee79312ea4bad816264375ca5bb2d (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=f789477e4c27430e8d7c180dee243ee8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:04.809448 9286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a7ee79312ea4bad816264375ca5bb2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:04.809448 9025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a7ee79312ea4bad816264375ca5bb2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:04.809653 11866 tablet_bootstrap.cc:492] T ba17d4d3e0754b2798eb95484d37229a P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:04.810906 11866 tablet_bootstrap.cc:654] T ba17d4d3e0754b2798eb95484d37229a P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:04.811609 11860 tablet_bootstrap.cc:492] T a3175dfc79d641a3bd70f2df5c579bdc P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:04.811834 9027 tablet_service.cc:1467] Processing CreateTablet for tablet ba17d4d3e0754b2798eb95484d37229a (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=f789477e4c27430e8d7c180dee243ee8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:04.811834 9285 tablet_service.cc:1467] Processing CreateTablet for tablet ba17d4d3e0754b2798eb95484d37229a (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=f789477e4c27430e8d7c180dee243ee8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:04.812112 9027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba17d4d3e0754b2798eb95484d37229a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:04.812125 9285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba17d4d3e0754b2798eb95484d37229a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:04.812600 11860 tablet_bootstrap.cc:654] T a3175dfc79d641a3bd70f2df5c579bdc P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:04.813722 11862 tablet_bootstrap.cc:492] T 1a7ee79312ea4bad816264375ca5bb2d P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:04.813808 11862 ts_tablet_manager.cc:1367] T 1a7ee79312ea4bad816264375ca5bb2d P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 02:00:04.814239 11862 raft_consensus.cc:348] T 1a7ee79312ea4bad816264375ca5bb2d P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } }
I20240503 02:00:04.814630 11862 raft_consensus.cc:374] T 1a7ee79312ea4bad816264375ca5bb2d P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:04.814666 11862 raft_consensus.cc:729] T 1a7ee79312ea4bad816264375ca5bb2d P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:04.814812 11862 consensus_queue.cc:260] T 1a7ee79312ea4bad816264375ca5bb2d P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } }
I20240503 02:00:04.815012 11851 consensus_queue.cc:1035] T 485833ae7cdf41ba9695f6502e1d83f7 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:04.815407 11862 ts_tablet_manager.cc:1398] T 1a7ee79312ea4bad816264375ca5bb2d P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:04.816537 11852 ts_tablet_manager.cc:902] T 0ee05e70ccbb496ca3cff08dc38112e8 P dec3e4a04d604f7eaaa3f077d081218a: Initiating tablet copy from peer 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:04.817231 11852 tablet_copy_client.cc:308] T 0ee05e70ccbb496ca3cff08dc38112e8 P dec3e4a04d604f7eaaa3f077d081218a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:34079
I20240503 02:00:04.817721 9199 tablet_copy_service.cc:140] P 2a01ace80baf4057b6bf1995cf88468b: Received BeginTabletCopySession request for tablet 0ee05e70ccbb496ca3cff08dc38112e8 from peer dec3e4a04d604f7eaaa3f077d081218a ({username='slave'} at 127.0.99.129:55593)
I20240503 02:00:04.817822 9199 tablet_copy_service.cc:161] P 2a01ace80baf4057b6bf1995cf88468b: Beginning new tablet copy session on tablet 0ee05e70ccbb496ca3cff08dc38112e8 from peer dec3e4a04d604f7eaaa3f077d081218a at {username='slave'} at 127.0.99.129:55593: session id = dec3e4a04d604f7eaaa3f077d081218a-0ee05e70ccbb496ca3cff08dc38112e8
I20240503 02:00:04.818711 9199 tablet_copy_source_session.cc:205] T 0ee05e70ccbb496ca3cff08dc38112e8 P 2a01ace80baf4057b6bf1995cf88468b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:04.820185 11861 tablet_bootstrap.cc:654] T a3175dfc79d641a3bd70f2df5c579bdc P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:04.820286 11868 tablet_bootstrap.cc:492] T 1a7ee79312ea4bad816264375ca5bb2d P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap starting.
I20240503 02:00:04.821035 11869 tablet_bootstrap.cc:492] T 1a7ee79312ea4bad816264375ca5bb2d P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:04.821131 11852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ee05e70ccbb496ca3cff08dc38112e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:04.821450 11875 tablet_bootstrap.cc:492] T a3175dfc79d641a3bd70f2df5c579bdc P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:04.822569 11875 tablet_bootstrap.cc:654] T a3175dfc79d641a3bd70f2df5c579bdc P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:04.823096 11869 tablet_bootstrap.cc:654] T 1a7ee79312ea4bad816264375ca5bb2d P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:04.823379 11868 tablet_bootstrap.cc:654] T 1a7ee79312ea4bad816264375ca5bb2d P e1a9251ce1cf4361b8b0ede9fbfc4023: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:04.824038 11852 tablet_copy_client.cc:783] T 0ee05e70ccbb496ca3cff08dc38112e8 P dec3e4a04d604f7eaaa3f077d081218a: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:04.824226 11852 tablet_copy_client.cc:647] T 0ee05e70ccbb496ca3cff08dc38112e8 P dec3e4a04d604f7eaaa3f077d081218a: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:04.825176 11861 tablet_bootstrap.cc:492] T a3175dfc79d641a3bd70f2df5c579bdc P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:04.825461 11861 ts_tablet_manager.cc:1367] T a3175dfc79d641a3bd70f2df5c579bdc P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20240503 02:00:04.825937 11861 raft_consensus.cc:348] T a3175dfc79d641a3bd70f2df5c579bdc P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:04.826121 11861 raft_consensus.cc:374] T a3175dfc79d641a3bd70f2df5c579bdc P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:04.826198 11861 raft_consensus.cc:729] T a3175dfc79d641a3bd70f2df5c579bdc P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:04.826382 11861 consensus_queue.cc:260] T a3175dfc79d641a3bd70f2df5c579bdc P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:04.827006 11866 tablet_bootstrap.cc:492] T ba17d4d3e0754b2798eb95484d37229a P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:04.827116 11866 ts_tablet_manager.cc:1367] T ba17d4d3e0754b2798eb95484d37229a P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20240503 02:00:04.827615 11866 raft_consensus.cc:348] T ba17d4d3e0754b2798eb95484d37229a P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:04.827701 11852 tablet_copy_client.cc:523] T 0ee05e70ccbb496ca3cff08dc38112e8 P dec3e4a04d604f7eaaa3f077d081218a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:04.827723 11866 raft_consensus.cc:374] T ba17d4d3e0754b2798eb95484d37229a P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:04.827751 11866 raft_consensus.cc:729] T ba17d4d3e0754b2798eb95484d37229a P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:04.827872 11866 consensus_queue.cc:260] T ba17d4d3e0754b2798eb95484d37229a P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:04.827911 11860 tablet_bootstrap.cc:492] T a3175dfc79d641a3bd70f2df5c579bdc P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:04.828019 11860 ts_tablet_manager.cc:1367] T a3175dfc79d641a3bd70f2df5c579bdc P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240503 02:00:04.828150 11866 ts_tablet_manager.cc:1398] T ba17d4d3e0754b2798eb95484d37229a P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:04.828541 11860 raft_consensus.cc:348] T a3175dfc79d641a3bd70f2df5c579bdc P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:04.828651 11860 raft_consensus.cc:374] T a3175dfc79d641a3bd70f2df5c579bdc P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:04.828681 11860 raft_consensus.cc:729] T a3175dfc79d641a3bd70f2df5c579bdc P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:04.828792 11860 consensus_queue.cc:260] T a3175dfc79d641a3bd70f2df5c579bdc P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:04.829068 11860 ts_tablet_manager.cc:1398] T a3175dfc79d641a3bd70f2df5c579bdc P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:04.829541 11852 tablet_bootstrap.cc:492] T 0ee05e70ccbb496ca3cff08dc38112e8 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:04.829756 11861 ts_tablet_manager.cc:1398] T a3175dfc79d641a3bd70f2df5c579bdc P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 02:00:04.830536 11869 tablet_bootstrap.cc:492] T 1a7ee79312ea4bad816264375ca5bb2d P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:04.830687 11869 ts_tablet_manager.cc:1367] T 1a7ee79312ea4bad816264375ca5bb2d P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 02:00:04.831216 11869 raft_consensus.cc:348] T 1a7ee79312ea4bad816264375ca5bb2d P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } }
I20240503 02:00:04.831427 11869 raft_consensus.cc:374] T 1a7ee79312ea4bad816264375ca5bb2d P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:04.831503 11869 raft_consensus.cc:729] T 1a7ee79312ea4bad816264375ca5bb2d P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:04.831696 11869 consensus_queue.cc:260] T 1a7ee79312ea4bad816264375ca5bb2d P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } }
I20240503 02:00:04.835353 9309 raft_consensus.cc:1261] T 46ad540deb624a25b97d3b967fe1f6f4 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 2 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:00:04.835994 11376 consensus_queue.cc:1035] T 46ad540deb624a25b97d3b967fe1f6f4 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:04.838893 8917 raft_consensus.cc:1261] T 46ad540deb624a25b97d3b967fe1f6f4 P 9942306855404cfd891adf227dd65eb2 [term 2 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:00:04.840327 11869 ts_tablet_manager.cc:1398] T 1a7ee79312ea4bad816264375ca5bb2d P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20240503 02:00:04.840795 11869 tablet_bootstrap.cc:492] T ba17d4d3e0754b2798eb95484d37229a P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:04.842615 11373 consensus_queue.cc:1035] T 46ad540deb624a25b97d3b967fe1f6f4 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:04.843139 11869 tablet_bootstrap.cc:654] T ba17d4d3e0754b2798eb95484d37229a P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:04.843837 11875 tablet_bootstrap.cc:492] T a3175dfc79d641a3bd70f2df5c579bdc P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:04.844058 11875 ts_tablet_manager.cc:1367] T a3175dfc79d641a3bd70f2df5c579bdc P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.000s
I20240503 02:00:04.844672 11875 raft_consensus.cc:348] T a3175dfc79d641a3bd70f2df5c579bdc P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:04.844864 11875 raft_consensus.cc:374] T a3175dfc79d641a3bd70f2df5c579bdc P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:04.844929 11875 raft_consensus.cc:729] T a3175dfc79d641a3bd70f2df5c579bdc P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:04.845129 11875 consensus_queue.cc:260] T a3175dfc79d641a3bd70f2df5c579bdc P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:04.845463 11875 ts_tablet_manager.cc:1398] T a3175dfc79d641a3bd70f2df5c579bdc P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:04.847064 11868 tablet_bootstrap.cc:492] T 1a7ee79312ea4bad816264375ca5bb2d P e1a9251ce1cf4361b8b0ede9fbfc4023: No bootstrap required, opened a new log
I20240503 02:00:04.847172 11868 ts_tablet_manager.cc:1367] T 1a7ee79312ea4bad816264375ca5bb2d P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.000s
I20240503 02:00:04.847648 11868 raft_consensus.cc:348] T 1a7ee79312ea4bad816264375ca5bb2d P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } }
I20240503 02:00:04.847774 11868 raft_consensus.cc:374] T 1a7ee79312ea4bad816264375ca5bb2d P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:04.847801 11868 raft_consensus.cc:729] T 1a7ee79312ea4bad816264375ca5bb2d P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1a9251ce1cf4361b8b0ede9fbfc4023, State: Initialized, Role: FOLLOWER
I20240503 02:00:04.847926 11868 consensus_queue.cc:260] T 1a7ee79312ea4bad816264375ca5bb2d P e1a9251ce1cf4361b8b0ede9fbfc4023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } }
I20240503 02:00:04.848186 11868 ts_tablet_manager.cc:1398] T 1a7ee79312ea4bad816264375ca5bb2d P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:04.848351 11868 tablet_bootstrap.cc:492] T ba17d4d3e0754b2798eb95484d37229a P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap starting.
I20240503 02:00:04.849408 11868 tablet_bootstrap.cc:654] T ba17d4d3e0754b2798eb95484d37229a P e1a9251ce1cf4361b8b0ede9fbfc4023: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:04.851452 11869 tablet_bootstrap.cc:492] T ba17d4d3e0754b2798eb95484d37229a P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:04.851539 6309 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:04.851622 11869 ts_tablet_manager.cc:1367] T ba17d4d3e0754b2798eb95484d37229a P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 02:00:04.851692 6309 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:04.852455 11869 raft_consensus.cc:348] T ba17d4d3e0754b2798eb95484d37229a P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:04.852633 11869 raft_consensus.cc:374] T ba17d4d3e0754b2798eb95484d37229a P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:04.852701 11869 raft_consensus.cc:729] T ba17d4d3e0754b2798eb95484d37229a P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:04.852861 11869 consensus_queue.cc:260] T ba17d4d3e0754b2798eb95484d37229a P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:04.853771 11869 ts_tablet_manager.cc:1398] T ba17d4d3e0754b2798eb95484d37229a P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:04.854032 6309 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet 478399c0161f439a9f7fd4b53d4506cb
I20240503 02:00:04.854825 11868 tablet_bootstrap.cc:492] T ba17d4d3e0754b2798eb95484d37229a P e1a9251ce1cf4361b8b0ede9fbfc4023: No bootstrap required, opened a new log
I20240503 02:00:04.854924 11868 ts_tablet_manager.cc:1367] T ba17d4d3e0754b2798eb95484d37229a P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:00:04.855074 9159 tablet_service.cc:1514] Processing DeleteTablet for tablet 478399c0161f439a9f7fd4b53d4506cb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:04 UTC) from {username='slave'} at 127.0.0.1:51940
I20240503 02:00:04.855116 9285 tablet_service.cc:1514] Processing DeleteTablet for tablet 478399c0161f439a9f7fd4b53d4506cb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:04 UTC) from {username='slave'} at 127.0.0.1:36004
I20240503 02:00:04.855397 11868 raft_consensus.cc:348] T ba17d4d3e0754b2798eb95484d37229a P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:04.855505 11868 raft_consensus.cc:374] T ba17d4d3e0754b2798eb95484d37229a P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:04.855538 11868 raft_consensus.cc:729] T ba17d4d3e0754b2798eb95484d37229a P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1a9251ce1cf4361b8b0ede9fbfc4023, State: Initialized, Role: FOLLOWER
I20240503 02:00:04.855664 11868 consensus_queue.cc:260] T ba17d4d3e0754b2798eb95484d37229a P e1a9251ce1cf4361b8b0ede9fbfc4023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:04.855715 11852 tablet_bootstrap.cc:492] T 0ee05e70ccbb496ca3cff08dc38112e8 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:04.855960 11868 ts_tablet_manager.cc:1398] T ba17d4d3e0754b2798eb95484d37229a P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:04.855980 11852 tablet_bootstrap.cc:492] T 0ee05e70ccbb496ca3cff08dc38112e8 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap complete.
I20240503 02:00:04.856148 11852 ts_tablet_manager.cc:1367] T 0ee05e70ccbb496ca3cff08dc38112e8 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.000s
I20240503 02:00:04.856150 8880 tablet_service.cc:1514] Processing DeleteTablet for tablet 478399c0161f439a9f7fd4b53d4506cb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:04 UTC) from {username='slave'} at 127.0.0.1:40296
I20240503 02:00:04.856530 11895 tablet_replica.cc:330] T 478399c0161f439a9f7fd4b53d4506cb P 2a01ace80baf4057b6bf1995cf88468b: stopping tablet replica
I20240503 02:00:04.856676 11895 raft_consensus.cc:2229] T 478399c0161f439a9f7fd4b53d4506cb P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:04.856662 11852 raft_consensus.cc:348] T 0ee05e70ccbb496ca3cff08dc38112e8 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: true } }
I20240503 02:00:04.856782 11852 raft_consensus.cc:729] T 0ee05e70ccbb496ca3cff08dc38112e8 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: LEARNER
I20240503 02:00:04.856882 11895 raft_consensus.cc:2258] T 478399c0161f439a9f7fd4b53d4506cb P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:04.856892 11852 consensus_queue.cc:260] T 0ee05e70ccbb496ca3cff08dc38112e8 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: true } }
I20240503 02:00:04.857160 11852 ts_tablet_manager.cc:1398] T 0ee05e70ccbb496ca3cff08dc38112e8 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:04.857502 11895 ts_tablet_manager.cc:1875] T 478399c0161f439a9f7fd4b53d4506cb P 2a01ace80baf4057b6bf1995cf88468b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:04.859066 9199 tablet_copy_service.cc:342] P 2a01ace80baf4057b6bf1995cf88468b: Request end of tablet copy session dec3e4a04d604f7eaaa3f077d081218a-0ee05e70ccbb496ca3cff08dc38112e8 received from {username='slave'} at 127.0.99.129:55593
I20240503 02:00:04.859153 9199 tablet_copy_service.cc:433] P 2a01ace80baf4057b6bf1995cf88468b: ending tablet copy session dec3e4a04d604f7eaaa3f077d081218a-0ee05e70ccbb496ca3cff08dc38112e8 on tablet 0ee05e70ccbb496ca3cff08dc38112e8 with peer dec3e4a04d604f7eaaa3f077d081218a
I20240503 02:00:04.859674 11897 tablet_replica.cc:330] T 478399c0161f439a9f7fd4b53d4506cb P 9942306855404cfd891adf227dd65eb2: stopping tablet replica
I20240503 02:00:04.859876 11897 raft_consensus.cc:2229] T 478399c0161f439a9f7fd4b53d4506cb P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:04.860065 11897 raft_consensus.cc:2258] T 478399c0161f439a9f7fd4b53d4506cb P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:04.860924 11897 ts_tablet_manager.cc:1875] T 478399c0161f439a9f7fd4b53d4506cb P 9942306855404cfd891adf227dd65eb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:04.862205 11828 ts_tablet_manager.cc:902] T 2e967e377984468d90799041c29882bd P 99ead02915954b4fab23b033ad6d546b: Initiating tablet copy from peer c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:04.862740 11828 tablet_copy_client.cc:308] T 2e967e377984468d90799041c29882bd P 99ead02915954b4fab23b033ad6d546b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:42655
I20240503 02:00:04.863278 6739 tablet_copy_service.cc:140] P c2ac2eb7cc264047b9ab32fe3a4fc025: Received BeginTabletCopySession request for tablet 2e967e377984468d90799041c29882bd from peer 99ead02915954b4fab23b033ad6d546b ({username='slave'} at 127.0.99.134:48173)
I20240503 02:00:04.863371 11896 tablet_replica.cc:330] T 478399c0161f439a9f7fd4b53d4506cb P e1a9251ce1cf4361b8b0ede9fbfc4023: stopping tablet replica
I20240503 02:00:04.863387 6739 tablet_copy_service.cc:161] P c2ac2eb7cc264047b9ab32fe3a4fc025: Beginning new tablet copy session on tablet 2e967e377984468d90799041c29882bd from peer 99ead02915954b4fab23b033ad6d546b at {username='slave'} at 127.0.99.134:48173: session id = 99ead02915954b4fab23b033ad6d546b-2e967e377984468d90799041c29882bd
I20240503 02:00:04.863574 11896 raft_consensus.cc:2229] T 478399c0161f439a9f7fd4b53d4506cb P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:04.863703 11896 raft_consensus.cc:2258] T 478399c0161f439a9f7fd4b53d4506cb P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:04.864365 6739 tablet_copy_source_session.cc:205] T 2e967e377984468d90799041c29882bd P c2ac2eb7cc264047b9ab32fe3a4fc025: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:04.864346 11897 ts_tablet_manager.cc:1888] T 478399c0161f439a9f7fd4b53d4506cb P 9942306855404cfd891adf227dd65eb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:04.864482 11897 log.cc:1196] T 478399c0161f439a9f7fd4b53d4506cb P 9942306855404cfd891adf227dd65eb2: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/478399c0161f439a9f7fd4b53d4506cb
I20240503 02:00:04.864476 11895 ts_tablet_manager.cc:1888] T 478399c0161f439a9f7fd4b53d4506cb P 2a01ace80baf4057b6bf1995cf88468b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:04.864750 11897 ts_tablet_manager.cc:1909] T 478399c0161f439a9f7fd4b53d4506cb P 9942306855404cfd891adf227dd65eb2: Deleting consensus metadata
I20240503 02:00:04.865417 11896 ts_tablet_manager.cc:1875] T 478399c0161f439a9f7fd4b53d4506cb P e1a9251ce1cf4361b8b0ede9fbfc4023: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:04.865449 11828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e967e377984468d90799041c29882bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:04.865639 6303 catalog_manager.cc:4830] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211): tablet 478399c0161f439a9f7fd4b53d4506cb (table rebalancer_dynamic_table_new_name [id=e49bcea404d2492ea04582ccfa57537b]) successfully deleted
I20240503 02:00:04.867218 11828 tablet_copy_client.cc:783] T 2e967e377984468d90799041c29882bd P 99ead02915954b4fab23b033ad6d546b: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:04.864548 11895 log.cc:1196] T 478399c0161f439a9f7fd4b53d4506cb P 2a01ace80baf4057b6bf1995cf88468b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/478399c0161f439a9f7fd4b53d4506cb
I20240503 02:00:04.867395 11828 tablet_copy_client.cc:647] T 2e967e377984468d90799041c29882bd P 99ead02915954b4fab23b033ad6d546b: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:04.867693 11895 ts_tablet_manager.cc:1909] T 478399c0161f439a9f7fd4b53d4506cb P 2a01ace80baf4057b6bf1995cf88468b: Deleting consensus metadata
I20240503 02:00:04.867921 11903 ts_tablet_manager.cc:902] T 067f54011a37432d99157884e70fc601 P 492448570cc94ed1879feb2ec3b9051d: Initiating tablet copy from peer 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:04.868562 11903 tablet_copy_client.cc:308] T 067f54011a37432d99157884e70fc601 P 492448570cc94ed1879feb2ec3b9051d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:33211
I20240503 02:00:04.868786 11828 tablet_copy_client.cc:523] T 2e967e377984468d90799041c29882bd P 99ead02915954b4fab23b033ad6d546b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:04.869189 6300 catalog_manager.cc:4830] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079): tablet 478399c0161f439a9f7fd4b53d4506cb (table rebalancer_dynamic_table_new_name [id=e49bcea404d2492ea04582ccfa57537b]) successfully deleted
I20240503 02:00:04.869602 11896 ts_tablet_manager.cc:1888] T 478399c0161f439a9f7fd4b53d4506cb P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:04.869686 11896 log.cc:1196] T 478399c0161f439a9f7fd4b53d4506cb P e1a9251ce1cf4361b8b0ede9fbfc4023: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/478399c0161f439a9f7fd4b53d4506cb
I20240503 02:00:04.869967 11896 ts_tablet_manager.cc:1909] T 478399c0161f439a9f7fd4b53d4506cb P e1a9251ce1cf4361b8b0ede9fbfc4023: Deleting consensus metadata
I20240503 02:00:04.870261 11828 tablet_bootstrap.cc:492] T 2e967e377984468d90799041c29882bd P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:04.870342 8937 tablet_copy_service.cc:140] P 9942306855404cfd891adf227dd65eb2: Received BeginTabletCopySession request for tablet 067f54011a37432d99157884e70fc601 from peer 492448570cc94ed1879feb2ec3b9051d ({username='slave'} at 127.0.99.132:43245)
I20240503 02:00:04.870453 8937 tablet_copy_service.cc:161] P 9942306855404cfd891adf227dd65eb2: Beginning new tablet copy session on tablet 067f54011a37432d99157884e70fc601 from peer 492448570cc94ed1879feb2ec3b9051d at {username='slave'} at 127.0.99.132:43245: session id = 492448570cc94ed1879feb2ec3b9051d-067f54011a37432d99157884e70fc601
I20240503 02:00:04.870704 6301 catalog_manager.cc:4830] TS e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): tablet 478399c0161f439a9f7fd4b53d4506cb (table rebalancer_dynamic_table_new_name [id=e49bcea404d2492ea04582ccfa57537b]) successfully deleted
I20240503 02:00:04.871443 8937 tablet_copy_source_session.cc:205] T 067f54011a37432d99157884e70fc601 P 9942306855404cfd891adf227dd65eb2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:04.872586 11903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 067f54011a37432d99157884e70fc601. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:04.874641 11907 ts_tablet_manager.cc:902] T 3416b6107e1848f299ac49ccd88ae7c8 P 9942306855404cfd891adf227dd65eb2: Initiating tablet copy from peer 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:04.875491 11903 tablet_copy_client.cc:783] T 067f54011a37432d99157884e70fc601 P 492448570cc94ed1879feb2ec3b9051d: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:04.875648 11903 tablet_copy_client.cc:647] T 067f54011a37432d99157884e70fc601 P 492448570cc94ed1879feb2ec3b9051d: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:04.875656 11795 ts_tablet_manager.cc:902] T 9c8a53980b00421d8c2bd3ae73f0c8cc P c2ac2eb7cc264047b9ab32fe3a4fc025: Initiating tablet copy from peer 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:04.876099 11795 tablet_copy_client.cc:308] T 9c8a53980b00421d8c2bd3ae73f0c8cc P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:33211
I20240503 02:00:04.876739 8937 tablet_copy_service.cc:140] P 9942306855404cfd891adf227dd65eb2: Received BeginTabletCopySession request for tablet 9c8a53980b00421d8c2bd3ae73f0c8cc from peer c2ac2eb7cc264047b9ab32fe3a4fc025 ({username='slave'} at 127.0.99.131:37759)
I20240503 02:00:04.876842 8937 tablet_copy_service.cc:161] P 9942306855404cfd891adf227dd65eb2: Beginning new tablet copy session on tablet 9c8a53980b00421d8c2bd3ae73f0c8cc from peer c2ac2eb7cc264047b9ab32fe3a4fc025 at {username='slave'} at 127.0.99.131:37759: session id = c2ac2eb7cc264047b9ab32fe3a4fc025-9c8a53980b00421d8c2bd3ae73f0c8cc
I20240503 02:00:04.877750 8937 tablet_copy_source_session.cc:205] T 9c8a53980b00421d8c2bd3ae73f0c8cc P 9942306855404cfd891adf227dd65eb2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:04.878571 11903 tablet_copy_client.cc:523] T 067f54011a37432d99157884e70fc601 P 492448570cc94ed1879feb2ec3b9051d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:04.878780 11795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c8a53980b00421d8c2bd3ae73f0c8cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:04.880321 11903 tablet_bootstrap.cc:492] T 067f54011a37432d99157884e70fc601 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:04.880932 11795 tablet_copy_client.cc:783] T 9c8a53980b00421d8c2bd3ae73f0c8cc P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:04.881099 11795 tablet_copy_client.cc:647] T 9c8a53980b00421d8c2bd3ae73f0c8cc P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:04.875183 11907 tablet_copy_client.cc:308] T 3416b6107e1848f299ac49ccd88ae7c8 P 9942306855404cfd891adf227dd65eb2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43863
I20240503 02:00:04.885020 6609 tablet_copy_service.cc:140] P 80fd18ef4d604c86b909d5c008a46b4f: Received BeginTabletCopySession request for tablet 3416b6107e1848f299ac49ccd88ae7c8 from peer 9942306855404cfd891adf227dd65eb2 ({username='slave'} at 127.0.99.133:50723)
I20240503 02:00:04.885134 6609 tablet_copy_service.cc:161] P 80fd18ef4d604c86b909d5c008a46b4f: Beginning new tablet copy session on tablet 3416b6107e1848f299ac49ccd88ae7c8 from peer 9942306855404cfd891adf227dd65eb2 at {username='slave'} at 127.0.99.133:50723: session id = 9942306855404cfd891adf227dd65eb2-3416b6107e1848f299ac49ccd88ae7c8
I20240503 02:00:04.886090 6609 tablet_copy_source_session.cc:205] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:04.887073 11907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3416b6107e1848f299ac49ccd88ae7c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:04.887656 11795 tablet_copy_client.cc:523] T 9c8a53980b00421d8c2bd3ae73f0c8cc P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:04.889744 11907 tablet_copy_client.cc:783] T 3416b6107e1848f299ac49ccd88ae7c8 P 9942306855404cfd891adf227dd65eb2: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:04.889961 11907 tablet_copy_client.cc:647] T 3416b6107e1848f299ac49ccd88ae7c8 P 9942306855404cfd891adf227dd65eb2: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:04.890229 11903 tablet_bootstrap.cc:492] T 067f54011a37432d99157884e70fc601 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:04.890578 11903 tablet_bootstrap.cc:492] T 067f54011a37432d99157884e70fc601 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap complete.
I20240503 02:00:04.890695 11795 tablet_bootstrap.cc:492] T 9c8a53980b00421d8c2bd3ae73f0c8cc P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:04.890779 11903 ts_tablet_manager.cc:1367] T 067f54011a37432d99157884e70fc601 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.002s
I20240503 02:00:04.891279 11903 raft_consensus.cc:348] T 067f54011a37432d99157884e70fc601 P 492448570cc94ed1879feb2ec3b9051d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { promote: true } }
I20240503 02:00:04.891415 11903 raft_consensus.cc:729] T 067f54011a37432d99157884e70fc601 P 492448570cc94ed1879feb2ec3b9051d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: LEARNER
I20240503 02:00:04.891538 11903 consensus_queue.cc:260] T 067f54011a37432d99157884e70fc601 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { promote: true } }
I20240503 02:00:04.891830 11903 ts_tablet_manager.cc:1398] T 067f54011a37432d99157884e70fc601 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:04.892735 8937 tablet_copy_service.cc:342] P 9942306855404cfd891adf227dd65eb2: Request end of tablet copy session 492448570cc94ed1879feb2ec3b9051d-067f54011a37432d99157884e70fc601 received from {username='slave'} at 127.0.99.132:43245
I20240503 02:00:04.892815 8937 tablet_copy_service.cc:433] P 9942306855404cfd891adf227dd65eb2: ending tablet copy session 492448570cc94ed1879feb2ec3b9051d-067f54011a37432d99157884e70fc601 on tablet 067f54011a37432d99157884e70fc601 with peer 492448570cc94ed1879feb2ec3b9051d
I20240503 02:00:04.893345 11907 tablet_copy_client.cc:523] T 3416b6107e1848f299ac49ccd88ae7c8 P 9942306855404cfd891adf227dd65eb2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:04.897181 11907 tablet_bootstrap.cc:492] T 3416b6107e1848f299ac49ccd88ae7c8 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:04.897482 11795 tablet_bootstrap.cc:492] T 9c8a53980b00421d8c2bd3ae73f0c8cc P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:04.897682 11795 tablet_bootstrap.cc:492] T 9c8a53980b00421d8c2bd3ae73f0c8cc P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap complete.
I20240503 02:00:04.897814 11795 ts_tablet_manager.cc:1367] T 9c8a53980b00421d8c2bd3ae73f0c8cc P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240503 02:00:04.897869 11828 tablet_bootstrap.cc:492] T 2e967e377984468d90799041c29882bd P 99ead02915954b4fab23b033ad6d546b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:04.898137 11828 tablet_bootstrap.cc:492] T 2e967e377984468d90799041c29882bd P 99ead02915954b4fab23b033ad6d546b: Bootstrap complete.
I20240503 02:00:04.898309 11828 ts_tablet_manager.cc:1367] T 2e967e377984468d90799041c29882bd P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.028s user 0.020s sys 0.005s
I20240503 02:00:04.898253 11795 raft_consensus.cc:348] T 9c8a53980b00421d8c2bd3ae73f0c8cc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: true } }
I20240503 02:00:04.898360 11795 raft_consensus.cc:729] T 9c8a53980b00421d8c2bd3ae73f0c8cc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: LEARNER
I20240503 02:00:04.898461 11795 consensus_queue.cc:260] T 9c8a53980b00421d8c2bd3ae73f0c8cc P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: true } }
I20240503 02:00:04.898731 11795 ts_tablet_manager.cc:1398] T 9c8a53980b00421d8c2bd3ae73f0c8cc P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:04.898723 11828 raft_consensus.cc:348] T 2e967e377984468d90799041c29882bd P 99ead02915954b4fab23b033ad6d546b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } }
I20240503 02:00:04.898852 11828 raft_consensus.cc:729] T 2e967e377984468d90799041c29882bd P 99ead02915954b4fab23b033ad6d546b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: LEARNER
I20240503 02:00:04.898964 11828 consensus_queue.cc:260] T 2e967e377984468d90799041c29882bd P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } }
I20240503 02:00:04.899247 8937 tablet_copy_service.cc:342] P 9942306855404cfd891adf227dd65eb2: Request end of tablet copy session c2ac2eb7cc264047b9ab32fe3a4fc025-9c8a53980b00421d8c2bd3ae73f0c8cc received from {username='slave'} at 127.0.99.131:37759
I20240503 02:00:04.899328 8937 tablet_copy_service.cc:433] P 9942306855404cfd891adf227dd65eb2: ending tablet copy session c2ac2eb7cc264047b9ab32fe3a4fc025-9c8a53980b00421d8c2bd3ae73f0c8cc on tablet 9c8a53980b00421d8c2bd3ae73f0c8cc with peer c2ac2eb7cc264047b9ab32fe3a4fc025
I20240503 02:00:04.899356 11514 raft_consensus.cc:482] T 1a7ee79312ea4bad816264375ca5bb2d P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:04.899430 11514 raft_consensus.cc:504] T 1a7ee79312ea4bad816264375ca5bb2d P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } }
I20240503 02:00:04.899811 11514 leader_election.cc:288] T 1a7ee79312ea4bad816264375ca5bb2d P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311)
I20240503 02:00:04.900265 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a7ee79312ea4bad816264375ca5bb2d" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 02:00:04.900465 6458 raft_consensus.cc:2454] T 1a7ee79312ea4bad816264375ca5bb2d P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:04.900741 11828 ts_tablet_manager.cc:1398] T 2e967e377984468d90799041c29882bd P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 02:00:04.900995 8981 leader_election.cc:302] T 1a7ee79312ea4bad816264375ca5bb2d P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99ead02915954b4fab23b033ad6d546b, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:04.901160 6739 tablet_copy_service.cc:342] P c2ac2eb7cc264047b9ab32fe3a4fc025: Request end of tablet copy session 99ead02915954b4fab23b033ad6d546b-2e967e377984468d90799041c29882bd received from {username='slave'} at 127.0.99.134:48173
I20240503 02:00:04.901149 9309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a7ee79312ea4bad816264375ca5bb2d" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" is_pre_election: true
I20240503 02:00:04.901232 6739 tablet_copy_service.cc:433] P c2ac2eb7cc264047b9ab32fe3a4fc025: ending tablet copy session 99ead02915954b4fab23b033ad6d546b-2e967e377984468d90799041c29882bd on tablet 2e967e377984468d90799041c29882bd with peer 99ead02915954b4fab23b033ad6d546b
I20240503 02:00:04.901270 9309 raft_consensus.cc:2454] T 1a7ee79312ea4bad816264375ca5bb2d P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:04.901346 10473 raft_consensus.cc:2781] T 1a7ee79312ea4bad816264375ca5bb2d P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:04.901387 10473 raft_consensus.cc:482] T 1a7ee79312ea4bad816264375ca5bb2d P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:04.901420 10473 raft_consensus.cc:3037] T 1a7ee79312ea4bad816264375ca5bb2d P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:04.902438 10473 raft_consensus.cc:504] T 1a7ee79312ea4bad816264375ca5bb2d P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } }
I20240503 02:00:04.902788 10473 leader_election.cc:288] T 1a7ee79312ea4bad816264375ca5bb2d P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Requested vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311)
I20240503 02:00:04.903378 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a7ee79312ea4bad816264375ca5bb2d" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 02:00:04.903507 6458 raft_consensus.cc:3037] T 1a7ee79312ea4bad816264375ca5bb2d P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:04.903575 9309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a7ee79312ea4bad816264375ca5bb2d" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023"
I20240503 02:00:04.903694 9309 raft_consensus.cc:3037] T 1a7ee79312ea4bad816264375ca5bb2d P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:04.904578 6458 raft_consensus.cc:2454] T 1a7ee79312ea4bad816264375ca5bb2d P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:04.904738 9309 raft_consensus.cc:2454] T 1a7ee79312ea4bad816264375ca5bb2d P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:04.904899 8981 leader_election.cc:302] T 1a7ee79312ea4bad816264375ca5bb2d P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99ead02915954b4fab23b033ad6d546b, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:04.905084 10473 raft_consensus.cc:2781] T 1a7ee79312ea4bad816264375ca5bb2d P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:04.905184 10473 raft_consensus.cc:686] T 1a7ee79312ea4bad816264375ca5bb2d P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Becoming Leader. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Running, Role: LEADER
I20240503 02:00:04.905311 10473 consensus_queue.cc:237] T 1a7ee79312ea4bad816264375ca5bb2d P 99ead02915954b4fab23b033ad6d546b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } }
I20240503 02:00:04.908563 6314 catalog_manager.cc:5484] T 1a7ee79312ea4bad816264375ca5bb2d P 99ead02915954b4fab23b033ad6d546b reported cstate change: term changed from 0 to 1, leader changed from <none> to 99ead02915954b4fab23b033ad6d546b (127.0.99.134). New cstate: current_term: 1 leader_uuid: "99ead02915954b4fab23b033ad6d546b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:04.917660 6720 raft_consensus.cc:1203] T 51b82ef8e4cc474387f6902f2e526ac0 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:00:04.927722 10345 raft_consensus.cc:1050] T d0d5dd42db3e4883af7fd16d8813b550 P 80fd18ef4d604c86b909d5c008a46b4f: attempting to promote NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 to VOTER
I20240503 02:00:04.928171 10345 consensus_queue.cc:237] T d0d5dd42db3e4883af7fd16d8813b550 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } }
I20240503 02:00:04.928670 11907 tablet_bootstrap.cc:492] T 3416b6107e1848f299ac49ccd88ae7c8 P 9942306855404cfd891adf227dd65eb2: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:04.928988 11907 tablet_bootstrap.cc:492] T 3416b6107e1848f299ac49ccd88ae7c8 P 9942306855404cfd891adf227dd65eb2: Bootstrap complete.
I20240503 02:00:04.929186 11907 ts_tablet_manager.cc:1367] T 3416b6107e1848f299ac49ccd88ae7c8 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.032s user 0.013s sys 0.010s
I20240503 02:00:04.929669 11907 raft_consensus.cc:348] T 3416b6107e1848f299ac49ccd88ae7c8 P 9942306855404cfd891adf227dd65eb2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } }
I20240503 02:00:04.929800 11907 raft_consensus.cc:729] T 3416b6107e1848f299ac49ccd88ae7c8 P 9942306855404cfd891adf227dd65eb2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: LEARNER
I20240503 02:00:04.929898 11907 consensus_queue.cc:260] T 3416b6107e1848f299ac49ccd88ae7c8 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } }
I20240503 02:00:04.930022 6849 raft_consensus.cc:1261] T d0d5dd42db3e4883af7fd16d8813b550 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:04.930018 6720 raft_consensus.cc:1261] T d0d5dd42db3e4883af7fd16d8813b550 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:04.930279 11907 ts_tablet_manager.cc:1398] T 3416b6107e1848f299ac49ccd88ae7c8 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 02:00:04.930375 9309 raft_consensus.cc:1261] T d0d5dd42db3e4883af7fd16d8813b550 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 LEARNER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:04.930815 9973 consensus_queue.cc:1035] T d0d5dd42db3e4883af7fd16d8813b550 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:04.931535 6609 tablet_copy_service.cc:342] P 80fd18ef4d604c86b909d5c008a46b4f: Request end of tablet copy session 9942306855404cfd891adf227dd65eb2-3416b6107e1848f299ac49ccd88ae7c8 received from {username='slave'} at 127.0.99.133:50723
I20240503 02:00:04.931612 6609 tablet_copy_service.cc:433] P 80fd18ef4d604c86b909d5c008a46b4f: ending tablet copy session 9942306855404cfd891adf227dd65eb2-3416b6107e1848f299ac49ccd88ae7c8 on tablet 3416b6107e1848f299ac49ccd88ae7c8 with peer 9942306855404cfd891adf227dd65eb2
I20240503 02:00:04.931715 9973 consensus_queue.cc:1035] T d0d5dd42db3e4883af7fd16d8813b550 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:04.931864 11492 consensus_queue.cc:1035] T d0d5dd42db3e4883af7fd16d8813b550 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:04.934862 9973 raft_consensus.cc:2932] T d0d5dd42db3e4883af7fd16d8813b550 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:04.936120 6720 raft_consensus.cc:2932] T d0d5dd42db3e4883af7fd16d8813b550 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:04.937336 9309 raft_consensus.cc:2932] T d0d5dd42db3e4883af7fd16d8813b550 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:04.937652 6849 raft_consensus.cc:2932] T d0d5dd42db3e4883af7fd16d8813b550 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:04.937789 6309 catalog_manager.cc:5484] T d0d5dd42db3e4883af7fd16d8813b550 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: config changed from index 4 to 5, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:04.942615 6582 consensus_queue.cc:237] T d0d5dd42db3e4883af7fd16d8813b550 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } }
I20240503 02:00:04.944226 6849 raft_consensus.cc:1261] T d0d5dd42db3e4883af7fd16d8813b550 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:04.944620 9309 raft_consensus.cc:1261] T d0d5dd42db3e4883af7fd16d8813b550 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:04.944960 11492 consensus_queue.cc:1035] T d0d5dd42db3e4883af7fd16d8813b550 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:04.945202 10345 consensus_queue.cc:1035] T d0d5dd42db3e4883af7fd16d8813b550 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:04.947578 11491 raft_consensus.cc:2932] T d0d5dd42db3e4883af7fd16d8813b550 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:04.948055 6849 raft_consensus.cc:2932] T d0d5dd42db3e4883af7fd16d8813b550 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:04.949326 6302 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d0d5dd42db3e4883af7fd16d8813b550 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:04.950484 9309 raft_consensus.cc:2932] T d0d5dd42db3e4883af7fd16d8813b550 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:04.950821 6312 catalog_manager.cc:5484] T d0d5dd42db3e4883af7fd16d8813b550 P 492448570cc94ed1879feb2ec3b9051d reported cstate change: config changed from index 5 to 6, VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:04.954357 6700 tablet_service.cc:1514] Processing DeleteTablet for tablet d0d5dd42db3e4883af7fd16d8813b550 with delete_type TABLET_DATA_TOMBSTONED (TS c2ac2eb7cc264047b9ab32fe3a4fc025 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:40336
I20240503 02:00:04.954730 11956 tablet_replica.cc:330] T d0d5dd42db3e4883af7fd16d8813b550 P c2ac2eb7cc264047b9ab32fe3a4fc025: stopping tablet replica
I20240503 02:00:04.954964 11956 raft_consensus.cc:2229] T d0d5dd42db3e4883af7fd16d8813b550 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:04.955258 11956 raft_consensus.cc:2258] T d0d5dd42db3e4883af7fd16d8813b550 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:04.955978 11956 ts_tablet_manager.cc:1875] T d0d5dd42db3e4883af7fd16d8813b550 P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:04.958427 6311 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:00:04.959854 11956 ts_tablet_manager.cc:1888] T d0d5dd42db3e4883af7fd16d8813b550 P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:00:04.960017 11956 log.cc:1196] T d0d5dd42db3e4883af7fd16d8813b550 P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/d0d5dd42db3e4883af7fd16d8813b550
I20240503 02:00:04.961091 6303 catalog_manager.cc:4830] TS c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655): tablet d0d5dd42db3e4883af7fd16d8813b550 (table rebalance_test_table_11 [id=805c97ce87c74e019deaa2acee6a5329]) successfully deleted
I20240503 02:00:04.966557 9285 tablet_service.cc:1467] Processing CreateTablet for tablet 10020e5222fc4c68be3d9a7957727d4a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2691a19595c54d3095907c7a788aada6]), partition=
I20240503 02:00:04.966902 9285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10020e5222fc4c68be3d9a7957727d4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:04.967201 9159 tablet_service.cc:1467] Processing CreateTablet for tablet 10020e5222fc4c68be3d9a7957727d4a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2691a19595c54d3095907c7a788aada6]), partition=
I20240503 02:00:04.967419 9159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10020e5222fc4c68be3d9a7957727d4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:04.969054 11868 tablet_bootstrap.cc:492] T 10020e5222fc4c68be3d9a7957727d4a P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap starting.
I20240503 02:00:04.969468 11875 tablet_bootstrap.cc:492] T 10020e5222fc4c68be3d9a7957727d4a P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:04.969651 6567 tablet_service.cc:1467] Processing CreateTablet for tablet 10020e5222fc4c68be3d9a7957727d4a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2691a19595c54d3095907c7a788aada6]), partition=
I20240503 02:00:04.969916 6567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10020e5222fc4c68be3d9a7957727d4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:04.970273 11875 tablet_bootstrap.cc:654] T 10020e5222fc4c68be3d9a7957727d4a P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:04.970324 11868 tablet_bootstrap.cc:654] T 10020e5222fc4c68be3d9a7957727d4a P e1a9251ce1cf4361b8b0ede9fbfc4023: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:04.971577 11868 tablet_bootstrap.cc:492] T 10020e5222fc4c68be3d9a7957727d4a P e1a9251ce1cf4361b8b0ede9fbfc4023: No bootstrap required, opened a new log
I20240503 02:00:04.971668 11868 ts_tablet_manager.cc:1367] T 10020e5222fc4c68be3d9a7957727d4a P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:04.972235 11868 raft_consensus.cc:348] T 10020e5222fc4c68be3d9a7957727d4a P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:04.972357 11868 raft_consensus.cc:374] T 10020e5222fc4c68be3d9a7957727d4a P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:04.972421 11868 raft_consensus.cc:729] T 10020e5222fc4c68be3d9a7957727d4a P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1a9251ce1cf4361b8b0ede9fbfc4023, State: Initialized, Role: FOLLOWER
I20240503 02:00:04.972506 11875 tablet_bootstrap.cc:492] T 10020e5222fc4c68be3d9a7957727d4a P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:04.972589 11875 ts_tablet_manager.cc:1367] T 10020e5222fc4c68be3d9a7957727d4a P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:04.973016 11875 raft_consensus.cc:348] T 10020e5222fc4c68be3d9a7957727d4a P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:04.973031 11868 consensus_queue.cc:260] T 10020e5222fc4c68be3d9a7957727d4a P e1a9251ce1cf4361b8b0ede9fbfc4023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:04.973120 11875 raft_consensus.cc:374] T 10020e5222fc4c68be3d9a7957727d4a P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:04.973158 11875 raft_consensus.cc:729] T 10020e5222fc4c68be3d9a7957727d4a P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:04.973266 11875 consensus_queue.cc:260] T 10020e5222fc4c68be3d9a7957727d4a P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:04.974364 9179 raft_consensus.cc:1203] T 7df4c083550046e883e5e61d3d4e1572 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 02:00:04.974608 11868 ts_tablet_manager.cc:1398] T 10020e5222fc4c68be3d9a7957727d4a P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:00:04.974463 11964 tablet_bootstrap.cc:492] T 10020e5222fc4c68be3d9a7957727d4a P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:04.975749 11964 tablet_bootstrap.cc:654] T 10020e5222fc4c68be3d9a7957727d4a P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:04.977263 11964 tablet_bootstrap.cc:492] T 10020e5222fc4c68be3d9a7957727d4a P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:04.977600 9711 raft_consensus.cc:1050] T 485d1d89a8b94505969d663be92124cd P c2ac2eb7cc264047b9ab32fe3a4fc025: attempting to promote NON_VOTER 9942306855404cfd891adf227dd65eb2 to VOTER
I20240503 02:00:04.977607 11964 ts_tablet_manager.cc:1367] T 10020e5222fc4c68be3d9a7957727d4a P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:04.978066 11875 ts_tablet_manager.cc:1398] T 10020e5222fc4c68be3d9a7957727d4a P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240503 02:00:04.978047 9711 consensus_queue.cc:237] T 485d1d89a8b94505969d663be92124cd P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } }
I20240503 02:00:04.978324 11964 raft_consensus.cc:348] T 10020e5222fc4c68be3d9a7957727d4a P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:04.978570 11964 raft_consensus.cc:374] T 10020e5222fc4c68be3d9a7957727d4a P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:04.978652 11964 raft_consensus.cc:729] T 10020e5222fc4c68be3d9a7957727d4a P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:04.978853 11964 consensus_queue.cc:260] T 10020e5222fc4c68be3d9a7957727d4a P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:04.979846 6458 raft_consensus.cc:1261] T 485d1d89a8b94505969d663be92124cd P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:04.980007 11964 ts_tablet_manager.cc:1398] T 10020e5222fc4c68be3d9a7957727d4a P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:04.980633 9984 consensus_queue.cc:1035] T 485d1d89a8b94505969d663be92124cd P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:04.980788 9039 raft_consensus.cc:1261] T 485d1d89a8b94505969d663be92124cd P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:04.981580 8916 raft_consensus.cc:1261] T 485d1d89a8b94505969d663be92124cd P 9942306855404cfd891adf227dd65eb2 [term 1 LEARNER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:04.981792 10349 consensus_queue.cc:1035] T 485d1d89a8b94505969d663be92124cd P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:04.982101 9711 consensus_queue.cc:1035] T 485d1d89a8b94505969d663be92124cd P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:04.985637 11872 raft_consensus.cc:482] T a3175dfc79d641a3bd70f2df5c579bdc P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:04.985487 9984 raft_consensus.cc:2932] T 485d1d89a8b94505969d663be92124cd P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:04.985731 11872 raft_consensus.cc:504] T a3175dfc79d641a3bd70f2df5c579bdc P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:04.986124 11872 leader_election.cc:288] T a3175dfc79d641a3bd70f2df5c579bdc P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:04.986157 6458 raft_consensus.cc:2932] T 485d1d89a8b94505969d663be92124cd P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:04.986653 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3175dfc79d641a3bd70f2df5c579bdc" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 02:00:04.986785 6849 raft_consensus.cc:2454] T a3175dfc79d641a3bd70f2df5c579bdc P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 0.
I20240503 02:00:04.987082 8853 leader_election.cc:302] T a3175dfc79d641a3bd70f2df5c579bdc P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:04.987246 11872 raft_consensus.cc:2781] T a3175dfc79d641a3bd70f2df5c579bdc P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:04.987293 11872 raft_consensus.cc:482] T a3175dfc79d641a3bd70f2df5c579bdc P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:04.987324 11872 raft_consensus.cc:3037] T a3175dfc79d641a3bd70f2df5c579bdc P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:04.988322 11872 raft_consensus.cc:504] T a3175dfc79d641a3bd70f2df5c579bdc P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:04.988730 11872 leader_election.cc:288] T a3175dfc79d641a3bd70f2df5c579bdc P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 election: Requested vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:04.989109 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3175dfc79d641a3bd70f2df5c579bdc" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 02:00:04.989185 9176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3175dfc79d641a3bd70f2df5c579bdc" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:04.989233 6849 raft_consensus.cc:3037] T a3175dfc79d641a3bd70f2df5c579bdc P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:04.989295 9176 raft_consensus.cc:3037] T a3175dfc79d641a3bd70f2df5c579bdc P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:04.990203 6849 raft_consensus.cc:2454] T a3175dfc79d641a3bd70f2df5c579bdc P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 1.
I20240503 02:00:04.990203 9176 raft_consensus.cc:2454] T a3175dfc79d641a3bd70f2df5c579bdc P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 1.
I20240503 02:00:04.990577 8850 leader_election.cc:302] T a3175dfc79d641a3bd70f2df5c579bdc P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:04.990866 9048 raft_consensus.cc:2932] T 485d1d89a8b94505969d663be92124cd P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:04.991925 8916 raft_consensus.cc:2932] T 485d1d89a8b94505969d663be92124cd P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:04.993278 9758 raft_consensus.cc:2781] T a3175dfc79d641a3bd70f2df5c579bdc P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:04.993369 9758 raft_consensus.cc:686] T a3175dfc79d641a3bd70f2df5c579bdc P 9942306855404cfd891adf227dd65eb2 [term 1 LEADER]: Becoming Leader. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Running, Role: LEADER
I20240503 02:00:04.993494 9758 consensus_queue.cc:237] T a3175dfc79d641a3bd70f2df5c579bdc P 9942306855404cfd891adf227dd65eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:04.993470 6312 catalog_manager.cc:5484] T 485d1d89a8b94505969d663be92124cd P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: config changed from index 7 to 8, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:04.996201 6314 catalog_manager.cc:5484] T a3175dfc79d641a3bd70f2df5c579bdc P 9942306855404cfd891adf227dd65eb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9942306855404cfd891adf227dd65eb2 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "9942306855404cfd891adf227dd65eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:04.996358 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3175dfc79d641a3bd70f2df5c579bdc" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:04.996510 9179 raft_consensus.cc:2362] T a3175dfc79d641a3bd70f2df5c579bdc P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 1. Re-sending same reply.
I20240503 02:00:05.002475 6720 consensus_queue.cc:237] T 485d1d89a8b94505969d663be92124cd P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } }
I20240503 02:00:05.004061 8916 raft_consensus.cc:1261] T 485d1d89a8b94505969d663be92124cd P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:05.004452 9984 consensus_queue.cc:1035] T 485d1d89a8b94505969d663be92124cd P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:05.004613 9048 raft_consensus.cc:1261] T 485d1d89a8b94505969d663be92124cd P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:05.005003 9984 consensus_queue.cc:1035] T 485d1d89a8b94505969d663be92124cd P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:05.006599 9984 raft_consensus.cc:2932] T 485d1d89a8b94505969d663be92124cd P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:05.006898 8916 raft_consensus.cc:2932] T 485d1d89a8b94505969d663be92124cd P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:05.009447 6303 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 485d1d89a8b94505969d663be92124cd with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:05.010399 9048 raft_consensus.cc:2932] T 485d1d89a8b94505969d663be92124cd P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:05.010753 6311 catalog_manager.cc:5484] T 485d1d89a8b94505969d663be92124cd P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: config changed from index 8 to 9, VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:05.013638 6436 tablet_service.cc:1514] Processing DeleteTablet for tablet 485d1d89a8b94505969d663be92124cd with delete_type TABLET_DATA_TOMBSTONED (TS dec3e4a04d604f7eaaa3f077d081218a not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:51338
I20240503 02:00:05.014020 11976 tablet_replica.cc:330] T 485d1d89a8b94505969d663be92124cd P dec3e4a04d604f7eaaa3f077d081218a: stopping tablet replica
I20240503 02:00:05.014237 11976 raft_consensus.cc:2229] T 485d1d89a8b94505969d663be92124cd P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:05.015054 11976 raft_consensus.cc:2258] T 485d1d89a8b94505969d663be92124cd P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:05.016103 11976 ts_tablet_manager.cc:1875] T 485d1d89a8b94505969d663be92124cd P dec3e4a04d604f7eaaa3f077d081218a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:05.018509 11976 ts_tablet_manager.cc:1888] T 485d1d89a8b94505969d663be92124cd P dec3e4a04d604f7eaaa3f077d081218a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 02:00:05.018675 11976 log.cc:1196] T 485d1d89a8b94505969d663be92124cd P dec3e4a04d604f7eaaa3f077d081218a: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/485d1d89a8b94505969d663be92124cd
I20240503 02:00:05.019251 6300 catalog_manager.cc:4830] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): tablet 485d1d89a8b94505969d663be92124cd (table rebalance_test_table_15 [id=4e740b0167ed439291b74d77653ec386]) successfully deleted
I20240503 02:00:05.026993 11967 raft_consensus.cc:482] T ba17d4d3e0754b2798eb95484d37229a P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:05.027170 11967 raft_consensus.cc:504] T ba17d4d3e0754b2798eb95484d37229a P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:05.027639 11967 leader_election.cc:288] T ba17d4d3e0754b2798eb95484d37229a P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157), e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311)
I20240503 02:00:05.028571 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba17d4d3e0754b2798eb95484d37229a" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:05.028702 9048 raft_consensus.cc:2454] T ba17d4d3e0754b2798eb95484d37229a P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:05.028930 9309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba17d4d3e0754b2798eb95484d37229a" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" is_pre_election: true
I20240503 02:00:05.029001 6654 leader_election.cc:302] T ba17d4d3e0754b2798eb95484d37229a P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99ead02915954b4fab23b033ad6d546b, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:05.029055 9309 raft_consensus.cc:2454] T ba17d4d3e0754b2798eb95484d37229a P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:05.029173 11967 raft_consensus.cc:2781] T ba17d4d3e0754b2798eb95484d37229a P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:05.029223 11967 raft_consensus.cc:482] T ba17d4d3e0754b2798eb95484d37229a P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:05.029258 11967 raft_consensus.cc:3037] T ba17d4d3e0754b2798eb95484d37229a P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:05.030478 11967 raft_consensus.cc:504] T ba17d4d3e0754b2798eb95484d37229a P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:05.030910 11967 leader_election.cc:288] T ba17d4d3e0754b2798eb95484d37229a P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Requested vote from peers 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157), e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311)
I20240503 02:00:05.031270 9309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba17d4d3e0754b2798eb95484d37229a" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023"
I20240503 02:00:05.031292 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba17d4d3e0754b2798eb95484d37229a" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:05.031404 9048 raft_consensus.cc:3037] T ba17d4d3e0754b2798eb95484d37229a P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:05.031406 9309 raft_consensus.cc:3037] T ba17d4d3e0754b2798eb95484d37229a P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:05.032420 9309 raft_consensus.cc:2454] T ba17d4d3e0754b2798eb95484d37229a P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:05.032425 9048 raft_consensus.cc:2454] T ba17d4d3e0754b2798eb95484d37229a P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:05.032753 6654 leader_election.cc:302] T ba17d4d3e0754b2798eb95484d37229a P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99ead02915954b4fab23b033ad6d546b, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:05.033795 11967 raft_consensus.cc:2781] T ba17d4d3e0754b2798eb95484d37229a P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:05.034133 11967 raft_consensus.cc:686] T ba17d4d3e0754b2798eb95484d37229a P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Becoming Leader. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Running, Role: LEADER
I20240503 02:00:05.034292 11967 consensus_queue.cc:237] T ba17d4d3e0754b2798eb95484d37229a P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:05.035753 6582 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cb873018448549ccbe3d14b784e47d26"
dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49914
I20240503 02:00:05.035877 6582 raft_consensus.cc:595] T cb873018448549ccbe3d14b784e47d26 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Received request to transfer leadership
I20240503 02:00:05.036569 6314 catalog_manager.cc:5484] T ba17d4d3e0754b2798eb95484d37229a P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:05.085218 11872 raft_consensus.cc:1050] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2: attempting to promote NON_VOTER dec3e4a04d604f7eaaa3f077d081218a to VOTER
I20240503 02:00:05.085662 11872 consensus_queue.cc:237] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } }
I20240503 02:00:05.087284 9048 raft_consensus.cc:1261] T 96c395771fd44056abacceb6c1acdb9f P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:05.087450 9179 raft_consensus.cc:1261] T 96c395771fd44056abacceb6c1acdb9f P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:05.087525 6456 raft_consensus.cc:1261] T 96c395771fd44056abacceb6c1acdb9f P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEARNER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:05.088052 11872 consensus_queue.cc:1035] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:05.088858 11073 consensus_queue.cc:1035] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:05.089412 11073 consensus_queue.cc:1035] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:05.091521 9176 raft_consensus.cc:1203] T 1fa274c98fc147f29344ed537e2ccf17 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 02:00:05.092453 11872 raft_consensus.cc:2932] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } }
I20240503 02:00:05.097803 6456 raft_consensus.cc:2932] T 96c395771fd44056abacceb6c1acdb9f P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } }
I20240503 02:00:05.099472 9048 raft_consensus.cc:2932] T 96c395771fd44056abacceb6c1acdb9f P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } }
I20240503 02:00:05.099670 9179 raft_consensus.cc:2932] T 96c395771fd44056abacceb6c1acdb9f P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } }
I20240503 02:00:05.101478 6311 catalog_manager.cc:5484] T 96c395771fd44056abacceb6c1acdb9f P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: config changed from index 2 to 3, dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9942306855404cfd891adf227dd65eb2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } }
I20240503 02:00:05.106029 8916 consensus_queue.cc:237] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } }
I20240503 02:00:05.107553 9048 raft_consensus.cc:1261] T 96c395771fd44056abacceb6c1acdb9f P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:05.107942 6456 raft_consensus.cc:1261] T 96c395771fd44056abacceb6c1acdb9f P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:05.107956 11357 consensus_queue.cc:1035] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:05.108350 11357 consensus_queue.cc:1035] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:05.109275 6849 consensus_queue.cc:237] T 8246488dec204bf6b09e868d7951ac5a P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } }
I20240503 02:00:05.110406 11073 raft_consensus.cc:2932] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } }
I20240503 02:00:05.110937 6456 raft_consensus.cc:1261] T 8246488dec204bf6b09e868d7951ac5a P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:05.111017 6720 raft_consensus.cc:1261] T 8246488dec204bf6b09e868d7951ac5a P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:05.111363 11888 consensus_queue.cc:1035] T 8246488dec204bf6b09e868d7951ac5a P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:00:05.111564 6785 consensus_peers.cc:485] T 8246488dec204bf6b09e868d7951ac5a P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8246488dec204bf6b09e868d7951ac5a. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:05.111953 11888 consensus_queue.cc:1035] T 8246488dec204bf6b09e868d7951ac5a P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:05.112782 9048 raft_consensus.cc:2932] T 96c395771fd44056abacceb6c1acdb9f P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } }
I20240503 02:00:05.112931 9039 raft_consensus.cc:1203] T 3dc213593e4048c99ccbe10161ff8e7f P 99ead02915954b4fab23b033ad6d546b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:00:05.113319 11888 raft_consensus.cc:2932] T 8246488dec204bf6b09e868d7951ac5a P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } } }
I20240503 02:00:05.113973 6456 raft_consensus.cc:2932] T 96c395771fd44056abacceb6c1acdb9f P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } }
I20240503 02:00:05.116360 6309 catalog_manager.cc:5484] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2 reported cstate change: config changed from index 3 to 4, VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "9942306855404cfd891adf227dd65eb2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:05.116972 6720 raft_consensus.cc:2932] T 8246488dec204bf6b09e868d7951ac5a P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } } }
I20240503 02:00:05.117681 6303 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 96c395771fd44056abacceb6c1acdb9f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:05.119910 9159 tablet_service.cc:1514] Processing DeleteTablet for tablet 96c395771fd44056abacceb6c1acdb9f with delete_type TABLET_DATA_TOMBSTONED (TS 2a01ace80baf4057b6bf1995cf88468b not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:51940
I20240503 02:00:05.120157 11895 tablet_replica.cc:330] T 96c395771fd44056abacceb6c1acdb9f P 2a01ace80baf4057b6bf1995cf88468b: stopping tablet replica
I20240503 02:00:05.120294 11895 raft_consensus.cc:2229] T 96c395771fd44056abacceb6c1acdb9f P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:05.119930 6456 raft_consensus.cc:2932] T 8246488dec204bf6b09e868d7951ac5a P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } } }
I20240503 02:00:05.120585 11895 raft_consensus.cc:2258] T 96c395771fd44056abacceb6c1acdb9f P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:05.123471 6309 catalog_manager.cc:5484] T 8246488dec204bf6b09e868d7951ac5a P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: config changed from index -1 to 4, NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } } }
I20240503 02:00:05.124713 11895 ts_tablet_manager.cc:1875] T 96c395771fd44056abacceb6c1acdb9f P 2a01ace80baf4057b6bf1995cf88468b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:05.127140 11895 ts_tablet_manager.cc:1888] T 96c395771fd44056abacceb6c1acdb9f P 2a01ace80baf4057b6bf1995cf88468b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:00:05.127269 11895 log.cc:1196] T 96c395771fd44056abacceb6c1acdb9f P 2a01ace80baf4057b6bf1995cf88468b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/96c395771fd44056abacceb6c1acdb9f
I20240503 02:00:05.127817 6300 catalog_manager.cc:4830] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079): tablet 96c395771fd44056abacceb6c1acdb9f (table rebalancer_extra_table_6 [id=0392495bb8a743099943f2bba5f15608]) successfully deleted
I20240503 02:00:05.137271 6582 raft_consensus.cc:1261] T c3485701bf3b48f5ae2e1568afeed133 P 80fd18ef4d604c86b909d5c008a46b4f [term 2 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:00:05.137897 11373 consensus_queue.cc:1035] T c3485701bf3b48f5ae2e1568afeed133 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:05.143642 9309 raft_consensus.cc:1203] T d897c21e630d4770832bb10f43778e82 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:00:05.147841 6720 raft_consensus.cc:1261] T c3485701bf3b48f5ae2e1568afeed133 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 2 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:00:05.148311 11373 consensus_queue.cc:1035] T c3485701bf3b48f5ae2e1568afeed133 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:05.151394 9309 raft_consensus.cc:1261] T c3485701bf3b48f5ae2e1568afeed133 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 2 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:00:05.154091 12032 consensus_queue.cc:1035] T c3485701bf3b48f5ae2e1568afeed133 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:05.180442 10679 raft_consensus.cc:482] T 10020e5222fc4c68be3d9a7957727d4a P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:05.180580 10679 raft_consensus.cc:504] T 10020e5222fc4c68be3d9a7957727d4a P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:05.180995 10679 leader_election.cc:288] T 10020e5222fc4c68be3d9a7957727d4a P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:05.181560 9309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10020e5222fc4c68be3d9a7957727d4a" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" is_pre_election: true
I20240503 02:00:05.181718 9309 raft_consensus.cc:2454] T 10020e5222fc4c68be3d9a7957727d4a P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:05.181993 9113 leader_election.cc:302] T 10020e5222fc4c68be3d9a7957727d4a P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, e1a9251ce1cf4361b8b0ede9fbfc4023; no voters:
I20240503 02:00:05.182184 10679 raft_consensus.cc:2781] T 10020e5222fc4c68be3d9a7957727d4a P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:05.182228 10679 raft_consensus.cc:482] T 10020e5222fc4c68be3d9a7957727d4a P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:05.182262 10679 raft_consensus.cc:3037] T 10020e5222fc4c68be3d9a7957727d4a P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:05.182233 6582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10020e5222fc4c68be3d9a7957727d4a" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 02:00:05.182358 6582 raft_consensus.cc:2454] T 10020e5222fc4c68be3d9a7957727d4a P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:05.183239 10679 raft_consensus.cc:504] T 10020e5222fc4c68be3d9a7957727d4a P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:05.183634 10679 leader_election.cc:288] T 10020e5222fc4c68be3d9a7957727d4a P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Requested vote from peers e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:05.184053 6582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10020e5222fc4c68be3d9a7957727d4a" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 02:00:05.184060 9309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10020e5222fc4c68be3d9a7957727d4a" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023"
I20240503 02:00:05.184187 6582 raft_consensus.cc:3037] T 10020e5222fc4c68be3d9a7957727d4a P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:05.184188 9309 raft_consensus.cc:3037] T 10020e5222fc4c68be3d9a7957727d4a P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:05.185124 6582 raft_consensus.cc:2454] T 10020e5222fc4c68be3d9a7957727d4a P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:05.185281 9309 raft_consensus.cc:2454] T 10020e5222fc4c68be3d9a7957727d4a P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:05.185397 9114 leader_election.cc:302] T 10020e5222fc4c68be3d9a7957727d4a P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 80fd18ef4d604c86b909d5c008a46b4f; no voters:
I20240503 02:00:05.185554 10679 raft_consensus.cc:2781] T 10020e5222fc4c68be3d9a7957727d4a P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:05.185621 10679 raft_consensus.cc:686] T 10020e5222fc4c68be3d9a7957727d4a P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Becoming Leader. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Running, Role: LEADER
I20240503 02:00:05.185751 10679 consensus_queue.cc:237] T 10020e5222fc4c68be3d9a7957727d4a P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:05.188055 6312 catalog_manager.cc:5484] T 10020e5222fc4c68be3d9a7957727d4a P 2a01ace80baf4057b6bf1995cf88468b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135). New cstate: current_term: 1 leader_uuid: "2a01ace80baf4057b6bf1995cf88468b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:05.221325 6582 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cb873018448549ccbe3d14b784e47d26"
dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49936
I20240503 02:00:05.221482 6582 raft_consensus.cc:595] T cb873018448549ccbe3d14b784e47d26 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Received request to transfer leadership
I20240503 02:00:05.251137 6720 raft_consensus.cc:1203] T 9c8a53980b00421d8c2bd3ae73f0c8cc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:00:05.264429 6720 consensus_queue.cc:237] T b91a933fb66b4075acb40c5b92ef4824 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } }
I20240503 02:00:05.265884 6582 raft_consensus.cc:1261] T b91a933fb66b4075acb40c5b92ef4824 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:05.265918 6456 raft_consensus.cc:1261] T b91a933fb66b4075acb40c5b92ef4824 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:05.266297 11967 consensus_queue.cc:1035] T b91a933fb66b4075acb40c5b92ef4824 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:00:05.266556 6656 consensus_peers.cc:485] T b91a933fb66b4075acb40c5b92ef4824 P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211): Couldn't send request to peer 9942306855404cfd891adf227dd65eb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b91a933fb66b4075acb40c5b92ef4824. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:05.266597 9984 consensus_queue.cc:1035] T b91a933fb66b4075acb40c5b92ef4824 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:05.268350 10349 raft_consensus.cc:2932] T b91a933fb66b4075acb40c5b92ef4824 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } } }
I20240503 02:00:05.268833 6582 raft_consensus.cc:2932] T b91a933fb66b4075acb40c5b92ef4824 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } } }
I20240503 02:00:05.271049 6314 catalog_manager.cc:5484] T b91a933fb66b4075acb40c5b92ef4824 P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: config changed from index -1 to 4, NON_VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:05.272238 6456 raft_consensus.cc:2932] T b91a933fb66b4075acb40c5b92ef4824 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } } }
I20240503 02:00:05.277253 6849 consensus_queue.cc:237] T 46ad540deb624a25b97d3b967fe1f6f4 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } }
I20240503 02:00:05.278659 8916 raft_consensus.cc:1261] T 46ad540deb624a25b97d3b967fe1f6f4 P 9942306855404cfd891adf227dd65eb2 [term 2 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:00:05.278717 9309 raft_consensus.cc:1261] T 46ad540deb624a25b97d3b967fe1f6f4 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 2 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:00:05.279214 11376 consensus_queue.cc:1035] T 46ad540deb624a25b97d3b967fe1f6f4 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:05.279532 11373 consensus_queue.cc:1035] T 46ad540deb624a25b97d3b967fe1f6f4 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:00:05.280241 9048 raft_consensus.cc:1203] T 2e967e377984468d90799041c29882bd P 99ead02915954b4fab23b033ad6d546b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:00:05.281594 12032 raft_consensus.cc:2932] T 46ad540deb624a25b97d3b967fe1f6f4 P 492448570cc94ed1879feb2ec3b9051d [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:05.282094 8916 raft_consensus.cc:2932] T 46ad540deb624a25b97d3b967fe1f6f4 P 9942306855404cfd891adf227dd65eb2 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:05.283381 6303 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 46ad540deb624a25b97d3b967fe1f6f4 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:05.284368 6314 catalog_manager.cc:5484] T 46ad540deb624a25b97d3b967fe1f6f4 P 492448570cc94ed1879feb2ec3b9051d reported cstate change: config changed from index 8 to 10, VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:05.284499 9309 raft_consensus.cc:2932] T 46ad540deb624a25b97d3b967fe1f6f4 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:05.287048 6436 tablet_service.cc:1514] Processing DeleteTablet for tablet 46ad540deb624a25b97d3b967fe1f6f4 with delete_type TABLET_DATA_TOMBSTONED (TS dec3e4a04d604f7eaaa3f077d081218a not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:51338
I20240503 02:00:05.287393 11976 tablet_replica.cc:330] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a: stopping tablet replica
I20240503 02:00:05.287403 8916 raft_consensus.cc:1203] T 3416b6107e1848f299ac49ccd88ae7c8 P 9942306855404cfd891adf227dd65eb2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 02:00:05.287565 11976 raft_consensus.cc:2229] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:05.287678 11976 raft_consensus.cc:2258] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:05.288559 11976 ts_tablet_manager.cc:1875] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:05.290724 11976 ts_tablet_manager.cc:1888] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 02:00:05.290814 11976 log.cc:1196] T 46ad540deb624a25b97d3b967fe1f6f4 P dec3e4a04d604f7eaaa3f077d081218a: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/46ad540deb624a25b97d3b967fe1f6f4
I20240503 02:00:05.291419 6300 catalog_manager.cc:4830] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): tablet 46ad540deb624a25b97d3b967fe1f6f4 (table rebalance_test_table_9 [id=c65135643094446abc8eba15be23cd99]) successfully deleted
I20240503 02:00:05.293871 9048 consensus_queue.cc:237] T 6a82dcc9be5b4c93b5fff03f789350ab P 99ead02915954b4fab23b033ad6d546b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: true } }
I20240503 02:00:05.295433 6849 raft_consensus.cc:1261] T 6a82dcc9be5b4c93b5fff03f789350ab P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer 99ead02915954b4fab23b033ad6d546b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:05.295583 9309 raft_consensus.cc:1261] T 6a82dcc9be5b4c93b5fff03f789350ab P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Refusing update from remote peer 99ead02915954b4fab23b033ad6d546b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:05.295835 11515 consensus_queue.cc:1035] T 6a82dcc9be5b4c93b5fff03f789350ab P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:00:05.296152 8981 consensus_peers.cc:485] T 6a82dcc9be5b4c93b5fff03f789350ab P 99ead02915954b4fab23b033ad6d546b -> Peer dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): Couldn't send request to peer dec3e4a04d604f7eaaa3f077d081218a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6a82dcc9be5b4c93b5fff03f789350ab. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:05.296207 11515 consensus_queue.cc:1035] T 6a82dcc9be5b4c93b5fff03f789350ab P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:05.298149 10473 raft_consensus.cc:2932] T 6a82dcc9be5b4c93b5fff03f789350ab P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: true } } }
I20240503 02:00:05.298584 9309 raft_consensus.cc:2932] T 6a82dcc9be5b4c93b5fff03f789350ab P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: true } } }
I20240503 02:00:05.298595 6849 raft_consensus.cc:2932] T 6a82dcc9be5b4c93b5fff03f789350ab P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: true } } }
I20240503 02:00:05.301599 6312 catalog_manager.cc:5484] T 6a82dcc9be5b4c93b5fff03f789350ab P e1a9251ce1cf4361b8b0ede9fbfc4023 reported cstate change: config changed from index -1 to 2, NON_VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "99ead02915954b4fab23b033ad6d546b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: true } } }
I20240503 02:00:05.311520 10473 consensus_queue.cc:1035] T 1a7ee79312ea4bad816264375ca5bb2d P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:05.315048 11514 consensus_queue.cc:1035] T 1a7ee79312ea4bad816264375ca5bb2d P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:05.324146 6314 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:05.328747 6582 raft_consensus.cc:1261] T 10020e5222fc4c68be3d9a7957727d4a P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer 2a01ace80baf4057b6bf1995cf88468b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:05.328789 9309 raft_consensus.cc:1261] T 10020e5222fc4c68be3d9a7957727d4a P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Refusing update from remote peer 2a01ace80baf4057b6bf1995cf88468b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:05.329207 11504 consensus_queue.cc:1035] T 10020e5222fc4c68be3d9a7957727d4a P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:05.329667 11504 consensus_queue.cc:1035] T 10020e5222fc4c68be3d9a7957727d4a P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:05.332317 6312 catalog_manager.cc:5576] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863) does not have the latest schema for tablet 10020e5222fc4c68be3d9a7957727d4a (table rebalancer_dynamic_table [id=2691a19595c54d3095907c7a788aada6]). Expected version 1 got 0
I20240503 02:00:05.333714 12172 tablet.cc:1722] T 10020e5222fc4c68be3d9a7957727d4a P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:05.334251 12172 tablet.cc:1620] T 10020e5222fc4c68be3d9a7957727d4a P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
I20240503 02:00:05.335422 6312 catalog_manager.cc:5576] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079) does not have the latest schema for tablet 10020e5222fc4c68be3d9a7957727d4a (table rebalancer_dynamic_table [id=2691a19595c54d3095907c7a788aada6]). Expected version 1 got 0
I20240503 02:00:05.336491 6309 catalog_manager.cc:5576] TS e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311) does not have the latest schema for tablet 10020e5222fc4c68be3d9a7957727d4a (table rebalancer_dynamic_table [id=2691a19595c54d3095907c7a788aada6]). Expected version 1 got 0
I20240503 02:00:05.338459 6309 catalog_manager.cc:5576] TS e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311) does not have the latest schema for tablet 10020e5222fc4c68be3d9a7957727d4a (table rebalancer_dynamic_table [id=2691a19595c54d3095907c7a788aada6]). Expected version 1 got 0
I20240503 02:00:05.339510 12173 tablet.cc:1722] T 10020e5222fc4c68be3d9a7957727d4a P e1a9251ce1cf4361b8b0ede9fbfc4023: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:05.339862 6309 catalog_manager.cc:5576] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863) does not have the latest schema for tablet 10020e5222fc4c68be3d9a7957727d4a (table rebalancer_dynamic_table [id=2691a19595c54d3095907c7a788aada6]). Expected version 1 got 0
I20240503 02:00:05.340054 12173 tablet.cc:1620] T 10020e5222fc4c68be3d9a7957727d4a P e1a9251ce1cf4361b8b0ede9fbfc4023: MemRowSet was empty: no flush needed.
I20240503 02:00:05.340495 12174 tablet.cc:1722] T 10020e5222fc4c68be3d9a7957727d4a P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:05.340796 12174 tablet.cc:1620] T 10020e5222fc4c68be3d9a7957727d4a P 80fd18ef4d604c86b909d5c008a46b4f: MemRowSet was empty: no flush needed.
I20240503 02:00:05.341544 6312 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=2691a19595c54d3095907c7a788aada6] alter complete (version 1)
I20240503 02:00:05.343545 11492 raft_consensus.cc:979] T cb873018448549ccbe3d14b784e47d26 P 80fd18ef4d604c86b909d5c008a46b4f: : Instructing follower 492448570cc94ed1879feb2ec3b9051d to start an election
I20240503 02:00:05.343633 11492 raft_consensus.cc:1067] T cb873018448549ccbe3d14b784e47d26 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Signalling peer 492448570cc94ed1879feb2ec3b9051d to start an election
I20240503 02:00:05.344301 6850 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cb873018448549ccbe3d14b784e47d26"
dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
from {username='slave'} at 127.0.99.130:55559
I20240503 02:00:05.344480 6850 raft_consensus.cc:482] T cb873018448549ccbe3d14b784e47d26 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:05.344535 6850 raft_consensus.cc:3037] T cb873018448549ccbe3d14b784e47d26 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:05.345572 6850 raft_consensus.cc:504] T cb873018448549ccbe3d14b784e47d26 P 492448570cc94ed1879feb2ec3b9051d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { promote: false } }
I20240503 02:00:05.346074 6850 leader_election.cc:288] T cb873018448549ccbe3d14b784e47d26 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 2 election: Requested vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:05.346400 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb873018448549ccbe3d14b784e47d26" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:05.346534 9048 raft_consensus.cc:3037] T cb873018448549ccbe3d14b784e47d26 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:05.346601 6582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb873018448549ccbe3d14b784e47d26" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 02:00:05.346700 6582 raft_consensus.cc:3032] T cb873018448549ccbe3d14b784e47d26 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:00:05.346735 6582 raft_consensus.cc:729] T cb873018448549ccbe3d14b784e47d26 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Running, Role: LEADER
I20240503 02:00:05.346868 6582 consensus_queue.cc:260] T cb873018448549ccbe3d14b784e47d26 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { promote: false } }
I20240503 02:00:05.346936 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb873018448549ccbe3d14b784e47d26" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:05.347038 6582 raft_consensus.cc:3037] T cb873018448549ccbe3d14b784e47d26 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:05.347054 8916 raft_consensus.cc:3037] T cb873018448549ccbe3d14b784e47d26 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:05.347568 9048 raft_consensus.cc:2454] T cb873018448549ccbe3d14b784e47d26 P 99ead02915954b4fab23b033ad6d546b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 2.
I20240503 02:00:05.348021 6582 raft_consensus.cc:2454] T cb873018448549ccbe3d14b784e47d26 P 80fd18ef4d604c86b909d5c008a46b4f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 2.
I20240503 02:00:05.348021 8916 raft_consensus.cc:2454] T cb873018448549ccbe3d14b784e47d26 P 9942306855404cfd891adf227dd65eb2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 2.
I20240503 02:00:05.349962 6456 raft_consensus.cc:1203] T 0ee05e70ccbb496ca3cff08dc38112e8 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:00:05.350214 6786 leader_election.cc:302] T cb873018448549ccbe3d14b784e47d26 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 80fd18ef4d604c86b909d5c008a46b4f, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:05.350420 12032 raft_consensus.cc:2781] T cb873018448549ccbe3d14b784e47d26 P 492448570cc94ed1879feb2ec3b9051d [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:05.350504 12032 raft_consensus.cc:686] T cb873018448549ccbe3d14b784e47d26 P 492448570cc94ed1879feb2ec3b9051d [term 2 LEADER]: Becoming Leader. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Running, Role: LEADER
I20240503 02:00:05.350648 12032 consensus_queue.cc:237] T cb873018448549ccbe3d14b784e47d26 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { promote: false } }
I20240503 02:00:05.353152 6309 catalog_manager.cc:5484] T cb873018448549ccbe3d14b784e47d26 P 492448570cc94ed1879feb2ec3b9051d reported cstate change: term changed from 1 to 2, leader changed from 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) to 492448570cc94ed1879feb2ec3b9051d (127.0.99.132). New cstate: current_term: 2 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:05.446262 6309 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:05.451987 12172 tablet.cc:1722] T 10020e5222fc4c68be3d9a7957727d4a P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:05.453040 12173 tablet.cc:1722] T 10020e5222fc4c68be3d9a7957727d4a P e1a9251ce1cf4361b8b0ede9fbfc4023: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:05.453343 12173 tablet.cc:1620] T 10020e5222fc4c68be3d9a7957727d4a P e1a9251ce1cf4361b8b0ede9fbfc4023: MemRowSet was empty: no flush needed.
I20240503 02:00:05.454367 12174 tablet.cc:1722] T 10020e5222fc4c68be3d9a7957727d4a P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:05.454680 12174 tablet.cc:1620] T 10020e5222fc4c68be3d9a7957727d4a P 80fd18ef4d604c86b909d5c008a46b4f: MemRowSet was empty: no flush needed.
I20240503 02:00:05.454993 12172 tablet.cc:1620] T 10020e5222fc4c68be3d9a7957727d4a P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
I20240503 02:00:05.461311 6309 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=2691a19595c54d3095907c7a788aada6] alter complete (version 2)
I20240503 02:00:05.466542 6850 raft_consensus.cc:1203] T 067f54011a37432d99157884e70fc601 P 492448570cc94ed1879feb2ec3b9051d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:00:05.473042 10679 raft_consensus.cc:1050] T 51b82ef8e4cc474387f6902f2e526ac0 P 2a01ace80baf4057b6bf1995cf88468b: attempting to promote NON_VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 to VOTER
I20240503 02:00:05.473444 10679 consensus_queue.cc:237] T 51b82ef8e4cc474387f6902f2e526ac0 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } }
I20240503 02:00:05.474866 6720 raft_consensus.cc:1261] T 51b82ef8e4cc474387f6902f2e526ac0 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEARNER]: Refusing update from remote peer 2a01ace80baf4057b6bf1995cf88468b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:05.475080 9048 raft_consensus.cc:1261] T 51b82ef8e4cc474387f6902f2e526ac0 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer 2a01ace80baf4057b6bf1995cf88468b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:05.475210 9309 raft_consensus.cc:1261] T 51b82ef8e4cc474387f6902f2e526ac0 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Refusing update from remote peer 2a01ace80baf4057b6bf1995cf88468b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:05.475497 10679 consensus_queue.cc:1035] T 51b82ef8e4cc474387f6902f2e526ac0 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:05.475765 11502 consensus_queue.cc:1035] T 51b82ef8e4cc474387f6902f2e526ac0 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:05.476048 11502 consensus_queue.cc:1035] T 51b82ef8e4cc474387f6902f2e526ac0 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:05.478314 11504 raft_consensus.cc:2932] T 51b82ef8e4cc474387f6902f2e526ac0 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } } }
I20240503 02:00:05.478891 6720 raft_consensus.cc:2932] T 51b82ef8e4cc474387f6902f2e526ac0 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } } }
I20240503 02:00:05.481102 9309 raft_consensus.cc:2932] T 51b82ef8e4cc474387f6902f2e526ac0 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } } }
I20240503 02:00:05.481765 6309 catalog_manager.cc:5484] T 51b82ef8e4cc474387f6902f2e526ac0 P 2a01ace80baf4057b6bf1995cf88468b reported cstate change: config changed from index 2 to 3, c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2a01ace80baf4057b6bf1995cf88468b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:05.484136 9048 raft_consensus.cc:2932] T 51b82ef8e4cc474387f6902f2e526ac0 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } } }
I20240503 02:00:05.487692 9176 consensus_queue.cc:237] T 51b82ef8e4cc474387f6902f2e526ac0 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } }
I20240503 02:00:05.489074 6720 raft_consensus.cc:1261] T 51b82ef8e4cc474387f6902f2e526ac0 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer 2a01ace80baf4057b6bf1995cf88468b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:05.489146 9048 raft_consensus.cc:1261] T 51b82ef8e4cc474387f6902f2e526ac0 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer 2a01ace80baf4057b6bf1995cf88468b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:05.489555 11502 consensus_queue.cc:1035] T 51b82ef8e4cc474387f6902f2e526ac0 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:05.490113 11502 consensus_queue.cc:1035] T 51b82ef8e4cc474387f6902f2e526ac0 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:05.490204 12276 ts_tablet_manager.cc:902] T 8246488dec204bf6b09e868d7951ac5a P e1a9251ce1cf4361b8b0ede9fbfc4023: Initiating tablet copy from peer 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:05.490968 12276 tablet_copy_client.cc:308] T 8246488dec204bf6b09e868d7951ac5a P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:42719
I20240503 02:00:05.491492 11502 raft_consensus.cc:2932] T 51b82ef8e4cc474387f6902f2e526ac0 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } } }
I20240503 02:00:05.492123 6871 tablet_copy_service.cc:140] P 492448570cc94ed1879feb2ec3b9051d: Received BeginTabletCopySession request for tablet 8246488dec204bf6b09e868d7951ac5a from peer e1a9251ce1cf4361b8b0ede9fbfc4023 ({username='slave'} at 127.0.99.136:48421)
I20240503 02:00:05.492224 6871 tablet_copy_service.cc:161] P 492448570cc94ed1879feb2ec3b9051d: Beginning new tablet copy session on tablet 8246488dec204bf6b09e868d7951ac5a from peer e1a9251ce1cf4361b8b0ede9fbfc4023 at {username='slave'} at 127.0.99.136:48421: session id = e1a9251ce1cf4361b8b0ede9fbfc4023-8246488dec204bf6b09e868d7951ac5a
I20240503 02:00:05.493227 6871 tablet_copy_source_session.cc:205] T 8246488dec204bf6b09e868d7951ac5a P 492448570cc94ed1879feb2ec3b9051d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:05.493845 6300 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 51b82ef8e4cc474387f6902f2e526ac0 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:05.494248 9984 consensus_queue.cc:1035] T ba17d4d3e0754b2798eb95484d37229a P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:05.494467 12276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8246488dec204bf6b09e868d7951ac5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:05.495417 6314 catalog_manager.cc:5484] T 51b82ef8e4cc474387f6902f2e526ac0 P 2a01ace80baf4057b6bf1995cf88468b reported cstate change: config changed from index 3 to 4, VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "2a01ace80baf4057b6bf1995cf88468b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:05.496845 12276 tablet_copy_client.cc:783] T 8246488dec204bf6b09e868d7951ac5a P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:05.497020 12276 tablet_copy_client.cc:647] T 8246488dec204bf6b09e868d7951ac5a P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:05.497248 9039 raft_consensus.cc:2932] T 51b82ef8e4cc474387f6902f2e526ac0 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } } }
I20240503 02:00:05.497550 9984 consensus_queue.cc:1035] T ba17d4d3e0754b2798eb95484d37229a P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:05.498214 12276 tablet_copy_client.cc:523] T 8246488dec204bf6b09e868d7951ac5a P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:05.499503 12276 tablet_bootstrap.cc:492] T 8246488dec204bf6b09e868d7951ac5a P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap starting.
I20240503 02:00:05.499761 9285 tablet_service.cc:1514] Processing DeleteTablet for tablet 51b82ef8e4cc474387f6902f2e526ac0 with delete_type TABLET_DATA_TOMBSTONED (TS e1a9251ce1cf4361b8b0ede9fbfc4023 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36004
I20240503 02:00:05.500290 6720 raft_consensus.cc:2932] T 51b82ef8e4cc474387f6902f2e526ac0 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } } }
I20240503 02:00:05.502473 12288 tablet_replica.cc:330] T 51b82ef8e4cc474387f6902f2e526ac0 P e1a9251ce1cf4361b8b0ede9fbfc4023: stopping tablet replica
I20240503 02:00:05.502658 12288 raft_consensus.cc:2229] T 51b82ef8e4cc474387f6902f2e526ac0 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:05.502815 12288 raft_consensus.cc:2258] T 51b82ef8e4cc474387f6902f2e526ac0 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:05.506191 12288 ts_tablet_manager.cc:1875] T 51b82ef8e4cc474387f6902f2e526ac0 P e1a9251ce1cf4361b8b0ede9fbfc4023: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:05.508545 12288 ts_tablet_manager.cc:1888] T 51b82ef8e4cc474387f6902f2e526ac0 P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:00:05.508678 12288 log.cc:1196] T 51b82ef8e4cc474387f6902f2e526ac0 P e1a9251ce1cf4361b8b0ede9fbfc4023: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/51b82ef8e4cc474387f6902f2e526ac0
I20240503 02:00:05.509260 6301 catalog_manager.cc:4830] TS e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): tablet 51b82ef8e4cc474387f6902f2e526ac0 (table rebalancer_extra_table_4 [id=c1969b96b61041869ddd81db8c84bf44]) successfully deleted
I20240503 02:00:05.526075 12276 tablet_bootstrap.cc:492] T 8246488dec204bf6b09e868d7951ac5a P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:05.526623 12276 tablet_bootstrap.cc:492] T 8246488dec204bf6b09e868d7951ac5a P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap complete.
I20240503 02:00:05.527004 12276 ts_tablet_manager.cc:1367] T 8246488dec204bf6b09e868d7951ac5a P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent bootstrapping tablet: real 0.028s user 0.020s sys 0.002s
I20240503 02:00:05.527496 12276 raft_consensus.cc:348] T 8246488dec204bf6b09e868d7951ac5a P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } }
I20240503 02:00:05.527627 12276 raft_consensus.cc:729] T 8246488dec204bf6b09e868d7951ac5a P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e1a9251ce1cf4361b8b0ede9fbfc4023, State: Initialized, Role: LEARNER
I20240503 02:00:05.527727 12276 consensus_queue.cc:260] T 8246488dec204bf6b09e868d7951ac5a P e1a9251ce1cf4361b8b0ede9fbfc4023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } }
I20240503 02:00:05.528074 12276 ts_tablet_manager.cc:1398] T 8246488dec204bf6b09e868d7951ac5a P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 02:00:05.528750 6871 tablet_copy_service.cc:342] P 492448570cc94ed1879feb2ec3b9051d: Request end of tablet copy session e1a9251ce1cf4361b8b0ede9fbfc4023-8246488dec204bf6b09e868d7951ac5a received from {username='slave'} at 127.0.99.136:48421
I20240503 02:00:05.528831 6871 tablet_copy_service.cc:433] P 492448570cc94ed1879feb2ec3b9051d: ending tablet copy session e1a9251ce1cf4361b8b0ede9fbfc4023-8246488dec204bf6b09e868d7951ac5a on tablet 8246488dec204bf6b09e868d7951ac5a with peer e1a9251ce1cf4361b8b0ede9fbfc4023
I20240503 02:00:05.548785 11357 consensus_queue.cc:1035] T a3175dfc79d641a3bd70f2df5c579bdc P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:05.553383 9758 consensus_queue.cc:1035] T a3175dfc79d641a3bd70f2df5c579bdc P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:05.559728 10345 raft_consensus.cc:1050] T 7df4c083550046e883e5e61d3d4e1572 P 80fd18ef4d604c86b909d5c008a46b4f: attempting to promote NON_VOTER 2a01ace80baf4057b6bf1995cf88468b to VOTER
I20240503 02:00:05.560186 10345 consensus_queue.cc:237] T 7df4c083550046e883e5e61d3d4e1572 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } }
I20240503 02:00:05.561936 6456 raft_consensus.cc:1261] T 7df4c083550046e883e5e61d3d4e1572 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:05.561978 9176 raft_consensus.cc:1261] T 7df4c083550046e883e5e61d3d4e1572 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEARNER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:05.563048 6312 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_extra_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:05.563230 11492 consensus_queue.cc:1035] T 7df4c083550046e883e5e61d3d4e1572 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:05.565102 12331 consensus_queue.cc:1035] T 7df4c083550046e883e5e61d3d4e1572 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:05.566242 8916 raft_consensus.cc:1261] T 7df4c083550046e883e5e61d3d4e1572 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:05.566740 12330 consensus_queue.cc:1035] T 7df4c083550046e883e5e61d3d4e1572 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:05.567664 6314 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:05.570852 11492 raft_consensus.cc:2932] T 7df4c083550046e883e5e61d3d4e1572 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:05.572106 6436 tablet_service.cc:1467] Processing CreateTablet for tablet 9cbb2be1fda74ebe8997123c70629ba3 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=9b52669a75314ef88150c5df1cac862c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:05.572458 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cbb2be1fda74ebe8997123c70629ba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:05.572625 6831 tablet_service.cc:1467] Processing CreateTablet for tablet 13756a6e44814fe4b09a6b0f087546c3 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=9b52669a75314ef88150c5df1cac862c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:05.572896 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13756a6e44814fe4b09a6b0f087546c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:05.573540 9158 tablet_service.cc:1467] Processing CreateTablet for tablet aa281822047841fab7c58b53d9733e49 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=9b52669a75314ef88150c5df1cac862c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:05.573556 9286 tablet_service.cc:1467] Processing CreateTablet for tablet 13756a6e44814fe4b09a6b0f087546c3 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=9b52669a75314ef88150c5df1cac862c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:05.573812 9286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13756a6e44814fe4b09a6b0f087546c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:05.573822 9158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa281822047841fab7c58b53d9733e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:05.575556 9285 tablet_service.cc:1467] Processing CreateTablet for tablet aa281822047841fab7c58b53d9733e49 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=9b52669a75314ef88150c5df1cac862c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:05.575810 9285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa281822047841fab7c58b53d9733e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:05.575884 9027 tablet_service.cc:1467] Processing CreateTablet for tablet aa281822047841fab7c58b53d9733e49 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=9b52669a75314ef88150c5df1cac862c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:05.576125 9027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa281822047841fab7c58b53d9733e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:05.576483 8880 tablet_service.cc:1467] Processing CreateTablet for tablet 9cbb2be1fda74ebe8997123c70629ba3 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=9b52669a75314ef88150c5df1cac862c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:05.576741 8880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cbb2be1fda74ebe8997123c70629ba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:05.578430 9025 tablet_service.cc:1467] Processing CreateTablet for tablet 13756a6e44814fe4b09a6b0f087546c3 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=9b52669a75314ef88150c5df1cac862c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:05.578686 9025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13756a6e44814fe4b09a6b0f087546c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:05.579358 12337 tablet_bootstrap.cc:492] T 13756a6e44814fe4b09a6b0f087546c3 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:05.580932 6314 catalog_manager.cc:5484] T 7df4c083550046e883e5e61d3d4e1572 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: config changed from index 7 to 8, 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:05.581025 9028 tablet_service.cc:1467] Processing CreateTablet for tablet 9cbb2be1fda74ebe8997123c70629ba3 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=9b52669a75314ef88150c5df1cac862c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:05.581277 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cbb2be1fda74ebe8997123c70629ba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:05.581192 9176 raft_consensus.cc:2932] T 7df4c083550046e883e5e61d3d4e1572 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:05.580487 12339 tablet_bootstrap.cc:492] T aa281822047841fab7c58b53d9733e49 P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap starting.
I20240503 02:00:05.582721 12337 tablet_bootstrap.cc:654] T 13756a6e44814fe4b09a6b0f087546c3 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:05.583573 12172 tablet.cc:1722] T 10020e5222fc4c68be3d9a7957727d4a P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:05.583791 12339 tablet_bootstrap.cc:654] T aa281822047841fab7c58b53d9733e49 P e1a9251ce1cf4361b8b0ede9fbfc4023: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:05.583999 10349 raft_consensus.cc:1050] T 3dc213593e4048c99ccbe10161ff8e7f P c2ac2eb7cc264047b9ab32fe3a4fc025: attempting to promote NON_VOTER 99ead02915954b4fab23b033ad6d546b to VOTER
I20240503 02:00:05.584441 10349 consensus_queue.cc:237] T 3dc213593e4048c99ccbe10161ff8e7f P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } }
I20240503 02:00:05.585507 12341 tablet_bootstrap.cc:492] T 9cbb2be1fda74ebe8997123c70629ba3 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:05.586407 6849 raft_consensus.cc:1261] T 3dc213593e4048c99ccbe10161ff8e7f P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:05.586534 12341 tablet_bootstrap.cc:654] T 9cbb2be1fda74ebe8997123c70629ba3 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:05.586822 9711 consensus_queue.cc:1035] T 3dc213593e4048c99ccbe10161ff8e7f P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:05.587620 9048 raft_consensus.cc:1261] T 3dc213593e4048c99ccbe10161ff8e7f P 99ead02915954b4fab23b033ad6d546b [term 1 LEARNER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:05.588825 12340 tablet_bootstrap.cc:492] T aa281822047841fab7c58b53d9733e49 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:05.588891 9711 consensus_queue.cc:1035] T 3dc213593e4048c99ccbe10161ff8e7f P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:05.589018 12338 tablet_bootstrap.cc:492] T aa281822047841fab7c58b53d9733e49 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:05.588959 8916 raft_consensus.cc:2932] T 7df4c083550046e883e5e61d3d4e1572 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:05.590382 12174 tablet.cc:1722] T 10020e5222fc4c68be3d9a7957727d4a P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:05.590708 6582 raft_consensus.cc:1261] T 3dc213593e4048c99ccbe10161ff8e7f P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:05.591253 9984 consensus_queue.cc:1035] T 3dc213593e4048c99ccbe10161ff8e7f P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:05.591356 12340 tablet_bootstrap.cc:654] T aa281822047841fab7c58b53d9733e49 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:05.592523 12338 tablet_bootstrap.cc:654] T aa281822047841fab7c58b53d9733e49 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:05.593746 12338 tablet_bootstrap.cc:492] T aa281822047841fab7c58b53d9733e49 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:05.594095 12338 ts_tablet_manager.cc:1367] T aa281822047841fab7c58b53d9733e49 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20240503 02:00:05.594568 12338 raft_consensus.cc:348] T aa281822047841fab7c58b53d9733e49 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:05.594875 12173 tablet.cc:1722] T 10020e5222fc4c68be3d9a7957727d4a P e1a9251ce1cf4361b8b0ede9fbfc4023: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:05.594920 6456 consensus_queue.cc:237] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } }
I20240503 02:00:05.595413 12341 tablet_bootstrap.cc:492] T 9cbb2be1fda74ebe8997123c70629ba3 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:05.595515 12341 ts_tablet_manager.cc:1367] T 9cbb2be1fda74ebe8997123c70629ba3 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 02:00:05.595721 12339 tablet_bootstrap.cc:492] T aa281822047841fab7c58b53d9733e49 P e1a9251ce1cf4361b8b0ede9fbfc4023: No bootstrap required, opened a new log
I20240503 02:00:05.595801 12339 ts_tablet_manager.cc:1367] T aa281822047841fab7c58b53d9733e49 P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240503 02:00:05.595978 12341 raft_consensus.cc:348] T 9cbb2be1fda74ebe8997123c70629ba3 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:05.596082 12341 raft_consensus.cc:374] T 9cbb2be1fda74ebe8997123c70629ba3 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:05.596112 12341 raft_consensus.cc:729] T 9cbb2be1fda74ebe8997123c70629ba3 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:05.596208 12341 consensus_queue.cc:260] T 9cbb2be1fda74ebe8997123c70629ba3 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:05.596982 6719 raft_consensus.cc:1261] T ac7fa0c3722a461c91fb237fd85794a3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:05.597700 11842 consensus_queue.cc:1035] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:05.597924 12341 ts_tablet_manager.cc:1398] T 9cbb2be1fda74ebe8997123c70629ba3 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20240503 02:00:05.597990 6392 consensus_peers.cc:485] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a -> Peer 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157): Couldn't send request to peer 99ead02915954b4fab23b033ad6d546b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ac7fa0c3722a461c91fb237fd85794a3. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:05.598706 6582 raft_consensus.cc:1261] T ac7fa0c3722a461c91fb237fd85794a3 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:05.598870 12339 raft_consensus.cc:348] T aa281822047841fab7c58b53d9733e49 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:05.599691 12337 tablet_bootstrap.cc:492] T 13756a6e44814fe4b09a6b0f087546c3 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:05.599781 12338 raft_consensus.cc:374] T aa281822047841fab7c58b53d9733e49 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:05.599844 12338 raft_consensus.cc:729] T aa281822047841fab7c58b53d9733e49 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:05.599975 12338 consensus_queue.cc:260] T aa281822047841fab7c58b53d9733e49 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:05.600157 12337 ts_tablet_manager.cc:1367] T 13756a6e44814fe4b09a6b0f087546c3 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.000s
I20240503 02:00:05.600972 12337 raft_consensus.cc:348] T 13756a6e44814fe4b09a6b0f087546c3 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:05.601444 12338 ts_tablet_manager.cc:1398] T aa281822047841fab7c58b53d9733e49 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.007s user 0.000s sys 0.001s
I20240503 02:00:05.601585 12337 raft_consensus.cc:374] T 13756a6e44814fe4b09a6b0f087546c3 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:05.601636 12337 raft_consensus.cc:729] T 13756a6e44814fe4b09a6b0f087546c3 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:05.601768 12339 raft_consensus.cc:374] T aa281822047841fab7c58b53d9733e49 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:05.601769 12337 consensus_queue.cc:260] T 13756a6e44814fe4b09a6b0f087546c3 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:05.601828 12339 raft_consensus.cc:729] T aa281822047841fab7c58b53d9733e49 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1a9251ce1cf4361b8b0ede9fbfc4023, State: Initialized, Role: FOLLOWER
I20240503 02:00:05.602043 12337 ts_tablet_manager.cc:1398] T 13756a6e44814fe4b09a6b0f087546c3 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:05.601994 12339 consensus_queue.cc:260] T aa281822047841fab7c58b53d9733e49 P e1a9251ce1cf4361b8b0ede9fbfc4023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:05.602155 12190 consensus_queue.cc:1035] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:05.602797 12339 ts_tablet_manager.cc:1398] T aa281822047841fab7c58b53d9733e49 P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:00:05.602871 12340 tablet_bootstrap.cc:492] T aa281822047841fab7c58b53d9733e49 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:05.602973 12340 ts_tablet_manager.cc:1367] T aa281822047841fab7c58b53d9733e49 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 02:00:05.603112 12339 tablet_bootstrap.cc:492] T 13756a6e44814fe4b09a6b0f087546c3 P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap starting.
I20240503 02:00:05.603438 12340 raft_consensus.cc:348] T aa281822047841fab7c58b53d9733e49 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:05.603554 12340 raft_consensus.cc:374] T aa281822047841fab7c58b53d9733e49 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:05.603590 12340 raft_consensus.cc:729] T aa281822047841fab7c58b53d9733e49 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:05.603742 12340 consensus_queue.cc:260] T aa281822047841fab7c58b53d9733e49 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:05.604079 12340 ts_tablet_manager.cc:1398] T aa281822047841fab7c58b53d9733e49 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:05.604683 12340 tablet_bootstrap.cc:492] T 13756a6e44814fe4b09a6b0f087546c3 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:05.605741 12340 tablet_bootstrap.cc:654] T 13756a6e44814fe4b09a6b0f087546c3 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:05.605890 6312 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=2691a19595c54d3095907c7a788aada6] alter complete (version 3)
I20240503 02:00:05.607298 6589 consensus_queue.cc:237] T 7df4c083550046e883e5e61d3d4e1572 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } }
I20240503 02:00:05.608819 12339 tablet_bootstrap.cc:654] T 13756a6e44814fe4b09a6b0f087546c3 P e1a9251ce1cf4361b8b0ede9fbfc4023: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:05.610318 6456 raft_consensus.cc:1261] T 7df4c083550046e883e5e61d3d4e1572 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:05.610858 9973 consensus_queue.cc:1035] T 7df4c083550046e883e5e61d3d4e1572 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:05.611757 6456 raft_consensus.cc:2932] T 7df4c083550046e883e5e61d3d4e1572 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:05.613082 9176 raft_consensus.cc:1261] T 7df4c083550046e883e5e61d3d4e1572 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:05.613493 9758 raft_consensus.cc:482] T 9cbb2be1fda74ebe8997123c70629ba3 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:05.613529 9973 consensus_queue.cc:1035] T 7df4c083550046e883e5e61d3d4e1572 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:05.613585 9758 raft_consensus.cc:504] T 9cbb2be1fda74ebe8997123c70629ba3 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:05.613941 9758 leader_election.cc:288] T 9cbb2be1fda74ebe8997123c70629ba3 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 02:00:05.614429 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cbb2be1fda74ebe8997123c70629ba3" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
W20240503 02:00:05.614928 8850 leader_election.cc:341] T 9cbb2be1fda74ebe8997123c70629ba3 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:00:05.615428 9047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cbb2be1fda74ebe8997123c70629ba3" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:05.615412 10345 raft_consensus.cc:2932] T 7df4c083550046e883e5e61d3d4e1572 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:05.615728 12336 tablet_bootstrap.cc:492] T 9cbb2be1fda74ebe8997123c70629ba3 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:05.616272 6458 raft_consensus.cc:2932] T 7df4c083550046e883e5e61d3d4e1572 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:05.616931 12340 tablet_bootstrap.cc:492] T 13756a6e44814fe4b09a6b0f087546c3 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:05.617020 12340 ts_tablet_manager.cc:1367] T 13756a6e44814fe4b09a6b0f087546c3 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 02:00:05.617492 12340 raft_consensus.cc:348] T 13756a6e44814fe4b09a6b0f087546c3 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:05.617595 12340 raft_consensus.cc:374] T 13756a6e44814fe4b09a6b0f087546c3 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:05.617626 12340 raft_consensus.cc:729] T 13756a6e44814fe4b09a6b0f087546c3 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:05.617751 12340 consensus_queue.cc:260] T 13756a6e44814fe4b09a6b0f087546c3 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:05.618036 12340 ts_tablet_manager.cc:1398] T 13756a6e44814fe4b09a6b0f087546c3 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:05.618263 12340 tablet_bootstrap.cc:492] T 9cbb2be1fda74ebe8997123c70629ba3 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:05.619292 12340 tablet_bootstrap.cc:654] T 9cbb2be1fda74ebe8997123c70629ba3 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:05.619287 11967 raft_consensus.cc:2932] T 3dc213593e4048c99ccbe10161ff8e7f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 99ead02915954b4fab23b033ad6d546b (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
W20240503 02:00:05.619928 8851 leader_election.cc:341] T 9cbb2be1fda74ebe8997123c70629ba3 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:00:05.620021 8851 leader_election.cc:302] T 9cbb2be1fda74ebe8997123c70629ba3 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9942306855404cfd891adf227dd65eb2; no voters: 99ead02915954b4fab23b033ad6d546b, dec3e4a04d604f7eaaa3f077d081218a
I20240503 02:00:05.620177 9758 raft_consensus.cc:2726] T 9cbb2be1fda74ebe8997123c70629ba3 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:00:05.621284 9176 raft_consensus.cc:2932] T 7df4c083550046e883e5e61d3d4e1572 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:05.621532 6302 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7df4c083550046e883e5e61d3d4e1572 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:05.621994 12339 tablet_bootstrap.cc:492] T 13756a6e44814fe4b09a6b0f087546c3 P e1a9251ce1cf4361b8b0ede9fbfc4023: No bootstrap required, opened a new log
I20240503 02:00:05.621886 6582 raft_consensus.cc:2932] T 3dc213593e4048c99ccbe10161ff8e7f P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 99ead02915954b4fab23b033ad6d546b (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:05.622081 12339 ts_tablet_manager.cc:1367] T 13756a6e44814fe4b09a6b0f087546c3 P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240503 02:00:05.622241 6315 catalog_manager.cc:5484] T 7df4c083550046e883e5e61d3d4e1572 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: config changed from index 8 to 9, VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:05.622552 12339 raft_consensus.cc:348] T 13756a6e44814fe4b09a6b0f087546c3 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:05.622648 12339 raft_consensus.cc:374] T 13756a6e44814fe4b09a6b0f087546c3 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:05.622675 12339 raft_consensus.cc:729] T 13756a6e44814fe4b09a6b0f087546c3 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1a9251ce1cf4361b8b0ede9fbfc4023, State: Initialized, Role: FOLLOWER
I20240503 02:00:05.622800 12339 consensus_queue.cc:260] T 13756a6e44814fe4b09a6b0f087546c3 P e1a9251ce1cf4361b8b0ede9fbfc4023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:05.623113 12339 ts_tablet_manager.cc:1398] T 13756a6e44814fe4b09a6b0f087546c3 P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:05.623637 12352 raft_consensus.cc:2932] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } } }
I20240503 02:00:05.625851 6582 raft_consensus.cc:2932] T ac7fa0c3722a461c91fb237fd85794a3 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } } }
I20240503 02:00:05.627925 6307 catalog_manager.cc:5484] T 3dc213593e4048c99ccbe10161ff8e7f P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: config changed from index 4 to 5, 99ead02915954b4fab23b033ad6d546b (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:05.628949 8880 tablet_service.cc:1514] Processing DeleteTablet for tablet 7df4c083550046e883e5e61d3d4e1572 with delete_type TABLET_DATA_TOMBSTONED (TS 9942306855404cfd891adf227dd65eb2 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:40296
I20240503 02:00:05.629489 12358 tablet_replica.cc:330] T 7df4c083550046e883e5e61d3d4e1572 P 9942306855404cfd891adf227dd65eb2: stopping tablet replica
I20240503 02:00:05.629711 12358 raft_consensus.cc:2229] T 7df4c083550046e883e5e61d3d4e1572 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:05.629987 12358 raft_consensus.cc:2258] T 7df4c083550046e883e5e61d3d4e1572 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:05.629948 6720 raft_consensus.cc:2932] T ac7fa0c3722a461c91fb237fd85794a3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } } }
I20240503 02:00:05.630813 12358 ts_tablet_manager.cc:1875] T 7df4c083550046e883e5e61d3d4e1572 P 9942306855404cfd891adf227dd65eb2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:05.633716 6309 catalog_manager.cc:5484] T ac7fa0c3722a461c91fb237fd85794a3 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: config changed from index -1 to 4, NON_VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } } }
I20240503 02:00:05.636706 12340 tablet_bootstrap.cc:492] T 9cbb2be1fda74ebe8997123c70629ba3 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:05.636605 9048 raft_consensus.cc:2932] T 3dc213593e4048c99ccbe10161ff8e7f P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 99ead02915954b4fab23b033ad6d546b (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:05.636816 12340 ts_tablet_manager.cc:1367] T 9cbb2be1fda74ebe8997123c70629ba3 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240503 02:00:05.637127 12358 ts_tablet_manager.cc:1888] T 7df4c083550046e883e5e61d3d4e1572 P 9942306855404cfd891adf227dd65eb2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 02:00:05.637219 12358 log.cc:1196] T 7df4c083550046e883e5e61d3d4e1572 P 9942306855404cfd891adf227dd65eb2: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/7df4c083550046e883e5e61d3d4e1572
I20240503 02:00:05.637257 12340 raft_consensus.cc:348] T 9cbb2be1fda74ebe8997123c70629ba3 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:05.637367 12340 raft_consensus.cc:374] T 9cbb2be1fda74ebe8997123c70629ba3 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:05.637401 12340 raft_consensus.cc:729] T 9cbb2be1fda74ebe8997123c70629ba3 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:05.637526 12340 consensus_queue.cc:260] T 9cbb2be1fda74ebe8997123c70629ba3 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:05.637794 12340 ts_tablet_manager.cc:1398] T 9cbb2be1fda74ebe8997123c70629ba3 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:05.638537 6303 catalog_manager.cc:4830] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211): tablet 7df4c083550046e883e5e61d3d4e1572 (table rebalance_test_table_15 [id=4e740b0167ed439291b74d77653ec386]) successfully deleted
I20240503 02:00:05.639173 6720 consensus_queue.cc:237] T 3dc213593e4048c99ccbe10161ff8e7f P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } }
I20240503 02:00:05.640939 6582 raft_consensus.cc:1261] T 3dc213593e4048c99ccbe10161ff8e7f P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:05.641641 9048 raft_consensus.cc:1261] T 3dc213593e4048c99ccbe10161ff8e7f P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:05.642046 9984 consensus_queue.cc:1035] T 3dc213593e4048c99ccbe10161ff8e7f P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:05.642498 9711 consensus_queue.cc:1035] T 3dc213593e4048c99ccbe10161ff8e7f P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:05.642608 12336 tablet_bootstrap.cc:654] T 9cbb2be1fda74ebe8997123c70629ba3 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:05.645172 12336 tablet_bootstrap.cc:492] T 9cbb2be1fda74ebe8997123c70629ba3 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:05.645076 9984 raft_consensus.cc:2932] T 3dc213593e4048c99ccbe10161ff8e7f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:05.645356 12336 ts_tablet_manager.cc:1367] T 9cbb2be1fda74ebe8997123c70629ba3 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.000s
I20240503 02:00:05.645951 12336 raft_consensus.cc:348] T 9cbb2be1fda74ebe8997123c70629ba3 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:05.646318 12336 raft_consensus.cc:374] T 9cbb2be1fda74ebe8997123c70629ba3 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:05.646402 12336 raft_consensus.cc:729] T 9cbb2be1fda74ebe8997123c70629ba3 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:05.646649 12336 consensus_queue.cc:260] T 9cbb2be1fda74ebe8997123c70629ba3 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:05.647076 12336 ts_tablet_manager.cc:1398] T 9cbb2be1fda74ebe8997123c70629ba3 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:05.647127 6303 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3dc213593e4048c99ccbe10161ff8e7f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:05.648204 6314 catalog_manager.cc:5484] T 3dc213593e4048c99ccbe10161ff8e7f P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: config changed from index 5 to 6, VOTER 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:05.648540 6582 raft_consensus.cc:2932] T 3dc213593e4048c99ccbe10161ff8e7f P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:05.652194 6831 tablet_service.cc:1514] Processing DeleteTablet for tablet 3dc213593e4048c99ccbe10161ff8e7f with delete_type TABLET_DATA_TOMBSTONED (TS 492448570cc94ed1879feb2ec3b9051d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52812
I20240503 02:00:05.652295 9048 raft_consensus.cc:2932] T 3dc213593e4048c99ccbe10161ff8e7f P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:05.652676 12365 tablet_replica.cc:330] T 3dc213593e4048c99ccbe10161ff8e7f P 492448570cc94ed1879feb2ec3b9051d: stopping tablet replica
I20240503 02:00:05.652956 12365 raft_consensus.cc:2229] T 3dc213593e4048c99ccbe10161ff8e7f P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:05.653095 12365 pending_rounds.cc:62] T 3dc213593e4048c99ccbe10161ff8e7f P 492448570cc94ed1879feb2ec3b9051d: Trying to abort 1 pending ops.
I20240503 02:00:05.653115 12365 pending_rounds.cc:68] T 3dc213593e4048c99ccbe10161ff8e7f P 492448570cc94ed1879feb2ec3b9051d: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023417776472690688 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "3dc213593e4048c99ccbe10161ff8e7f" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } } }
I20240503 02:00:05.653227 12365 raft_consensus.cc:2894] T 3dc213593e4048c99ccbe10161ff8e7f P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240503 02:00:05.653285 12365 raft_consensus.cc:2866] T 3dc213593e4048c99ccbe10161ff8e7f P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 02:00:05.653327 12365 raft_consensus.cc:2258] T 3dc213593e4048c99ccbe10161ff8e7f P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:05.653985 12365 ts_tablet_manager.cc:1875] T 3dc213593e4048c99ccbe10161ff8e7f P 492448570cc94ed1879feb2ec3b9051d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:05.655881 6720 consensus_queue.cc:237] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } }
I20240503 02:00:05.656714 12365 ts_tablet_manager.cc:1888] T 3dc213593e4048c99ccbe10161ff8e7f P 492448570cc94ed1879feb2ec3b9051d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:00:05.656808 12365 log.cc:1196] T 3dc213593e4048c99ccbe10161ff8e7f P 492448570cc94ed1879feb2ec3b9051d: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/3dc213593e4048c99ccbe10161ff8e7f
I20240503 02:00:05.657903 6303 catalog_manager.cc:4830] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719): tablet 3dc213593e4048c99ccbe10161ff8e7f (table rebalance_test_table_19 [id=250f0720cafa4425b45a0360ebe04a03]) successfully deleted
I20240503 02:00:05.658041 9176 raft_consensus.cc:1261] T 1feaf29d8c6b4842829ce66e0f4ec23b P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:05.658169 6458 raft_consensus.cc:1261] T 1feaf29d8c6b4842829ce66e0f4ec23b P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:05.658461 9984 consensus_queue.cc:1035] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 02:00:05.658684 6654 consensus_peers.cc:485] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1feaf29d8c6b4842829ce66e0f4ec23b. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:05.658699 10349 consensus_queue.cc:1035] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:05.660606 9984 raft_consensus.cc:2932] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } } }
I20240503 02:00:05.661617 9176 raft_consensus.cc:2932] T 1feaf29d8c6b4842829ce66e0f4ec23b P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } } }
I20240503 02:00:05.664464 12369 ts_tablet_manager.cc:902] T b91a933fb66b4075acb40c5b92ef4824 P 9942306855404cfd891adf227dd65eb2: Initiating tablet copy from peer c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:05.665127 12369 tablet_copy_client.cc:308] T b91a933fb66b4075acb40c5b92ef4824 P 9942306855404cfd891adf227dd65eb2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:42655
I20240503 02:00:05.665107 6458 raft_consensus.cc:2932] T 1feaf29d8c6b4842829ce66e0f4ec23b P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } } }
I20240503 02:00:05.665761 6739 tablet_copy_service.cc:140] P c2ac2eb7cc264047b9ab32fe3a4fc025: Received BeginTabletCopySession request for tablet b91a933fb66b4075acb40c5b92ef4824 from peer 9942306855404cfd891adf227dd65eb2 ({username='slave'} at 127.0.99.133:46077)
I20240503 02:00:05.665865 6739 tablet_copy_service.cc:161] P c2ac2eb7cc264047b9ab32fe3a4fc025: Beginning new tablet copy session on tablet b91a933fb66b4075acb40c5b92ef4824 from peer 9942306855404cfd891adf227dd65eb2 at {username='slave'} at 127.0.99.133:46077: session id = 9942306855404cfd891adf227dd65eb2-b91a933fb66b4075acb40c5b92ef4824
I20240503 02:00:05.666774 6739 tablet_copy_source_session.cc:205] T b91a933fb66b4075acb40c5b92ef4824 P c2ac2eb7cc264047b9ab32fe3a4fc025: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:05.667516 6307 catalog_manager.cc:5484] T 1feaf29d8c6b4842829ce66e0f4ec23b P 2a01ace80baf4057b6bf1995cf88468b reported cstate change: config changed from index 6 to 7, NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } } }
I20240503 02:00:05.668758 12369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b91a933fb66b4075acb40c5b92ef4824. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:05.670315 12032 raft_consensus.cc:1050] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d: attempting to promote NON_VOTER 2a01ace80baf4057b6bf1995cf88468b to VOTER
I20240503 02:00:05.670748 12032 consensus_queue.cc:237] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } }
I20240503 02:00:05.672554 6582 raft_consensus.cc:1261] T 1fa274c98fc147f29344ed537e2ccf17 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:05.673099 11376 consensus_queue.cc:1035] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:05.673532 12369 tablet_copy_client.cc:783] T b91a933fb66b4075acb40c5b92ef4824 P 9942306855404cfd891adf227dd65eb2: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:05.673544 9176 raft_consensus.cc:1261] T 1fa274c98fc147f29344ed537e2ccf17 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEARNER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:05.673902 11376 consensus_queue.cc:1035] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:05.673979 12369 tablet_copy_client.cc:647] T b91a933fb66b4075acb40c5b92ef4824 P 9942306855404cfd891adf227dd65eb2: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:05.674288 9309 raft_consensus.cc:1261] T 1fa274c98fc147f29344ed537e2ccf17 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:05.674809 11376 consensus_queue.cc:1035] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:05.675122 12369 tablet_copy_client.cc:523] T b91a933fb66b4075acb40c5b92ef4824 P 9942306855404cfd891adf227dd65eb2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:05.676854 12369 tablet_bootstrap.cc:492] T b91a933fb66b4075acb40c5b92ef4824 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:05.677731 11376 raft_consensus.cc:2932] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:05.679863 6582 raft_consensus.cc:2932] T 1fa274c98fc147f29344ed537e2ccf17 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:05.680696 9309 raft_consensus.cc:2932] T 1fa274c98fc147f29344ed537e2ccf17 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:05.682909 6314 catalog_manager.cc:5484] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d reported cstate change: config changed from index 7 to 8, 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:05.683655 9176 raft_consensus.cc:2932] T 1fa274c98fc147f29344ed537e2ccf17 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:05.686180 6849 consensus_queue.cc:237] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } }
I20240503 02:00:05.687767 9179 raft_consensus.cc:1261] T 1fa274c98fc147f29344ed537e2ccf17 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:05.688234 12032 consensus_queue.cc:1035] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:05.688797 9309 raft_consensus.cc:1261] T 1fa274c98fc147f29344ed537e2ccf17 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:05.689914 12032 consensus_queue.cc:1035] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:05.690842 11373 raft_consensus.cc:2932] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:05.691262 9179 raft_consensus.cc:2932] T 1fa274c98fc147f29344ed537e2ccf17 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:05.693208 6303 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1fa274c98fc147f29344ed537e2ccf17 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:05.694342 6309 catalog_manager.cc:5484] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d reported cstate change: config changed from index 8 to 9, VOTER 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:05.694919 9309 raft_consensus.cc:2932] T 1fa274c98fc147f29344ed537e2ccf17 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:05.697654 6567 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fa274c98fc147f29344ed537e2ccf17 with delete_type TABLET_DATA_TOMBSTONED (TS 80fd18ef4d604c86b909d5c008a46b4f not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:59846
I20240503 02:00:05.698114 12390 tablet_replica.cc:330] T 1fa274c98fc147f29344ed537e2ccf17 P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 02:00:05.698261 12390 raft_consensus.cc:2229] T 1fa274c98fc147f29344ed537e2ccf17 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:05.698506 12390 raft_consensus.cc:2258] T 1fa274c98fc147f29344ed537e2ccf17 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:05.699393 12390 ts_tablet_manager.cc:1875] T 1fa274c98fc147f29344ed537e2ccf17 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:05.702055 12390 ts_tablet_manager.cc:1888] T 1fa274c98fc147f29344ed537e2ccf17 P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 02:00:05.702229 12390 log.cc:1196] T 1fa274c98fc147f29344ed537e2ccf17 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/1fa274c98fc147f29344ed537e2ccf17
I20240503 02:00:05.702844 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet 1fa274c98fc147f29344ed537e2ccf17 (table rebalance_test_table_1 [id=6e24d367bc1d434584d8a7363cf538b6]) successfully deleted
I20240503 02:00:05.711396 6307 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:05.711575 6307 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:05.712993 12369 tablet_bootstrap.cc:492] T b91a933fb66b4075acb40c5b92ef4824 P 9942306855404cfd891adf227dd65eb2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:05.713809 12369 tablet_bootstrap.cc:492] T b91a933fb66b4075acb40c5b92ef4824 P 9942306855404cfd891adf227dd65eb2: Bootstrap complete.
I20240503 02:00:05.713974 6307 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet 10020e5222fc4c68be3d9a7957727d4a
I20240503 02:00:05.714025 12369 ts_tablet_manager.cc:1367] T b91a933fb66b4075acb40c5b92ef4824 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.037s user 0.025s sys 0.000s
I20240503 02:00:05.714725 9158 tablet_service.cc:1514] Processing DeleteTablet for tablet 10020e5222fc4c68be3d9a7957727d4a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:05 UTC) from {username='slave'} at 127.0.0.1:51940
I20240503 02:00:05.714946 6567 tablet_service.cc:1514] Processing DeleteTablet for tablet 10020e5222fc4c68be3d9a7957727d4a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:05 UTC) from {username='slave'} at 127.0.0.1:59846
I20240503 02:00:05.714972 9286 tablet_service.cc:1514] Processing DeleteTablet for tablet 10020e5222fc4c68be3d9a7957727d4a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:05 UTC) from {username='slave'} at 127.0.0.1:36004
I20240503 02:00:05.715132 12390 tablet_replica.cc:330] T 10020e5222fc4c68be3d9a7957727d4a P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 02:00:05.715133 12401 tablet_replica.cc:330] T 10020e5222fc4c68be3d9a7957727d4a P 2a01ace80baf4057b6bf1995cf88468b: stopping tablet replica
I20240503 02:00:05.714516 12369 raft_consensus.cc:348] T b91a933fb66b4075acb40c5b92ef4824 P 9942306855404cfd891adf227dd65eb2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } }
I20240503 02:00:05.715220 12369 raft_consensus.cc:729] T b91a933fb66b4075acb40c5b92ef4824 P 9942306855404cfd891adf227dd65eb2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: LEARNER
I20240503 02:00:05.715261 12390 raft_consensus.cc:2229] T 10020e5222fc4c68be3d9a7957727d4a P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:05.715265 12401 raft_consensus.cc:2229] T 10020e5222fc4c68be3d9a7957727d4a P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:05.715325 12369 consensus_queue.cc:260] T b91a933fb66b4075acb40c5b92ef4824 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } }
I20240503 02:00:05.715389 12390 raft_consensus.cc:2258] T 10020e5222fc4c68be3d9a7957727d4a P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:05.715543 12401 raft_consensus.cc:2258] T 10020e5222fc4c68be3d9a7957727d4a P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:05.715565 12288 tablet_replica.cc:330] T 10020e5222fc4c68be3d9a7957727d4a P e1a9251ce1cf4361b8b0ede9fbfc4023: stopping tablet replica
I20240503 02:00:05.715693 12288 raft_consensus.cc:2229] T 10020e5222fc4c68be3d9a7957727d4a P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:05.715821 12288 raft_consensus.cc:2258] T 10020e5222fc4c68be3d9a7957727d4a P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:05.716454 12288 ts_tablet_manager.cc:1875] T 10020e5222fc4c68be3d9a7957727d4a P e1a9251ce1cf4361b8b0ede9fbfc4023: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:05.716495 12390 ts_tablet_manager.cc:1875] T 10020e5222fc4c68be3d9a7957727d4a P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:05.716581 12401 ts_tablet_manager.cc:1875] T 10020e5222fc4c68be3d9a7957727d4a P 2a01ace80baf4057b6bf1995cf88468b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:05.718271 12369 ts_tablet_manager.cc:1398] T b91a933fb66b4075acb40c5b92ef4824 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 02:00:05.718837 12288 ts_tablet_manager.cc:1888] T 10020e5222fc4c68be3d9a7957727d4a P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:05.718914 12288 log.cc:1196] T 10020e5222fc4c68be3d9a7957727d4a P e1a9251ce1cf4361b8b0ede9fbfc4023: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/10020e5222fc4c68be3d9a7957727d4a
I20240503 02:00:05.718909 12390 ts_tablet_manager.cc:1888] T 10020e5222fc4c68be3d9a7957727d4a P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:05.718983 12390 log.cc:1196] T 10020e5222fc4c68be3d9a7957727d4a P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/10020e5222fc4c68be3d9a7957727d4a
I20240503 02:00:05.719177 12288 ts_tablet_manager.cc:1909] T 10020e5222fc4c68be3d9a7957727d4a P e1a9251ce1cf4361b8b0ede9fbfc4023: Deleting consensus metadata
I20240503 02:00:05.719229 12390 ts_tablet_manager.cc:1909] T 10020e5222fc4c68be3d9a7957727d4a P 80fd18ef4d604c86b909d5c008a46b4f: Deleting consensus metadata
I20240503 02:00:05.719296 12401 ts_tablet_manager.cc:1888] T 10020e5222fc4c68be3d9a7957727d4a P 2a01ace80baf4057b6bf1995cf88468b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:05.719362 12401 log.cc:1196] T 10020e5222fc4c68be3d9a7957727d4a P 2a01ace80baf4057b6bf1995cf88468b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/10020e5222fc4c68be3d9a7957727d4a
I20240503 02:00:05.720728 12401 ts_tablet_manager.cc:1909] T 10020e5222fc4c68be3d9a7957727d4a P 2a01ace80baf4057b6bf1995cf88468b: Deleting consensus metadata
I20240503 02:00:05.721669 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet 10020e5222fc4c68be3d9a7957727d4a (table rebalancer_dynamic_table_new_name [id=2691a19595c54d3095907c7a788aada6]) successfully deleted
I20240503 02:00:05.722084 6301 catalog_manager.cc:4830] TS e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): tablet 10020e5222fc4c68be3d9a7957727d4a (table rebalancer_dynamic_table_new_name [id=2691a19595c54d3095907c7a788aada6]) successfully deleted
I20240503 02:00:05.722920 6300 catalog_manager.cc:4830] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079): tablet 10020e5222fc4c68be3d9a7957727d4a (table rebalancer_dynamic_table_new_name [id=2691a19595c54d3095907c7a788aada6]) successfully deleted
I20240503 02:00:05.725464 6739 tablet_copy_service.cc:342] P c2ac2eb7cc264047b9ab32fe3a4fc025: Request end of tablet copy session 9942306855404cfd891adf227dd65eb2-b91a933fb66b4075acb40c5b92ef4824 received from {username='slave'} at 127.0.99.133:46077
I20240503 02:00:05.725557 6739 tablet_copy_service.cc:433] P c2ac2eb7cc264047b9ab32fe3a4fc025: ending tablet copy session 9942306855404cfd891adf227dd65eb2-b91a933fb66b4075acb40c5b92ef4824 on tablet b91a933fb66b4075acb40c5b92ef4824 with peer 9942306855404cfd891adf227dd65eb2
I20240503 02:00:05.725533 6849 consensus_queue.cc:237] T c3485701bf3b48f5ae2e1568afeed133 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } }
I20240503 02:00:05.725806 12330 raft_consensus.cc:1050] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f: attempting to promote NON_VOTER 9942306855404cfd891adf227dd65eb2 to VOTER
I20240503 02:00:05.726269 12330 consensus_queue.cc:237] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } }
I20240503 02:00:05.727375 10473 raft_consensus.cc:482] T aa281822047841fab7c58b53d9733e49 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:05.727484 10473 raft_consensus.cc:504] T aa281822047841fab7c58b53d9733e49 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:05.727829 10473 leader_election.cc:288] T aa281822047841fab7c58b53d9733e49 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:05.727896 9308 raft_consensus.cc:1261] T 3416b6107e1848f299ac49ccd88ae7c8 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:05.728251 8917 raft_consensus.cc:1261] T 3416b6107e1848f299ac49ccd88ae7c8 P 9942306855404cfd891adf227dd65eb2 [term 1 LEARNER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:05.728251 9176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa281822047841fab7c58b53d9733e49" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:05.728349 9309 raft_consensus.cc:1261] T c3485701bf3b48f5ae2e1568afeed133 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 2 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:00:05.728404 9176 raft_consensus.cc:2454] T aa281822047841fab7c58b53d9733e49 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:05.728556 10345 consensus_queue.cc:1035] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:05.728729 8981 leader_election.cc:302] T aa281822047841fab7c58b53d9733e49 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:05.728963 12032 consensus_queue.cc:1035] T c3485701bf3b48f5ae2e1568afeed133 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:05.730684 12330 consensus_queue.cc:1035] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:05.731283 12032 raft_consensus.cc:2932] T c3485701bf3b48f5ae2e1568afeed133 P 492448570cc94ed1879feb2ec3b9051d [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:05.731824 9711 raft_consensus.cc:1050] T d897c21e630d4770832bb10f43778e82 P c2ac2eb7cc264047b9ab32fe3a4fc025: attempting to promote NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 to VOTER
I20240503 02:00:05.732261 9711 consensus_queue.cc:237] T d897c21e630d4770832bb10f43778e82 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } }
I20240503 02:00:05.732997 12331 raft_consensus.cc:2932] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:05.733520 9309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa281822047841fab7c58b53d9733e49" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" is_pre_election: true
I20240503 02:00:05.733649 9309 raft_consensus.cc:2454] T aa281822047841fab7c58b53d9733e49 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:05.734031 9308 raft_consensus.cc:2932] T c3485701bf3b48f5ae2e1568afeed133 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:05.734380 6720 raft_consensus.cc:1261] T c3485701bf3b48f5ae2e1568afeed133 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 2 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:00:05.734385 10473 raft_consensus.cc:2781] T aa281822047841fab7c58b53d9733e49 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:05.734469 10473 raft_consensus.cc:482] T aa281822047841fab7c58b53d9733e49 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:05.734506 10473 raft_consensus.cc:3037] T aa281822047841fab7c58b53d9733e49 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:05.735296 9309 raft_consensus.cc:2932] T 3416b6107e1848f299ac49ccd88ae7c8 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:05.735527 10473 raft_consensus.cc:504] T aa281822047841fab7c58b53d9733e49 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:05.735829 6303 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c3485701bf3b48f5ae2e1568afeed133 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:05.735939 10473 leader_election.cc:288] T aa281822047841fab7c58b53d9733e49 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Requested vote from peers e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:05.736227 6850 raft_consensus.cc:1261] T 3416b6107e1848f299ac49ccd88ae7c8 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:05.736558 6849 raft_consensus.cc:1261] T d897c21e630d4770832bb10f43778e82 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:05.737169 9176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa281822047841fab7c58b53d9733e49" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:05.737296 9176 raft_consensus.cc:3037] T aa281822047841fab7c58b53d9733e49 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:05.737195 8917 raft_consensus.cc:2932] T 3416b6107e1848f299ac49ccd88ae7c8 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:05.737576 6309 catalog_manager.cc:5484] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: config changed from index 7 to 8, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:05.737686 6458 raft_consensus.cc:1261] T d897c21e630d4770832bb10f43778e82 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:05.738332 9176 raft_consensus.cc:2454] T aa281822047841fab7c58b53d9733e49 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:05.738322 9984 consensus_queue.cc:1035] T d897c21e630d4770832bb10f43778e82 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:05.738664 8981 leader_election.cc:302] T aa281822047841fab7c58b53d9733e49 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:05.739367 10473 raft_consensus.cc:2781] T aa281822047841fab7c58b53d9733e49 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:05.739456 10473 raft_consensus.cc:686] T aa281822047841fab7c58b53d9733e49 P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Becoming Leader. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Running, Role: LEADER
I20240503 02:00:05.739579 10473 consensus_queue.cc:237] T aa281822047841fab7c58b53d9733e49 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:05.740121 9309 raft_consensus.cc:1261] T d897c21e630d4770832bb10f43778e82 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 LEARNER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:05.740547 9984 consensus_queue.cc:1035] T d897c21e630d4770832bb10f43778e82 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:05.741173 6315 catalog_manager.cc:5484] T c3485701bf3b48f5ae2e1568afeed133 P e1a9251ce1cf4361b8b0ede9fbfc4023 reported cstate change: config changed from index 5 to 7, VOTER 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:05.741775 6309 catalog_manager.cc:5484] T aa281822047841fab7c58b53d9733e49 P 99ead02915954b4fab23b033ad6d546b reported cstate change: term changed from 0 to 1, leader changed from <none> to 99ead02915954b4fab23b033ad6d546b (127.0.99.134). New cstate: current_term: 1 leader_uuid: "99ead02915954b4fab23b033ad6d546b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:05.741997 6582 consensus_queue.cc:237] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } }
I20240503 02:00:05.744256 9310 raft_consensus.cc:1261] T 3416b6107e1848f299ac49ccd88ae7c8 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:05.745085 8916 raft_consensus.cc:1261] T 3416b6107e1848f299ac49ccd88ae7c8 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:05.745530 10345 consensus_queue.cc:1035] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:05.746328 12380 consensus_queue.cc:1035] T c3485701bf3b48f5ae2e1568afeed133 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:05.746428 9984 consensus_queue.cc:1035] T d897c21e630d4770832bb10f43778e82 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:05.747573 6720 raft_consensus.cc:2932] T c3485701bf3b48f5ae2e1568afeed133 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:05.750283 10345 consensus_queue.cc:1035] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:05.751716 6567 tablet_service.cc:1514] Processing DeleteTablet for tablet c3485701bf3b48f5ae2e1568afeed133 with delete_type TABLET_DATA_TOMBSTONED (TS 80fd18ef4d604c86b909d5c008a46b4f not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:59846
I20240503 02:00:05.752043 12390 tablet_replica.cc:330] T c3485701bf3b48f5ae2e1568afeed133 P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 02:00:05.752200 12390 raft_consensus.cc:2229] T c3485701bf3b48f5ae2e1568afeed133 P 80fd18ef4d604c86b909d5c008a46b4f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:05.752316 12390 raft_consensus.cc:2258] T c3485701bf3b48f5ae2e1568afeed133 P 80fd18ef4d604c86b909d5c008a46b4f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:05.753144 12390 ts_tablet_manager.cc:1875] T c3485701bf3b48f5ae2e1568afeed133 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:05.755470 12390 ts_tablet_manager.cc:1888] T c3485701bf3b48f5ae2e1568afeed133 P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 02:00:05.755565 12390 log.cc:1196] T c3485701bf3b48f5ae2e1568afeed133 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/c3485701bf3b48f5ae2e1568afeed133
I20240503 02:00:05.756095 9309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa281822047841fab7c58b53d9733e49" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023"
I20240503 02:00:05.756124 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet c3485701bf3b48f5ae2e1568afeed133 (table rebalance_test_table_15 [id=4e740b0167ed439291b74d77653ec386]) successfully deleted
I20240503 02:00:05.756225 9309 raft_consensus.cc:3037] T aa281822047841fab7c58b53d9733e49 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:05.757149 9309 raft_consensus.cc:2454] T aa281822047841fab7c58b53d9733e49 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:05.757879 12330 raft_consensus.cc:2932] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:05.758574 9308 raft_consensus.cc:2932] T 3416b6107e1848f299ac49ccd88ae7c8 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:05.759814 8916 raft_consensus.cc:2932] T 3416b6107e1848f299ac49ccd88ae7c8 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:05.760236 6302 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3416b6107e1848f299ac49ccd88ae7c8 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:05.760857 9984 raft_consensus.cc:2932] T d897c21e630d4770832bb10f43778e82 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:05.761240 6309 catalog_manager.cc:5484] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: config changed from index 8 to 9, VOTER 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:05.763249 6849 raft_consensus.cc:2932] T d897c21e630d4770832bb10f43778e82 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:05.763855 6307 catalog_manager.cc:5484] T d897c21e630d4770832bb10f43778e82 P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: config changed from index 4 to 5, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:05.766543 6458 raft_consensus.cc:2932] T d897c21e630d4770832bb10f43778e82 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:05.768405 6831 tablet_service.cc:1514] Processing DeleteTablet for tablet 3416b6107e1848f299ac49ccd88ae7c8 with delete_type TABLET_DATA_TOMBSTONED (TS 492448570cc94ed1879feb2ec3b9051d not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:52812
I20240503 02:00:05.768270 9308 raft_consensus.cc:2932] T d897c21e630d4770832bb10f43778e82 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:05.768622 12365 tablet_replica.cc:330] T 3416b6107e1848f299ac49ccd88ae7c8 P 492448570cc94ed1879feb2ec3b9051d: stopping tablet replica
I20240503 02:00:05.768759 12365 raft_consensus.cc:2229] T 3416b6107e1848f299ac49ccd88ae7c8 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:05.768939 12365 raft_consensus.cc:2258] T 3416b6107e1848f299ac49ccd88ae7c8 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:05.769579 12365 ts_tablet_manager.cc:1875] T 3416b6107e1848f299ac49ccd88ae7c8 P 492448570cc94ed1879feb2ec3b9051d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:05.774044 12365 ts_tablet_manager.cc:1888] T 3416b6107e1848f299ac49ccd88ae7c8 P 492448570cc94ed1879feb2ec3b9051d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 02:00:05.774214 12365 log.cc:1196] T 3416b6107e1848f299ac49ccd88ae7c8 P 492448570cc94ed1879feb2ec3b9051d: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/3416b6107e1848f299ac49ccd88ae7c8
I20240503 02:00:05.775295 6303 catalog_manager.cc:4830] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719): tablet 3416b6107e1848f299ac49ccd88ae7c8 (table rebalance_test_table_7 [id=e0e5e479de824fb18516cfe56d5892c6]) successfully deleted
I20240503 02:00:05.789489 6582 raft_consensus.cc:1261] T cb873018448549ccbe3d14b784e47d26 P 80fd18ef4d604c86b909d5c008a46b4f [term 2 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:00:05.790163 11373 consensus_queue.cc:1035] T cb873018448549ccbe3d14b784e47d26 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:05.804879 6720 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d897c21e630d4770832bb10f43778e82"
dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44686
I20240503 02:00:05.805186 6720 raft_consensus.cc:595] T d897c21e630d4770832bb10f43778e82 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:00:05.815651 6314 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:00:05.816172 8916 raft_consensus.cc:1261] T cb873018448549ccbe3d14b784e47d26 P 9942306855404cfd891adf227dd65eb2 [term 2 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:00:05.816701 11373 consensus_queue.cc:1035] T cb873018448549ccbe3d14b784e47d26 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:05.821823 9048 raft_consensus.cc:1261] T cb873018448549ccbe3d14b784e47d26 P 99ead02915954b4fab23b033ad6d546b [term 2 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:00:05.822458 11373 consensus_queue.cc:1035] T cb873018448549ccbe3d14b784e47d26 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:05.824441 9286 tablet_service.cc:1467] Processing CreateTablet for tablet be6fc55aca6842329a3deb7c2b557b66 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=fac49b5a5e3845cebcdbfaeb3b0eb0c5]), partition=
I20240503 02:00:05.824729 9286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be6fc55aca6842329a3deb7c2b557b66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:05.824767 6567 tablet_service.cc:1467] Processing CreateTablet for tablet be6fc55aca6842329a3deb7c2b557b66 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=fac49b5a5e3845cebcdbfaeb3b0eb0c5]), partition=
I20240503 02:00:05.825024 6567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be6fc55aca6842329a3deb7c2b557b66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:05.826715 12339 tablet_bootstrap.cc:492] T be6fc55aca6842329a3deb7c2b557b66 P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap starting.
I20240503 02:00:05.827720 12339 tablet_bootstrap.cc:654] T be6fc55aca6842329a3deb7c2b557b66 P e1a9251ce1cf4361b8b0ede9fbfc4023: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:05.827703 8880 tablet_service.cc:1467] Processing CreateTablet for tablet be6fc55aca6842329a3deb7c2b557b66 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=fac49b5a5e3845cebcdbfaeb3b0eb0c5]), partition=
I20240503 02:00:05.827926 8880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be6fc55aca6842329a3deb7c2b557b66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:05.830003 12341 tablet_bootstrap.cc:492] T be6fc55aca6842329a3deb7c2b557b66 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:05.830932 12341 tablet_bootstrap.cc:654] T be6fc55aca6842329a3deb7c2b557b66 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:05.832273 11357 raft_consensus.cc:1050] T 9c8a53980b00421d8c2bd3ae73f0c8cc P 9942306855404cfd891adf227dd65eb2: attempting to promote NON_VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 to VOTER
I20240503 02:00:05.832756 12339 tablet_bootstrap.cc:492] T be6fc55aca6842329a3deb7c2b557b66 P e1a9251ce1cf4361b8b0ede9fbfc4023: No bootstrap required, opened a new log
I20240503 02:00:05.832720 11357 consensus_queue.cc:237] T 9c8a53980b00421d8c2bd3ae73f0c8cc P 9942306855404cfd891adf227dd65eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } }
I20240503 02:00:05.832834 12339 ts_tablet_manager.cc:1367] T be6fc55aca6842329a3deb7c2b557b66 P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:00:05.833268 12339 raft_consensus.cc:348] T be6fc55aca6842329a3deb7c2b557b66 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:05.833374 12339 raft_consensus.cc:374] T be6fc55aca6842329a3deb7c2b557b66 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:05.833407 12339 raft_consensus.cc:729] T be6fc55aca6842329a3deb7c2b557b66 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1a9251ce1cf4361b8b0ede9fbfc4023, State: Initialized, Role: FOLLOWER
I20240503 02:00:05.833527 12339 consensus_queue.cc:260] T be6fc55aca6842329a3deb7c2b557b66 P e1a9251ce1cf4361b8b0ede9fbfc4023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:05.834378 9048 raft_consensus.cc:1261] T 9c8a53980b00421d8c2bd3ae73f0c8cc P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:05.834406 12341 tablet_bootstrap.cc:492] T be6fc55aca6842329a3deb7c2b557b66 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:05.834488 12341 ts_tablet_manager.cc:1367] T be6fc55aca6842329a3deb7c2b557b66 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:00:05.834779 9758 consensus_queue.cc:1035] T 9c8a53980b00421d8c2bd3ae73f0c8cc P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:05.834919 12341 raft_consensus.cc:348] T be6fc55aca6842329a3deb7c2b557b66 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:05.835000 12341 raft_consensus.cc:374] T be6fc55aca6842329a3deb7c2b557b66 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:05.835029 12341 raft_consensus.cc:729] T be6fc55aca6842329a3deb7c2b557b66 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:05.835134 12341 consensus_queue.cc:260] T be6fc55aca6842329a3deb7c2b557b66 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:05.835430 12341 ts_tablet_manager.cc:1398] T be6fc55aca6842329a3deb7c2b557b66 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:05.835724 6458 raft_consensus.cc:1261] T 9c8a53980b00421d8c2bd3ae73f0c8cc P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:05.836223 6720 raft_consensus.cc:1261] T 9c8a53980b00421d8c2bd3ae73f0c8cc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEARNER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:05.836653 9758 consensus_queue.cc:1035] T 9c8a53980b00421d8c2bd3ae73f0c8cc P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:05.836920 11357 consensus_queue.cc:1035] T 9c8a53980b00421d8c2bd3ae73f0c8cc P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:05.837072 12339 ts_tablet_manager.cc:1398] T be6fc55aca6842329a3deb7c2b557b66 P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240503 02:00:05.838688 11357 raft_consensus.cc:2932] T 9c8a53980b00421d8c2bd3ae73f0c8cc P 9942306855404cfd891adf227dd65eb2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } } }
I20240503 02:00:05.839071 12444 tablet_bootstrap.cc:492] T be6fc55aca6842329a3deb7c2b557b66 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:05.839273 6720 raft_consensus.cc:2932] T 9c8a53980b00421d8c2bd3ae73f0c8cc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } } }
I20240503 02:00:05.840790 12444 tablet_bootstrap.cc:654] T be6fc55aca6842329a3deb7c2b557b66 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:05.842177 6458 raft_consensus.cc:2932] T 9c8a53980b00421d8c2bd3ae73f0c8cc P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } } }
I20240503 02:00:05.842854 12444 tablet_bootstrap.cc:492] T be6fc55aca6842329a3deb7c2b557b66 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:05.842963 12444 ts_tablet_manager.cc:1367] T be6fc55aca6842329a3deb7c2b557b66 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:05.843480 12444 raft_consensus.cc:348] T be6fc55aca6842329a3deb7c2b557b66 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:05.843690 12444 raft_consensus.cc:374] T be6fc55aca6842329a3deb7c2b557b66 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:05.843770 12444 raft_consensus.cc:729] T be6fc55aca6842329a3deb7c2b557b66 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:05.843927 12444 consensus_queue.cc:260] T be6fc55aca6842329a3deb7c2b557b66 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:05.844503 6307 catalog_manager.cc:5484] T 9c8a53980b00421d8c2bd3ae73f0c8cc P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: config changed from index 2 to 3, c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9942306855404cfd891adf227dd65eb2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } } }
I20240503 02:00:05.845005 12444 ts_tablet_manager.cc:1398] T be6fc55aca6842329a3deb7c2b557b66 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:05.846807 9048 raft_consensus.cc:2932] T 9c8a53980b00421d8c2bd3ae73f0c8cc P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } } }
I20240503 02:00:05.852766 8916 consensus_queue.cc:237] T 9c8a53980b00421d8c2bd3ae73f0c8cc P 9942306855404cfd891adf227dd65eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } }
I20240503 02:00:05.854326 9048 raft_consensus.cc:1261] T 9c8a53980b00421d8c2bd3ae73f0c8cc P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:05.854734 6720 raft_consensus.cc:1261] T 9c8a53980b00421d8c2bd3ae73f0c8cc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:05.854795 9758 consensus_queue.cc:1035] T 9c8a53980b00421d8c2bd3ae73f0c8cc P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:05.855067 11073 consensus_queue.cc:1035] T 9c8a53980b00421d8c2bd3ae73f0c8cc P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:05.856855 11073 raft_consensus.cc:2932] T 9c8a53980b00421d8c2bd3ae73f0c8cc P 9942306855404cfd891adf227dd65eb2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } } }
I20240503 02:00:05.857514 6720 raft_consensus.cc:2932] T 9c8a53980b00421d8c2bd3ae73f0c8cc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } } }
I20240503 02:00:05.858271 9048 raft_consensus.cc:2932] T 9c8a53980b00421d8c2bd3ae73f0c8cc P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } } }
I20240503 02:00:05.859642 6303 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9c8a53980b00421d8c2bd3ae73f0c8cc with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:05.860010 6307 catalog_manager.cc:5484] T 9c8a53980b00421d8c2bd3ae73f0c8cc P 9942306855404cfd891adf227dd65eb2 reported cstate change: config changed from index 3 to 4, VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "9942306855404cfd891adf227dd65eb2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:05.863211 6436 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c8a53980b00421d8c2bd3ae73f0c8cc with delete_type TABLET_DATA_TOMBSTONED (TS dec3e4a04d604f7eaaa3f077d081218a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:51338
I20240503 02:00:05.863613 12462 tablet_replica.cc:330] T 9c8a53980b00421d8c2bd3ae73f0c8cc P dec3e4a04d604f7eaaa3f077d081218a: stopping tablet replica
I20240503 02:00:05.864986 12462 raft_consensus.cc:2229] T 9c8a53980b00421d8c2bd3ae73f0c8cc P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:05.865164 12462 raft_consensus.cc:2258] T 9c8a53980b00421d8c2bd3ae73f0c8cc P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:05.865831 12462 ts_tablet_manager.cc:1875] T 9c8a53980b00421d8c2bd3ae73f0c8cc P dec3e4a04d604f7eaaa3f077d081218a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:05.868515 12462 ts_tablet_manager.cc:1888] T 9c8a53980b00421d8c2bd3ae73f0c8cc P dec3e4a04d604f7eaaa3f077d081218a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:00:05.868606 12462 log.cc:1196] T 9c8a53980b00421d8c2bd3ae73f0c8cc P dec3e4a04d604f7eaaa3f077d081218a: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/9c8a53980b00421d8c2bd3ae73f0c8cc
I20240503 02:00:05.869171 6300 catalog_manager.cc:4830] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): tablet 9c8a53980b00421d8c2bd3ae73f0c8cc (table rebalancer_extra_table_2 [id=54f1891be3f54f19821aad5d1dfbd85d]) successfully deleted
I20240503 02:00:05.893136 10349 raft_consensus.cc:1050] T 2e967e377984468d90799041c29882bd P c2ac2eb7cc264047b9ab32fe3a4fc025: attempting to promote NON_VOTER 99ead02915954b4fab23b033ad6d546b to VOTER
I20240503 02:00:05.893577 10349 consensus_queue.cc:237] T 2e967e377984468d90799041c29882bd P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } }
I20240503 02:00:05.894943 6458 raft_consensus.cc:1261] T 2e967e377984468d90799041c29882bd P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:05.894973 9048 raft_consensus.cc:1261] T 2e967e377984468d90799041c29882bd P 99ead02915954b4fab23b033ad6d546b [term 1 LEARNER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:05.894973 6582 raft_consensus.cc:1261] T 2e967e377984468d90799041c29882bd P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:05.895367 9984 consensus_queue.cc:1035] T 2e967e377984468d90799041c29882bd P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:05.895581 11967 consensus_queue.cc:1035] T 2e967e377984468d90799041c29882bd P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:05.895876 9711 consensus_queue.cc:1035] T 2e967e377984468d90799041c29882bd P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:05.898478 10349 raft_consensus.cc:2932] T 2e967e377984468d90799041c29882bd P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 99ead02915954b4fab23b033ad6d546b (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:05.898890 6582 raft_consensus.cc:2932] T 2e967e377984468d90799041c29882bd P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 99ead02915954b4fab23b033ad6d546b (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:05.901154 6458 raft_consensus.cc:2932] T 2e967e377984468d90799041c29882bd P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 99ead02915954b4fab23b033ad6d546b (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:05.902024 6314 catalog_manager.cc:5484] T 2e967e377984468d90799041c29882bd P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: config changed from index 4 to 5, 99ead02915954b4fab23b033ad6d546b (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:05.902042 9048 raft_consensus.cc:2932] T 2e967e377984468d90799041c29882bd P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 99ead02915954b4fab23b033ad6d546b (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:05.905550 10473 raft_consensus.cc:482] T 13756a6e44814fe4b09a6b0f087546c3 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:05.905650 10473 raft_consensus.cc:504] T 13756a6e44814fe4b09a6b0f087546c3 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:05.906018 10473 leader_election.cc:288] T 13756a6e44814fe4b09a6b0f087546c3 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:05.906446 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13756a6e44814fe4b09a6b0f087546c3" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 02:00:05.906589 6849 raft_consensus.cc:2454] T 13756a6e44814fe4b09a6b0f087546c3 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:05.906993 8984 leader_election.cc:302] T 13756a6e44814fe4b09a6b0f087546c3 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:05.906983 9308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13756a6e44814fe4b09a6b0f087546c3" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" is_pre_election: true
I20240503 02:00:05.907112 9308 raft_consensus.cc:2454] T 13756a6e44814fe4b09a6b0f087546c3 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:05.907337 11515 raft_consensus.cc:2781] T 13756a6e44814fe4b09a6b0f087546c3 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:05.907397 11515 raft_consensus.cc:482] T 13756a6e44814fe4b09a6b0f087546c3 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:05.907431 11515 raft_consensus.cc:3037] T 13756a6e44814fe4b09a6b0f087546c3 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:05.907449 6720 consensus_queue.cc:237] T 2e967e377984468d90799041c29882bd P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } }
I20240503 02:00:05.908457 11515 raft_consensus.cc:504] T 13756a6e44814fe4b09a6b0f087546c3 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:05.908727 6458 raft_consensus.cc:1261] T 2e967e377984468d90799041c29882bd P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:05.908809 11515 leader_election.cc:288] T 13756a6e44814fe4b09a6b0f087546c3 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Requested vote from peers e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:05.908814 9048 raft_consensus.cc:1261] T 2e967e377984468d90799041c29882bd P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:05.909121 9308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13756a6e44814fe4b09a6b0f087546c3" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023"
I20240503 02:00:05.909242 9308 raft_consensus.cc:3037] T 13756a6e44814fe4b09a6b0f087546c3 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:05.909253 9711 consensus_queue.cc:1035] T 2e967e377984468d90799041c29882bd P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:05.909839 12497 ts_tablet_manager.cc:902] T 6a82dcc9be5b4c93b5fff03f789350ab P dec3e4a04d604f7eaaa3f077d081218a: Initiating tablet copy from peer 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:05.910039 9308 raft_consensus.cc:2454] T 13756a6e44814fe4b09a6b0f087546c3 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:05.910421 8982 leader_election.cc:302] T 13756a6e44814fe4b09a6b0f087546c3 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99ead02915954b4fab23b033ad6d546b, e1a9251ce1cf4361b8b0ede9fbfc4023; no voters:
I20240503 02:00:05.910522 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13756a6e44814fe4b09a6b0f087546c3" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 02:00:05.910640 6849 raft_consensus.cc:3037] T 13756a6e44814fe4b09a6b0f087546c3 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:05.910830 9711 consensus_queue.cc:1035] T 2e967e377984468d90799041c29882bd P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:05.910948 10473 raft_consensus.cc:2781] T 13756a6e44814fe4b09a6b0f087546c3 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:05.911032 10473 raft_consensus.cc:686] T 13756a6e44814fe4b09a6b0f087546c3 P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Becoming Leader. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Running, Role: LEADER
I20240503 02:00:05.911152 10473 consensus_queue.cc:237] T 13756a6e44814fe4b09a6b0f087546c3 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:05.911682 6849 raft_consensus.cc:2454] T 13756a6e44814fe4b09a6b0f087546c3 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:05.912490 12497 tablet_copy_client.cc:308] T 6a82dcc9be5b4c93b5fff03f789350ab P dec3e4a04d604f7eaaa3f077d081218a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:46157
I20240503 02:00:05.913046 9068 tablet_copy_service.cc:140] P 99ead02915954b4fab23b033ad6d546b: Received BeginTabletCopySession request for tablet 6a82dcc9be5b4c93b5fff03f789350ab from peer dec3e4a04d604f7eaaa3f077d081218a ({username='slave'} at 127.0.99.129:53227)
I20240503 02:00:05.913162 9068 tablet_copy_service.cc:161] P 99ead02915954b4fab23b033ad6d546b: Beginning new tablet copy session on tablet 6a82dcc9be5b4c93b5fff03f789350ab from peer dec3e4a04d604f7eaaa3f077d081218a at {username='slave'} at 127.0.99.129:53227: session id = dec3e4a04d604f7eaaa3f077d081218a-6a82dcc9be5b4c93b5fff03f789350ab
I20240503 02:00:05.913098 9711 raft_consensus.cc:2932] T 2e967e377984468d90799041c29882bd P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:05.913137 6312 catalog_manager.cc:5484] T 13756a6e44814fe4b09a6b0f087546c3 P 99ead02915954b4fab23b033ad6d546b reported cstate change: term changed from 0 to 1, leader changed from <none> to 99ead02915954b4fab23b033ad6d546b (127.0.99.134). New cstate: current_term: 1 leader_uuid: "99ead02915954b4fab23b033ad6d546b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:05.914970 9068 tablet_copy_source_session.cc:205] T 6a82dcc9be5b4c93b5fff03f789350ab P 99ead02915954b4fab23b033ad6d546b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:05.915887 6314 catalog_manager.cc:5484] T 2e967e377984468d90799041c29882bd P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: config changed from index 5 to 6, VOTER 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:05.916124 12497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a82dcc9be5b4c93b5fff03f789350ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:05.916079 6458 raft_consensus.cc:2932] T 2e967e377984468d90799041c29882bd P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:05.916448 6303 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2e967e377984468d90799041c29882bd with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:05.916986 9048 raft_consensus.cc:2932] T 2e967e377984468d90799041c29882bd P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:05.919775 6567 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e967e377984468d90799041c29882bd with delete_type TABLET_DATA_TOMBSTONED (TS 80fd18ef4d604c86b909d5c008a46b4f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59846
I20240503 02:00:05.920042 12390 tablet_replica.cc:330] T 2e967e377984468d90799041c29882bd P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 02:00:05.920189 12390 raft_consensus.cc:2229] T 2e967e377984468d90799041c29882bd P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:05.920367 12390 raft_consensus.cc:2258] T 2e967e377984468d90799041c29882bd P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:05.920329 12497 tablet_copy_client.cc:783] T 6a82dcc9be5b4c93b5fff03f789350ab P dec3e4a04d604f7eaaa3f077d081218a: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:05.920975 12497 tablet_copy_client.cc:647] T 6a82dcc9be5b4c93b5fff03f789350ab P dec3e4a04d604f7eaaa3f077d081218a: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:05.921224 12390 ts_tablet_manager.cc:1875] T 2e967e377984468d90799041c29882bd P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:05.924664 12390 ts_tablet_manager.cc:1888] T 2e967e377984468d90799041c29882bd P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:00:05.924806 12390 log.cc:1196] T 2e967e377984468d90799041c29882bd P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/2e967e377984468d90799041c29882bd
I20240503 02:00:05.924957 12497 tablet_copy_client.cc:523] T 6a82dcc9be5b4c93b5fff03f789350ab P dec3e4a04d604f7eaaa3f077d081218a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:05.925779 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet 2e967e377984468d90799041c29882bd (table rebalance_test_table_7 [id=e0e5e479de824fb18516cfe56d5892c6]) successfully deleted
I20240503 02:00:05.925766 6458 consensus_queue.cc:237] T 51c0137930464635be1e6a97f5e9d952 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: true } }
I20240503 02:00:05.926499 12497 tablet_bootstrap.cc:492] T 6a82dcc9be5b4c93b5fff03f789350ab P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:05.927174 8916 raft_consensus.cc:1261] T 51c0137930464635be1e6a97f5e9d952 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:05.927197 9048 raft_consensus.cc:1261] T 51c0137930464635be1e6a97f5e9d952 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:00:05.927397 6394 consensus_peers.cc:485] T 51c0137930464635be1e6a97f5e9d952 P dec3e4a04d604f7eaaa3f077d081218a -> Peer c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655): Couldn't send request to peer c2ac2eb7cc264047b9ab32fe3a4fc025. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 51c0137930464635be1e6a97f5e9d952. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:05.927551 12352 consensus_queue.cc:1035] T 51c0137930464635be1e6a97f5e9d952 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:05.927783 12190 consensus_queue.cc:1035] T 51c0137930464635be1e6a97f5e9d952 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:05.929925 12352 raft_consensus.cc:2932] T 51c0137930464635be1e6a97f5e9d952 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: true } } }
I20240503 02:00:05.930434 9048 raft_consensus.cc:2932] T 51c0137930464635be1e6a97f5e9d952 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: true } } }
I20240503 02:00:05.930670 10679 raft_consensus.cc:1050] T 0ee05e70ccbb496ca3cff08dc38112e8 P 2a01ace80baf4057b6bf1995cf88468b: attempting to promote NON_VOTER dec3e4a04d604f7eaaa3f077d081218a to VOTER
I20240503 02:00:05.931068 10679 consensus_queue.cc:237] T 0ee05e70ccbb496ca3cff08dc38112e8 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } }
I20240503 02:00:05.931507 11357 raft_consensus.cc:482] T be6fc55aca6842329a3deb7c2b557b66 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:05.931596 11357 raft_consensus.cc:504] T be6fc55aca6842329a3deb7c2b557b66 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:05.931942 11357 leader_election.cc:288] T be6fc55aca6842329a3deb7c2b557b66 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:05.932600 6720 raft_consensus.cc:1261] T 0ee05e70ccbb496ca3cff08dc38112e8 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer 2a01ace80baf4057b6bf1995cf88468b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:05.932864 9308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be6fc55aca6842329a3deb7c2b557b66" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" is_pre_election: true
I20240503 02:00:05.933001 9308 raft_consensus.cc:2454] T be6fc55aca6842329a3deb7c2b557b66 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 0.
I20240503 02:00:05.933176 6582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be6fc55aca6842329a3deb7c2b557b66" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 02:00:05.933118 6307 catalog_manager.cc:5484] T 51c0137930464635be1e6a97f5e9d952 P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: config changed from index -1 to 2, NON_VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:05.933303 6582 raft_consensus.cc:2454] T be6fc55aca6842329a3deb7c2b557b66 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 0.
I20240503 02:00:05.933324 11502 consensus_queue.cc:1035] T 0ee05e70ccbb496ca3cff08dc38112e8 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:05.934020 8851 leader_election.cc:302] T be6fc55aca6842329a3deb7c2b557b66 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9942306855404cfd891adf227dd65eb2, e1a9251ce1cf4361b8b0ede9fbfc4023; no voters:
I20240503 02:00:05.934345 11357 raft_consensus.cc:2781] T be6fc55aca6842329a3deb7c2b557b66 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:05.934415 11357 raft_consensus.cc:482] T be6fc55aca6842329a3deb7c2b557b66 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:05.934394 6458 raft_consensus.cc:1261] T 0ee05e70ccbb496ca3cff08dc38112e8 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEARNER]: Refusing update from remote peer 2a01ace80baf4057b6bf1995cf88468b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:05.934450 11357 raft_consensus.cc:3037] T be6fc55aca6842329a3deb7c2b557b66 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:05.934792 12267 consensus_queue.cc:1035] T 0ee05e70ccbb496ca3cff08dc38112e8 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:05.935494 11357 raft_consensus.cc:504] T be6fc55aca6842329a3deb7c2b557b66 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:05.935858 11357 leader_election.cc:288] T be6fc55aca6842329a3deb7c2b557b66 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 election: Requested vote from peers e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:05.936599 9308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be6fc55aca6842329a3deb7c2b557b66" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023"
I20240503 02:00:05.936722 9308 raft_consensus.cc:3037] T be6fc55aca6842329a3deb7c2b557b66 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:05.937098 6582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be6fc55aca6842329a3deb7c2b557b66" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 02:00:05.937220 6582 raft_consensus.cc:3037] T be6fc55aca6842329a3deb7c2b557b66 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:05.937515 8916 raft_consensus.cc:2932] T 51c0137930464635be1e6a97f5e9d952 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: true } } }
I20240503 02:00:05.937698 9308 raft_consensus.cc:2454] T be6fc55aca6842329a3deb7c2b557b66 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 1.
I20240503 02:00:05.937844 9048 raft_consensus.cc:1261] T 0ee05e70ccbb496ca3cff08dc38112e8 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer 2a01ace80baf4057b6bf1995cf88468b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:05.938295 6582 raft_consensus.cc:2454] T be6fc55aca6842329a3deb7c2b557b66 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 1.
I20240503 02:00:05.938287 12267 consensus_queue.cc:1035] T 0ee05e70ccbb496ca3cff08dc38112e8 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:05.938970 8852 leader_election.cc:302] T be6fc55aca6842329a3deb7c2b557b66 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:05.939296 11872 raft_consensus.cc:2781] T be6fc55aca6842329a3deb7c2b557b66 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:05.939373 11872 raft_consensus.cc:686] T be6fc55aca6842329a3deb7c2b557b66 P 9942306855404cfd891adf227dd65eb2 [term 1 LEADER]: Becoming Leader. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Running, Role: LEADER
I20240503 02:00:05.939518 11872 consensus_queue.cc:237] T be6fc55aca6842329a3deb7c2b557b66 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:05.939761 12497 tablet_bootstrap.cc:492] T 6a82dcc9be5b4c93b5fff03f789350ab P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:05.940014 12497 tablet_bootstrap.cc:492] T 6a82dcc9be5b4c93b5fff03f789350ab P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap complete.
I20240503 02:00:05.940184 12497 ts_tablet_manager.cc:1367] T 6a82dcc9be5b4c93b5fff03f789350ab P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.001s
I20240503 02:00:05.940697 12497 raft_consensus.cc:348] T 6a82dcc9be5b4c93b5fff03f789350ab P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: true } }
I20240503 02:00:05.940806 12497 raft_consensus.cc:729] T 6a82dcc9be5b4c93b5fff03f789350ab P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: LEARNER
I20240503 02:00:05.940902 12497 consensus_queue.cc:260] T 6a82dcc9be5b4c93b5fff03f789350ab P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: true } }
I20240503 02:00:05.941313 12497 ts_tablet_manager.cc:1398] T 6a82dcc9be5b4c93b5fff03f789350ab P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 02:00:05.941815 9068 tablet_copy_service.cc:342] P 99ead02915954b4fab23b033ad6d546b: Request end of tablet copy session dec3e4a04d604f7eaaa3f077d081218a-6a82dcc9be5b4c93b5fff03f789350ab received from {username='slave'} at 127.0.99.129:53227
I20240503 02:00:05.941921 9068 tablet_copy_service.cc:433] P 99ead02915954b4fab23b033ad6d546b: ending tablet copy session dec3e4a04d604f7eaaa3f077d081218a-6a82dcc9be5b4c93b5fff03f789350ab on tablet 6a82dcc9be5b4c93b5fff03f789350ab with peer dec3e4a04d604f7eaaa3f077d081218a
I20240503 02:00:05.941886 6312 catalog_manager.cc:5484] T be6fc55aca6842329a3deb7c2b557b66 P 9942306855404cfd891adf227dd65eb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9942306855404cfd891adf227dd65eb2 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "9942306855404cfd891adf227dd65eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:05.944061 12267 raft_consensus.cc:2932] T 0ee05e70ccbb496ca3cff08dc38112e8 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } }
I20240503 02:00:05.946136 9047 raft_consensus.cc:2932] T 0ee05e70ccbb496ca3cff08dc38112e8 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } }
I20240503 02:00:05.946851 6458 raft_consensus.cc:2932] T 0ee05e70ccbb496ca3cff08dc38112e8 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } }
I20240503 02:00:05.947135 6720 raft_consensus.cc:2932] T 0ee05e70ccbb496ca3cff08dc38112e8 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } }
I20240503 02:00:05.947383 6312 catalog_manager.cc:5484] T 0ee05e70ccbb496ca3cff08dc38112e8 P 2a01ace80baf4057b6bf1995cf88468b reported cstate change: config changed from index 2 to 3, dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2a01ace80baf4057b6bf1995cf88468b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:05.955011 8916 consensus_queue.cc:237] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: true } }
I20240503 02:00:05.956501 6849 raft_consensus.cc:1261] T 1c721b33983b4c47afd8faa8f155ca84 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:05.956596 9308 raft_consensus.cc:1261] T 1c721b33983b4c47afd8faa8f155ca84 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:00:05.956801 8850 consensus_peers.cc:485] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2 -> Peer dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): Couldn't send request to peer dec3e4a04d604f7eaaa3f077d081218a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1c721b33983b4c47afd8faa8f155ca84. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:05.956969 11357 consensus_queue.cc:1035] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:05.957340 9758 consensus_queue.cc:1035] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:05.959054 11357 raft_consensus.cc:2932] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: true } } }
I20240503 02:00:05.959758 9308 raft_consensus.cc:2932] T 1c721b33983b4c47afd8faa8f155ca84 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: true } } }
I20240503 02:00:05.960952 6849 raft_consensus.cc:2932] T 1c721b33983b4c47afd8faa8f155ca84 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: true } } }
I20240503 02:00:05.962064 6315 catalog_manager.cc:5484] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2 reported cstate change: config changed from index -1 to 2, NON_VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "9942306855404cfd891adf227dd65eb2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:05.999228 9308 raft_consensus.cc:1203] T 8246488dec204bf6b09e868d7951ac5a P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:00:06.033277 11872 raft_consensus.cc:1050] T 067f54011a37432d99157884e70fc601 P 9942306855404cfd891adf227dd65eb2: attempting to promote NON_VOTER 492448570cc94ed1879feb2ec3b9051d to VOTER
I20240503 02:00:06.033725 11872 consensus_queue.cc:237] T 067f54011a37432d99157884e70fc601 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { promote: false } }
I20240503 02:00:06.035267 9308 raft_consensus.cc:1261] T 067f54011a37432d99157884e70fc601 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:06.035283 9176 raft_consensus.cc:1261] T 067f54011a37432d99157884e70fc601 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:06.035305 6849 raft_consensus.cc:1261] T 067f54011a37432d99157884e70fc601 P 492448570cc94ed1879feb2ec3b9051d [term 1 LEARNER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:06.035684 11872 consensus_queue.cc:1035] T 067f54011a37432d99157884e70fc601 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:06.035859 9758 consensus_queue.cc:1035] T 067f54011a37432d99157884e70fc601 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:06.036011 11073 consensus_queue.cc:1035] T 067f54011a37432d99157884e70fc601 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:06.038591 9758 raft_consensus.cc:2932] T 067f54011a37432d99157884e70fc601 P 9942306855404cfd891adf227dd65eb2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { promote: false } } }
I20240503 02:00:06.040671 6849 raft_consensus.cc:2932] T 067f54011a37432d99157884e70fc601 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { promote: false } } }
I20240503 02:00:06.040696 9308 raft_consensus.cc:2932] T 067f54011a37432d99157884e70fc601 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { promote: false } } }
I20240503 02:00:06.041781 9176 raft_consensus.cc:2932] T 067f54011a37432d99157884e70fc601 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { promote: false } } }
I20240503 02:00:06.042259 6315 catalog_manager.cc:5484] T 067f54011a37432d99157884e70fc601 P 9942306855404cfd891adf227dd65eb2 reported cstate change: config changed from index 2 to 3, 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9942306855404cfd891adf227dd65eb2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:06.045837 6315 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:06.047170 8917 consensus_queue.cc:237] T 067f54011a37432d99157884e70fc601 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { promote: false } }
I20240503 02:00:06.048585 9308 raft_consensus.cc:1261] T 067f54011a37432d99157884e70fc601 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:06.048669 6849 raft_consensus.cc:1261] T 067f54011a37432d99157884e70fc601 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:06.049036 11357 consensus_queue.cc:1035] T 067f54011a37432d99157884e70fc601 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:06.049288 11073 consensus_queue.cc:1035] T 067f54011a37432d99157884e70fc601 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:06.051460 6582 raft_consensus.cc:1261] T be6fc55aca6842329a3deb7c2b557b66 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:06.051493 9309 raft_consensus.cc:1261] T be6fc55aca6842329a3deb7c2b557b66 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:06.051869 11357 consensus_queue.cc:1035] T be6fc55aca6842329a3deb7c2b557b66 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:06.052171 11073 consensus_queue.cc:1035] T be6fc55aca6842329a3deb7c2b557b66 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:06.052557 11872 raft_consensus.cc:2932] T 067f54011a37432d99157884e70fc601 P 9942306855404cfd891adf227dd65eb2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { promote: false } } }
I20240503 02:00:06.054262 6303 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 067f54011a37432d99157884e70fc601 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:06.055076 6849 raft_consensus.cc:2932] T 067f54011a37432d99157884e70fc601 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { promote: false } } }
I20240503 02:00:06.055457 6314 catalog_manager.cc:5484] T 067f54011a37432d99157884e70fc601 P 9942306855404cfd891adf227dd65eb2 reported cstate change: config changed from index 3 to 4, VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "9942306855404cfd891adf227dd65eb2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:06.055977 12174 tablet.cc:1722] T be6fc55aca6842329a3deb7c2b557b66 P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:06.056305 12174 tablet.cc:1620] T be6fc55aca6842329a3deb7c2b557b66 P 80fd18ef4d604c86b909d5c008a46b4f: MemRowSet was empty: no flush needed.
I20240503 02:00:06.056268 9308 raft_consensus.cc:2932] T 067f54011a37432d99157884e70fc601 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { promote: false } } }
I20240503 02:00:06.056615 12576 tablet.cc:1722] T be6fc55aca6842329a3deb7c2b557b66 P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:06.056902 12576 tablet.cc:1620] T be6fc55aca6842329a3deb7c2b557b66 P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
I20240503 02:00:06.057593 6307 catalog_manager.cc:5576] TS e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311) does not have the latest schema for tablet be6fc55aca6842329a3deb7c2b557b66 (table rebalancer_dynamic_table [id=fac49b5a5e3845cebcdbfaeb3b0eb0c5]). Expected version 1 got 0
I20240503 02:00:06.059722 9158 tablet_service.cc:1514] Processing DeleteTablet for tablet 067f54011a37432d99157884e70fc601 with delete_type TABLET_DATA_TOMBSTONED (TS 2a01ace80baf4057b6bf1995cf88468b not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:51940
I20240503 02:00:06.060568 12401 tablet_replica.cc:330] T 067f54011a37432d99157884e70fc601 P 2a01ace80baf4057b6bf1995cf88468b: stopping tablet replica
I20240503 02:00:06.060716 12401 raft_consensus.cc:2229] T 067f54011a37432d99157884e70fc601 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:06.060856 12401 raft_consensus.cc:2258] T 067f54011a37432d99157884e70fc601 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:06.060993 6315 catalog_manager.cc:5576] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863) does not have the latest schema for tablet be6fc55aca6842329a3deb7c2b557b66 (table rebalancer_dynamic_table [id=fac49b5a5e3845cebcdbfaeb3b0eb0c5]). Expected version 1 got 0
I20240503 02:00:06.061105 12173 tablet.cc:1722] T be6fc55aca6842329a3deb7c2b557b66 P e1a9251ce1cf4361b8b0ede9fbfc4023: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:06.061437 12173 tablet.cc:1620] T be6fc55aca6842329a3deb7c2b557b66 P e1a9251ce1cf4361b8b0ede9fbfc4023: MemRowSet was empty: no flush needed.
I20240503 02:00:06.065783 6314 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=fac49b5a5e3845cebcdbfaeb3b0eb0c5] alter complete (version 1)
I20240503 02:00:06.065866 12401 ts_tablet_manager.cc:1875] T 067f54011a37432d99157884e70fc601 P 2a01ace80baf4057b6bf1995cf88468b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:06.068126 12401 ts_tablet_manager.cc:1888] T 067f54011a37432d99157884e70fc601 P 2a01ace80baf4057b6bf1995cf88468b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:00:06.068208 12401 log.cc:1196] T 067f54011a37432d99157884e70fc601 P 2a01ace80baf4057b6bf1995cf88468b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/067f54011a37432d99157884e70fc601
I20240503 02:00:06.068893 12190 raft_consensus.cc:482] T 9cbb2be1fda74ebe8997123c70629ba3 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:06.068995 12190 raft_consensus.cc:504] T 9cbb2be1fda74ebe8997123c70629ba3 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:06.069422 12190 leader_election.cc:288] T 9cbb2be1fda74ebe8997123c70629ba3 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:06.069777 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cbb2be1fda74ebe8997123c70629ba3" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:06.069849 6300 catalog_manager.cc:4830] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079): tablet 067f54011a37432d99157884e70fc601 (table rebalancer_extra_table_2 [id=54f1891be3f54f19821aad5d1dfbd85d]) successfully deleted
I20240503 02:00:06.069926 9048 raft_consensus.cc:2454] T 9cbb2be1fda74ebe8997123c70629ba3 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:06.070220 6392 leader_election.cc:302] T 9cbb2be1fda74ebe8997123c70629ba3 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99ead02915954b4fab23b033ad6d546b, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:06.070561 12190 raft_consensus.cc:2781] T 9cbb2be1fda74ebe8997123c70629ba3 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:06.070626 12190 raft_consensus.cc:482] T 9cbb2be1fda74ebe8997123c70629ba3 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:06.070664 12190 raft_consensus.cc:3037] T 9cbb2be1fda74ebe8997123c70629ba3 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:06.071707 12190 raft_consensus.cc:504] T 9cbb2be1fda74ebe8997123c70629ba3 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:06.072113 12190 leader_election.cc:288] T 9cbb2be1fda74ebe8997123c70629ba3 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Requested vote from peers 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:06.072188 8917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cbb2be1fda74ebe8997123c70629ba3" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:06.072310 8917 raft_consensus.cc:2454] T 9cbb2be1fda74ebe8997123c70629ba3 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:06.073213 8917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cbb2be1fda74ebe8997123c70629ba3" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:06.073323 8917 raft_consensus.cc:3037] T 9cbb2be1fda74ebe8997123c70629ba3 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:06.073449 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cbb2be1fda74ebe8997123c70629ba3" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:06.073561 9048 raft_consensus.cc:3037] T 9cbb2be1fda74ebe8997123c70629ba3 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:06.074214 8917 raft_consensus.cc:2454] T 9cbb2be1fda74ebe8997123c70629ba3 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:06.074482 9048 raft_consensus.cc:2454] T 9cbb2be1fda74ebe8997123c70629ba3 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:06.074865 6392 leader_election.cc:302] T 9cbb2be1fda74ebe8997123c70629ba3 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99ead02915954b4fab23b033ad6d546b, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:06.075004 6720 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d897c21e630d4770832bb10f43778e82"
dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44746
I20240503 02:00:06.075134 6720 raft_consensus.cc:595] T d897c21e630d4770832bb10f43778e82 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:00:06.075640 12190 raft_consensus.cc:2781] T 9cbb2be1fda74ebe8997123c70629ba3 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:06.075729 12190 raft_consensus.cc:686] T 9cbb2be1fda74ebe8997123c70629ba3 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Becoming Leader. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Running, Role: LEADER
I20240503 02:00:06.075865 12190 consensus_queue.cc:237] T 9cbb2be1fda74ebe8997123c70629ba3 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:06.079201 6315 catalog_manager.cc:5484] T 9cbb2be1fda74ebe8997123c70629ba3 P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: term changed from 0 to 1, leader changed from <none> to dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129). New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:06.085775 12587 ts_tablet_manager.cc:902] T ac7fa0c3722a461c91fb237fd85794a3 P 99ead02915954b4fab23b033ad6d546b: Initiating tablet copy from peer dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 02:00:06.086772 12587 tablet_copy_client.cc:308] T ac7fa0c3722a461c91fb237fd85794a3 P 99ead02915954b4fab23b033ad6d546b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:33093
I20240503 02:00:06.087323 6478 tablet_copy_service.cc:140] P dec3e4a04d604f7eaaa3f077d081218a: Received BeginTabletCopySession request for tablet ac7fa0c3722a461c91fb237fd85794a3 from peer 99ead02915954b4fab23b033ad6d546b ({username='slave'} at 127.0.99.134:54509)
I20240503 02:00:06.087440 6478 tablet_copy_service.cc:161] P dec3e4a04d604f7eaaa3f077d081218a: Beginning new tablet copy session on tablet ac7fa0c3722a461c91fb237fd85794a3 from peer 99ead02915954b4fab23b033ad6d546b at {username='slave'} at 127.0.99.134:54509: session id = 99ead02915954b4fab23b033ad6d546b-ac7fa0c3722a461c91fb237fd85794a3
I20240503 02:00:06.088583 6478 tablet_copy_source_session.cc:205] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:06.089777 9176 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0ee05e70ccbb496ca3cff08dc38112e8"
dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52178
I20240503 02:00:06.089879 12587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac7fa0c3722a461c91fb237fd85794a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:06.089970 9176 raft_consensus.cc:595] T 0ee05e70ccbb496ca3cff08dc38112e8 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Received request to transfer leadership
I20240503 02:00:06.092355 12587 tablet_copy_client.cc:783] T ac7fa0c3722a461c91fb237fd85794a3 P 99ead02915954b4fab23b033ad6d546b: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:06.092563 12587 tablet_copy_client.cc:647] T ac7fa0c3722a461c91fb237fd85794a3 P 99ead02915954b4fab23b033ad6d546b: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:06.093813 12587 tablet_copy_client.cc:523] T ac7fa0c3722a461c91fb237fd85794a3 P 99ead02915954b4fab23b033ad6d546b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:06.095224 12587 tablet_bootstrap.cc:492] T ac7fa0c3722a461c91fb237fd85794a3 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:06.119011 12587 tablet_bootstrap.cc:492] T ac7fa0c3722a461c91fb237fd85794a3 P 99ead02915954b4fab23b033ad6d546b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:06.119307 12587 tablet_bootstrap.cc:492] T ac7fa0c3722a461c91fb237fd85794a3 P 99ead02915954b4fab23b033ad6d546b: Bootstrap complete.
I20240503 02:00:06.119490 12587 ts_tablet_manager.cc:1367] T ac7fa0c3722a461c91fb237fd85794a3 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.024s user 0.019s sys 0.005s
I20240503 02:00:06.119935 12587 raft_consensus.cc:348] T ac7fa0c3722a461c91fb237fd85794a3 P 99ead02915954b4fab23b033ad6d546b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } }
I20240503 02:00:06.120064 12587 raft_consensus.cc:729] T ac7fa0c3722a461c91fb237fd85794a3 P 99ead02915954b4fab23b033ad6d546b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: LEARNER
I20240503 02:00:06.120167 12587 consensus_queue.cc:260] T ac7fa0c3722a461c91fb237fd85794a3 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: true } }
I20240503 02:00:06.120534 12587 ts_tablet_manager.cc:1398] T ac7fa0c3722a461c91fb237fd85794a3 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.002s sys 0.001s
I20240503 02:00:06.121055 6478 tablet_copy_service.cc:342] P dec3e4a04d604f7eaaa3f077d081218a: Request end of tablet copy session 99ead02915954b4fab23b033ad6d546b-ac7fa0c3722a461c91fb237fd85794a3 received from {username='slave'} at 127.0.99.134:54509
I20240503 02:00:06.121140 6478 tablet_copy_service.cc:433] P dec3e4a04d604f7eaaa3f077d081218a: ending tablet copy session 99ead02915954b4fab23b033ad6d546b-ac7fa0c3722a461c91fb237fd85794a3 on tablet ac7fa0c3722a461c91fb237fd85794a3 with peer 99ead02915954b4fab23b033ad6d546b
I20240503 02:00:06.122752 11514 consensus_queue.cc:1035] T aa281822047841fab7c58b53d9733e49 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:06.123813 12619 ts_tablet_manager.cc:902] T 1feaf29d8c6b4842829ce66e0f4ec23b P e1a9251ce1cf4361b8b0ede9fbfc4023: Initiating tablet copy from peer c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:06.124436 12619 tablet_copy_client.cc:308] T 1feaf29d8c6b4842829ce66e0f4ec23b P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:42655
I20240503 02:00:06.124943 6739 tablet_copy_service.cc:140] P c2ac2eb7cc264047b9ab32fe3a4fc025: Received BeginTabletCopySession request for tablet 1feaf29d8c6b4842829ce66e0f4ec23b from peer e1a9251ce1cf4361b8b0ede9fbfc4023 ({username='slave'} at 127.0.99.136:40237)
I20240503 02:00:06.125051 6739 tablet_copy_service.cc:161] P c2ac2eb7cc264047b9ab32fe3a4fc025: Beginning new tablet copy session on tablet 1feaf29d8c6b4842829ce66e0f4ec23b from peer e1a9251ce1cf4361b8b0ede9fbfc4023 at {username='slave'} at 127.0.99.136:40237: session id = e1a9251ce1cf4361b8b0ede9fbfc4023-1feaf29d8c6b4842829ce66e0f4ec23b
I20240503 02:00:06.125993 6739 tablet_copy_source_session.cc:205] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:06.127219 11515 consensus_queue.cc:1035] T aa281822047841fab7c58b53d9733e49 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:06.131822 12619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1feaf29d8c6b4842829ce66e0f4ec23b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:06.134292 12619 tablet_copy_client.cc:783] T 1feaf29d8c6b4842829ce66e0f4ec23b P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:06.134490 12619 tablet_copy_client.cc:647] T 1feaf29d8c6b4842829ce66e0f4ec23b P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:06.135774 12619 tablet_copy_client.cc:523] T 1feaf29d8c6b4842829ce66e0f4ec23b P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:06.137717 12619 tablet_bootstrap.cc:492] T 1feaf29d8c6b4842829ce66e0f4ec23b P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap starting.
I20240503 02:00:06.165203 12619 tablet_bootstrap.cc:492] T 1feaf29d8c6b4842829ce66e0f4ec23b P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:06.165531 12619 tablet_bootstrap.cc:492] T 1feaf29d8c6b4842829ce66e0f4ec23b P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap complete.
I20240503 02:00:06.165733 12619 ts_tablet_manager.cc:1367] T 1feaf29d8c6b4842829ce66e0f4ec23b P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent bootstrapping tablet: real 0.028s user 0.025s sys 0.000s
I20240503 02:00:06.166236 12619 raft_consensus.cc:348] T 1feaf29d8c6b4842829ce66e0f4ec23b P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } }
I20240503 02:00:06.166400 12619 raft_consensus.cc:729] T 1feaf29d8c6b4842829ce66e0f4ec23b P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e1a9251ce1cf4361b8b0ede9fbfc4023, State: Initialized, Role: LEARNER
I20240503 02:00:06.166508 12619 consensus_queue.cc:260] T 1feaf29d8c6b4842829ce66e0f4ec23b P e1a9251ce1cf4361b8b0ede9fbfc4023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } }
I20240503 02:00:06.166792 12619 ts_tablet_manager.cc:1398] T 1feaf29d8c6b4842829ce66e0f4ec23b P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:06.167409 6739 tablet_copy_service.cc:342] P c2ac2eb7cc264047b9ab32fe3a4fc025: Request end of tablet copy session e1a9251ce1cf4361b8b0ede9fbfc4023-1feaf29d8c6b4842829ce66e0f4ec23b received from {username='slave'} at 127.0.99.136:40237
I20240503 02:00:06.167495 6739 tablet_copy_service.cc:433] P c2ac2eb7cc264047b9ab32fe3a4fc025: ending tablet copy session e1a9251ce1cf4361b8b0ede9fbfc4023-1feaf29d8c6b4842829ce66e0f4ec23b on tablet 1feaf29d8c6b4842829ce66e0f4ec23b with peer e1a9251ce1cf4361b8b0ede9fbfc4023
I20240503 02:00:06.171343 8917 raft_consensus.cc:1203] T b91a933fb66b4075acb40c5b92ef4824 P 9942306855404cfd891adf227dd65eb2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:00:06.175706 6720 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d897c21e630d4770832bb10f43778e82"
dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44782
I20240503 02:00:06.175861 6720 raft_consensus.cc:595] T d897c21e630d4770832bb10f43778e82 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:00:06.178020 6315 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:06.184002 12576 tablet.cc:1722] T be6fc55aca6842329a3deb7c2b557b66 P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:06.184362 12576 tablet.cc:1620] T be6fc55aca6842329a3deb7c2b557b66 P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
I20240503 02:00:06.184741 12174 tablet.cc:1722] T be6fc55aca6842329a3deb7c2b557b66 P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:06.185035 12174 tablet.cc:1620] T be6fc55aca6842329a3deb7c2b557b66 P 80fd18ef4d604c86b909d5c008a46b4f: MemRowSet was empty: no flush needed.
I20240503 02:00:06.185384 12173 tablet.cc:1722] T be6fc55aca6842329a3deb7c2b557b66 P e1a9251ce1cf4361b8b0ede9fbfc4023: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:06.185674 12173 tablet.cc:1620] T be6fc55aca6842329a3deb7c2b557b66 P e1a9251ce1cf4361b8b0ede9fbfc4023: MemRowSet was empty: no flush needed.
I20240503 02:00:06.193236 6315 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=fac49b5a5e3845cebcdbfaeb3b0eb0c5] alter complete (version 2)
I20240503 02:00:06.197350 9176 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0ee05e70ccbb496ca3cff08dc38112e8"
dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52186
I20240503 02:00:06.197515 9176 raft_consensus.cc:595] T 0ee05e70ccbb496ca3cff08dc38112e8 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Received request to transfer leadership
I20240503 02:00:06.219172 10349 raft_consensus.cc:979] T d897c21e630d4770832bb10f43778e82 P c2ac2eb7cc264047b9ab32fe3a4fc025: : Instructing follower 492448570cc94ed1879feb2ec3b9051d to start an election
I20240503 02:00:06.219296 10349 raft_consensus.cc:1067] T d897c21e630d4770832bb10f43778e82 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Signalling peer 492448570cc94ed1879feb2ec3b9051d to start an election
I20240503 02:00:06.219913 6849 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d897c21e630d4770832bb10f43778e82"
dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
from {username='slave'} at 127.0.99.131:34433
I20240503 02:00:06.220070 6849 raft_consensus.cc:482] T d897c21e630d4770832bb10f43778e82 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:06.220136 6849 raft_consensus.cc:3037] T d897c21e630d4770832bb10f43778e82 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:06.221263 6849 raft_consensus.cc:504] T d897c21e630d4770832bb10f43778e82 P 492448570cc94ed1879feb2ec3b9051d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } }
I20240503 02:00:06.221735 6849 leader_election.cc:288] T d897c21e630d4770832bb10f43778e82 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 2 election: Requested vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311)
I20240503 02:00:06.222220 6720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d897c21e630d4770832bb10f43778e82" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 02:00:06.222345 6720 raft_consensus.cc:3032] T d897c21e630d4770832bb10f43778e82 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:00:06.222386 6720 raft_consensus.cc:729] T d897c21e630d4770832bb10f43778e82 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Running, Role: LEADER
I20240503 02:00:06.222553 6720 consensus_queue.cc:260] T d897c21e630d4770832bb10f43778e82 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } }
I20240503 02:00:06.222739 6720 raft_consensus.cc:3037] T d897c21e630d4770832bb10f43778e82 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:06.223441 9309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d897c21e630d4770832bb10f43778e82" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023"
I20240503 02:00:06.223563 9309 raft_consensus.cc:3037] T d897c21e630d4770832bb10f43778e82 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:06.223811 6720 raft_consensus.cc:2454] T d897c21e630d4770832bb10f43778e82 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 2.
I20240503 02:00:06.224458 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d897c21e630d4770832bb10f43778e82" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 02:00:06.224587 6458 raft_consensus.cc:3037] T d897c21e630d4770832bb10f43778e82 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:06.224576 9309 raft_consensus.cc:2454] T d897c21e630d4770832bb10f43778e82 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 2.
I20240503 02:00:06.224985 6787 leader_election.cc:302] T d897c21e630d4770832bb10f43778e82 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, c2ac2eb7cc264047b9ab32fe3a4fc025, e1a9251ce1cf4361b8b0ede9fbfc4023; no voters:
I20240503 02:00:06.225194 11373 raft_consensus.cc:2781] T d897c21e630d4770832bb10f43778e82 P 492448570cc94ed1879feb2ec3b9051d [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:06.225283 11373 raft_consensus.cc:686] T d897c21e630d4770832bb10f43778e82 P 492448570cc94ed1879feb2ec3b9051d [term 2 LEADER]: Becoming Leader. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Running, Role: LEADER
I20240503 02:00:06.225423 11373 consensus_queue.cc:237] T d897c21e630d4770832bb10f43778e82 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } }
I20240503 02:00:06.225561 6458 raft_consensus.cc:2454] T d897c21e630d4770832bb10f43778e82 P dec3e4a04d604f7eaaa3f077d081218a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 2.
I20240503 02:00:06.228196 6312 catalog_manager.cc:5484] T d897c21e630d4770832bb10f43778e82 P 492448570cc94ed1879feb2ec3b9051d reported cstate change: term changed from 1 to 2, leader changed from c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) to 492448570cc94ed1879feb2ec3b9051d (127.0.99.132). New cstate: current_term: 2 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:06.277972 9176 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0ee05e70ccbb496ca3cff08dc38112e8"
dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52194
I20240503 02:00:06.278141 9176 raft_consensus.cc:595] T 0ee05e70ccbb496ca3cff08dc38112e8 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Received request to transfer leadership
I20240503 02:00:06.300297 6312 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:06.305769 12576 tablet.cc:1722] T be6fc55aca6842329a3deb7c2b557b66 P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:06.306255 12174 tablet.cc:1722] T be6fc55aca6842329a3deb7c2b557b66 P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:06.307545 12173 tablet.cc:1722] T be6fc55aca6842329a3deb7c2b557b66 P e1a9251ce1cf4361b8b0ede9fbfc4023: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:06.310822 12733 ts_tablet_manager.cc:902] T 51c0137930464635be1e6a97f5e9d952 P c2ac2eb7cc264047b9ab32fe3a4fc025: Initiating tablet copy from peer dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 02:00:06.311403 12733 tablet_copy_client.cc:308] T 51c0137930464635be1e6a97f5e9d952 P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:33093
I20240503 02:00:06.311517 6312 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=fac49b5a5e3845cebcdbfaeb3b0eb0c5] alter complete (version 3)
I20240503 02:00:06.312479 6478 tablet_copy_service.cc:140] P dec3e4a04d604f7eaaa3f077d081218a: Received BeginTabletCopySession request for tablet 51c0137930464635be1e6a97f5e9d952 from peer c2ac2eb7cc264047b9ab32fe3a4fc025 ({username='slave'} at 127.0.99.131:59575)
I20240503 02:00:06.312604 6478 tablet_copy_service.cc:161] P dec3e4a04d604f7eaaa3f077d081218a: Beginning new tablet copy session on tablet 51c0137930464635be1e6a97f5e9d952 from peer c2ac2eb7cc264047b9ab32fe3a4fc025 at {username='slave'} at 127.0.99.131:59575: session id = c2ac2eb7cc264047b9ab32fe3a4fc025-51c0137930464635be1e6a97f5e9d952
I20240503 02:00:06.313546 6478 tablet_copy_source_session.cc:205] T 51c0137930464635be1e6a97f5e9d952 P dec3e4a04d604f7eaaa3f077d081218a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:06.314462 12733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51c0137930464635be1e6a97f5e9d952. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:06.316843 12733 tablet_copy_client.cc:783] T 51c0137930464635be1e6a97f5e9d952 P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:06.317020 12733 tablet_copy_client.cc:647] T 51c0137930464635be1e6a97f5e9d952 P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:06.318095 12733 tablet_copy_client.cc:523] T 51c0137930464635be1e6a97f5e9d952 P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:06.319296 12733 tablet_bootstrap.cc:492] T 51c0137930464635be1e6a97f5e9d952 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:06.322706 12733 tablet_bootstrap.cc:492] T 51c0137930464635be1e6a97f5e9d952 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:06.322932 12733 tablet_bootstrap.cc:492] T 51c0137930464635be1e6a97f5e9d952 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap complete.
I20240503 02:00:06.323079 12733 ts_tablet_manager.cc:1367] T 51c0137930464635be1e6a97f5e9d952 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 02:00:06.323500 12733 raft_consensus.cc:348] T 51c0137930464635be1e6a97f5e9d952 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: true } }
I20240503 02:00:06.323616 12733 raft_consensus.cc:729] T 51c0137930464635be1e6a97f5e9d952 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: LEARNER
I20240503 02:00:06.323701 12733 consensus_queue.cc:260] T 51c0137930464635be1e6a97f5e9d952 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: true } }
I20240503 02:00:06.324016 12733 ts_tablet_manager.cc:1398] T 51c0137930464635be1e6a97f5e9d952 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:06.324555 6478 tablet_copy_service.cc:342] P dec3e4a04d604f7eaaa3f077d081218a: Request end of tablet copy session c2ac2eb7cc264047b9ab32fe3a4fc025-51c0137930464635be1e6a97f5e9d952 received from {username='slave'} at 127.0.99.131:59575
I20240503 02:00:06.324739 6478 tablet_copy_service.cc:433] P dec3e4a04d604f7eaaa3f077d081218a: ending tablet copy session c2ac2eb7cc264047b9ab32fe3a4fc025-51c0137930464635be1e6a97f5e9d952 on tablet 51c0137930464635be1e6a97f5e9d952 with peer c2ac2eb7cc264047b9ab32fe3a4fc025
I20240503 02:00:06.328503 6458 raft_consensus.cc:1203] T 6a82dcc9be5b4c93b5fff03f789350ab P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:00:06.370437 6849 consensus_queue.cc:237] T cb873018448549ccbe3d14b784e47d26 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { promote: false } }
I20240503 02:00:06.371709 9048 raft_consensus.cc:1261] T cb873018448549ccbe3d14b784e47d26 P 99ead02915954b4fab23b033ad6d546b [term 2 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:00:06.371734 8917 raft_consensus.cc:1261] T cb873018448549ccbe3d14b784e47d26 P 9942306855404cfd891adf227dd65eb2 [term 2 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:00:06.372285 11376 consensus_queue.cc:1035] T cb873018448549ccbe3d14b784e47d26 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:06.372520 11373 consensus_queue.cc:1035] T cb873018448549ccbe3d14b784e47d26 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:06.374493 11376 raft_consensus.cc:2932] T cb873018448549ccbe3d14b784e47d26 P 492448570cc94ed1879feb2ec3b9051d [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { promote: false } } }
I20240503 02:00:06.377888 11504 raft_consensus.cc:979] T 0ee05e70ccbb496ca3cff08dc38112e8 P 2a01ace80baf4057b6bf1995cf88468b: : Instructing follower dec3e4a04d604f7eaaa3f077d081218a to start an election
I20240503 02:00:06.378000 11504 raft_consensus.cc:1067] T 0ee05e70ccbb496ca3cff08dc38112e8 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Signalling peer dec3e4a04d604f7eaaa3f077d081218a to start an election
I20240503 02:00:06.378667 6458 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0ee05e70ccbb496ca3cff08dc38112e8"
dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
from {username='slave'} at 127.0.99.135:56449
I20240503 02:00:06.378831 6458 raft_consensus.cc:482] T 0ee05e70ccbb496ca3cff08dc38112e8 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:06.378892 6458 raft_consensus.cc:3037] T 0ee05e70ccbb496ca3cff08dc38112e8 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:06.380003 6458 raft_consensus.cc:504] T 0ee05e70ccbb496ca3cff08dc38112e8 P dec3e4a04d604f7eaaa3f077d081218a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } }
I20240503 02:00:06.380563 6458 leader_election.cc:288] T 0ee05e70ccbb496ca3cff08dc38112e8 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 2 election: Requested vote from peers 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:06.381359 9176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ee05e70ccbb496ca3cff08dc38112e8" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:06.381489 9176 raft_consensus.cc:3032] T 0ee05e70ccbb496ca3cff08dc38112e8 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:00:06.381525 9176 raft_consensus.cc:729] T 0ee05e70ccbb496ca3cff08dc38112e8 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Running, Role: LEADER
I20240503 02:00:06.381680 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ee05e70ccbb496ca3cff08dc38112e8" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:06.381690 9176 consensus_queue.cc:260] T 0ee05e70ccbb496ca3cff08dc38112e8 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } }
I20240503 02:00:06.381803 9048 raft_consensus.cc:3037] T 0ee05e70ccbb496ca3cff08dc38112e8 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:06.381889 9176 raft_consensus.cc:3037] T 0ee05e70ccbb496ca3cff08dc38112e8 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:06.382843 9176 raft_consensus.cc:2454] T 0ee05e70ccbb496ca3cff08dc38112e8 P 2a01ace80baf4057b6bf1995cf88468b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 2.
I20240503 02:00:06.382863 9048 raft_consensus.cc:2454] T 0ee05e70ccbb496ca3cff08dc38112e8 P 99ead02915954b4fab23b033ad6d546b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 2.
I20240503 02:00:06.383781 8917 raft_consensus.cc:2932] T cb873018448549ccbe3d14b784e47d26 P 9942306855404cfd891adf227dd65eb2 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { promote: false } } }
I20240503 02:00:06.384336 6303 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cb873018448549ccbe3d14b784e47d26 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:06.385207 6720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ee05e70ccbb496ca3cff08dc38112e8" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 02:00:06.385262 6392 leader_election.cc:302] T 0ee05e70ccbb496ca3cff08dc38112e8 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 99ead02915954b4fab23b033ad6d546b, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:06.385336 6720 raft_consensus.cc:3037] T 0ee05e70ccbb496ca3cff08dc38112e8 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:06.385483 11851 raft_consensus.cc:2781] T 0ee05e70ccbb496ca3cff08dc38112e8 P dec3e4a04d604f7eaaa3f077d081218a [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:06.385556 11851 raft_consensus.cc:686] T 0ee05e70ccbb496ca3cff08dc38112e8 P dec3e4a04d604f7eaaa3f077d081218a [term 2 LEADER]: Becoming Leader. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Running, Role: LEADER
I20240503 02:00:06.385699 11851 consensus_queue.cc:237] T 0ee05e70ccbb496ca3cff08dc38112e8 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } }
I20240503 02:00:06.386320 6720 raft_consensus.cc:2454] T 0ee05e70ccbb496ca3cff08dc38112e8 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 2.
I20240503 02:00:06.387388 12032 raft_consensus.cc:1050] T 8246488dec204bf6b09e868d7951ac5a P 492448570cc94ed1879feb2ec3b9051d: attempting to promote NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 to VOTER
I20240503 02:00:06.387745 12032 consensus_queue.cc:237] T 8246488dec204bf6b09e868d7951ac5a P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } }
I20240503 02:00:06.387753 6315 catalog_manager.cc:5484] T 0ee05e70ccbb496ca3cff08dc38112e8 P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: term changed from 1 to 2, leader changed from 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) to dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129). New cstate: current_term: 2 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:06.389403 6314 catalog_manager.cc:5484] T cb873018448549ccbe3d14b784e47d26 P 492448570cc94ed1879feb2ec3b9051d reported cstate change: config changed from index 3 to 5, VOTER 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:06.392583 6458 raft_consensus.cc:1261] T 8246488dec204bf6b09e868d7951ac5a P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:06.392700 6567 tablet_service.cc:1514] Processing DeleteTablet for tablet cb873018448549ccbe3d14b784e47d26 with delete_type TABLET_DATA_TOMBSTONED (TS 80fd18ef4d604c86b909d5c008a46b4f not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:59846
I20240503 02:00:06.392918 12390 tablet_replica.cc:330] T cb873018448549ccbe3d14b784e47d26 P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 02:00:06.392969 12742 consensus_queue.cc:1035] T 8246488dec204bf6b09e868d7951ac5a P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:06.393074 12390 raft_consensus.cc:2229] T cb873018448549ccbe3d14b784e47d26 P 80fd18ef4d604c86b909d5c008a46b4f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:06.393200 12390 raft_consensus.cc:2258] T cb873018448549ccbe3d14b784e47d26 P 80fd18ef4d604c86b909d5c008a46b4f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:06.393791 12390 ts_tablet_manager.cc:1875] T cb873018448549ccbe3d14b784e47d26 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:06.393903 6720 raft_consensus.cc:1261] T 8246488dec204bf6b09e868d7951ac5a P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:06.394757 9309 raft_consensus.cc:1261] T 8246488dec204bf6b09e868d7951ac5a P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 LEARNER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:06.396721 12390 ts_tablet_manager.cc:1888] T cb873018448549ccbe3d14b784e47d26 P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240503 02:00:06.396798 12390 log.cc:1196] T cb873018448549ccbe3d14b784e47d26 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/cb873018448549ccbe3d14b784e47d26
I20240503 02:00:06.396070 11373 consensus_queue.cc:1035] T 8246488dec204bf6b09e868d7951ac5a P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:06.397302 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet cb873018448549ccbe3d14b784e47d26 (table rebalancer_extra_table_6 [id=0392495bb8a743099943f2bba5f15608]) successfully deleted
I20240503 02:00:06.397683 9048 raft_consensus.cc:2932] T cb873018448549ccbe3d14b784e47d26 P 99ead02915954b4fab23b033ad6d546b [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { promote: false } } }
I20240503 02:00:06.398960 11376 consensus_queue.cc:1035] T 8246488dec204bf6b09e868d7951ac5a P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:06.402993 11376 raft_consensus.cc:2932] T 8246488dec204bf6b09e868d7951ac5a P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:06.403362 9309 raft_consensus.cc:2932] T 8246488dec204bf6b09e868d7951ac5a P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:06.405467 6458 raft_consensus.cc:2932] T 8246488dec204bf6b09e868d7951ac5a P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:06.406545 6312 catalog_manager.cc:5484] T 8246488dec204bf6b09e868d7951ac5a P 492448570cc94ed1879feb2ec3b9051d reported cstate change: config changed from index 4 to 5, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:06.412143 6849 consensus_queue.cc:237] T 8246488dec204bf6b09e868d7951ac5a P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } }
I20240503 02:00:06.413445 9308 raft_consensus.cc:1261] T 8246488dec204bf6b09e868d7951ac5a P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:06.413934 11373 consensus_queue.cc:1035] T 8246488dec204bf6b09e868d7951ac5a P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:06.414487 6458 raft_consensus.cc:1261] T 8246488dec204bf6b09e868d7951ac5a P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:06.415900 6315 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:06.416030 6315 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:06.416659 11373 consensus_queue.cc:1035] T 8246488dec204bf6b09e868d7951ac5a P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:06.417039 6720 raft_consensus.cc:2932] T 8246488dec204bf6b09e868d7951ac5a P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:06.417091 12740 raft_consensus.cc:2932] T 8246488dec204bf6b09e868d7951ac5a P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:06.419198 9309 raft_consensus.cc:2932] T 8246488dec204bf6b09e868d7951ac5a P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:06.419723 6315 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet be6fc55aca6842329a3deb7c2b557b66
I20240503 02:00:06.420343 9286 tablet_service.cc:1514] Processing DeleteTablet for tablet be6fc55aca6842329a3deb7c2b557b66 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:06 UTC) from {username='slave'} at 127.0.0.1:36004
I20240503 02:00:06.420987 6567 tablet_service.cc:1514] Processing DeleteTablet for tablet be6fc55aca6842329a3deb7c2b557b66 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:06 UTC) from {username='slave'} at 127.0.0.1:59846
I20240503 02:00:06.421310 12390 tablet_replica.cc:330] T be6fc55aca6842329a3deb7c2b557b66 P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 02:00:06.421361 8880 tablet_service.cc:1514] Processing DeleteTablet for tablet be6fc55aca6842329a3deb7c2b557b66 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:06 UTC) from {username='slave'} at 127.0.0.1:40296
I20240503 02:00:06.421447 12390 raft_consensus.cc:2229] T be6fc55aca6842329a3deb7c2b557b66 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:06.421573 12390 raft_consensus.cc:2258] T be6fc55aca6842329a3deb7c2b557b66 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:06.421993 6314 catalog_manager.cc:5484] T 8246488dec204bf6b09e868d7951ac5a P 492448570cc94ed1879feb2ec3b9051d reported cstate change: config changed from index 5 to 6, VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:06.422748 6303 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8246488dec204bf6b09e868d7951ac5a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:06.423152 12746 tablet_replica.cc:330] T be6fc55aca6842329a3deb7c2b557b66 P e1a9251ce1cf4361b8b0ede9fbfc4023: stopping tablet replica
I20240503 02:00:06.423341 12746 raft_consensus.cc:2229] T be6fc55aca6842329a3deb7c2b557b66 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:06.423476 12746 raft_consensus.cc:2258] T be6fc55aca6842329a3deb7c2b557b66 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:06.423269 6458 raft_consensus.cc:2932] T 8246488dec204bf6b09e868d7951ac5a P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:06.424036 12746 ts_tablet_manager.cc:1875] T be6fc55aca6842329a3deb7c2b557b66 P e1a9251ce1cf4361b8b0ede9fbfc4023: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:06.424162 12390 ts_tablet_manager.cc:1875] T be6fc55aca6842329a3deb7c2b557b66 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:06.424523 6700 tablet_service.cc:1514] Processing DeleteTablet for tablet 8246488dec204bf6b09e868d7951ac5a with delete_type TABLET_DATA_TOMBSTONED (TS c2ac2eb7cc264047b9ab32fe3a4fc025 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:40336
I20240503 02:00:06.427199 12390 ts_tablet_manager.cc:1888] T be6fc55aca6842329a3deb7c2b557b66 P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:06.427278 12390 log.cc:1196] T be6fc55aca6842329a3deb7c2b557b66 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/be6fc55aca6842329a3deb7c2b557b66
I20240503 02:00:06.427506 12390 ts_tablet_manager.cc:1909] T be6fc55aca6842329a3deb7c2b557b66 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting consensus metadata
I20240503 02:00:06.427640 12746 ts_tablet_manager.cc:1888] T be6fc55aca6842329a3deb7c2b557b66 P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:06.427757 12746 log.cc:1196] T be6fc55aca6842329a3deb7c2b557b66 P e1a9251ce1cf4361b8b0ede9fbfc4023: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/be6fc55aca6842329a3deb7c2b557b66
I20240503 02:00:06.428037 12746 ts_tablet_manager.cc:1909] T be6fc55aca6842329a3deb7c2b557b66 P e1a9251ce1cf4361b8b0ede9fbfc4023: Deleting consensus metadata
I20240503 02:00:06.428308 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet be6fc55aca6842329a3deb7c2b557b66 (table rebalancer_dynamic_table_new_name [id=fac49b5a5e3845cebcdbfaeb3b0eb0c5]) successfully deleted
I20240503 02:00:06.428421 12747 tablet_replica.cc:330] T be6fc55aca6842329a3deb7c2b557b66 P 9942306855404cfd891adf227dd65eb2: stopping tablet replica
I20240503 02:00:06.428565 12747 raft_consensus.cc:2229] T be6fc55aca6842329a3deb7c2b557b66 P 9942306855404cfd891adf227dd65eb2 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:06.428788 12747 raft_consensus.cc:2258] T be6fc55aca6842329a3deb7c2b557b66 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:06.428934 6301 catalog_manager.cc:4830] TS e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): tablet be6fc55aca6842329a3deb7c2b557b66 (table rebalancer_dynamic_table_new_name [id=fac49b5a5e3845cebcdbfaeb3b0eb0c5]) successfully deleted
I20240503 02:00:06.429085 12748 tablet_replica.cc:330] T 8246488dec204bf6b09e868d7951ac5a P c2ac2eb7cc264047b9ab32fe3a4fc025: stopping tablet replica
I20240503 02:00:06.429190 12748 raft_consensus.cc:2229] T 8246488dec204bf6b09e868d7951ac5a P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:06.429328 12748 raft_consensus.cc:2258] T 8246488dec204bf6b09e868d7951ac5a P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:06.429497 12747 ts_tablet_manager.cc:1875] T be6fc55aca6842329a3deb7c2b557b66 P 9942306855404cfd891adf227dd65eb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:06.430004 12748 ts_tablet_manager.cc:1875] T 8246488dec204bf6b09e868d7951ac5a P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:06.431516 12747 ts_tablet_manager.cc:1888] T be6fc55aca6842329a3deb7c2b557b66 P 9942306855404cfd891adf227dd65eb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:06.431656 12747 log.cc:1196] T be6fc55aca6842329a3deb7c2b557b66 P 9942306855404cfd891adf227dd65eb2: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/be6fc55aca6842329a3deb7c2b557b66
I20240503 02:00:06.431957 12747 ts_tablet_manager.cc:1909] T be6fc55aca6842329a3deb7c2b557b66 P 9942306855404cfd891adf227dd65eb2: Deleting consensus metadata
I20240503 02:00:06.432181 12748 ts_tablet_manager.cc:1888] T 8246488dec204bf6b09e868d7951ac5a P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:00:06.432247 12748 log.cc:1196] T 8246488dec204bf6b09e868d7951ac5a P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/8246488dec204bf6b09e868d7951ac5a
I20240503 02:00:06.432971 6303 catalog_manager.cc:4830] TS c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655): tablet 8246488dec204bf6b09e868d7951ac5a (table rebalance_test_table_19 [id=250f0720cafa4425b45a0360ebe04a03]) successfully deleted
I20240503 02:00:06.433127 6303 catalog_manager.cc:4830] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211): tablet be6fc55aca6842329a3deb7c2b557b66 (table rebalancer_dynamic_table_new_name [id=fac49b5a5e3845cebcdbfaeb3b0eb0c5]) successfully deleted
I20240503 02:00:06.483251 11515 consensus_queue.cc:1035] T 13756a6e44814fe4b09a6b0f087546c3 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:06.487349 11514 consensus_queue.cc:1035] T 13756a6e44814fe4b09a6b0f087546c3 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:06.501252 11851 consensus_queue.cc:1035] T 9cbb2be1fda74ebe8997123c70629ba3 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:06.505887 12190 consensus_queue.cc:1035] T 9cbb2be1fda74ebe8997123c70629ba3 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:06.522881 6314 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:00:06.529436 9286 tablet_service.cc:1467] Processing CreateTablet for tablet e831f37dd1884641a81c0a513092955f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e2ff6c9d20d9476b9c546ad715bb2066]), partition=
I20240503 02:00:06.529603 6436 tablet_service.cc:1467] Processing CreateTablet for tablet e831f37dd1884641a81c0a513092955f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e2ff6c9d20d9476b9c546ad715bb2066]), partition=
I20240503 02:00:06.529716 9286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e831f37dd1884641a81c0a513092955f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:06.529808 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e831f37dd1884641a81c0a513092955f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:06.530331 9158 tablet_service.cc:1467] Processing CreateTablet for tablet e831f37dd1884641a81c0a513092955f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e2ff6c9d20d9476b9c546ad715bb2066]), partition=
I20240503 02:00:06.530527 9158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e831f37dd1884641a81c0a513092955f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:06.532359 12786 tablet_bootstrap.cc:492] T e831f37dd1884641a81c0a513092955f P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:06.532663 12787 tablet_bootstrap.cc:492] T e831f37dd1884641a81c0a513092955f P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:06.533528 12788 tablet_bootstrap.cc:492] T e831f37dd1884641a81c0a513092955f P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap starting.
I20240503 02:00:06.533695 12787 tablet_bootstrap.cc:654] T e831f37dd1884641a81c0a513092955f P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:06.534489 12786 tablet_bootstrap.cc:654] T e831f37dd1884641a81c0a513092955f P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:06.535647 12787 tablet_bootstrap.cc:492] T e831f37dd1884641a81c0a513092955f P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:06.535666 12788 tablet_bootstrap.cc:654] T e831f37dd1884641a81c0a513092955f P e1a9251ce1cf4361b8b0ede9fbfc4023: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:06.535861 12787 ts_tablet_manager.cc:1367] T e831f37dd1884641a81c0a513092955f P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:06.536409 12787 raft_consensus.cc:348] T e831f37dd1884641a81c0a513092955f P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } }
I20240503 02:00:06.536545 12787 raft_consensus.cc:374] T e831f37dd1884641a81c0a513092955f P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:06.536581 12787 raft_consensus.cc:729] T e831f37dd1884641a81c0a513092955f P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:06.536716 12787 consensus_queue.cc:260] T e831f37dd1884641a81c0a513092955f P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } }
I20240503 02:00:06.537012 12787 ts_tablet_manager.cc:1398] T e831f37dd1884641a81c0a513092955f P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:06.537647 12786 tablet_bootstrap.cc:492] T e831f37dd1884641a81c0a513092955f P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:06.537741 12786 ts_tablet_manager.cc:1367] T e831f37dd1884641a81c0a513092955f P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240503 02:00:06.538128 12786 raft_consensus.cc:348] T e831f37dd1884641a81c0a513092955f P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } }
I20240503 02:00:06.538219 12786 raft_consensus.cc:374] T e831f37dd1884641a81c0a513092955f P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:06.538245 12786 raft_consensus.cc:729] T e831f37dd1884641a81c0a513092955f P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:06.538368 12786 consensus_queue.cc:260] T e831f37dd1884641a81c0a513092955f P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } }
I20240503 02:00:06.538645 12786 ts_tablet_manager.cc:1398] T e831f37dd1884641a81c0a513092955f P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:06.538970 9047 raft_consensus.cc:1203] T ac7fa0c3722a461c91fb237fd85794a3 P 99ead02915954b4fab23b033ad6d546b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:00:06.540902 12788 tablet_bootstrap.cc:492] T e831f37dd1884641a81c0a513092955f P e1a9251ce1cf4361b8b0ede9fbfc4023: No bootstrap required, opened a new log
I20240503 02:00:06.540997 12788 ts_tablet_manager.cc:1367] T e831f37dd1884641a81c0a513092955f P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 02:00:06.541417 12788 raft_consensus.cc:348] T e831f37dd1884641a81c0a513092955f P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } }
I20240503 02:00:06.541517 12788 raft_consensus.cc:374] T e831f37dd1884641a81c0a513092955f P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:06.541538 12788 raft_consensus.cc:729] T e831f37dd1884641a81c0a513092955f P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1a9251ce1cf4361b8b0ede9fbfc4023, State: Initialized, Role: FOLLOWER
I20240503 02:00:06.541723 12788 consensus_queue.cc:260] T e831f37dd1884641a81c0a513092955f P e1a9251ce1cf4361b8b0ede9fbfc4023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } }
I20240503 02:00:06.542604 12788 ts_tablet_manager.cc:1398] T e831f37dd1884641a81c0a513092955f P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:06.570899 12813 ts_tablet_manager.cc:902] T 1c721b33983b4c47afd8faa8f155ca84 P dec3e4a04d604f7eaaa3f077d081218a: Initiating tablet copy from peer 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:06.571440 12813 tablet_copy_client.cc:308] T 1c721b33983b4c47afd8faa8f155ca84 P dec3e4a04d604f7eaaa3f077d081218a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:33211
I20240503 02:00:06.572060 8937 tablet_copy_service.cc:140] P 9942306855404cfd891adf227dd65eb2: Received BeginTabletCopySession request for tablet 1c721b33983b4c47afd8faa8f155ca84 from peer dec3e4a04d604f7eaaa3f077d081218a ({username='slave'} at 127.0.99.129:38483)
I20240503 02:00:06.572185 8937 tablet_copy_service.cc:161] P 9942306855404cfd891adf227dd65eb2: Beginning new tablet copy session on tablet 1c721b33983b4c47afd8faa8f155ca84 from peer dec3e4a04d604f7eaaa3f077d081218a at {username='slave'} at 127.0.99.129:38483: session id = dec3e4a04d604f7eaaa3f077d081218a-1c721b33983b4c47afd8faa8f155ca84
I20240503 02:00:06.573230 8937 tablet_copy_source_session.cc:205] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:06.574126 12813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c721b33983b4c47afd8faa8f155ca84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:06.576962 12813 tablet_copy_client.cc:783] T 1c721b33983b4c47afd8faa8f155ca84 P dec3e4a04d604f7eaaa3f077d081218a: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:06.577138 12813 tablet_copy_client.cc:647] T 1c721b33983b4c47afd8faa8f155ca84 P dec3e4a04d604f7eaaa3f077d081218a: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:06.578630 12813 tablet_copy_client.cc:523] T 1c721b33983b4c47afd8faa8f155ca84 P dec3e4a04d604f7eaaa3f077d081218a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:06.581735 12813 tablet_bootstrap.cc:492] T 1c721b33983b4c47afd8faa8f155ca84 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:06.584573 6315 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_extra_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:06.586207 12813 tablet_bootstrap.cc:492] T 1c721b33983b4c47afd8faa8f155ca84 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:06.586671 12813 tablet_bootstrap.cc:492] T 1c721b33983b4c47afd8faa8f155ca84 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap complete.
I20240503 02:00:06.586850 12813 ts_tablet_manager.cc:1367] T 1c721b33983b4c47afd8faa8f155ca84 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 02:00:06.587615 9309 raft_consensus.cc:1203] T 1feaf29d8c6b4842829ce66e0f4ec23b P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 02:00:06.588191 12813 raft_consensus.cc:348] T 1c721b33983b4c47afd8faa8f155ca84 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: true } }
I20240503 02:00:06.588591 12813 raft_consensus.cc:729] T 1c721b33983b4c47afd8faa8f155ca84 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: LEARNER
I20240503 02:00:06.588804 12813 consensus_queue.cc:260] T 1c721b33983b4c47afd8faa8f155ca84 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: true } }
I20240503 02:00:06.589107 12813 ts_tablet_manager.cc:1398] T 1c721b33983b4c47afd8faa8f155ca84 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:06.590525 8937 tablet_copy_service.cc:342] P 9942306855404cfd891adf227dd65eb2: Request end of tablet copy session dec3e4a04d604f7eaaa3f077d081218a-1c721b33983b4c47afd8faa8f155ca84 received from {username='slave'} at 127.0.99.129:38483
I20240503 02:00:06.590610 8937 tablet_copy_service.cc:433] P 9942306855404cfd891adf227dd65eb2: ending tablet copy session dec3e4a04d604f7eaaa3f077d081218a-1c721b33983b4c47afd8faa8f155ca84 on tablet 1c721b33983b4c47afd8faa8f155ca84 with peer dec3e4a04d604f7eaaa3f077d081218a
I20240503 02:00:06.592696 6831 tablet_service.cc:1467] Processing CreateTablet for tablet b2e62c043adf4a8d9024279d786f596c (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=e035403de1c6423e98f3bb44dcb47f8a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:06.593014 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2e62c043adf4a8d9024279d786f596c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:06.593098 9285 tablet_service.cc:1467] Processing CreateTablet for tablet beeea71f02ae4c89a1af13d3584ef70a (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=e035403de1c6423e98f3bb44dcb47f8a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:06.593371 9285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet beeea71f02ae4c89a1af13d3584ef70a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:06.594882 12829 tablet_bootstrap.cc:492] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:06.595131 9286 tablet_service.cc:1467] Processing CreateTablet for tablet b2e62c043adf4a8d9024279d786f596c (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=e035403de1c6423e98f3bb44dcb47f8a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:06.595381 9286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2e62c043adf4a8d9024279d786f596c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:06.595839 12829 tablet_bootstrap.cc:654] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:06.597668 9028 tablet_service.cc:1467] Processing CreateTablet for tablet beeea71f02ae4c89a1af13d3584ef70a (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=e035403de1c6423e98f3bb44dcb47f8a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:06.597962 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet beeea71f02ae4c89a1af13d3584ef70a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:06.600575 12788 tablet_bootstrap.cc:492] T beeea71f02ae4c89a1af13d3584ef70a P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap starting.
I20240503 02:00:06.601778 12788 tablet_bootstrap.cc:654] T beeea71f02ae4c89a1af13d3584ef70a P e1a9251ce1cf4361b8b0ede9fbfc4023: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:06.603523 12788 tablet_bootstrap.cc:492] T beeea71f02ae4c89a1af13d3584ef70a P e1a9251ce1cf4361b8b0ede9fbfc4023: No bootstrap required, opened a new log
I20240503 02:00:06.603617 12788 ts_tablet_manager.cc:1367] T beeea71f02ae4c89a1af13d3584ef70a P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:06.603945 9158 tablet_service.cc:1467] Processing CreateTablet for tablet b2e62c043adf4a8d9024279d786f596c (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=e035403de1c6423e98f3bb44dcb47f8a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:06.604066 12788 raft_consensus.cc:348] T beeea71f02ae4c89a1af13d3584ef70a P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } }
I20240503 02:00:06.604177 12788 raft_consensus.cc:374] T beeea71f02ae4c89a1af13d3584ef70a P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:06.604209 12788 raft_consensus.cc:729] T beeea71f02ae4c89a1af13d3584ef70a P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1a9251ce1cf4361b8b0ede9fbfc4023, State: Initialized, Role: FOLLOWER
I20240503 02:00:06.604243 9158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2e62c043adf4a8d9024279d786f596c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:06.604327 12788 consensus_queue.cc:260] T beeea71f02ae4c89a1af13d3584ef70a P e1a9251ce1cf4361b8b0ede9fbfc4023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } }
I20240503 02:00:06.604624 9159 tablet_service.cc:1467] Processing CreateTablet for tablet beeea71f02ae4c89a1af13d3584ef70a (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=e035403de1c6423e98f3bb44dcb47f8a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:06.604728 12788 ts_tablet_manager.cc:1398] T beeea71f02ae4c89a1af13d3584ef70a P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:06.604843 9159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet beeea71f02ae4c89a1af13d3584ef70a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:06.604888 12788 tablet_bootstrap.cc:492] T b2e62c043adf4a8d9024279d786f596c P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap starting.
I20240503 02:00:06.604996 9157 tablet_service.cc:1467] Processing CreateTablet for tablet b20ff3a6f74f4640afebfd4059b58699 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=e035403de1c6423e98f3bb44dcb47f8a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:06.605217 9157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b20ff3a6f74f4640afebfd4059b58699. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:06.606220 12788 tablet_bootstrap.cc:654] T b2e62c043adf4a8d9024279d786f596c P e1a9251ce1cf4361b8b0ede9fbfc4023: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:06.605551 9028 tablet_service.cc:1467] Processing CreateTablet for tablet b20ff3a6f74f4640afebfd4059b58699 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=e035403de1c6423e98f3bb44dcb47f8a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:06.606954 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b20ff3a6f74f4640afebfd4059b58699. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:06.607007 12787 tablet_bootstrap.cc:492] T beeea71f02ae4c89a1af13d3584ef70a P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:06.607887 12787 tablet_bootstrap.cc:654] T beeea71f02ae4c89a1af13d3584ef70a P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:06.608403 12835 tablet_bootstrap.cc:492] T beeea71f02ae4c89a1af13d3584ef70a P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:06.608887 6700 tablet_service.cc:1467] Processing CreateTablet for tablet b20ff3a6f74f4640afebfd4059b58699 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=e035403de1c6423e98f3bb44dcb47f8a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:06.608980 12787 tablet_bootstrap.cc:492] T beeea71f02ae4c89a1af13d3584ef70a P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:06.609056 12787 ts_tablet_manager.cc:1367] T beeea71f02ae4c89a1af13d3584ef70a P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:06.609161 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b20ff3a6f74f4640afebfd4059b58699. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:06.609537 12787 raft_consensus.cc:348] T beeea71f02ae4c89a1af13d3584ef70a P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } }
I20240503 02:00:06.609639 12787 raft_consensus.cc:374] T beeea71f02ae4c89a1af13d3584ef70a P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:06.609668 12787 raft_consensus.cc:729] T beeea71f02ae4c89a1af13d3584ef70a P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:06.609776 12787 consensus_queue.cc:260] T beeea71f02ae4c89a1af13d3584ef70a P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } }
I20240503 02:00:06.610049 12787 ts_tablet_manager.cc:1398] T beeea71f02ae4c89a1af13d3584ef70a P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:06.610273 12787 tablet_bootstrap.cc:492] T b20ff3a6f74f4640afebfd4059b58699 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:06.610997 12787 tablet_bootstrap.cc:654] T b20ff3a6f74f4640afebfd4059b58699 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:06.611267 12840 tablet_bootstrap.cc:492] T b20ff3a6f74f4640afebfd4059b58699 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:06.612522 12835 tablet_bootstrap.cc:654] T beeea71f02ae4c89a1af13d3584ef70a P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:06.612918 12787 tablet_bootstrap.cc:492] T b20ff3a6f74f4640afebfd4059b58699 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:06.613003 12787 ts_tablet_manager.cc:1367] T b20ff3a6f74f4640afebfd4059b58699 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:06.613986 12840 tablet_bootstrap.cc:654] T b20ff3a6f74f4640afebfd4059b58699 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:06.614223 12787 raft_consensus.cc:348] T b20ff3a6f74f4640afebfd4059b58699 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:06.614440 12787 raft_consensus.cc:374] T b20ff3a6f74f4640afebfd4059b58699 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:06.614521 12787 raft_consensus.cc:729] T b20ff3a6f74f4640afebfd4059b58699 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:06.614712 12788 tablet_bootstrap.cc:492] T b2e62c043adf4a8d9024279d786f596c P e1a9251ce1cf4361b8b0ede9fbfc4023: No bootstrap required, opened a new log
I20240503 02:00:06.614689 12787 consensus_queue.cc:260] T b20ff3a6f74f4640afebfd4059b58699 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:06.614807 12788 ts_tablet_manager.cc:1367] T b2e62c043adf4a8d9024279d786f596c P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 02:00:06.615293 12788 raft_consensus.cc:348] T b2e62c043adf4a8d9024279d786f596c P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } }
I20240503 02:00:06.615394 12788 raft_consensus.cc:374] T b2e62c043adf4a8d9024279d786f596c P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:06.615425 12788 raft_consensus.cc:729] T b2e62c043adf4a8d9024279d786f596c P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1a9251ce1cf4361b8b0ede9fbfc4023, State: Initialized, Role: FOLLOWER
I20240503 02:00:06.615538 12788 consensus_queue.cc:260] T b2e62c043adf4a8d9024279d786f596c P e1a9251ce1cf4361b8b0ede9fbfc4023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } }
I20240503 02:00:06.615700 12835 tablet_bootstrap.cc:492] T beeea71f02ae4c89a1af13d3584ef70a P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:06.615805 12788 ts_tablet_manager.cc:1398] T b2e62c043adf4a8d9024279d786f596c P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:06.615800 12835 ts_tablet_manager.cc:1367] T beeea71f02ae4c89a1af13d3584ef70a P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.001s
I20240503 02:00:06.616549 12835 raft_consensus.cc:348] T beeea71f02ae4c89a1af13d3584ef70a P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } }
I20240503 02:00:06.616720 12829 tablet_bootstrap.cc:492] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:06.616731 12835 raft_consensus.cc:374] T beeea71f02ae4c89a1af13d3584ef70a P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:06.616829 12829 ts_tablet_manager.cc:1367] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20240503 02:00:06.616849 12835 raft_consensus.cc:729] T beeea71f02ae4c89a1af13d3584ef70a P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:06.617024 12835 consensus_queue.cc:260] T beeea71f02ae4c89a1af13d3584ef70a P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } }
I20240503 02:00:06.617257 12829 raft_consensus.cc:348] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } }
I20240503 02:00:06.617347 12829 raft_consensus.cc:374] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:06.617345 12835 ts_tablet_manager.cc:1398] T beeea71f02ae4c89a1af13d3584ef70a P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:06.617375 12829 raft_consensus.cc:729] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:06.617514 12835 tablet_bootstrap.cc:492] T b20ff3a6f74f4640afebfd4059b58699 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:06.617491 12829 consensus_queue.cc:260] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } }
I20240503 02:00:06.617731 12829 ts_tablet_manager.cc:1398] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:06.618552 12835 tablet_bootstrap.cc:654] T b20ff3a6f74f4640afebfd4059b58699 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:06.620208 12840 tablet_bootstrap.cc:492] T b20ff3a6f74f4640afebfd4059b58699 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:06.620405 12840 ts_tablet_manager.cc:1367] T b20ff3a6f74f4640afebfd4059b58699 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 02:00:06.621088 12840 raft_consensus.cc:348] T b20ff3a6f74f4640afebfd4059b58699 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:06.621249 12840 raft_consensus.cc:374] T b20ff3a6f74f4640afebfd4059b58699 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:06.621292 12840 raft_consensus.cc:729] T b20ff3a6f74f4640afebfd4059b58699 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:06.621428 12840 consensus_queue.cc:260] T b20ff3a6f74f4640afebfd4059b58699 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:06.621763 12840 ts_tablet_manager.cc:1398] T b20ff3a6f74f4640afebfd4059b58699 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:06.623742 12835 tablet_bootstrap.cc:492] T b20ff3a6f74f4640afebfd4059b58699 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:06.623870 12835 ts_tablet_manager.cc:1367] T b20ff3a6f74f4640afebfd4059b58699 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20240503 02:00:06.624271 6458 raft_consensus.cc:1261] T d897c21e630d4770832bb10f43778e82 P dec3e4a04d604f7eaaa3f077d081218a [term 2 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:00:06.624325 12835 raft_consensus.cc:348] T b20ff3a6f74f4640afebfd4059b58699 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:06.624475 12835 raft_consensus.cc:374] T b20ff3a6f74f4640afebfd4059b58699 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:06.624511 12835 raft_consensus.cc:729] T b20ff3a6f74f4640afebfd4059b58699 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:06.624629 12835 consensus_queue.cc:260] T b20ff3a6f74f4640afebfd4059b58699 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:06.625066 12742 consensus_queue.cc:1035] T d897c21e630d4770832bb10f43778e82 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:06.625511 12835 ts_tablet_manager.cc:1398] T b20ff3a6f74f4640afebfd4059b58699 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:06.626983 12787 ts_tablet_manager.cc:1398] T b20ff3a6f74f4640afebfd4059b58699 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.014s user 0.001s sys 0.000s
I20240503 02:00:06.627851 12787 tablet_bootstrap.cc:492] T b2e62c043adf4a8d9024279d786f596c P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:06.629141 12787 tablet_bootstrap.cc:654] T b2e62c043adf4a8d9024279d786f596c P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:06.630730 12787 tablet_bootstrap.cc:492] T b2e62c043adf4a8d9024279d786f596c P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:06.630837 12787 ts_tablet_manager.cc:1367] T b2e62c043adf4a8d9024279d786f596c P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:06.631343 12787 raft_consensus.cc:348] T b2e62c043adf4a8d9024279d786f596c P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } }
I20240503 02:00:06.631459 12787 raft_consensus.cc:374] T b2e62c043adf4a8d9024279d786f596c P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:06.631493 12787 raft_consensus.cc:729] T b2e62c043adf4a8d9024279d786f596c P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:06.631677 12787 consensus_queue.cc:260] T b2e62c043adf4a8d9024279d786f596c P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } }
I20240503 02:00:06.632126 12787 ts_tablet_manager.cc:1398] T b2e62c043adf4a8d9024279d786f596c P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 02:00:06.660447 10349 raft_consensus.cc:1050] T b91a933fb66b4075acb40c5b92ef4824 P c2ac2eb7cc264047b9ab32fe3a4fc025: attempting to promote NON_VOTER 9942306855404cfd891adf227dd65eb2 to VOTER
I20240503 02:00:06.660881 10349 consensus_queue.cc:237] T b91a933fb66b4075acb40c5b92ef4824 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } }
I20240503 02:00:06.662174 6582 raft_consensus.cc:1261] T b91a933fb66b4075acb40c5b92ef4824 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:06.662626 11967 consensus_queue.cc:1035] T b91a933fb66b4075acb40c5b92ef4824 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:06.662748 6456 raft_consensus.cc:1261] T b91a933fb66b4075acb40c5b92ef4824 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:06.663072 8917 raft_consensus.cc:1261] T b91a933fb66b4075acb40c5b92ef4824 P 9942306855404cfd891adf227dd65eb2 [term 1 LEARNER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:06.663209 11967 consensus_queue.cc:1035] T b91a933fb66b4075acb40c5b92ef4824 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:06.663457 10349 consensus_queue.cc:1035] T b91a933fb66b4075acb40c5b92ef4824 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:06.665354 10349 raft_consensus.cc:2932] T b91a933fb66b4075acb40c5b92ef4824 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:06.665851 8917 raft_consensus.cc:2932] T b91a933fb66b4075acb40c5b92ef4824 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:06.666093 6456 raft_consensus.cc:2932] T b91a933fb66b4075acb40c5b92ef4824 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:06.668057 9309 raft_consensus.cc:1261] T d897c21e630d4770832bb10f43778e82 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 2 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:00:06.668808 12742 consensus_queue.cc:1035] T d897c21e630d4770832bb10f43778e82 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:06.669224 6314 catalog_manager.cc:5484] T b91a933fb66b4075acb40c5b92ef4824 P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: config changed from index 4 to 5, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:06.669508 6582 raft_consensus.cc:2932] T b91a933fb66b4075acb40c5b92ef4824 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:06.672526 6720 consensus_queue.cc:237] T b91a933fb66b4075acb40c5b92ef4824 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } }
I20240503 02:00:06.674396 6720 raft_consensus.cc:1261] T d897c21e630d4770832bb10f43778e82 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 2 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:00:06.674924 12741 consensus_queue.cc:1035] T d897c21e630d4770832bb10f43778e82 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:06.675706 8917 raft_consensus.cc:1261] T b91a933fb66b4075acb40c5b92ef4824 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:06.676225 11967 consensus_queue.cc:1035] T b91a933fb66b4075acb40c5b92ef4824 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:06.679195 6582 raft_consensus.cc:1261] T b91a933fb66b4075acb40c5b92ef4824 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:06.679405 6849 consensus_queue.cc:237] T d897c21e630d4770832bb10f43778e82 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } }
I20240503 02:00:06.679740 10349 consensus_queue.cc:1035] T b91a933fb66b4075acb40c5b92ef4824 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:06.679682 11967 raft_consensus.cc:2932] T b91a933fb66b4075acb40c5b92ef4824 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:06.681088 6458 raft_consensus.cc:1261] T d897c21e630d4770832bb10f43778e82 P dec3e4a04d604f7eaaa3f077d081218a [term 2 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:00:06.681650 6303 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b91a933fb66b4075acb40c5b92ef4824 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:06.681968 12741 consensus_queue.cc:1035] T d897c21e630d4770832bb10f43778e82 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:06.682211 9309 raft_consensus.cc:1261] T d897c21e630d4770832bb10f43778e82 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 2 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:00:06.682212 8917 raft_consensus.cc:2932] T b91a933fb66b4075acb40c5b92ef4824 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:06.684599 12032 consensus_queue.cc:1035] T d897c21e630d4770832bb10f43778e82 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:06.684693 6582 raft_consensus.cc:2932] T b91a933fb66b4075acb40c5b92ef4824 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:06.685154 12032 raft_consensus.cc:2932] T d897c21e630d4770832bb10f43778e82 P 492448570cc94ed1879feb2ec3b9051d [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:06.685906 6312 catalog_manager.cc:5484] T b91a933fb66b4075acb40c5b92ef4824 P 9942306855404cfd891adf227dd65eb2 reported cstate change: config changed from index 5 to 6, VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:06.687585 6456 raft_consensus.cc:2932] T d897c21e630d4770832bb10f43778e82 P dec3e4a04d604f7eaaa3f077d081218a [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:06.688553 6303 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d897c21e630d4770832bb10f43778e82 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:06.690867 6315 catalog_manager.cc:5484] T d897c21e630d4770832bb10f43778e82 P 492448570cc94ed1879feb2ec3b9051d reported cstate change: config changed from index 5 to 7, VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:06.692458 6436 tablet_service.cc:1514] Processing DeleteTablet for tablet b91a933fb66b4075acb40c5b92ef4824 with delete_type TABLET_DATA_TOMBSTONED (TS dec3e4a04d604f7eaaa3f077d081218a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51338
I20240503 02:00:06.692868 12877 tablet_replica.cc:330] T b91a933fb66b4075acb40c5b92ef4824 P dec3e4a04d604f7eaaa3f077d081218a: stopping tablet replica
I20240503 02:00:06.693132 12877 raft_consensus.cc:2229] T b91a933fb66b4075acb40c5b92ef4824 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:06.693400 12877 raft_consensus.cc:2258] T b91a933fb66b4075acb40c5b92ef4824 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:06.694204 6700 tablet_service.cc:1514] Processing DeleteTablet for tablet d897c21e630d4770832bb10f43778e82 with delete_type TABLET_DATA_TOMBSTONED (TS c2ac2eb7cc264047b9ab32fe3a4fc025 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:40336
I20240503 02:00:06.694468 12748 tablet_replica.cc:330] T d897c21e630d4770832bb10f43778e82 P c2ac2eb7cc264047b9ab32fe3a4fc025: stopping tablet replica
I20240503 02:00:06.694607 12748 raft_consensus.cc:2229] T d897c21e630d4770832bb10f43778e82 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:06.694610 12877 ts_tablet_manager.cc:1875] T b91a933fb66b4075acb40c5b92ef4824 P dec3e4a04d604f7eaaa3f077d081218a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:06.694708 12748 raft_consensus.cc:2258] T d897c21e630d4770832bb10f43778e82 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:06.694875 9309 raft_consensus.cc:2932] T d897c21e630d4770832bb10f43778e82 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:06.696117 12748 ts_tablet_manager.cc:1875] T d897c21e630d4770832bb10f43778e82 P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:06.697198 12877 ts_tablet_manager.cc:1888] T b91a933fb66b4075acb40c5b92ef4824 P dec3e4a04d604f7eaaa3f077d081218a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:00:06.697288 12877 log.cc:1196] T b91a933fb66b4075acb40c5b92ef4824 P dec3e4a04d604f7eaaa3f077d081218a: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/b91a933fb66b4075acb40c5b92ef4824
I20240503 02:00:06.697789 6300 catalog_manager.cc:4830] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): tablet b91a933fb66b4075acb40c5b92ef4824 (table rebalance_test_table_5 [id=9b77b848294749e09f0854d24da5d9f5]) successfully deleted
I20240503 02:00:06.698143 12748 ts_tablet_manager.cc:1888] T d897c21e630d4770832bb10f43778e82 P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 02:00:06.698235 12748 log.cc:1196] T d897c21e630d4770832bb10f43778e82 P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/d897c21e630d4770832bb10f43778e82
I20240503 02:00:06.698807 6303 catalog_manager.cc:4830] TS c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655): tablet d897c21e630d4770832bb10f43778e82 (table rebalance_test_table_5 [id=9b77b848294749e09f0854d24da5d9f5]) successfully deleted
I20240503 02:00:06.721499 10473 raft_consensus.cc:482] T beeea71f02ae4c89a1af13d3584ef70a P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:06.721655 10473 raft_consensus.cc:504] T beeea71f02ae4c89a1af13d3584ef70a P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } }
I20240503 02:00:06.722134 10473 leader_election.cc:288] T beeea71f02ae4c89a1af13d3584ef70a P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311)
I20240503 02:00:06.722620 9176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beeea71f02ae4c89a1af13d3584ef70a" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:06.722625 9309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beeea71f02ae4c89a1af13d3584ef70a" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" is_pre_election: true
I20240503 02:00:06.722783 9309 raft_consensus.cc:2454] T beeea71f02ae4c89a1af13d3584ef70a P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:06.722792 9176 raft_consensus.cc:2454] T beeea71f02ae4c89a1af13d3584ef70a P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:06.723207 8982 leader_election.cc:302] T beeea71f02ae4c89a1af13d3584ef70a P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99ead02915954b4fab23b033ad6d546b, e1a9251ce1cf4361b8b0ede9fbfc4023; no voters:
I20240503 02:00:06.723385 10473 raft_consensus.cc:2781] T beeea71f02ae4c89a1af13d3584ef70a P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:06.723450 10473 raft_consensus.cc:482] T beeea71f02ae4c89a1af13d3584ef70a P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:06.723505 10473 raft_consensus.cc:3037] T beeea71f02ae4c89a1af13d3584ef70a P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:06.724642 10473 raft_consensus.cc:504] T beeea71f02ae4c89a1af13d3584ef70a P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } }
I20240503 02:00:06.725129 10473 leader_election.cc:288] T beeea71f02ae4c89a1af13d3584ef70a P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Requested vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311)
I20240503 02:00:06.725427 9176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beeea71f02ae4c89a1af13d3584ef70a" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:06.725560 9176 raft_consensus.cc:3037] T beeea71f02ae4c89a1af13d3584ef70a P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:06.725657 9309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beeea71f02ae4c89a1af13d3584ef70a" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023"
I20240503 02:00:06.725831 9309 raft_consensus.cc:3037] T beeea71f02ae4c89a1af13d3584ef70a P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:06.726514 9176 raft_consensus.cc:2454] T beeea71f02ae4c89a1af13d3584ef70a P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:06.726686 9309 raft_consensus.cc:2454] T beeea71f02ae4c89a1af13d3584ef70a P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:06.726807 8981 leader_election.cc:302] T beeea71f02ae4c89a1af13d3584ef70a P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:06.726984 10473 raft_consensus.cc:2781] T beeea71f02ae4c89a1af13d3584ef70a P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:06.727059 10473 raft_consensus.cc:686] T beeea71f02ae4c89a1af13d3584ef70a P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Becoming Leader. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Running, Role: LEADER
I20240503 02:00:06.727258 10473 consensus_queue.cc:237] T beeea71f02ae4c89a1af13d3584ef70a P 99ead02915954b4fab23b033ad6d546b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } }
I20240503 02:00:06.729422 6307 catalog_manager.cc:5484] T beeea71f02ae4c89a1af13d3584ef70a P 99ead02915954b4fab23b033ad6d546b reported cstate change: term changed from 0 to 1, leader changed from <none> to 99ead02915954b4fab23b033ad6d546b (127.0.99.134). New cstate: current_term: 1 leader_uuid: "99ead02915954b4fab23b033ad6d546b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:06.752710 11459 raft_consensus.cc:482] T e831f37dd1884641a81c0a513092955f P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:06.752874 11459 raft_consensus.cc:504] T e831f37dd1884641a81c0a513092955f P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } }
I20240503 02:00:06.753870 11459 leader_election.cc:288] T e831f37dd1884641a81c0a513092955f P e1a9251ce1cf4361b8b0ede9fbfc4023 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:06.754485 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e831f37dd1884641a81c0a513092955f" candidate_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 02:00:06.754657 6456 raft_consensus.cc:2454] T e831f37dd1884641a81c0a513092955f P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1a9251ce1cf4361b8b0ede9fbfc4023 in term 0.
I20240503 02:00:06.755057 9243 leader_election.cc:302] T e831f37dd1884641a81c0a513092955f P e1a9251ce1cf4361b8b0ede9fbfc4023 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dec3e4a04d604f7eaaa3f077d081218a, e1a9251ce1cf4361b8b0ede9fbfc4023; no voters:
I20240503 02:00:06.755264 11459 raft_consensus.cc:2781] T e831f37dd1884641a81c0a513092955f P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:06.755326 11459 raft_consensus.cc:482] T e831f37dd1884641a81c0a513092955f P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:06.755362 11459 raft_consensus.cc:3037] T e831f37dd1884641a81c0a513092955f P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:06.756181 11459 raft_consensus.cc:504] T e831f37dd1884641a81c0a513092955f P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } }
I20240503 02:00:06.756577 11459 leader_election.cc:288] T e831f37dd1884641a81c0a513092955f P e1a9251ce1cf4361b8b0ede9fbfc4023 [CANDIDATE]: Term 1 election: Requested vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:06.756973 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e831f37dd1884641a81c0a513092955f" candidate_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 02:00:06.757097 6456 raft_consensus.cc:3037] T e831f37dd1884641a81c0a513092955f P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:06.757709 9176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e831f37dd1884641a81c0a513092955f" candidate_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:06.757709 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e831f37dd1884641a81c0a513092955f" candidate_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:06.757858 9176 raft_consensus.cc:2454] T e831f37dd1884641a81c0a513092955f P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1a9251ce1cf4361b8b0ede9fbfc4023 in term 0.
I20240503 02:00:06.758026 6456 raft_consensus.cc:2454] T e831f37dd1884641a81c0a513092955f P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1a9251ce1cf4361b8b0ede9fbfc4023 in term 1.
I20240503 02:00:06.758616 9243 leader_election.cc:302] T e831f37dd1884641a81c0a513092955f P e1a9251ce1cf4361b8b0ede9fbfc4023 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dec3e4a04d604f7eaaa3f077d081218a, e1a9251ce1cf4361b8b0ede9fbfc4023; no voters: 2a01ace80baf4057b6bf1995cf88468b
I20240503 02:00:06.758922 6720 raft_consensus.cc:1203] T 51c0137930464635be1e6a97f5e9d952 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:00:06.759284 11459 raft_consensus.cc:2781] T e831f37dd1884641a81c0a513092955f P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:06.759474 11459 raft_consensus.cc:686] T e831f37dd1884641a81c0a513092955f P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 LEADER]: Becoming Leader. State: Replica: e1a9251ce1cf4361b8b0ede9fbfc4023, State: Running, Role: LEADER
I20240503 02:00:06.759737 11459 consensus_queue.cc:237] T e831f37dd1884641a81c0a513092955f P e1a9251ce1cf4361b8b0ede9fbfc4023 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } }
I20240503 02:00:06.762413 6314 catalog_manager.cc:5484] T e831f37dd1884641a81c0a513092955f P e1a9251ce1cf4361b8b0ede9fbfc4023 reported cstate change: term changed from 0 to 1, leader changed from <none> to e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:06.792313 6720 raft_consensus.cc:1261] T 0ee05e70ccbb496ca3cff08dc38112e8 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 2 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:00:06.792807 12190 consensus_queue.cc:1035] T 0ee05e70ccbb496ca3cff08dc38112e8 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:06.840209 12880 raft_consensus.cc:482] T b2e62c043adf4a8d9024279d786f596c P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:06.840333 12880 raft_consensus.cc:504] T b2e62c043adf4a8d9024279d786f596c P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } }
I20240503 02:00:06.840821 12880 leader_election.cc:288] T b2e62c043adf4a8d9024279d786f596c P e1a9251ce1cf4361b8b0ede9fbfc4023 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:06.841254 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2e62c043adf4a8d9024279d786f596c" candidate_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 02:00:06.841261 9176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2e62c043adf4a8d9024279d786f596c" candidate_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:06.841408 9176 raft_consensus.cc:2454] T b2e62c043adf4a8d9024279d786f596c P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1a9251ce1cf4361b8b0ede9fbfc4023 in term 0.
I20240503 02:00:06.841409 6849 raft_consensus.cc:2454] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1a9251ce1cf4361b8b0ede9fbfc4023 in term 0.
I20240503 02:00:06.841715 9246 leader_election.cc:302] T b2e62c043adf4a8d9024279d786f596c P e1a9251ce1cf4361b8b0ede9fbfc4023 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, e1a9251ce1cf4361b8b0ede9fbfc4023; no voters:
I20240503 02:00:06.841897 12880 raft_consensus.cc:2781] T b2e62c043adf4a8d9024279d786f596c P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:06.841965 12880 raft_consensus.cc:482] T b2e62c043adf4a8d9024279d786f596c P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:06.841996 12880 raft_consensus.cc:3037] T b2e62c043adf4a8d9024279d786f596c P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:06.842864 12880 raft_consensus.cc:504] T b2e62c043adf4a8d9024279d786f596c P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } }
I20240503 02:00:06.843286 12880 leader_election.cc:288] T b2e62c043adf4a8d9024279d786f596c P e1a9251ce1cf4361b8b0ede9fbfc4023 [CANDIDATE]: Term 1 election: Requested vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:06.843614 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2e62c043adf4a8d9024279d786f596c" candidate_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 02:00:06.843705 9176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2e62c043adf4a8d9024279d786f596c" candidate_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:06.843742 6849 raft_consensus.cc:3037] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:06.843851 9176 raft_consensus.cc:3037] T b2e62c043adf4a8d9024279d786f596c P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:06.844677 6849 raft_consensus.cc:2454] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1a9251ce1cf4361b8b0ede9fbfc4023 in term 1.
I20240503 02:00:06.844779 9176 raft_consensus.cc:2454] T b2e62c043adf4a8d9024279d786f596c P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1a9251ce1cf4361b8b0ede9fbfc4023 in term 1.
I20240503 02:00:06.844985 9246 leader_election.cc:302] T b2e62c043adf4a8d9024279d786f596c P e1a9251ce1cf4361b8b0ede9fbfc4023 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, e1a9251ce1cf4361b8b0ede9fbfc4023; no voters:
I20240503 02:00:06.845213 12880 raft_consensus.cc:2781] T b2e62c043adf4a8d9024279d786f596c P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:06.845286 12880 raft_consensus.cc:686] T b2e62c043adf4a8d9024279d786f596c P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 LEADER]: Becoming Leader. State: Replica: e1a9251ce1cf4361b8b0ede9fbfc4023, State: Running, Role: LEADER
I20240503 02:00:06.845465 12880 consensus_queue.cc:237] T b2e62c043adf4a8d9024279d786f596c P e1a9251ce1cf4361b8b0ede9fbfc4023 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } }
I20240503 02:00:06.848121 6314 catalog_manager.cc:5484] T b2e62c043adf4a8d9024279d786f596c P e1a9251ce1cf4361b8b0ede9fbfc4023 reported cstate change: term changed from 0 to 1, leader changed from <none> to e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:06.883683 6314 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:06.888959 6456 raft_consensus.cc:1261] T e831f37dd1884641a81c0a513092955f P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer e1a9251ce1cf4361b8b0ede9fbfc4023: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:06.889493 9179 raft_consensus.cc:1261] T 0ee05e70ccbb496ca3cff08dc38112e8 P 2a01ace80baf4057b6bf1995cf88468b [term 2 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:00:06.889550 9176 raft_consensus.cc:3037] T e831f37dd1884641a81c0a513092955f P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:06.889843 11459 consensus_queue.cc:1035] T e831f37dd1884641a81c0a513092955f P e1a9251ce1cf4361b8b0ede9fbfc4023 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:06.890672 9176 raft_consensus.cc:1261] T e831f37dd1884641a81c0a513092955f P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer e1a9251ce1cf4361b8b0ede9fbfc4023: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:06.892305 12792 consensus_queue.cc:1035] T 0ee05e70ccbb496ca3cff08dc38112e8 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:06.892560 11459 consensus_queue.cc:1035] T e831f37dd1884641a81c0a513092955f P e1a9251ce1cf4361b8b0ede9fbfc4023 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:06.895573 6314 catalog_manager.cc:5576] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079) does not have the latest schema for tablet e831f37dd1884641a81c0a513092955f (table rebalancer_dynamic_table [id=e2ff6c9d20d9476b9c546ad715bb2066]). Expected version 1 got 0
I20240503 02:00:06.896329 12911 tablet.cc:1722] T e831f37dd1884641a81c0a513092955f P e1a9251ce1cf4361b8b0ede9fbfc4023: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:06.896629 9047 raft_consensus.cc:1261] T 0ee05e70ccbb496ca3cff08dc38112e8 P 99ead02915954b4fab23b033ad6d546b [term 2 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:00:06.897199 12911 tablet.cc:1620] T e831f37dd1884641a81c0a513092955f P e1a9251ce1cf4361b8b0ede9fbfc4023: MemRowSet was empty: no flush needed.
I20240503 02:00:06.898095 11851 consensus_queue.cc:1035] T 0ee05e70ccbb496ca3cff08dc38112e8 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:06.898774 12914 tablet.cc:1722] T e831f37dd1884641a81c0a513092955f P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:06.899107 12914 tablet.cc:1620] T e831f37dd1884641a81c0a513092955f P dec3e4a04d604f7eaaa3f077d081218a: MemRowSet was empty: no flush needed.
I20240503 02:00:06.902359 6314 catalog_manager.cc:5576] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093) does not have the latest schema for tablet e831f37dd1884641a81c0a513092955f (table rebalancer_dynamic_table [id=e2ff6c9d20d9476b9c546ad715bb2066]). Expected version 1 got 0
I20240503 02:00:06.905014 6307 catalog_manager.cc:5576] TS e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311) does not have the latest schema for tablet e831f37dd1884641a81c0a513092955f (table rebalancer_dynamic_table [id=e2ff6c9d20d9476b9c546ad715bb2066]). Expected version 1 got 0
I20240503 02:00:06.907505 6314 catalog_manager.cc:5576] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079) does not have the latest schema for tablet e831f37dd1884641a81c0a513092955f (table rebalancer_dynamic_table [id=e2ff6c9d20d9476b9c546ad715bb2066]). Expected version 1 got 0
I20240503 02:00:06.907907 12912 tablet.cc:1722] T e831f37dd1884641a81c0a513092955f P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:06.908324 12912 tablet.cc:1620] T e831f37dd1884641a81c0a513092955f P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
I20240503 02:00:06.908942 6307 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=e2ff6c9d20d9476b9c546ad715bb2066] alter complete (version 1)
I20240503 02:00:06.918114 12907 raft_consensus.cc:482] T b20ff3a6f74f4640afebfd4059b58699 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:06.918231 11514 raft_consensus.cc:1050] T 6a82dcc9be5b4c93b5fff03f789350ab P 99ead02915954b4fab23b033ad6d546b: attempting to promote NON_VOTER dec3e4a04d604f7eaaa3f077d081218a to VOTER
I20240503 02:00:06.918231 12907 raft_consensus.cc:504] T b20ff3a6f74f4640afebfd4059b58699 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:06.918676 12907 leader_election.cc:288] T b20ff3a6f74f4640afebfd4059b58699 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:06.918695 11514 consensus_queue.cc:237] T 6a82dcc9be5b4c93b5fff03f789350ab P 99ead02915954b4fab23b033ad6d546b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } }
I20240503 02:00:06.919224 6720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b20ff3a6f74f4640afebfd4059b58699" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 02:00:06.919381 6720 raft_consensus.cc:2454] T b20ff3a6f74f4640afebfd4059b58699 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:06.919664 9115 leader_election.cc:302] T b20ff3a6f74f4640afebfd4059b58699 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:06.919812 9047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b20ff3a6f74f4640afebfd4059b58699" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:06.919930 9047 raft_consensus.cc:2454] T b20ff3a6f74f4640afebfd4059b58699 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:06.919936 12907 raft_consensus.cc:2781] T b20ff3a6f74f4640afebfd4059b58699 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:06.919991 12907 raft_consensus.cc:482] T b20ff3a6f74f4640afebfd4059b58699 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:06.920023 12907 raft_consensus.cc:3037] T b20ff3a6f74f4640afebfd4059b58699 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:06.920132 9309 raft_consensus.cc:1261] T 6a82dcc9be5b4c93b5fff03f789350ab P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Refusing update from remote peer 99ead02915954b4fab23b033ad6d546b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:06.920315 6456 raft_consensus.cc:1261] T 6a82dcc9be5b4c93b5fff03f789350ab P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEARNER]: Refusing update from remote peer 99ead02915954b4fab23b033ad6d546b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:06.920570 11515 consensus_queue.cc:1035] T 6a82dcc9be5b4c93b5fff03f789350ab P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:06.921203 12907 raft_consensus.cc:504] T b20ff3a6f74f4640afebfd4059b58699 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:06.921308 6849 raft_consensus.cc:1261] T 6a82dcc9be5b4c93b5fff03f789350ab P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer 99ead02915954b4fab23b033ad6d546b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:06.921375 10473 consensus_queue.cc:1035] T 6a82dcc9be5b4c93b5fff03f789350ab P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:06.921589 12907 leader_election.cc:288] T b20ff3a6f74f4640afebfd4059b58699 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Requested vote from peers 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:06.921813 11515 consensus_queue.cc:1035] T 6a82dcc9be5b4c93b5fff03f789350ab P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:06.921981 6720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b20ff3a6f74f4640afebfd4059b58699" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 02:00:06.922096 6720 raft_consensus.cc:3037] T b20ff3a6f74f4640afebfd4059b58699 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:06.923117 6720 raft_consensus.cc:2454] T b20ff3a6f74f4640afebfd4059b58699 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:06.923717 9047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b20ff3a6f74f4640afebfd4059b58699" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:06.923823 9047 raft_consensus.cc:3037] T b20ff3a6f74f4640afebfd4059b58699 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:06.924096 9115 leader_election.cc:302] T b20ff3a6f74f4640afebfd4059b58699 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:06.924261 12907 raft_consensus.cc:2781] T b20ff3a6f74f4640afebfd4059b58699 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:06.924321 12907 raft_consensus.cc:686] T b20ff3a6f74f4640afebfd4059b58699 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Becoming Leader. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Running, Role: LEADER
I20240503 02:00:06.924485 12907 consensus_queue.cc:237] T b20ff3a6f74f4640afebfd4059b58699 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:06.924827 9047 raft_consensus.cc:2454] T b20ff3a6f74f4640afebfd4059b58699 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:06.925062 11515 raft_consensus.cc:2932] T 6a82dcc9be5b4c93b5fff03f789350ab P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } }
I20240503 02:00:06.925434 6849 raft_consensus.cc:2932] T 6a82dcc9be5b4c93b5fff03f789350ab P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } }
I20240503 02:00:06.926962 6307 catalog_manager.cc:5484] T b20ff3a6f74f4640afebfd4059b58699 P 2a01ace80baf4057b6bf1995cf88468b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135). New cstate: current_term: 1 leader_uuid: "2a01ace80baf4057b6bf1995cf88468b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:06.926961 9309 raft_consensus.cc:2932] T 6a82dcc9be5b4c93b5fff03f789350ab P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } }
I20240503 02:00:06.928807 6314 catalog_manager.cc:5484] T 6a82dcc9be5b4c93b5fff03f789350ab P 492448570cc94ed1879feb2ec3b9051d reported cstate change: config changed from index 2 to 3, dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "99ead02915954b4fab23b033ad6d546b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } }
I20240503 02:00:06.929687 6456 raft_consensus.cc:2932] T 6a82dcc9be5b4c93b5fff03f789350ab P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } }
I20240503 02:00:06.935750 9047 consensus_queue.cc:237] T 6a82dcc9be5b4c93b5fff03f789350ab P 99ead02915954b4fab23b033ad6d546b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } }
I20240503 02:00:06.936996 6456 raft_consensus.cc:1261] T 6a82dcc9be5b4c93b5fff03f789350ab P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 99ead02915954b4fab23b033ad6d546b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:06.937045 6849 raft_consensus.cc:1261] T 6a82dcc9be5b4c93b5fff03f789350ab P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer 99ead02915954b4fab23b033ad6d546b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:06.937436 12922 consensus_queue.cc:1035] T 6a82dcc9be5b4c93b5fff03f789350ab P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:06.937651 10473 consensus_queue.cc:1035] T 6a82dcc9be5b4c93b5fff03f789350ab P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:06.939436 11515 raft_consensus.cc:2932] T 6a82dcc9be5b4c93b5fff03f789350ab P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } }
I20240503 02:00:06.939834 6458 raft_consensus.cc:2932] T 6a82dcc9be5b4c93b5fff03f789350ab P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } }
I20240503 02:00:06.941749 6301 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6a82dcc9be5b4c93b5fff03f789350ab with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:06.942967 6307 catalog_manager.cc:5484] T 6a82dcc9be5b4c93b5fff03f789350ab P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: config changed from index 3 to 4, VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "99ead02915954b4fab23b033ad6d546b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } }
I20240503 02:00:06.944005 6849 raft_consensus.cc:2932] T 6a82dcc9be5b4c93b5fff03f789350ab P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } }
I20240503 02:00:06.946261 9286 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a82dcc9be5b4c93b5fff03f789350ab with delete_type TABLET_DATA_TOMBSTONED (TS e1a9251ce1cf4361b8b0ede9fbfc4023 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36004
I20240503 02:00:06.946847 12937 tablet_replica.cc:330] T 6a82dcc9be5b4c93b5fff03f789350ab P e1a9251ce1cf4361b8b0ede9fbfc4023: stopping tablet replica
I20240503 02:00:06.947019 12937 raft_consensus.cc:2229] T 6a82dcc9be5b4c93b5fff03f789350ab P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:06.947268 12937 raft_consensus.cc:2258] T 6a82dcc9be5b4c93b5fff03f789350ab P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:06.948001 12937 ts_tablet_manager.cc:1875] T 6a82dcc9be5b4c93b5fff03f789350ab P e1a9251ce1cf4361b8b0ede9fbfc4023: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:06.950698 12937 ts_tablet_manager.cc:1888] T 6a82dcc9be5b4c93b5fff03f789350ab P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:00:06.950831 12937 log.cc:1196] T 6a82dcc9be5b4c93b5fff03f789350ab P e1a9251ce1cf4361b8b0ede9fbfc4023: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/6a82dcc9be5b4c93b5fff03f789350ab
I20240503 02:00:06.951401 6301 catalog_manager.cc:4830] TS e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): tablet 6a82dcc9be5b4c93b5fff03f789350ab (table rebalancer_extra_table_4 [id=c1969b96b61041869ddd81db8c84bf44]) successfully deleted
I20240503 02:00:06.980679 11851 raft_consensus.cc:1050] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a: attempting to promote NON_VOTER 99ead02915954b4fab23b033ad6d546b to VOTER
I20240503 02:00:06.981108 11851 consensus_queue.cc:237] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } }
I20240503 02:00:06.983276 6720 raft_consensus.cc:1261] T ac7fa0c3722a461c91fb237fd85794a3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:06.983358 6582 raft_consensus.cc:1261] T ac7fa0c3722a461c91fb237fd85794a3 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:06.983803 12792 consensus_queue.cc:1035] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:06.984226 12190 consensus_queue.cc:1035] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:06.986831 9047 raft_consensus.cc:1261] T ac7fa0c3722a461c91fb237fd85794a3 P 99ead02915954b4fab23b033ad6d546b [term 1 LEARNER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:06.987334 12959 consensus_queue.cc:1035] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:06.988020 12792 raft_consensus.cc:2932] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 99ead02915954b4fab23b033ad6d546b (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:06.990335 6582 raft_consensus.cc:2932] T ac7fa0c3722a461c91fb237fd85794a3 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 99ead02915954b4fab23b033ad6d546b (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:06.992565 6307 catalog_manager.cc:5484] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: config changed from index 4 to 5, 99ead02915954b4fab23b033ad6d546b (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:06.993762 6720 raft_consensus.cc:2932] T ac7fa0c3722a461c91fb237fd85794a3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 99ead02915954b4fab23b033ad6d546b (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:06.994233 9047 raft_consensus.cc:2932] T ac7fa0c3722a461c91fb237fd85794a3 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 99ead02915954b4fab23b033ad6d546b (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:06.999338 6458 consensus_queue.cc:237] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } }
I20240503 02:00:07.000728 6582 raft_consensus.cc:1261] T ac7fa0c3722a461c91fb237fd85794a3 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:07.000728 9047 raft_consensus.cc:1261] T ac7fa0c3722a461c91fb237fd85794a3 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:07.001374 12792 consensus_queue.cc:1035] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:07.001603 12190 consensus_queue.cc:1035] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:07.003427 12190 raft_consensus.cc:2932] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:07.005003 6300 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ac7fa0c3722a461c91fb237fd85794a3 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:07.005139 9047 raft_consensus.cc:2932] T ac7fa0c3722a461c91fb237fd85794a3 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:07.005781 6582 raft_consensus.cc:2932] T ac7fa0c3722a461c91fb237fd85794a3 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } }
I20240503 02:00:07.007872 6315 catalog_manager.cc:5484] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: config changed from index 5 to 6, VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:07.011263 6700 tablet_service.cc:1514] Processing DeleteTablet for tablet ac7fa0c3722a461c91fb237fd85794a3 with delete_type TABLET_DATA_TOMBSTONED (TS c2ac2eb7cc264047b9ab32fe3a4fc025 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:40336
I20240503 02:00:07.011663 12748 tablet_replica.cc:330] T ac7fa0c3722a461c91fb237fd85794a3 P c2ac2eb7cc264047b9ab32fe3a4fc025: stopping tablet replica
I20240503 02:00:07.011839 12748 raft_consensus.cc:2229] T ac7fa0c3722a461c91fb237fd85794a3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:07.012032 12748 raft_consensus.cc:2258] T ac7fa0c3722a461c91fb237fd85794a3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:07.012554 9984 raft_consensus.cc:1050] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025: attempting to promote NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 to VOTER
I20240503 02:00:07.012936 9984 consensus_queue.cc:237] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } }
I20240503 02:00:07.013011 12748 ts_tablet_manager.cc:1875] T ac7fa0c3722a461c91fb237fd85794a3 P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:07.014780 9179 raft_consensus.cc:1261] T 1feaf29d8c6b4842829ce66e0f4ec23b P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:07.014863 6456 raft_consensus.cc:1261] T 1feaf29d8c6b4842829ce66e0f4ec23b P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:07.015188 12748 ts_tablet_manager.cc:1888] T ac7fa0c3722a461c91fb237fd85794a3 P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:00:07.015255 12748 log.cc:1196] T ac7fa0c3722a461c91fb237fd85794a3 P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/ac7fa0c3722a461c91fb237fd85794a3
I20240503 02:00:07.015306 12872 consensus_queue.cc:1035] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:07.015575 11967 consensus_queue.cc:1035] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:07.015776 9309 raft_consensus.cc:1261] T 1feaf29d8c6b4842829ce66e0f4ec23b P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 LEARNER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:07.016163 11967 consensus_queue.cc:1035] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:07.017585 6303 catalog_manager.cc:4830] TS c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655): tablet ac7fa0c3722a461c91fb237fd85794a3 (table rebalance_test_table_3 [id=72f29547da0b4a95974018c2782cb359]) successfully deleted
I20240503 02:00:07.017920 6312 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:07.019690 11967 raft_consensus.cc:2932] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:07.020294 9309 raft_consensus.cc:2932] T 1feaf29d8c6b4842829ce66e0f4ec23b P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:07.022354 9179 raft_consensus.cc:2932] T 1feaf29d8c6b4842829ce66e0f4ec23b P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:07.023222 6312 catalog_manager.cc:5484] T 1feaf29d8c6b4842829ce66e0f4ec23b P e1a9251ce1cf4361b8b0ede9fbfc4023 reported cstate change: config changed from index 7 to 8, e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:07.024025 12911 tablet.cc:1722] T e831f37dd1884641a81c0a513092955f P e1a9251ce1cf4361b8b0ede9fbfc4023: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:07.024367 12911 tablet.cc:1620] T e831f37dd1884641a81c0a513092955f P e1a9251ce1cf4361b8b0ede9fbfc4023: MemRowSet was empty: no flush needed.
I20240503 02:00:07.024370 12914 tablet.cc:1722] T e831f37dd1884641a81c0a513092955f P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:07.024684 12914 tablet.cc:1620] T e831f37dd1884641a81c0a513092955f P dec3e4a04d604f7eaaa3f077d081218a: MemRowSet was empty: no flush needed.
I20240503 02:00:07.027874 6720 consensus_queue.cc:237] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } }
I20240503 02:00:07.029978 9176 raft_consensus.cc:1261] T 1feaf29d8c6b4842829ce66e0f4ec23b P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:07.030447 12872 consensus_queue.cc:1035] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:07.033071 6312 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=e2ff6c9d20d9476b9c546ad715bb2066] alter complete (version 2)
I20240503 02:00:07.034083 9984 raft_consensus.cc:2932] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:07.034605 9176 raft_consensus.cc:2932] T 1feaf29d8c6b4842829ce66e0f4ec23b P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:07.034797 9309 raft_consensus.cc:1261] T 1feaf29d8c6b4842829ce66e0f4ec23b P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:07.035791 6303 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1feaf29d8c6b4842829ce66e0f4ec23b with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:07.036466 10349 consensus_queue.cc:1035] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:07.037582 9309 raft_consensus.cc:2932] T 1feaf29d8c6b4842829ce66e0f4ec23b P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } }
I20240503 02:00:07.039014 12912 tablet.cc:1722] T e831f37dd1884641a81c0a513092955f P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:07.039476 12912 tablet.cc:1620] T e831f37dd1884641a81c0a513092955f P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
I20240503 02:00:07.041136 6314 catalog_manager.cc:5484] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: config changed from index 8 to 9, VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:07.043731 6436 tablet_service.cc:1514] Processing DeleteTablet for tablet 1feaf29d8c6b4842829ce66e0f4ec23b with delete_type TABLET_DATA_TOMBSTONED (TS dec3e4a04d604f7eaaa3f077d081218a not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:51338
I20240503 02:00:07.044256 12877 tablet_replica.cc:330] T 1feaf29d8c6b4842829ce66e0f4ec23b P dec3e4a04d604f7eaaa3f077d081218a: stopping tablet replica
I20240503 02:00:07.044456 12877 raft_consensus.cc:2229] T 1feaf29d8c6b4842829ce66e0f4ec23b P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:07.044624 12877 pending_rounds.cc:62] T 1feaf29d8c6b4842829ce66e0f4ec23b P dec3e4a04d604f7eaaa3f077d081218a: Trying to abort 1 pending ops.
I20240503 02:00:07.044654 12877 pending_rounds.cc:68] T 1feaf29d8c6b4842829ce66e0f4ec23b P dec3e4a04d604f7eaaa3f077d081218a: Aborting op as it isn't in flight: id { term: 1 index: 8 } timestamp: 7023417782323978240 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "1feaf29d8c6b4842829ce66e0f4ec23b" old_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } } } new_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: false } } } }
I20240503 02:00:07.044806 12877 raft_consensus.cc:2894] T 1feaf29d8c6b4842829ce66e0f4ec23b P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Aborting config change with OpId 1.8: Aborted: Op aborted
I20240503 02:00:07.044867 12877 raft_consensus.cc:2866] T 1feaf29d8c6b4842829ce66e0f4ec23b P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 02:00:07.044907 12877 raft_consensus.cc:2258] T 1feaf29d8c6b4842829ce66e0f4ec23b P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:07.046010 12877 ts_tablet_manager.cc:1875] T 1feaf29d8c6b4842829ce66e0f4ec23b P dec3e4a04d604f7eaaa3f077d081218a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:07.048408 12877 ts_tablet_manager.cc:1888] T 1feaf29d8c6b4842829ce66e0f4ec23b P dec3e4a04d604f7eaaa3f077d081218a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 02:00:07.048493 12877 log.cc:1196] T 1feaf29d8c6b4842829ce66e0f4ec23b P dec3e4a04d604f7eaaa3f077d081218a: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/1feaf29d8c6b4842829ce66e0f4ec23b
I20240503 02:00:07.049077 6300 catalog_manager.cc:4830] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): tablet 1feaf29d8c6b4842829ce66e0f4ec23b (table rebalance_test_table_13 [id=029e2476b8c44a8d8cdb98005eb84a4b]) successfully deleted
I20240503 02:00:07.139091 6314 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:07.146059 12911 tablet.cc:1722] T e831f37dd1884641a81c0a513092955f P e1a9251ce1cf4361b8b0ede9fbfc4023: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:07.147073 12912 tablet.cc:1722] T e831f37dd1884641a81c0a513092955f P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:07.148088 12914 tablet.cc:1722] T e831f37dd1884641a81c0a513092955f P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:07.152509 6315 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=e2ff6c9d20d9476b9c546ad715bb2066] alter complete (version 3)
I20240503 02:00:07.185577 6456 raft_consensus.cc:1203] T 1c721b33983b4c47afd8faa8f155ca84 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:00:07.203635 12792 raft_consensus.cc:1050] T 51c0137930464635be1e6a97f5e9d952 P dec3e4a04d604f7eaaa3f077d081218a: attempting to promote NON_VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 to VOTER
I20240503 02:00:07.204275 12792 consensus_queue.cc:237] T 51c0137930464635be1e6a97f5e9d952 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } }
I20240503 02:00:07.205847 6720 raft_consensus.cc:1261] T 51c0137930464635be1e6a97f5e9d952 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEARNER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:07.206038 9047 raft_consensus.cc:1261] T 51c0137930464635be1e6a97f5e9d952 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:07.206277 12959 consensus_queue.cc:1035] T 51c0137930464635be1e6a97f5e9d952 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:07.206982 12959 consensus_queue.cc:1035] T 51c0137930464635be1e6a97f5e9d952 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:07.209138 8917 raft_consensus.cc:1261] T 51c0137930464635be1e6a97f5e9d952 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:07.209560 12959 consensus_queue.cc:1035] T 51c0137930464635be1e6a97f5e9d952 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:07.211107 12959 raft_consensus.cc:2932] T 51c0137930464635be1e6a97f5e9d952 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } } }
I20240503 02:00:07.213174 6720 raft_consensus.cc:2932] T 51c0137930464635be1e6a97f5e9d952 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } } }
I20240503 02:00:07.214812 9047 raft_consensus.cc:2932] T 51c0137930464635be1e6a97f5e9d952 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } } }
I20240503 02:00:07.215301 6315 catalog_manager.cc:5484] T 51c0137930464635be1e6a97f5e9d952 P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: config changed from index 2 to 3, c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:07.217782 8917 raft_consensus.cc:2932] T 51c0137930464635be1e6a97f5e9d952 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } } }
I20240503 02:00:07.219820 6456 consensus_queue.cc:237] T 51c0137930464635be1e6a97f5e9d952 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } }
I20240503 02:00:07.221184 9047 raft_consensus.cc:1261] T 51c0137930464635be1e6a97f5e9d952 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:07.221290 6720 raft_consensus.cc:1261] T 51c0137930464635be1e6a97f5e9d952 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:07.221637 12190 consensus_queue.cc:1035] T 51c0137930464635be1e6a97f5e9d952 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:07.221827 12792 consensus_queue.cc:1035] T 51c0137930464635be1e6a97f5e9d952 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:07.223547 12959 raft_consensus.cc:2932] T 51c0137930464635be1e6a97f5e9d952 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } } }
I20240503 02:00:07.223920 6720 raft_consensus.cc:2932] T 51c0137930464635be1e6a97f5e9d952 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } } }
I20240503 02:00:07.223955 9047 raft_consensus.cc:2932] T 51c0137930464635be1e6a97f5e9d952 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } } }
I20240503 02:00:07.227017 6312 catalog_manager.cc:5484] T 51c0137930464635be1e6a97f5e9d952 P 99ead02915954b4fab23b033ad6d546b reported cstate change: config changed from index 3 to 4, VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } } }
I20240503 02:00:07.227227 6300 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 51c0137930464635be1e6a97f5e9d952 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:07.229769 8880 tablet_service.cc:1514] Processing DeleteTablet for tablet 51c0137930464635be1e6a97f5e9d952 with delete_type TABLET_DATA_TOMBSTONED (TS 9942306855404cfd891adf227dd65eb2 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40296
I20240503 02:00:07.230705 13023 tablet_replica.cc:330] T 51c0137930464635be1e6a97f5e9d952 P 9942306855404cfd891adf227dd65eb2: stopping tablet replica
I20240503 02:00:07.230872 13023 raft_consensus.cc:2229] T 51c0137930464635be1e6a97f5e9d952 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:07.231032 13023 raft_consensus.cc:2258] T 51c0137930464635be1e6a97f5e9d952 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:07.231681 13023 ts_tablet_manager.cc:1875] T 51c0137930464635be1e6a97f5e9d952 P 9942306855404cfd891adf227dd65eb2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:07.233819 13023 ts_tablet_manager.cc:1888] T 51c0137930464635be1e6a97f5e9d952 P 9942306855404cfd891adf227dd65eb2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:00:07.233898 13023 log.cc:1196] T 51c0137930464635be1e6a97f5e9d952 P 9942306855404cfd891adf227dd65eb2: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/51c0137930464635be1e6a97f5e9d952
I20240503 02:00:07.234431 6303 catalog_manager.cc:4830] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211): tablet 51c0137930464635be1e6a97f5e9d952 (table rebalancer_extra_table_0 [id=296df7c9486144058745436d9b814a78]) successfully deleted
I20240503 02:00:07.255769 6315 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:07.255936 6315 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:07.258100 6315 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet e831f37dd1884641a81c0a513092955f
I20240503 02:00:07.258780 6436 tablet_service.cc:1514] Processing DeleteTablet for tablet e831f37dd1884641a81c0a513092955f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:07 UTC) from {username='slave'} at 127.0.0.1:51338
I20240503 02:00:07.258777 9286 tablet_service.cc:1514] Processing DeleteTablet for tablet e831f37dd1884641a81c0a513092955f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:07 UTC) from {username='slave'} at 127.0.0.1:36004
I20240503 02:00:07.258944 9158 tablet_service.cc:1514] Processing DeleteTablet for tablet e831f37dd1884641a81c0a513092955f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:07 UTC) from {username='slave'} at 127.0.0.1:51940
I20240503 02:00:07.258992 12937 tablet_replica.cc:330] T e831f37dd1884641a81c0a513092955f P e1a9251ce1cf4361b8b0ede9fbfc4023: stopping tablet replica
I20240503 02:00:07.258987 12877 tablet_replica.cc:330] T e831f37dd1884641a81c0a513092955f P dec3e4a04d604f7eaaa3f077d081218a: stopping tablet replica
I20240503 02:00:07.259116 12877 raft_consensus.cc:2229] T e831f37dd1884641a81c0a513092955f P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:07.259119 12937 raft_consensus.cc:2229] T e831f37dd1884641a81c0a513092955f P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:07.259251 12877 raft_consensus.cc:2258] T e831f37dd1884641a81c0a513092955f P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:07.259294 13024 tablet_replica.cc:330] T e831f37dd1884641a81c0a513092955f P 2a01ace80baf4057b6bf1995cf88468b: stopping tablet replica
I20240503 02:00:07.259413 13024 raft_consensus.cc:2229] T e831f37dd1884641a81c0a513092955f P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:07.259423 12937 raft_consensus.cc:2258] T e831f37dd1884641a81c0a513092955f P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:07.259531 13024 raft_consensus.cc:2258] T e831f37dd1884641a81c0a513092955f P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:07.259891 12877 ts_tablet_manager.cc:1875] T e831f37dd1884641a81c0a513092955f P dec3e4a04d604f7eaaa3f077d081218a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:07.260016 12937 ts_tablet_manager.cc:1875] T e831f37dd1884641a81c0a513092955f P e1a9251ce1cf4361b8b0ede9fbfc4023: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:07.260913 13024 ts_tablet_manager.cc:1875] T e831f37dd1884641a81c0a513092955f P 2a01ace80baf4057b6bf1995cf88468b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:07.262163 12937 ts_tablet_manager.cc:1888] T e831f37dd1884641a81c0a513092955f P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:07.262174 12877 ts_tablet_manager.cc:1888] T e831f37dd1884641a81c0a513092955f P dec3e4a04d604f7eaaa3f077d081218a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:07.262236 12937 log.cc:1196] T e831f37dd1884641a81c0a513092955f P e1a9251ce1cf4361b8b0ede9fbfc4023: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/e831f37dd1884641a81c0a513092955f
I20240503 02:00:07.262243 12877 log.cc:1196] T e831f37dd1884641a81c0a513092955f P dec3e4a04d604f7eaaa3f077d081218a: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/e831f37dd1884641a81c0a513092955f
I20240503 02:00:07.262491 12937 ts_tablet_manager.cc:1909] T e831f37dd1884641a81c0a513092955f P e1a9251ce1cf4361b8b0ede9fbfc4023: Deleting consensus metadata
I20240503 02:00:07.262501 12877 ts_tablet_manager.cc:1909] T e831f37dd1884641a81c0a513092955f P dec3e4a04d604f7eaaa3f077d081218a: Deleting consensus metadata
I20240503 02:00:07.263214 13024 ts_tablet_manager.cc:1888] T e831f37dd1884641a81c0a513092955f P 2a01ace80baf4057b6bf1995cf88468b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:07.263320 13024 log.cc:1196] T e831f37dd1884641a81c0a513092955f P 2a01ace80baf4057b6bf1995cf88468b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/e831f37dd1884641a81c0a513092955f
I20240503 02:00:07.263423 6301 catalog_manager.cc:4830] TS e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): tablet e831f37dd1884641a81c0a513092955f (table rebalancer_dynamic_table_new_name [id=e2ff6c9d20d9476b9c546ad715bb2066]) successfully deleted
I20240503 02:00:07.263572 13024 ts_tablet_manager.cc:1909] T e831f37dd1884641a81c0a513092955f P 2a01ace80baf4057b6bf1995cf88468b: Deleting consensus metadata
I20240503 02:00:07.263639 6300 catalog_manager.cc:4830] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): tablet e831f37dd1884641a81c0a513092955f (table rebalancer_dynamic_table_new_name [id=e2ff6c9d20d9476b9c546ad715bb2066]) successfully deleted
I20240503 02:00:07.264302 6300 catalog_manager.cc:4830] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079): tablet e831f37dd1884641a81c0a513092955f (table rebalancer_dynamic_table_new_name [id=e2ff6c9d20d9476b9c546ad715bb2066]) successfully deleted
I20240503 02:00:07.296708 12922 consensus_queue.cc:1035] T beeea71f02ae4c89a1af13d3584ef70a P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:07.300697 11514 consensus_queue.cc:1035] T beeea71f02ae4c89a1af13d3584ef70a P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:07.322901 12880 consensus_queue.cc:1035] T b2e62c043adf4a8d9024279d786f596c P e1a9251ce1cf4361b8b0ede9fbfc4023 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:07.329497 12913 consensus_queue.cc:1035] T b2e62c043adf4a8d9024279d786f596c P e1a9251ce1cf4361b8b0ede9fbfc4023 [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:07.329699 12907 consensus_queue.cc:1035] T b20ff3a6f74f4640afebfd4059b58699 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:07.335542 12907 consensus_queue.cc:1035] T b20ff3a6f74f4640afebfd4059b58699 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:07.359985 6307 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:00:07.365482 9158 tablet_service.cc:1467] Processing CreateTablet for tablet 911e4b8bd8844cae911e94b8309eac97 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c47890e2f27546a28895df8613566824]), partition=
I20240503 02:00:07.365782 9158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 911e4b8bd8844cae911e94b8309eac97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:07.365779 9028 tablet_service.cc:1467] Processing CreateTablet for tablet 911e4b8bd8844cae911e94b8309eac97 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c47890e2f27546a28895df8613566824]), partition=
I20240503 02:00:07.366031 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 911e4b8bd8844cae911e94b8309eac97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:07.368408 8880 tablet_service.cc:1467] Processing CreateTablet for tablet 911e4b8bd8844cae911e94b8309eac97 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c47890e2f27546a28895df8613566824]), partition=
I20240503 02:00:07.368607 13060 tablet_bootstrap.cc:492] T 911e4b8bd8844cae911e94b8309eac97 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:07.368671 8880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 911e4b8bd8844cae911e94b8309eac97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:07.368777 13059 tablet_bootstrap.cc:492] T 911e4b8bd8844cae911e94b8309eac97 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:07.369704 13060 tablet_bootstrap.cc:654] T 911e4b8bd8844cae911e94b8309eac97 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:07.370285 13059 tablet_bootstrap.cc:654] T 911e4b8bd8844cae911e94b8309eac97 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:07.370893 13062 tablet_bootstrap.cc:492] T 911e4b8bd8844cae911e94b8309eac97 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:07.371779 13062 tablet_bootstrap.cc:654] T 911e4b8bd8844cae911e94b8309eac97 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:07.373346 13062 tablet_bootstrap.cc:492] T 911e4b8bd8844cae911e94b8309eac97 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:07.373481 13062 ts_tablet_manager.cc:1367] T 911e4b8bd8844cae911e94b8309eac97 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:07.373538 13060 tablet_bootstrap.cc:492] T 911e4b8bd8844cae911e94b8309eac97 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:07.373631 13060 ts_tablet_manager.cc:1367] T 911e4b8bd8844cae911e94b8309eac97 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:00:07.373916 13062 raft_consensus.cc:348] T 911e4b8bd8844cae911e94b8309eac97 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:07.374025 13062 raft_consensus.cc:374] T 911e4b8bd8844cae911e94b8309eac97 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:07.374053 13062 raft_consensus.cc:729] T 911e4b8bd8844cae911e94b8309eac97 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:07.374202 13062 consensus_queue.cc:260] T 911e4b8bd8844cae911e94b8309eac97 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:07.374049 13060 raft_consensus.cc:348] T 911e4b8bd8844cae911e94b8309eac97 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:07.374375 13060 raft_consensus.cc:374] T 911e4b8bd8844cae911e94b8309eac97 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:07.374480 13060 raft_consensus.cc:729] T 911e4b8bd8844cae911e94b8309eac97 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:07.374675 13060 consensus_queue.cc:260] T 911e4b8bd8844cae911e94b8309eac97 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:07.375305 13062 ts_tablet_manager.cc:1398] T 911e4b8bd8844cae911e94b8309eac97 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:07.375690 13060 ts_tablet_manager.cc:1398] T 911e4b8bd8844cae911e94b8309eac97 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:07.381369 13059 tablet_bootstrap.cc:492] T 911e4b8bd8844cae911e94b8309eac97 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:07.382316 13059 ts_tablet_manager.cc:1367] T 911e4b8bd8844cae911e94b8309eac97 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 02:00:07.383175 13059 raft_consensus.cc:348] T 911e4b8bd8844cae911e94b8309eac97 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:07.383368 13059 raft_consensus.cc:374] T 911e4b8bd8844cae911e94b8309eac97 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:07.383463 13059 raft_consensus.cc:729] T 911e4b8bd8844cae911e94b8309eac97 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:07.383654 13059 consensus_queue.cc:260] T 911e4b8bd8844cae911e94b8309eac97 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:07.384078 13059 ts_tablet_manager.cc:1398] T 911e4b8bd8844cae911e94b8309eac97 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:07.424973 12907 raft_consensus.cc:482] T 911e4b8bd8844cae911e94b8309eac97 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:07.425163 12907 raft_consensus.cc:504] T 911e4b8bd8844cae911e94b8309eac97 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:07.425638 12907 leader_election.cc:288] T 911e4b8bd8844cae911e94b8309eac97 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:07.426088 9047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "911e4b8bd8844cae911e94b8309eac97" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:07.426128 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "911e4b8bd8844cae911e94b8309eac97" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:07.426234 9047 raft_consensus.cc:2454] T 911e4b8bd8844cae911e94b8309eac97 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:07.426270 8916 raft_consensus.cc:2454] T 911e4b8bd8844cae911e94b8309eac97 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:07.426581 9115 leader_election.cc:302] T 911e4b8bd8844cae911e94b8309eac97 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:07.426761 12907 raft_consensus.cc:2781] T 911e4b8bd8844cae911e94b8309eac97 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:07.426818 12907 raft_consensus.cc:482] T 911e4b8bd8844cae911e94b8309eac97 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:07.426863 12907 raft_consensus.cc:3037] T 911e4b8bd8844cae911e94b8309eac97 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:07.427781 12907 raft_consensus.cc:504] T 911e4b8bd8844cae911e94b8309eac97 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:07.428123 12907 leader_election.cc:288] T 911e4b8bd8844cae911e94b8309eac97 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Requested vote from peers 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:07.428498 9047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "911e4b8bd8844cae911e94b8309eac97" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:07.428526 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "911e4b8bd8844cae911e94b8309eac97" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:07.428629 9047 raft_consensus.cc:3037] T 911e4b8bd8844cae911e94b8309eac97 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:07.428671 8916 raft_consensus.cc:3037] T 911e4b8bd8844cae911e94b8309eac97 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:07.429445 8916 raft_consensus.cc:2454] T 911e4b8bd8844cae911e94b8309eac97 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:07.429481 9047 raft_consensus.cc:2454] T 911e4b8bd8844cae911e94b8309eac97 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:07.429733 9115 leader_election.cc:302] T 911e4b8bd8844cae911e94b8309eac97 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:07.429880 12907 raft_consensus.cc:2781] T 911e4b8bd8844cae911e94b8309eac97 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:07.429967 12907 raft_consensus.cc:686] T 911e4b8bd8844cae911e94b8309eac97 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Becoming Leader. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Running, Role: LEADER
I20240503 02:00:07.430094 12907 consensus_queue.cc:237] T 911e4b8bd8844cae911e94b8309eac97 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:07.432214 6307 catalog_manager.cc:5484] T 911e4b8bd8844cae911e94b8309eac97 P 2a01ace80baf4057b6bf1995cf88468b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135). New cstate: current_term: 1 leader_uuid: "2a01ace80baf4057b6bf1995cf88468b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:07.433672 6458 consensus_queue.cc:237] T 0ee05e70ccbb496ca3cff08dc38112e8 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } }
I20240503 02:00:07.435169 6719 raft_consensus.cc:1261] T 0ee05e70ccbb496ca3cff08dc38112e8 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 2 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:00:07.435381 9047 raft_consensus.cc:1261] T 0ee05e70ccbb496ca3cff08dc38112e8 P 99ead02915954b4fab23b033ad6d546b [term 2 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:00:07.435787 12959 consensus_queue.cc:1035] T 0ee05e70ccbb496ca3cff08dc38112e8 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:07.435940 12190 consensus_queue.cc:1035] T 0ee05e70ccbb496ca3cff08dc38112e8 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:07.437737 11851 raft_consensus.cc:2932] T 0ee05e70ccbb496ca3cff08dc38112e8 P dec3e4a04d604f7eaaa3f077d081218a [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } }
I20240503 02:00:07.438221 9047 raft_consensus.cc:2932] T 0ee05e70ccbb496ca3cff08dc38112e8 P 99ead02915954b4fab23b033ad6d546b [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } }
I20240503 02:00:07.439867 6719 raft_consensus.cc:2932] T 0ee05e70ccbb496ca3cff08dc38112e8 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } }
I20240503 02:00:07.440330 6300 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0ee05e70ccbb496ca3cff08dc38112e8 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:07.441221 6307 catalog_manager.cc:5484] T 0ee05e70ccbb496ca3cff08dc38112e8 P 99ead02915954b4fab23b033ad6d546b reported cstate change: config changed from index 3 to 5, VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) evicted. New cstate: current_term: 2 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } }
I20240503 02:00:07.443930 9158 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ee05e70ccbb496ca3cff08dc38112e8 with delete_type TABLET_DATA_TOMBSTONED (TS 2a01ace80baf4057b6bf1995cf88468b not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:51940
I20240503 02:00:07.444190 13024 tablet_replica.cc:330] T 0ee05e70ccbb496ca3cff08dc38112e8 P 2a01ace80baf4057b6bf1995cf88468b: stopping tablet replica
I20240503 02:00:07.444315 13024 raft_consensus.cc:2229] T 0ee05e70ccbb496ca3cff08dc38112e8 P 2a01ace80baf4057b6bf1995cf88468b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:07.444453 13024 raft_consensus.cc:2258] T 0ee05e70ccbb496ca3cff08dc38112e8 P 2a01ace80baf4057b6bf1995cf88468b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:07.445019 13024 ts_tablet_manager.cc:1875] T 0ee05e70ccbb496ca3cff08dc38112e8 P 2a01ace80baf4057b6bf1995cf88468b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:07.447654 13024 ts_tablet_manager.cc:1888] T 0ee05e70ccbb496ca3cff08dc38112e8 P 2a01ace80baf4057b6bf1995cf88468b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240503 02:00:07.447733 13024 log.cc:1196] T 0ee05e70ccbb496ca3cff08dc38112e8 P 2a01ace80baf4057b6bf1995cf88468b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/0ee05e70ccbb496ca3cff08dc38112e8
I20240503 02:00:07.448263 6300 catalog_manager.cc:4830] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079): tablet 0ee05e70ccbb496ca3cff08dc38112e8 (table rebalancer_extra_table_8 [id=404345ac1896484ca34a8c78c6770131]) successfully deleted
I20240503 02:00:07.477831 6312 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_extra_table_20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:07.484884 6436 tablet_service.cc:1467] Processing CreateTablet for tablet 7da8f614a70e4177b70e7c2f31d22a10 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=dd81a75146044b9d9fe1ec1f86a7aebd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:07.485147 8880 tablet_service.cc:1467] Processing CreateTablet for tablet 7da8f614a70e4177b70e7c2f31d22a10 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=dd81a75146044b9d9fe1ec1f86a7aebd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:07.485244 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7da8f614a70e4177b70e7c2f31d22a10. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:07.485388 8880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7da8f614a70e4177b70e7c2f31d22a10. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:07.485548 9286 tablet_service.cc:1467] Processing CreateTablet for tablet 7da8f614a70e4177b70e7c2f31d22a10 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=dd81a75146044b9d9fe1ec1f86a7aebd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:07.485843 9286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7da8f614a70e4177b70e7c2f31d22a10. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:07.487783 6700 tablet_service.cc:1467] Processing CreateTablet for tablet 5f51fe5f799f4e479b62f5917a2c2606 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=dd81a75146044b9d9fe1ec1f86a7aebd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:07.488085 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f51fe5f799f4e479b62f5917a2c2606. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:07.488176 13062 tablet_bootstrap.cc:492] T 7da8f614a70e4177b70e7c2f31d22a10 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:07.488811 8887 tablet_service.cc:1467] Processing CreateTablet for tablet b1740b877c1e4372a0337d99f5bcab0e (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=dd81a75146044b9d9fe1ec1f86a7aebd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:07.489050 8887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1740b877c1e4372a0337d99f5bcab0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:07.489205 13062 tablet_bootstrap.cc:654] T 7da8f614a70e4177b70e7c2f31d22a10 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:07.489904 13067 tablet_bootstrap.cc:492] T 7da8f614a70e4177b70e7c2f31d22a10 P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap starting.
I20240503 02:00:07.490481 8897 tablet_service.cc:1467] Processing CreateTablet for tablet 5f51fe5f799f4e479b62f5917a2c2606 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=dd81a75146044b9d9fe1ec1f86a7aebd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:07.490712 8897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f51fe5f799f4e479b62f5917a2c2606. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:07.490825 13067 tablet_bootstrap.cc:654] T 7da8f614a70e4177b70e7c2f31d22a10 P e1a9251ce1cf4361b8b0ede9fbfc4023: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:07.491295 13062 tablet_bootstrap.cc:492] T 7da8f614a70e4177b70e7c2f31d22a10 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:07.491376 13062 ts_tablet_manager.cc:1367] T 7da8f614a70e4177b70e7c2f31d22a10 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:07.491760 13062 raft_consensus.cc:348] T 7da8f614a70e4177b70e7c2f31d22a10 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:07.491863 13062 raft_consensus.cc:374] T 7da8f614a70e4177b70e7c2f31d22a10 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:07.491925 13062 raft_consensus.cc:729] T 7da8f614a70e4177b70e7c2f31d22a10 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:07.491884 9286 tablet_service.cc:1467] Processing CreateTablet for tablet 5f51fe5f799f4e479b62f5917a2c2606 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=dd81a75146044b9d9fe1ec1f86a7aebd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:07.492139 9286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f51fe5f799f4e479b62f5917a2c2606. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:07.492128 13066 tablet_bootstrap.cc:492] T 7da8f614a70e4177b70e7c2f31d22a10 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:07.492101 13062 consensus_queue.cc:260] T 7da8f614a70e4177b70e7c2f31d22a10 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:07.492496 13062 ts_tablet_manager.cc:1398] T 7da8f614a70e4177b70e7c2f31d22a10 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:07.492648 13062 tablet_bootstrap.cc:492] T b1740b877c1e4372a0337d99f5bcab0e P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:07.492799 9285 tablet_service.cc:1467] Processing CreateTablet for tablet b1740b877c1e4372a0337d99f5bcab0e (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=dd81a75146044b9d9fe1ec1f86a7aebd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:07.493013 9285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1740b877c1e4372a0337d99f5bcab0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:07.493103 13066 tablet_bootstrap.cc:654] T 7da8f614a70e4177b70e7c2f31d22a10 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:07.493516 13062 tablet_bootstrap.cc:654] T b1740b877c1e4372a0337d99f5bcab0e P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:07.494571 13062 tablet_bootstrap.cc:492] T b1740b877c1e4372a0337d99f5bcab0e P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:07.494652 13062 ts_tablet_manager.cc:1367] T b1740b877c1e4372a0337d99f5bcab0e P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:07.495086 13062 raft_consensus.cc:348] T b1740b877c1e4372a0337d99f5bcab0e P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:07.495188 13062 raft_consensus.cc:374] T b1740b877c1e4372a0337d99f5bcab0e P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:07.495188 9158 tablet_service.cc:1467] Processing CreateTablet for tablet b1740b877c1e4372a0337d99f5bcab0e (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=dd81a75146044b9d9fe1ec1f86a7aebd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:07.495255 13062 raft_consensus.cc:729] T b1740b877c1e4372a0337d99f5bcab0e P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:07.495362 13062 consensus_queue.cc:260] T b1740b877c1e4372a0337d99f5bcab0e P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:07.495452 9158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1740b877c1e4372a0337d99f5bcab0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:07.495724 13062 ts_tablet_manager.cc:1398] T b1740b877c1e4372a0337d99f5bcab0e P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:07.496078 13067 tablet_bootstrap.cc:492] T 7da8f614a70e4177b70e7c2f31d22a10 P e1a9251ce1cf4361b8b0ede9fbfc4023: No bootstrap required, opened a new log
I20240503 02:00:07.496162 13067 ts_tablet_manager.cc:1367] T 7da8f614a70e4177b70e7c2f31d22a10 P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:00:07.496608 13067 raft_consensus.cc:348] T 7da8f614a70e4177b70e7c2f31d22a10 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:07.496752 13067 raft_consensus.cc:374] T 7da8f614a70e4177b70e7c2f31d22a10 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:07.496788 13067 raft_consensus.cc:729] T 7da8f614a70e4177b70e7c2f31d22a10 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1a9251ce1cf4361b8b0ede9fbfc4023, State: Initialized, Role: FOLLOWER
I20240503 02:00:07.496894 13067 consensus_queue.cc:260] T 7da8f614a70e4177b70e7c2f31d22a10 P e1a9251ce1cf4361b8b0ede9fbfc4023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:07.497150 13067 ts_tablet_manager.cc:1398] T 7da8f614a70e4177b70e7c2f31d22a10 P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:07.497282 13067 tablet_bootstrap.cc:492] T 5f51fe5f799f4e479b62f5917a2c2606 P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap starting.
I20240503 02:00:07.497526 13059 tablet_bootstrap.cc:492] T b1740b877c1e4372a0337d99f5bcab0e P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:07.498016 13062 tablet_bootstrap.cc:492] T 5f51fe5f799f4e479b62f5917a2c2606 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:07.498574 13067 tablet_bootstrap.cc:654] T 5f51fe5f799f4e479b62f5917a2c2606 P e1a9251ce1cf4361b8b0ede9fbfc4023: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:07.498760 13069 tablet_bootstrap.cc:492] T 5f51fe5f799f4e479b62f5917a2c2606 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:07.499102 13062 tablet_bootstrap.cc:654] T 5f51fe5f799f4e479b62f5917a2c2606 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:07.499711 13069 tablet_bootstrap.cc:654] T 5f51fe5f799f4e479b62f5917a2c2606 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:07.501472 13066 tablet_bootstrap.cc:492] T 7da8f614a70e4177b70e7c2f31d22a10 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:07.501593 13066 ts_tablet_manager.cc:1367] T 7da8f614a70e4177b70e7c2f31d22a10 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 02:00:07.502100 13066 raft_consensus.cc:348] T 7da8f614a70e4177b70e7c2f31d22a10 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:07.502208 13066 raft_consensus.cc:374] T 7da8f614a70e4177b70e7c2f31d22a10 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:07.502238 13066 raft_consensus.cc:729] T 7da8f614a70e4177b70e7c2f31d22a10 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:07.502360 13066 consensus_queue.cc:260] T 7da8f614a70e4177b70e7c2f31d22a10 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:07.502596 13069 tablet_bootstrap.cc:492] T 5f51fe5f799f4e479b62f5917a2c2606 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:07.502609 13066 ts_tablet_manager.cc:1398] T 7da8f614a70e4177b70e7c2f31d22a10 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:07.502710 13069 ts_tablet_manager.cc:1367] T 5f51fe5f799f4e479b62f5917a2c2606 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:07.503199 13069 raft_consensus.cc:348] T 5f51fe5f799f4e479b62f5917a2c2606 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:07.503315 13069 raft_consensus.cc:374] T 5f51fe5f799f4e479b62f5917a2c2606 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:07.503346 13069 raft_consensus.cc:729] T 5f51fe5f799f4e479b62f5917a2c2606 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:07.503459 13069 consensus_queue.cc:260] T 5f51fe5f799f4e479b62f5917a2c2606 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:07.503831 13069 ts_tablet_manager.cc:1398] T 5f51fe5f799f4e479b62f5917a2c2606 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:07.506178 13059 tablet_bootstrap.cc:654] T b1740b877c1e4372a0337d99f5bcab0e P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:07.507310 13059 tablet_bootstrap.cc:492] T b1740b877c1e4372a0337d99f5bcab0e P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:07.507408 13059 ts_tablet_manager.cc:1367] T b1740b877c1e4372a0337d99f5bcab0e P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 02:00:07.507846 13059 raft_consensus.cc:348] T b1740b877c1e4372a0337d99f5bcab0e P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:07.507957 13059 raft_consensus.cc:374] T b1740b877c1e4372a0337d99f5bcab0e P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:07.507990 13059 raft_consensus.cc:729] T b1740b877c1e4372a0337d99f5bcab0e P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:07.508106 13059 consensus_queue.cc:260] T b1740b877c1e4372a0337d99f5bcab0e P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:07.508414 13059 ts_tablet_manager.cc:1398] T b1740b877c1e4372a0337d99f5bcab0e P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:07.509416 13018 raft_consensus.cc:482] T 7da8f614a70e4177b70e7c2f31d22a10 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:07.509508 13018 raft_consensus.cc:504] T 7da8f614a70e4177b70e7c2f31d22a10 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:07.509671 13067 tablet_bootstrap.cc:492] T 5f51fe5f799f4e479b62f5917a2c2606 P e1a9251ce1cf4361b8b0ede9fbfc4023: No bootstrap required, opened a new log
I20240503 02:00:07.509750 13067 ts_tablet_manager.cc:1367] T 5f51fe5f799f4e479b62f5917a2c2606 P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 02:00:07.509917 13018 leader_election.cc:288] T 7da8f614a70e4177b70e7c2f31d22a10 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 02:00:07.510170 13067 raft_consensus.cc:348] T 5f51fe5f799f4e479b62f5917a2c2606 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:07.510280 13067 raft_consensus.cc:374] T 5f51fe5f799f4e479b62f5917a2c2606 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:07.510313 13067 raft_consensus.cc:729] T 5f51fe5f799f4e479b62f5917a2c2606 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1a9251ce1cf4361b8b0ede9fbfc4023, State: Initialized, Role: FOLLOWER
I20240503 02:00:07.510392 9309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7da8f614a70e4177b70e7c2f31d22a10" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" is_pre_election: true
I20240503 02:00:07.510470 13062 tablet_bootstrap.cc:492] T 5f51fe5f799f4e479b62f5917a2c2606 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:07.510452 13067 consensus_queue.cc:260] T 5f51fe5f799f4e479b62f5917a2c2606 P e1a9251ce1cf4361b8b0ede9fbfc4023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:07.510536 13062 ts_tablet_manager.cc:1367] T 5f51fe5f799f4e479b62f5917a2c2606 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 02:00:07.510485 9309 raft_consensus.cc:2454] T 7da8f614a70e4177b70e7c2f31d22a10 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 0.
I20240503 02:00:07.510744 13067 ts_tablet_manager.cc:1398] T 5f51fe5f799f4e479b62f5917a2c2606 P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:07.510820 8851 leader_election.cc:302] T 7da8f614a70e4177b70e7c2f31d22a10 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9942306855404cfd891adf227dd65eb2, e1a9251ce1cf4361b8b0ede9fbfc4023; no voters:
I20240503 02:00:07.510916 13067 tablet_bootstrap.cc:492] T b1740b877c1e4372a0337d99f5bcab0e P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap starting.
I20240503 02:00:07.510958 13062 raft_consensus.cc:348] T 5f51fe5f799f4e479b62f5917a2c2606 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:07.511056 13062 raft_consensus.cc:374] T 5f51fe5f799f4e479b62f5917a2c2606 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:07.511082 13062 raft_consensus.cc:729] T 5f51fe5f799f4e479b62f5917a2c2606 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:07.511235 13062 consensus_queue.cc:260] T 5f51fe5f799f4e479b62f5917a2c2606 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:07.511217 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7da8f614a70e4177b70e7c2f31d22a10" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 02:00:07.511350 6458 raft_consensus.cc:2454] T 7da8f614a70e4177b70e7c2f31d22a10 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 0.
I20240503 02:00:07.511384 11872 raft_consensus.cc:2781] T 7da8f614a70e4177b70e7c2f31d22a10 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:07.511427 11872 raft_consensus.cc:482] T 7da8f614a70e4177b70e7c2f31d22a10 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:07.511462 11872 raft_consensus.cc:3037] T 7da8f614a70e4177b70e7c2f31d22a10 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:07.511507 13062 ts_tablet_manager.cc:1398] T 5f51fe5f799f4e479b62f5917a2c2606 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:07.511868 13067 tablet_bootstrap.cc:654] T b1740b877c1e4372a0337d99f5bcab0e P e1a9251ce1cf4361b8b0ede9fbfc4023: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:07.512678 11872 raft_consensus.cc:504] T 7da8f614a70e4177b70e7c2f31d22a10 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:07.513037 11872 leader_election.cc:288] T 7da8f614a70e4177b70e7c2f31d22a10 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 election: Requested vote from peers e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 02:00:07.513100 13067 tablet_bootstrap.cc:492] T b1740b877c1e4372a0337d99f5bcab0e P e1a9251ce1cf4361b8b0ede9fbfc4023: No bootstrap required, opened a new log
I20240503 02:00:07.513182 13067 ts_tablet_manager.cc:1367] T b1740b877c1e4372a0337d99f5bcab0e P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:07.513399 9309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7da8f614a70e4177b70e7c2f31d22a10" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023"
I20240503 02:00:07.513511 9309 raft_consensus.cc:3037] T 7da8f614a70e4177b70e7c2f31d22a10 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:07.513653 13067 raft_consensus.cc:348] T b1740b877c1e4372a0337d99f5bcab0e P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:07.513806 13067 raft_consensus.cc:374] T b1740b877c1e4372a0337d99f5bcab0e P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:07.513851 13067 raft_consensus.cc:729] T b1740b877c1e4372a0337d99f5bcab0e P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1a9251ce1cf4361b8b0ede9fbfc4023, State: Initialized, Role: FOLLOWER
I20240503 02:00:07.513963 13067 consensus_queue.cc:260] T b1740b877c1e4372a0337d99f5bcab0e P e1a9251ce1cf4361b8b0ede9fbfc4023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:07.514290 13067 ts_tablet_manager.cc:1398] T b1740b877c1e4372a0337d99f5bcab0e P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:07.514510 9309 raft_consensus.cc:2454] T 7da8f614a70e4177b70e7c2f31d22a10 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 1.
I20240503 02:00:07.514807 8851 leader_election.cc:302] T 7da8f614a70e4177b70e7c2f31d22a10 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9942306855404cfd891adf227dd65eb2, e1a9251ce1cf4361b8b0ede9fbfc4023; no voters:
I20240503 02:00:07.514961 11872 raft_consensus.cc:2781] T 7da8f614a70e4177b70e7c2f31d22a10 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:07.515048 11872 raft_consensus.cc:686] T 7da8f614a70e4177b70e7c2f31d22a10 P 9942306855404cfd891adf227dd65eb2 [term 1 LEADER]: Becoming Leader. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Running, Role: LEADER
I20240503 02:00:07.515192 11872 consensus_queue.cc:237] T 7da8f614a70e4177b70e7c2f31d22a10 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:07.515241 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7da8f614a70e4177b70e7c2f31d22a10" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 02:00:07.515352 6458 raft_consensus.cc:3037] T 7da8f614a70e4177b70e7c2f31d22a10 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:07.516296 6458 raft_consensus.cc:2454] T 7da8f614a70e4177b70e7c2f31d22a10 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 1.
I20240503 02:00:07.517833 6307 catalog_manager.cc:5484] T 7da8f614a70e4177b70e7c2f31d22a10 P 9942306855404cfd891adf227dd65eb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9942306855404cfd891adf227dd65eb2 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "9942306855404cfd891adf227dd65eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:07.541674 6307 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:07.546020 9047 raft_consensus.cc:1261] T 911e4b8bd8844cae911e94b8309eac97 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer 2a01ace80baf4057b6bf1995cf88468b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:07.546042 8916 raft_consensus.cc:1261] T 911e4b8bd8844cae911e94b8309eac97 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Refusing update from remote peer 2a01ace80baf4057b6bf1995cf88468b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:07.546496 10679 consensus_queue.cc:1035] T 911e4b8bd8844cae911e94b8309eac97 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:07.546794 10679 consensus_queue.cc:1035] T 911e4b8bd8844cae911e94b8309eac97 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:07.549782 6307 catalog_manager.cc:5576] TS 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157) does not have the latest schema for tablet 911e4b8bd8844cae911e94b8309eac97 (table rebalancer_dynamic_table [id=c47890e2f27546a28895df8613566824]). Expected version 1 got 0
I20240503 02:00:07.550043 12912 tablet.cc:1722] T 911e4b8bd8844cae911e94b8309eac97 P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:07.551815 6307 catalog_manager.cc:5576] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211) does not have the latest schema for tablet 911e4b8bd8844cae911e94b8309eac97 (table rebalancer_dynamic_table [id=c47890e2f27546a28895df8613566824]). Expected version 1 got 0
I20240503 02:00:07.553493 6312 catalog_manager.cc:5576] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079) does not have the latest schema for tablet 911e4b8bd8844cae911e94b8309eac97 (table rebalancer_dynamic_table [id=c47890e2f27546a28895df8613566824]). Expected version 1 got 0
I20240503 02:00:07.555483 6312 catalog_manager.cc:5576] TS 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157) does not have the latest schema for tablet 911e4b8bd8844cae911e94b8309eac97 (table rebalancer_dynamic_table [id=c47890e2f27546a28895df8613566824]). Expected version 1 got 0
I20240503 02:00:07.557616 12912 tablet.cc:1620] T 911e4b8bd8844cae911e94b8309eac97 P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
I20240503 02:00:07.560086 6314 catalog_manager.cc:5576] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211) does not have the latest schema for tablet 911e4b8bd8844cae911e94b8309eac97 (table rebalancer_dynamic_table [id=c47890e2f27546a28895df8613566824]). Expected version 1 got 0
I20240503 02:00:07.560596 13097 tablet.cc:1722] T 911e4b8bd8844cae911e94b8309eac97 P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:07.560873 6315 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=c47890e2f27546a28895df8613566824] alter complete (version 1)
I20240503 02:00:07.561035 13097 tablet.cc:1620] T 911e4b8bd8844cae911e94b8309eac97 P 99ead02915954b4fab23b033ad6d546b: MemRowSet was empty: no flush needed.
I20240503 02:00:07.565346 13099 tablet.cc:1722] T 911e4b8bd8844cae911e94b8309eac97 P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:07.565716 13099 tablet.cc:1620] T 911e4b8bd8844cae911e94b8309eac97 P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
I20240503 02:00:07.597786 13018 raft_consensus.cc:482] T 5f51fe5f799f4e479b62f5917a2c2606 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:07.597927 13018 raft_consensus.cc:504] T 5f51fe5f799f4e479b62f5917a2c2606 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:07.598328 13018 leader_election.cc:288] T 5f51fe5f799f4e479b62f5917a2c2606 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311)
I20240503 02:00:07.598727 9309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f51fe5f799f4e479b62f5917a2c2606" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" is_pre_election: true
I20240503 02:00:07.598735 6719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f51fe5f799f4e479b62f5917a2c2606" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 02:00:07.598868 6719 raft_consensus.cc:2454] T 5f51fe5f799f4e479b62f5917a2c2606 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 0.
I20240503 02:00:07.598870 9309 raft_consensus.cc:2454] T 5f51fe5f799f4e479b62f5917a2c2606 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 0.
I20240503 02:00:07.599187 8853 leader_election.cc:302] T 5f51fe5f799f4e479b62f5917a2c2606 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9942306855404cfd891adf227dd65eb2, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:07.599375 13018 raft_consensus.cc:2781] T 5f51fe5f799f4e479b62f5917a2c2606 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:07.599435 13018 raft_consensus.cc:482] T 5f51fe5f799f4e479b62f5917a2c2606 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:07.599470 13018 raft_consensus.cc:3037] T 5f51fe5f799f4e479b62f5917a2c2606 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:07.600554 13018 raft_consensus.cc:504] T 5f51fe5f799f4e479b62f5917a2c2606 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:07.600906 13018 leader_election.cc:288] T 5f51fe5f799f4e479b62f5917a2c2606 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 election: Requested vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311)
I20240503 02:00:07.601239 6719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f51fe5f799f4e479b62f5917a2c2606" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 02:00:07.601307 9309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f51fe5f799f4e479b62f5917a2c2606" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023"
I20240503 02:00:07.601372 6719 raft_consensus.cc:3037] T 5f51fe5f799f4e479b62f5917a2c2606 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:07.601418 9309 raft_consensus.cc:3037] T 5f51fe5f799f4e479b62f5917a2c2606 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:07.602509 9309 raft_consensus.cc:2454] T 5f51fe5f799f4e479b62f5917a2c2606 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 1.
I20240503 02:00:07.602512 6719 raft_consensus.cc:2454] T 5f51fe5f799f4e479b62f5917a2c2606 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 1.
I20240503 02:00:07.602967 8851 leader_election.cc:302] T 5f51fe5f799f4e479b62f5917a2c2606 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9942306855404cfd891adf227dd65eb2, e1a9251ce1cf4361b8b0ede9fbfc4023; no voters:
I20240503 02:00:07.603164 13018 raft_consensus.cc:2781] T 5f51fe5f799f4e479b62f5917a2c2606 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:07.603260 13018 raft_consensus.cc:686] T 5f51fe5f799f4e479b62f5917a2c2606 P 9942306855404cfd891adf227dd65eb2 [term 1 LEADER]: Becoming Leader. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Running, Role: LEADER
I20240503 02:00:07.603421 13018 consensus_queue.cc:237] T 5f51fe5f799f4e479b62f5917a2c2606 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:07.605567 6314 catalog_manager.cc:5484] T 5f51fe5f799f4e479b62f5917a2c2606 P 9942306855404cfd891adf227dd65eb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9942306855404cfd891adf227dd65eb2 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "9942306855404cfd891adf227dd65eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:07.664572 6314 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:07.670047 12912 tablet.cc:1722] T 911e4b8bd8844cae911e94b8309eac97 P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:07.670403 12912 tablet.cc:1620] T 911e4b8bd8844cae911e94b8309eac97 P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
I20240503 02:00:07.671249 13097 tablet.cc:1722] T 911e4b8bd8844cae911e94b8309eac97 P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:07.671658 13097 tablet.cc:1620] T 911e4b8bd8844cae911e94b8309eac97 P 99ead02915954b4fab23b033ad6d546b: MemRowSet was empty: no flush needed.
I20240503 02:00:07.671700 13099 tablet.cc:1722] T 911e4b8bd8844cae911e94b8309eac97 P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:07.675235 13099 tablet.cc:1620] T 911e4b8bd8844cae911e94b8309eac97 P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
I20240503 02:00:07.675714 6315 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=c47890e2f27546a28895df8613566824] alter complete (version 2)
I20240503 02:00:07.708285 13018 raft_consensus.cc:1050] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2: attempting to promote NON_VOTER dec3e4a04d604f7eaaa3f077d081218a to VOTER
I20240503 02:00:07.708786 13018 consensus_queue.cc:237] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } }
I20240503 02:00:07.710166 9309 raft_consensus.cc:1261] T 1c721b33983b4c47afd8faa8f155ca84 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:07.710166 6850 raft_consensus.cc:1261] T 1c721b33983b4c47afd8faa8f155ca84 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:07.710259 6456 raft_consensus.cc:1261] T 1c721b33983b4c47afd8faa8f155ca84 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEARNER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:07.710657 13018 consensus_queue.cc:1035] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:07.710878 13102 consensus_queue.cc:1035] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:07.711062 11872 consensus_queue.cc:1035] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:07.713083 13018 raft_consensus.cc:2932] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } }
I20240503 02:00:07.714751 6850 raft_consensus.cc:2932] T 1c721b33983b4c47afd8faa8f155ca84 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } }
I20240503 02:00:07.714898 6456 raft_consensus.cc:2932] T 1c721b33983b4c47afd8faa8f155ca84 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } }
I20240503 02:00:07.715976 6314 catalog_manager.cc:5484] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2 reported cstate change: config changed from index 2 to 3, dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9942306855404cfd891adf227dd65eb2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:07.716534 9309 raft_consensus.cc:2932] T 1c721b33983b4c47afd8faa8f155ca84 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } }
I20240503 02:00:07.721390 8916 consensus_queue.cc:237] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } }
I20240503 02:00:07.722671 6456 raft_consensus.cc:1261] T 1c721b33983b4c47afd8faa8f155ca84 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:07.723016 12907 raft_consensus.cc:482] T b1740b877c1e4372a0337d99f5bcab0e P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:07.723114 12907 raft_consensus.cc:504] T b1740b877c1e4372a0337d99f5bcab0e P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:07.723098 13102 consensus_queue.cc:1035] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:07.723496 12907 leader_election.cc:288] T b1740b877c1e4372a0337d99f5bcab0e P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:07.723874 9309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1740b877c1e4372a0337d99f5bcab0e" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" is_pre_election: true
I20240503 02:00:07.724006 9309 raft_consensus.cc:2454] T b1740b877c1e4372a0337d99f5bcab0e P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:07.724234 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1740b877c1e4372a0337d99f5bcab0e" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:07.724292 9113 leader_election.cc:302] T b1740b877c1e4372a0337d99f5bcab0e P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, e1a9251ce1cf4361b8b0ede9fbfc4023; no voters:
I20240503 02:00:07.724354 8916 raft_consensus.cc:2454] T b1740b877c1e4372a0337d99f5bcab0e P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:07.724483 12907 raft_consensus.cc:2781] T b1740b877c1e4372a0337d99f5bcab0e P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:07.724524 12907 raft_consensus.cc:482] T b1740b877c1e4372a0337d99f5bcab0e P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:07.724563 12907 raft_consensus.cc:3037] T b1740b877c1e4372a0337d99f5bcab0e P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:07.725451 6850 raft_consensus.cc:1261] T 1c721b33983b4c47afd8faa8f155ca84 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:07.725667 12907 raft_consensus.cc:504] T b1740b877c1e4372a0337d99f5bcab0e P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:07.725833 13102 consensus_queue.cc:1035] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:07.726018 12907 leader_election.cc:288] T b1740b877c1e4372a0337d99f5bcab0e P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Requested vote from peers e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:07.726487 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1740b877c1e4372a0337d99f5bcab0e" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:07.726595 8916 raft_consensus.cc:3037] T b1740b877c1e4372a0337d99f5bcab0e P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:07.727167 13017 raft_consensus.cc:2932] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } }
I20240503 02:00:07.727600 8916 raft_consensus.cc:2454] T b1740b877c1e4372a0337d99f5bcab0e P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:07.727605 6456 raft_consensus.cc:2932] T 1c721b33983b4c47afd8faa8f155ca84 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } }
I20240503 02:00:07.728734 9309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1740b877c1e4372a0337d99f5bcab0e" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023"
I20240503 02:00:07.728855 9309 raft_consensus.cc:3037] T b1740b877c1e4372a0337d99f5bcab0e P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:07.729231 9115 leader_election.cc:302] T b1740b877c1e4372a0337d99f5bcab0e P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:07.729614 12907 raft_consensus.cc:2781] T b1740b877c1e4372a0337d99f5bcab0e P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:07.729694 12907 raft_consensus.cc:686] T b1740b877c1e4372a0337d99f5bcab0e P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Becoming Leader. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Running, Role: LEADER
I20240503 02:00:07.729826 12907 consensus_queue.cc:237] T b1740b877c1e4372a0337d99f5bcab0e P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:07.730026 9309 raft_consensus.cc:2454] T b1740b877c1e4372a0337d99f5bcab0e P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:07.730127 6850 raft_consensus.cc:2932] T 1c721b33983b4c47afd8faa8f155ca84 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } }
I20240503 02:00:07.730926 6303 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1c721b33983b4c47afd8faa8f155ca84 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:07.731868 6307 catalog_manager.cc:5484] T 1c721b33983b4c47afd8faa8f155ca84 P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: config changed from index 3 to 4, VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "9942306855404cfd891adf227dd65eb2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } }
I20240503 02:00:07.732035 6312 catalog_manager.cc:5484] T b1740b877c1e4372a0337d99f5bcab0e P 2a01ace80baf4057b6bf1995cf88468b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135). New cstate: current_term: 1 leader_uuid: "2a01ace80baf4057b6bf1995cf88468b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:07.734669 9285 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c721b33983b4c47afd8faa8f155ca84 with delete_type TABLET_DATA_TOMBSTONED (TS e1a9251ce1cf4361b8b0ede9fbfc4023 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36004
I20240503 02:00:07.734934 12937 tablet_replica.cc:330] T 1c721b33983b4c47afd8faa8f155ca84 P e1a9251ce1cf4361b8b0ede9fbfc4023: stopping tablet replica
I20240503 02:00:07.735090 12937 raft_consensus.cc:2229] T 1c721b33983b4c47afd8faa8f155ca84 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:07.735234 12937 raft_consensus.cc:2258] T 1c721b33983b4c47afd8faa8f155ca84 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:07.735780 12937 ts_tablet_manager.cc:1875] T 1c721b33983b4c47afd8faa8f155ca84 P e1a9251ce1cf4361b8b0ede9fbfc4023: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:07.737715 12937 ts_tablet_manager.cc:1888] T 1c721b33983b4c47afd8faa8f155ca84 P e1a9251ce1cf4361b8b0ede9fbfc4023: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:00:07.737802 12937 log.cc:1196] T 1c721b33983b4c47afd8faa8f155ca84 P e1a9251ce1cf4361b8b0ede9fbfc4023: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/1c721b33983b4c47afd8faa8f155ca84
I20240503 02:00:07.738338 6301 catalog_manager.cc:4830] TS e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): tablet 1c721b33983b4c47afd8faa8f155ca84 (table rebalancer_extra_table_10 [id=e43866993a7646be8fc624a6f2150d34]) successfully deleted
I20240503 02:00:07.778030 6312 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:07.786031 12912 tablet.cc:1722] T 911e4b8bd8844cae911e94b8309eac97 P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:07.786252 13097 tablet.cc:1722] T 911e4b8bd8844cae911e94b8309eac97 P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:07.791038 13099 tablet.cc:1722] T 911e4b8bd8844cae911e94b8309eac97 P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:07.791599 6315 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=c47890e2f27546a28895df8613566824] alter complete (version 3)
I20240503 02:00:07.892256 6567 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:00:07.893062 6831 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:00:07.895215 6313 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:07.895363 6313 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:07.897693 6313 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet 911e4b8bd8844cae911e94b8309eac97
I20240503 02:00:07.898259 9028 tablet_service.cc:1514] Processing DeleteTablet for tablet 911e4b8bd8844cae911e94b8309eac97 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:07 UTC) from {username='slave'} at 127.0.0.1:52584
I20240503 02:00:07.898301 9158 tablet_service.cc:1514] Processing DeleteTablet for tablet 911e4b8bd8844cae911e94b8309eac97 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:07 UTC) from {username='slave'} at 127.0.0.1:51940
I20240503 02:00:07.898605 13166 tablet_replica.cc:330] T 911e4b8bd8844cae911e94b8309eac97 P 99ead02915954b4fab23b033ad6d546b: stopping tablet replica
I20240503 02:00:07.898806 13166 raft_consensus.cc:2229] T 911e4b8bd8844cae911e94b8309eac97 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:07.898947 13166 raft_consensus.cc:2258] T 911e4b8bd8844cae911e94b8309eac97 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:07.899653 13024 tablet_replica.cc:330] T 911e4b8bd8844cae911e94b8309eac97 P 2a01ace80baf4057b6bf1995cf88468b: stopping tablet replica
I20240503 02:00:07.899792 13024 raft_consensus.cc:2229] T 911e4b8bd8844cae911e94b8309eac97 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:07.899893 13166 ts_tablet_manager.cc:1875] T 911e4b8bd8844cae911e94b8309eac97 P 99ead02915954b4fab23b033ad6d546b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:07.899991 13024 raft_consensus.cc:2258] T 911e4b8bd8844cae911e94b8309eac97 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:07.900537 8897 tablet_service.cc:1514] Processing DeleteTablet for tablet 911e4b8bd8844cae911e94b8309eac97 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:07 UTC) from {username='slave'} at 127.0.0.1:40296
I20240503 02:00:07.900926 13024 ts_tablet_manager.cc:1875] T 911e4b8bd8844cae911e94b8309eac97 P 2a01ace80baf4057b6bf1995cf88468b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:07.902999 13024 ts_tablet_manager.cc:1888] T 911e4b8bd8844cae911e94b8309eac97 P 2a01ace80baf4057b6bf1995cf88468b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:07.903095 13024 log.cc:1196] T 911e4b8bd8844cae911e94b8309eac97 P 2a01ace80baf4057b6bf1995cf88468b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/911e4b8bd8844cae911e94b8309eac97
I20240503 02:00:07.903359 13024 ts_tablet_manager.cc:1909] T 911e4b8bd8844cae911e94b8309eac97 P 2a01ace80baf4057b6bf1995cf88468b: Deleting consensus metadata
I20240503 02:00:07.903712 13167 tablet_replica.cc:330] T 911e4b8bd8844cae911e94b8309eac97 P 9942306855404cfd891adf227dd65eb2: stopping tablet replica
I20240503 02:00:07.903896 13167 raft_consensus.cc:2229] T 911e4b8bd8844cae911e94b8309eac97 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:07.904107 13167 raft_consensus.cc:2258] T 911e4b8bd8844cae911e94b8309eac97 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:07.904156 6300 catalog_manager.cc:4830] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079): tablet 911e4b8bd8844cae911e94b8309eac97 (table rebalancer_dynamic_table_new_name [id=c47890e2f27546a28895df8613566824]) successfully deleted
I20240503 02:00:07.904192 13166 ts_tablet_manager.cc:1888] T 911e4b8bd8844cae911e94b8309eac97 P 99ead02915954b4fab23b033ad6d546b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:07.904274 13166 log.cc:1196] T 911e4b8bd8844cae911e94b8309eac97 P 99ead02915954b4fab23b033ad6d546b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/911e4b8bd8844cae911e94b8309eac97
I20240503 02:00:07.904522 8897 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:00:07.904596 13166 ts_tablet_manager.cc:1909] T 911e4b8bd8844cae911e94b8309eac97 P 99ead02915954b4fab23b033ad6d546b: Deleting consensus metadata
I20240503 02:00:07.904930 13167 ts_tablet_manager.cc:1875] T 911e4b8bd8844cae911e94b8309eac97 P 9942306855404cfd891adf227dd65eb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:07.906244 9158 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:00:07.907305 13167 ts_tablet_manager.cc:1888] T 911e4b8bd8844cae911e94b8309eac97 P 9942306855404cfd891adf227dd65eb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:07.907374 13167 log.cc:1196] T 911e4b8bd8844cae911e94b8309eac97 P 9942306855404cfd891adf227dd65eb2: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/911e4b8bd8844cae911e94b8309eac97
I20240503 02:00:07.907451 6301 catalog_manager.cc:4830] TS 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157): tablet 911e4b8bd8844cae911e94b8309eac97 (table rebalancer_dynamic_table_new_name [id=c47890e2f27546a28895df8613566824]) successfully deleted
I20240503 02:00:07.907613 13167 ts_tablet_manager.cc:1909] T 911e4b8bd8844cae911e94b8309eac97 P 9942306855404cfd891adf227dd65eb2: Deleting consensus metadata
I20240503 02:00:07.908273 6303 catalog_manager.cc:4830] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211): tablet 911e4b8bd8844cae911e94b8309eac97 (table rebalancer_dynamic_table_new_name [id=c47890e2f27546a28895df8613566824]) successfully deleted
I20240503 02:00:07.910313 9028 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:00:07.912083 6436 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:00:07.922621 6700 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:00:07.924525 13102 consensus_queue.cc:1035] T 7da8f614a70e4177b70e7c2f31d22a10 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:07.933236 13102 consensus_queue.cc:1035] T 7da8f614a70e4177b70e7c2f31d22a10 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:07.939512 9285 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:07.998425 9048 consensus_queue.cc:237] T beeea71f02ae4c89a1af13d3584ef70a P 99ead02915954b4fab23b033ad6d546b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: true } }
I20240503 02:00:07.999856 9179 raft_consensus.cc:1261] T beeea71f02ae4c89a1af13d3584ef70a P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer 99ead02915954b4fab23b033ad6d546b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:07.999903 9309 raft_consensus.cc:1261] T beeea71f02ae4c89a1af13d3584ef70a P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Refusing update from remote peer 99ead02915954b4fab23b033ad6d546b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:00:08.000188 8983 consensus_peers.cc:485] T beeea71f02ae4c89a1af13d3584ef70a P 99ead02915954b4fab23b033ad6d546b -> Peer 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): Couldn't send request to peer 80fd18ef4d604c86b909d5c008a46b4f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: beeea71f02ae4c89a1af13d3584ef70a. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:08.000262 12922 consensus_queue.cc:1035] T beeea71f02ae4c89a1af13d3584ef70a P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:08.000516 11514 consensus_queue.cc:1035] T beeea71f02ae4c89a1af13d3584ef70a P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:08.002462 6313 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:00:08.002453 11514 raft_consensus.cc:2932] T beeea71f02ae4c89a1af13d3584ef70a P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: true } } }
I20240503 02:00:08.005640 9179 raft_consensus.cc:2932] T beeea71f02ae4c89a1af13d3584ef70a P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: true } } }
I20240503 02:00:08.005667 9309 raft_consensus.cc:2932] T beeea71f02ae4c89a1af13d3584ef70a P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: true } } }
I20240503 02:00:08.008772 6313 catalog_manager.cc:5484] T beeea71f02ae4c89a1af13d3584ef70a P e1a9251ce1cf4361b8b0ede9fbfc4023 reported cstate change: config changed from index -1 to 2, NON_VOTER 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "99ead02915954b4fab23b033ad6d546b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: true } } }
I20240503 02:00:08.010614 9158 tablet_service.cc:1467] Processing CreateTablet for tablet 3d3463f1ff2b41a396bb1b039ddf1946 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=954039feb1f843f7adbb2096b370accb]), partition=
I20240503 02:00:08.010820 9285 tablet_service.cc:1467] Processing CreateTablet for tablet 3d3463f1ff2b41a396bb1b039ddf1946 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=954039feb1f843f7adbb2096b370accb]), partition=
I20240503 02:00:08.010818 8897 tablet_service.cc:1467] Processing CreateTablet for tablet 3d3463f1ff2b41a396bb1b039ddf1946 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=954039feb1f843f7adbb2096b370accb]), partition=
I20240503 02:00:08.010897 9158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d3463f1ff2b41a396bb1b039ddf1946. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:08.011037 9285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d3463f1ff2b41a396bb1b039ddf1946. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:08.011070 8897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d3463f1ff2b41a396bb1b039ddf1946. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:08.013464 13181 tablet_bootstrap.cc:492] T 3d3463f1ff2b41a396bb1b039ddf1946 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:08.013622 13180 tablet_bootstrap.cc:492] T 3d3463f1ff2b41a396bb1b039ddf1946 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:08.013721 13067 tablet_bootstrap.cc:492] T 3d3463f1ff2b41a396bb1b039ddf1946 P e1a9251ce1cf4361b8b0ede9fbfc4023: Bootstrap starting.
I20240503 02:00:08.014509 13181 tablet_bootstrap.cc:654] T 3d3463f1ff2b41a396bb1b039ddf1946 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:08.014561 13067 tablet_bootstrap.cc:654] T 3d3463f1ff2b41a396bb1b039ddf1946 P e1a9251ce1cf4361b8b0ede9fbfc4023: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:08.014562 13180 tablet_bootstrap.cc:654] T 3d3463f1ff2b41a396bb1b039ddf1946 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:08.016311 13102 consensus_queue.cc:1035] T 5f51fe5f799f4e479b62f5917a2c2606 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:08.021356 13102 consensus_queue.cc:1035] T 5f51fe5f799f4e479b62f5917a2c2606 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:08.022287 13180 tablet_bootstrap.cc:492] T 3d3463f1ff2b41a396bb1b039ddf1946 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:08.022387 13180 ts_tablet_manager.cc:1367] T 3d3463f1ff2b41a396bb1b039ddf1946 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 02:00:08.022857 13180 raft_consensus.cc:348] T 3d3463f1ff2b41a396bb1b039ddf1946 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } }
I20240503 02:00:08.022961 13180 raft_consensus.cc:374] T 3d3463f1ff2b41a396bb1b039ddf1946 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:08.022989 13180 raft_consensus.cc:729] T 3d3463f1ff2b41a396bb1b039ddf1946 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:08.023105 13180 consensus_queue.cc:260] T 3d3463f1ff2b41a396bb1b039ddf1946 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } }
I20240503 02:00:08.023396 13180 ts_tablet_manager.cc:1398] T 3d3463f1ff2b41a396bb1b039ddf1946 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:08.026024 13067 tablet_bootstrap.cc:492] T 3d3463f1ff2b41a396bb1b039ddf1946 P e1a9251ce1cf4361b8b0ede9fbfc4023: No bootstrap required, opened a new log
I20240503 02:00:08.026132 13067 ts_tablet_manager.cc:1367] T 3d3463f1ff2b41a396bb1b039ddf1946 P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 02:00:08.026620 13067 raft_consensus.cc:348] T 3d3463f1ff2b41a396bb1b039ddf1946 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } }
I20240503 02:00:08.026734 13067 raft_consensus.cc:374] T 3d3463f1ff2b41a396bb1b039ddf1946 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:08.026765 13067 raft_consensus.cc:729] T 3d3463f1ff2b41a396bb1b039ddf1946 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1a9251ce1cf4361b8b0ede9fbfc4023, State: Initialized, Role: FOLLOWER
I20240503 02:00:08.026880 13067 consensus_queue.cc:260] T 3d3463f1ff2b41a396bb1b039ddf1946 P e1a9251ce1cf4361b8b0ede9fbfc4023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } }
I20240503 02:00:08.027168 13067 ts_tablet_manager.cc:1398] T 3d3463f1ff2b41a396bb1b039ddf1946 P e1a9251ce1cf4361b8b0ede9fbfc4023: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:08.030642 8916 consensus_queue.cc:237] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: true } }
I20240503 02:00:08.032332 6456 raft_consensus.cc:1261] T 96c395771fd44056abacceb6c1acdb9f P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:08.032806 13181 tablet_bootstrap.cc:492] T 3d3463f1ff2b41a396bb1b039ddf1946 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:08.032857 13102 consensus_queue.cc:1035] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:08.033027 13181 ts_tablet_manager.cc:1367] T 3d3463f1ff2b41a396bb1b039ddf1946 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
W20240503 02:00:08.033602 8853 consensus_peers.cc:485] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2 -> Peer c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655): Couldn't send request to peer c2ac2eb7cc264047b9ab32fe3a4fc025. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 96c395771fd44056abacceb6c1acdb9f. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:08.033560 13181 raft_consensus.cc:348] T 3d3463f1ff2b41a396bb1b039ddf1946 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } }
I20240503 02:00:08.033665 13181 raft_consensus.cc:374] T 3d3463f1ff2b41a396bb1b039ddf1946 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:08.033812 9048 raft_consensus.cc:1261] T 96c395771fd44056abacceb6c1acdb9f P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:08.034220 13102 consensus_queue.cc:1035] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:08.034590 13181 raft_consensus.cc:729] T 3d3463f1ff2b41a396bb1b039ddf1946 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:08.034816 13181 consensus_queue.cc:260] T 3d3463f1ff2b41a396bb1b039ddf1946 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } }
I20240503 02:00:08.035202 13181 ts_tablet_manager.cc:1398] T 3d3463f1ff2b41a396bb1b039ddf1946 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:08.036073 11872 raft_consensus.cc:2932] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: true } } }
I20240503 02:00:08.036582 9048 raft_consensus.cc:2932] T 96c395771fd44056abacceb6c1acdb9f P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: true } } }
I20240503 02:00:08.040035 6456 raft_consensus.cc:2932] T 96c395771fd44056abacceb6c1acdb9f P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: true } } }
I20240503 02:00:08.040561 6314 catalog_manager.cc:5484] T 96c395771fd44056abacceb6c1acdb9f P 99ead02915954b4fab23b033ad6d546b reported cstate change: config changed from index 4 to 5, NON_VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "9942306855404cfd891adf227dd65eb2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: true } } }
I20240503 02:00:08.051596 9308 consensus_queue.cc:237] T b2e62c043adf4a8d9024279d786f596c P e1a9251ce1cf4361b8b0ede9fbfc4023 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } }
I20240503 02:00:08.053411 9179 raft_consensus.cc:1261] T b2e62c043adf4a8d9024279d786f596c P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer e1a9251ce1cf4361b8b0ede9fbfc4023: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:08.054270 13201 consensus_queue.cc:1035] T b2e62c043adf4a8d9024279d786f596c P e1a9251ce1cf4361b8b0ede9fbfc4023 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:08.054872 6850 raft_consensus.cc:1261] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer e1a9251ce1cf4361b8b0ede9fbfc4023: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:08.056768 13201 consensus_queue.cc:1035] T b2e62c043adf4a8d9024279d786f596c P e1a9251ce1cf4361b8b0ede9fbfc4023 [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:08.057793 13168 raft_consensus.cc:2932] T b2e62c043adf4a8d9024279d786f596c P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } } }
I20240503 02:00:08.058616 6850 raft_consensus.cc:2932] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } } }
I20240503 02:00:08.062433 9179 raft_consensus.cc:2932] T b2e62c043adf4a8d9024279d786f596c P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } } }
W20240503 02:00:08.065229 9246 consensus_peers.cc:485] T b2e62c043adf4a8d9024279d786f596c P e1a9251ce1cf4361b8b0ede9fbfc4023 -> Peer 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211): Couldn't send request to peer 9942306855404cfd891adf227dd65eb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b2e62c043adf4a8d9024279d786f596c. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:08.066805 6307 catalog_manager.cc:5484] T b2e62c043adf4a8d9024279d786f596c P 2a01ace80baf4057b6bf1995cf88468b reported cstate change: config changed from index -1 to 2, NON_VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } } }
I20240503 02:00:08.074081 6719 consensus_queue.cc:237] T cea1ea50fa434d31b42451f84f26bc76 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { promote: true } }
I20240503 02:00:08.075672 9308 raft_consensus.cc:1261] T cea1ea50fa434d31b42451f84f26bc76 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:08.075712 9048 raft_consensus.cc:1261] T cea1ea50fa434d31b42451f84f26bc76 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:00:08.076017 6656 consensus_peers.cc:485] T cea1ea50fa434d31b42451f84f26bc76 P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719): Couldn't send request to peer 492448570cc94ed1879feb2ec3b9051d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cea1ea50fa434d31b42451f84f26bc76. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:08.076267 13209 consensus_queue.cc:1035] T cea1ea50fa434d31b42451f84f26bc76 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:08.076498 12872 consensus_queue.cc:1035] T cea1ea50fa434d31b42451f84f26bc76 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:08.078694 13209 raft_consensus.cc:2932] T cea1ea50fa434d31b42451f84f26bc76 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { promote: true } } }
I20240503 02:00:08.079003 9048 raft_consensus.cc:2932] T cea1ea50fa434d31b42451f84f26bc76 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { promote: true } } }
I20240503 02:00:08.079102 9308 raft_consensus.cc:2932] T cea1ea50fa434d31b42451f84f26bc76 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { promote: true } } }
I20240503 02:00:08.082130 6313 catalog_manager.cc:5484] T cea1ea50fa434d31b42451f84f26bc76 P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: config changed from index -1 to 2, NON_VOTER 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:08.090395 9179 consensus_queue.cc:237] T b20ff3a6f74f4640afebfd4059b58699 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: true } }
I20240503 02:00:08.091773 6719 raft_consensus.cc:1261] T b20ff3a6f74f4640afebfd4059b58699 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer 2a01ace80baf4057b6bf1995cf88468b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:00:08.091933 9112 consensus_peers.cc:485] T b20ff3a6f74f4640afebfd4059b58699 P 2a01ace80baf4057b6bf1995cf88468b -> Peer dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): Couldn't send request to peer dec3e4a04d604f7eaaa3f077d081218a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b20ff3a6f74f4640afebfd4059b58699. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:08.091929 9048 raft_consensus.cc:1261] T b20ff3a6f74f4640afebfd4059b58699 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer 2a01ace80baf4057b6bf1995cf88468b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:08.092257 12907 consensus_queue.cc:1035] T b20ff3a6f74f4640afebfd4059b58699 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:08.093077 13098 consensus_queue.cc:1035] T b20ff3a6f74f4640afebfd4059b58699 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240503 02:00:08.094653 13098 raft_consensus.cc:2932] T b20ff3a6f74f4640afebfd4059b58699 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: true } } }
I20240503 02:00:08.095209 9048 raft_consensus.cc:2932] T b20ff3a6f74f4640afebfd4059b58699 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: true } } }
I20240503 02:00:08.095821 6719 raft_consensus.cc:2932] T b20ff3a6f74f4640afebfd4059b58699 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: true } } }
I20240503 02:00:08.098109 6314 catalog_manager.cc:5484] T b20ff3a6f74f4640afebfd4059b58699 P 99ead02915954b4fab23b033ad6d546b reported cstate change: config changed from index -1 to 2, NON_VOTER dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "2a01ace80baf4057b6bf1995cf88468b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: true } } }
I20240503 02:00:08.107373 9048 consensus_queue.cc:237] T aa281822047841fab7c58b53d9733e49 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: true } }
I20240503 02:00:08.108980 9308 raft_consensus.cc:1261] T aa281822047841fab7c58b53d9733e49 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Refusing update from remote peer 99ead02915954b4fab23b033ad6d546b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:08.109042 9179 raft_consensus.cc:1261] T aa281822047841fab7c58b53d9733e49 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer 99ead02915954b4fab23b033ad6d546b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:00:08.109064 8984 consensus_peers.cc:485] T aa281822047841fab7c58b53d9733e49 P 99ead02915954b4fab23b033ad6d546b -> Peer c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655): Couldn't send request to peer c2ac2eb7cc264047b9ab32fe3a4fc025. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aa281822047841fab7c58b53d9733e49. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:08.109408 12922 consensus_queue.cc:1035] T aa281822047841fab7c58b53d9733e49 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:08.109616 13179 consensus_queue.cc:1035] T aa281822047841fab7c58b53d9733e49 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:08.111399 13178 raft_consensus.cc:2932] T aa281822047841fab7c58b53d9733e49 P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: true } } }
I20240503 02:00:08.111865 9179 raft_consensus.cc:2932] T aa281822047841fab7c58b53d9733e49 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: true } } }
I20240503 02:00:08.112946 9308 raft_consensus.cc:2932] T aa281822047841fab7c58b53d9733e49 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: true } } }
I20240503 02:00:08.115483 6307 catalog_manager.cc:5484] T aa281822047841fab7c58b53d9733e49 P 99ead02915954b4fab23b033ad6d546b reported cstate change: config changed from index -1 to 2, NON_VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "99ead02915954b4fab23b033ad6d546b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:08.119498 10679 raft_consensus.cc:482] T 3d3463f1ff2b41a396bb1b039ddf1946 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:08.119616 10679 raft_consensus.cc:504] T 3d3463f1ff2b41a396bb1b039ddf1946 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } }
I20240503 02:00:08.120036 10679 leader_election.cc:288] T 3d3463f1ff2b41a396bb1b039ddf1946 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311)
I20240503 02:00:08.120476 9308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d3463f1ff2b41a396bb1b039ddf1946" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" is_pre_election: true
I20240503 02:00:08.120627 9308 raft_consensus.cc:2454] T 3d3463f1ff2b41a396bb1b039ddf1946 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:08.120620 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d3463f1ff2b41a396bb1b039ddf1946" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:08.120765 8916 raft_consensus.cc:2454] T 3d3463f1ff2b41a396bb1b039ddf1946 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:08.121021 9115 leader_election.cc:302] T 3d3463f1ff2b41a396bb1b039ddf1946 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:08.121223 10679 raft_consensus.cc:2781] T 3d3463f1ff2b41a396bb1b039ddf1946 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:08.121294 10679 raft_consensus.cc:482] T 3d3463f1ff2b41a396bb1b039ddf1946 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:08.121330 10679 raft_consensus.cc:3037] T 3d3463f1ff2b41a396bb1b039ddf1946 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:08.122354 10679 raft_consensus.cc:504] T 3d3463f1ff2b41a396bb1b039ddf1946 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } }
I20240503 02:00:08.122690 10679 leader_election.cc:288] T 3d3463f1ff2b41a396bb1b039ddf1946 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Requested vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311)
I20240503 02:00:08.123081 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d3463f1ff2b41a396bb1b039ddf1946" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:08.123217 8916 raft_consensus.cc:3037] T 3d3463f1ff2b41a396bb1b039ddf1946 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:08.123198 9308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d3463f1ff2b41a396bb1b039ddf1946" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023"
I20240503 02:00:08.123317 9308 raft_consensus.cc:3037] T 3d3463f1ff2b41a396bb1b039ddf1946 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:08.124181 9308 raft_consensus.cc:2454] T 3d3463f1ff2b41a396bb1b039ddf1946 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:08.124200 8916 raft_consensus.cc:2454] T 3d3463f1ff2b41a396bb1b039ddf1946 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:08.124475 6456 consensus_queue.cc:237] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } }
I20240503 02:00:08.124577 9115 leader_election.cc:302] T 3d3463f1ff2b41a396bb1b039ddf1946 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:08.124781 10679 raft_consensus.cc:2781] T 3d3463f1ff2b41a396bb1b039ddf1946 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:08.124908 10679 raft_consensus.cc:686] T 3d3463f1ff2b41a396bb1b039ddf1946 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Becoming Leader. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Running, Role: LEADER
I20240503 02:00:08.125034 10679 consensus_queue.cc:237] T 3d3463f1ff2b41a396bb1b039ddf1946 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } }
I20240503 02:00:08.126245 9048 raft_consensus.cc:1261] T ac7fa0c3722a461c91fb237fd85794a3 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:08.126246 6582 raft_consensus.cc:1261] T ac7fa0c3722a461c91fb237fd85794a3 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:08.126632 12190 consensus_queue.cc:1035] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:08.126845 12959 consensus_queue.cc:1035] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:08.127775 6314 catalog_manager.cc:5484] T 3d3463f1ff2b41a396bb1b039ddf1946 P 2a01ace80baf4057b6bf1995cf88468b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135). New cstate: current_term: 1 leader_uuid: "2a01ace80baf4057b6bf1995cf88468b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } health_report { overall_health: UNKNOWN } } }
W20240503 02:00:08.128789 6391 consensus_peers.cc:485] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a -> Peer 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079): Couldn't send request to peer 2a01ace80baf4057b6bf1995cf88468b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ac7fa0c3722a461c91fb237fd85794a3. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:08.128971 12959 raft_consensus.cc:2932] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } } }
I20240503 02:00:08.129967 6582 raft_consensus.cc:2932] T ac7fa0c3722a461c91fb237fd85794a3 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } } }
I20240503 02:00:08.132113 6314 catalog_manager.cc:5484] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: config changed from index 6 to 7, NON_VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:08.133029 9048 raft_consensus.cc:2932] T ac7fa0c3722a461c91fb237fd85794a3 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } } }
I20240503 02:00:08.141841 8916 consensus_queue.cc:237] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: true } }
I20240503 02:00:08.143218 6456 raft_consensus.cc:1261] T 1c721b33983b4c47afd8faa8f155ca84 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:08.143265 6850 raft_consensus.cc:1261] T 1c721b33983b4c47afd8faa8f155ca84 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 02:00:08.143435 8853 consensus_peers.cc:485] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2 -> Peer c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655): Couldn't send request to peer c2ac2eb7cc264047b9ab32fe3a4fc025. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1c721b33983b4c47afd8faa8f155ca84. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:08.143620 13018 consensus_queue.cc:1035] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:08.143853 11872 consensus_queue.cc:1035] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:08.146181 13017 raft_consensus.cc:2932] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: true } } }
I20240503 02:00:08.146580 6850 raft_consensus.cc:2932] T 1c721b33983b4c47afd8faa8f155ca84 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: true } } }
I20240503 02:00:08.146694 6456 raft_consensus.cc:2932] T 1c721b33983b4c47afd8faa8f155ca84 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: true } } }
I20240503 02:00:08.149812 6313 catalog_manager.cc:5484] T 1c721b33983b4c47afd8faa8f155ca84 P 492448570cc94ed1879feb2ec3b9051d reported cstate change: config changed from index 4 to 5, NON_VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "9942306855404cfd891adf227dd65eb2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: true } } }
I20240503 02:00:08.153924 10679 consensus_queue.cc:1035] T b1740b877c1e4372a0337d99f5bcab0e P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:08.158141 13098 consensus_queue.cc:1035] T b1740b877c1e4372a0337d99f5bcab0e P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:08.176975 9048 consensus_queue.cc:237] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } }
I20240503 02:00:08.178526 6456 raft_consensus.cc:1261] T bd8b9532bd744ecdbe8da66a1225724b P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 99ead02915954b4fab23b033ad6d546b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:00:08.178599 8982 consensus_peers.cc:485] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bd8b9532bd744ecdbe8da66a1225724b. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:08.178892 9179 raft_consensus.cc:1261] T bd8b9532bd744ecdbe8da66a1225724b P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer 99ead02915954b4fab23b033ad6d546b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:08.179312 13179 consensus_queue.cc:1035] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:08.179677 11514 consensus_queue.cc:1035] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:08.181716 13179 raft_consensus.cc:2932] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } } }
I20240503 02:00:08.182240 6456 raft_consensus.cc:2932] T bd8b9532bd744ecdbe8da66a1225724b P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } } }
I20240503 02:00:08.182603 9179 raft_consensus.cc:2932] T bd8b9532bd744ecdbe8da66a1225724b P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } } }
I20240503 02:00:08.185312 6314 catalog_manager.cc:5484] T bd8b9532bd744ecdbe8da66a1225724b P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: config changed from index -1 to 2, NON_VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "99ead02915954b4fab23b033ad6d546b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { promote: true } } }
I20240503 02:00:08.196075 9048 consensus_queue.cc:237] T e52373099af3428497d176e3fcd8ef27 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: true } }
W20240503 02:00:08.197892 8983 consensus_peers.cc:485] T e52373099af3428497d176e3fcd8ef27 P 99ead02915954b4fab23b033ad6d546b -> Peer 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): Couldn't send request to peer 80fd18ef4d604c86b909d5c008a46b4f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e52373099af3428497d176e3fcd8ef27. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:08.197906 6719 raft_consensus.cc:1261] T e52373099af3428497d176e3fcd8ef27 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer 99ead02915954b4fab23b033ad6d546b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:08.198037 9179 raft_consensus.cc:1261] T e52373099af3428497d176e3fcd8ef27 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer 99ead02915954b4fab23b033ad6d546b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:08.198295 13179 consensus_queue.cc:1035] T e52373099af3428497d176e3fcd8ef27 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:08.198554 12922 consensus_queue.cc:1035] T e52373099af3428497d176e3fcd8ef27 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:08.200757 13179 raft_consensus.cc:2932] T e52373099af3428497d176e3fcd8ef27 P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: true } } }
I20240503 02:00:08.202518 9179 raft_consensus.cc:2932] T e52373099af3428497d176e3fcd8ef27 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: true } } }
I20240503 02:00:08.202631 6719 raft_consensus.cc:2932] T e52373099af3428497d176e3fcd8ef27 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: true } } }
I20240503 02:00:08.205932 6313 catalog_manager.cc:5484] T e52373099af3428497d176e3fcd8ef27 P 99ead02915954b4fab23b033ad6d546b reported cstate change: config changed from index -1 to 2, NON_VOTER 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "99ead02915954b4fab23b033ad6d546b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:08.219238 6720 consensus_queue.cc:237] T d73ea6e550a04c04a2695b058e38c566 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } }
I20240503 02:00:08.220734 6582 raft_consensus.cc:1261] T d73ea6e550a04c04a2695b058e38c566 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:08.220880 6456 raft_consensus.cc:1261] T d73ea6e550a04c04a2695b058e38c566 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:00:08.221031 6656 consensus_peers.cc:485] T d73ea6e550a04c04a2695b058e38c566 P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211): Couldn't send request to peer 9942306855404cfd891adf227dd65eb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d73ea6e550a04c04a2695b058e38c566. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:08.221251 13209 consensus_queue.cc:1035] T d73ea6e550a04c04a2695b058e38c566 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:08.221524 10349 consensus_queue.cc:1035] T d73ea6e550a04c04a2695b058e38c566 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:08.223392 12872 raft_consensus.cc:2932] T d73ea6e550a04c04a2695b058e38c566 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } } }
I20240503 02:00:08.223804 6582 raft_consensus.cc:2932] T d73ea6e550a04c04a2695b058e38c566 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } } }
I20240503 02:00:08.223984 6456 raft_consensus.cc:2932] T d73ea6e550a04c04a2695b058e38c566 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } } }
I20240503 02:00:08.226964 6314 catalog_manager.cc:5484] T d73ea6e550a04c04a2695b058e38c566 P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: config changed from index -1 to 4, NON_VOTER 9942306855404cfd891adf227dd65eb2 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:08.234865 6456 consensus_queue.cc:237] T 9cbb2be1fda74ebe8997123c70629ba3 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: true } }
I20240503 02:00:08.236398 9048 raft_consensus.cc:1261] T 9cbb2be1fda74ebe8997123c70629ba3 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:00:08.236534 6393 consensus_peers.cc:485] T 9cbb2be1fda74ebe8997123c70629ba3 P dec3e4a04d604f7eaaa3f077d081218a -> Peer 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): Couldn't send request to peer 80fd18ef4d604c86b909d5c008a46b4f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9cbb2be1fda74ebe8997123c70629ba3. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:08.236588 8916 raft_consensus.cc:1261] T 9cbb2be1fda74ebe8997123c70629ba3 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:08.236828 13231 consensus_queue.cc:1035] T 9cbb2be1fda74ebe8997123c70629ba3 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:08.237048 12190 consensus_queue.cc:1035] T 9cbb2be1fda74ebe8997123c70629ba3 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:08.238878 12959 raft_consensus.cc:2932] T 9cbb2be1fda74ebe8997123c70629ba3 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: true } } }
I20240503 02:00:08.239398 9048 raft_consensus.cc:2932] T 9cbb2be1fda74ebe8997123c70629ba3 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: true } } }
I20240503 02:00:08.241571 8916 raft_consensus.cc:2932] T 9cbb2be1fda74ebe8997123c70629ba3 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: true } } }
I20240503 02:00:08.242002 6307 catalog_manager.cc:5484] T 9cbb2be1fda74ebe8997123c70629ba3 P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: config changed from index -1 to 2, NON_VOTER 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:08.242734 6313 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_extra_table_22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:08.252282 6314 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:08.252576 9158 tablet_service.cc:1467] Processing CreateTablet for tablet c70e6cff9fa440219007a0af1c43cdb9 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=9acd567582744849a2fa10540b380dbe]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:08.252804 9028 tablet_service.cc:1467] Processing CreateTablet for tablet 539e7e4ef5a742c094bd5e4949493ac6 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=9acd567582744849a2fa10540b380dbe]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:08.252880 9158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c70e6cff9fa440219007a0af1c43cdb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:08.253008 8897 tablet_service.cc:1467] Processing CreateTablet for tablet c70e6cff9fa440219007a0af1c43cdb9 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=9acd567582744849a2fa10540b380dbe]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:08.253075 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 539e7e4ef5a742c094bd5e4949493ac6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:08.253249 8897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c70e6cff9fa440219007a0af1c43cdb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:08.254154 6567 tablet_service.cc:1467] Processing CreateTablet for tablet 539e7e4ef5a742c094bd5e4949493ac6 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=9acd567582744849a2fa10540b380dbe]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:08.254417 6567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 539e7e4ef5a742c094bd5e4949493ac6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:08.255440 8887 tablet_service.cc:1467] Processing CreateTablet for tablet 539e7e4ef5a742c094bd5e4949493ac6 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=9acd567582744849a2fa10540b380dbe]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:08.255607 6831 tablet_service.cc:1467] Processing CreateTablet for tablet c70e6cff9fa440219007a0af1c43cdb9 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=9acd567582744849a2fa10540b380dbe]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:08.255712 8887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 539e7e4ef5a742c094bd5e4949493ac6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:08.255878 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c70e6cff9fa440219007a0af1c43cdb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:08.258111 6569 tablet_service.cc:1467] Processing CreateTablet for tablet 5d82d1279c8f4eea9383cfbe7705aeb8 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=9acd567582744849a2fa10540b380dbe]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:08.258423 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d82d1279c8f4eea9383cfbe7705aeb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:08.258706 13279 tablet_bootstrap.cc:492] T 539e7e4ef5a742c094bd5e4949493ac6 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:08.258844 9028 tablet_service.cc:1467] Processing CreateTablet for tablet 5d82d1279c8f4eea9383cfbe7705aeb8 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=9acd567582744849a2fa10540b380dbe]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:08.259074 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d82d1279c8f4eea9383cfbe7705aeb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:08.259697 13279 tablet_bootstrap.cc:654] T 539e7e4ef5a742c094bd5e4949493ac6 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:08.260424 13181 tablet_bootstrap.cc:492] T c70e6cff9fa440219007a0af1c43cdb9 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:08.261253 13281 tablet_bootstrap.cc:492] T c70e6cff9fa440219007a0af1c43cdb9 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:08.261417 13180 tablet_bootstrap.cc:492] T c70e6cff9fa440219007a0af1c43cdb9 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:08.261443 13181 tablet_bootstrap.cc:654] T c70e6cff9fa440219007a0af1c43cdb9 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:08.262208 13281 tablet_bootstrap.cc:654] T c70e6cff9fa440219007a0af1c43cdb9 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:08.262462 13180 tablet_bootstrap.cc:654] T c70e6cff9fa440219007a0af1c43cdb9 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:08.262907 13280 tablet_bootstrap.cc:492] T 539e7e4ef5a742c094bd5e4949493ac6 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:08.263590 13181 tablet_bootstrap.cc:492] T c70e6cff9fa440219007a0af1c43cdb9 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:08.263690 13181 ts_tablet_manager.cc:1367] T c70e6cff9fa440219007a0af1c43cdb9 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:08.263705 13280 tablet_bootstrap.cc:654] T 539e7e4ef5a742c094bd5e4949493ac6 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:08.264178 13181 raft_consensus.cc:348] T c70e6cff9fa440219007a0af1c43cdb9 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:08.264297 13181 raft_consensus.cc:374] T c70e6cff9fa440219007a0af1c43cdb9 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:08.264330 13181 raft_consensus.cc:729] T c70e6cff9fa440219007a0af1c43cdb9 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:08.264480 13181 consensus_queue.cc:260] T c70e6cff9fa440219007a0af1c43cdb9 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:08.264777 13181 ts_tablet_manager.cc:1398] T c70e6cff9fa440219007a0af1c43cdb9 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:08.265328 13281 tablet_bootstrap.cc:492] T c70e6cff9fa440219007a0af1c43cdb9 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:08.265429 13281 ts_tablet_manager.cc:1367] T c70e6cff9fa440219007a0af1c43cdb9 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:08.265620 9158 tablet_service.cc:1467] Processing CreateTablet for tablet 5d82d1279c8f4eea9383cfbe7705aeb8 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=9acd567582744849a2fa10540b380dbe]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:08.265897 9158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d82d1279c8f4eea9383cfbe7705aeb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:08.265905 13281 raft_consensus.cc:348] T c70e6cff9fa440219007a0af1c43cdb9 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:08.266100 13281 raft_consensus.cc:374] T c70e6cff9fa440219007a0af1c43cdb9 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:08.266184 13281 raft_consensus.cc:729] T c70e6cff9fa440219007a0af1c43cdb9 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:08.266369 13281 consensus_queue.cc:260] T c70e6cff9fa440219007a0af1c43cdb9 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:08.266939 9309 raft_consensus.cc:1261] T 3d3463f1ff2b41a396bb1b039ddf1946 P e1a9251ce1cf4361b8b0ede9fbfc4023 [term 1 FOLLOWER]: Refusing update from remote peer 2a01ace80baf4057b6bf1995cf88468b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:08.267410 12907 consensus_queue.cc:1035] T 3d3463f1ff2b41a396bb1b039ddf1946 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:08.267678 8916 raft_consensus.cc:1261] T 3d3463f1ff2b41a396bb1b039ddf1946 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Refusing update from remote peer 2a01ace80baf4057b6bf1995cf88468b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:08.268000 13180 tablet_bootstrap.cc:492] T c70e6cff9fa440219007a0af1c43cdb9 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:08.268095 13180 ts_tablet_manager.cc:1367] T c70e6cff9fa440219007a0af1c43cdb9 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:00:08.268261 12907 consensus_queue.cc:1035] T 3d3463f1ff2b41a396bb1b039ddf1946 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:08.268802 13180 raft_consensus.cc:348] T c70e6cff9fa440219007a0af1c43cdb9 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:08.268924 13180 raft_consensus.cc:374] T c70e6cff9fa440219007a0af1c43cdb9 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:08.268963 13180 raft_consensus.cc:729] T c70e6cff9fa440219007a0af1c43cdb9 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:08.269086 13180 consensus_queue.cc:260] T c70e6cff9fa440219007a0af1c43cdb9 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:08.270025 13181 tablet_bootstrap.cc:492] T 5d82d1279c8f4eea9383cfbe7705aeb8 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:08.270702 13180 ts_tablet_manager.cc:1398] T c70e6cff9fa440219007a0af1c43cdb9 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:00:08.270875 13180 tablet_bootstrap.cc:492] T 539e7e4ef5a742c094bd5e4949493ac6 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:08.271034 13181 tablet_bootstrap.cc:654] T 5d82d1279c8f4eea9383cfbe7705aeb8 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:08.272747 13180 tablet_bootstrap.cc:654] T 539e7e4ef5a742c094bd5e4949493ac6 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:08.272830 6313 catalog_manager.cc:5576] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211) does not have the latest schema for tablet 3d3463f1ff2b41a396bb1b039ddf1946 (table rebalancer_dynamic_table [id=954039feb1f843f7adbb2096b370accb]). Expected version 1 got 0
I20240503 02:00:08.273232 13181 tablet_bootstrap.cc:492] T 5d82d1279c8f4eea9383cfbe7705aeb8 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:08.273317 13181 ts_tablet_manager.cc:1367] T 5d82d1279c8f4eea9383cfbe7705aeb8 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:08.273821 13181 raft_consensus.cc:348] T 5d82d1279c8f4eea9383cfbe7705aeb8 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:08.273941 13181 raft_consensus.cc:374] T 5d82d1279c8f4eea9383cfbe7705aeb8 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:08.273972 13181 raft_consensus.cc:729] T 5d82d1279c8f4eea9383cfbe7705aeb8 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:08.274102 13181 consensus_queue.cc:260] T 5d82d1279c8f4eea9383cfbe7705aeb8 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:08.274421 13181 ts_tablet_manager.cc:1398] T 5d82d1279c8f4eea9383cfbe7705aeb8 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:08.274900 6314 catalog_manager.cc:5576] TS e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311) does not have the latest schema for tablet 3d3463f1ff2b41a396bb1b039ddf1946 (table rebalancer_dynamic_table [id=954039feb1f843f7adbb2096b370accb]). Expected version 1 got 0
I20240503 02:00:08.275831 12912 tablet.cc:1722] T 3d3463f1ff2b41a396bb1b039ddf1946 P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:08.275926 6456 consensus_queue.cc:237] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } }
I20240503 02:00:08.276060 13280 tablet_bootstrap.cc:492] T 539e7e4ef5a742c094bd5e4949493ac6 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:08.276157 12912 tablet.cc:1620] T 3d3463f1ff2b41a396bb1b039ddf1946 P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
I20240503 02:00:08.276234 13281 ts_tablet_manager.cc:1398] T c70e6cff9fa440219007a0af1c43cdb9 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20240503 02:00:08.276512 13280 ts_tablet_manager.cc:1367] T 539e7e4ef5a742c094bd5e4949493ac6 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 02:00:08.277215 13279 tablet_bootstrap.cc:492] T 539e7e4ef5a742c094bd5e4949493ac6 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:08.277338 13279 ts_tablet_manager.cc:1367] T 539e7e4ef5a742c094bd5e4949493ac6 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240503 02:00:08.277235 13280 raft_consensus.cc:348] T 539e7e4ef5a742c094bd5e4949493ac6 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:08.277382 13280 raft_consensus.cc:374] T 539e7e4ef5a742c094bd5e4949493ac6 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:08.277451 13280 raft_consensus.cc:729] T 539e7e4ef5a742c094bd5e4949493ac6 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:08.277735 13280 consensus_queue.cc:260] T 539e7e4ef5a742c094bd5e4949493ac6 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:08.277793 13279 raft_consensus.cc:348] T 539e7e4ef5a742c094bd5e4949493ac6 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:08.277890 13279 raft_consensus.cc:374] T 539e7e4ef5a742c094bd5e4949493ac6 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:08.277917 13279 raft_consensus.cc:729] T 539e7e4ef5a742c094bd5e4949493ac6 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:08.278033 13279 consensus_queue.cc:260] T 539e7e4ef5a742c094bd5e4949493ac6 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:08.278148 13280 ts_tablet_manager.cc:1398] T 539e7e4ef5a742c094bd5e4949493ac6 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:08.278312 13279 ts_tablet_manager.cc:1398] T 539e7e4ef5a742c094bd5e4949493ac6 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:08.278489 13279 tablet_bootstrap.cc:492] T 5d82d1279c8f4eea9383cfbe7705aeb8 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:08.278465 13099 tablet.cc:1722] T 3d3463f1ff2b41a396bb1b039ddf1946 P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:08.278784 13099 tablet.cc:1620] T 3d3463f1ff2b41a396bb1b039ddf1946 P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
I20240503 02:00:08.278841 13280 tablet_bootstrap.cc:492] T 5d82d1279c8f4eea9383cfbe7705aeb8 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:08.279506 13279 tablet_bootstrap.cc:654] T 5d82d1279c8f4eea9383cfbe7705aeb8 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:08.280051 6313 catalog_manager.cc:5576] TS e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311) does not have the latest schema for tablet 3d3463f1ff2b41a396bb1b039ddf1946 (table rebalancer_dynamic_table [id=954039feb1f843f7adbb2096b370accb]). Expected version 1 got 0
I20240503 02:00:08.281280 9047 raft_consensus.cc:1261] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:08.281672 12959 consensus_queue.cc:1035] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:08.281989 6720 raft_consensus.cc:1261] T 2d7244b0dc4d419cb5c3f3677acd70c7 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:08.283365 6314 catalog_manager.cc:5576] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211) does not have the latest schema for tablet 3d3463f1ff2b41a396bb1b039ddf1946 (table rebalancer_dynamic_table [id=954039feb1f843f7adbb2096b370accb]). Expected version 1 got 0
I20240503 02:00:08.283550 12190 consensus_queue.cc:1035] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:08.285223 13180 tablet_bootstrap.cc:492] T 539e7e4ef5a742c094bd5e4949493ac6 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:08.285261 6313 catalog_manager.cc:5576] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079) does not have the latest schema for tablet 3d3463f1ff2b41a396bb1b039ddf1946 (table rebalancer_dynamic_table [id=954039feb1f843f7adbb2096b370accb]). Expected version 1 got 0
I20240503 02:00:08.285331 13180 ts_tablet_manager.cc:1367] T 539e7e4ef5a742c094bd5e4949493ac6 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.002s
I20240503 02:00:08.285768 13180 raft_consensus.cc:348] T 539e7e4ef5a742c094bd5e4949493ac6 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:08.285879 13180 raft_consensus.cc:374] T 539e7e4ef5a742c094bd5e4949493ac6 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:08.285912 13180 raft_consensus.cc:729] T 539e7e4ef5a742c094bd5e4949493ac6 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:08.286016 13180 consensus_queue.cc:260] T 539e7e4ef5a742c094bd5e4949493ac6 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:08.286300 13180 ts_tablet_manager.cc:1398] T 539e7e4ef5a742c094bd5e4949493ac6 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:08.287390 13280 tablet_bootstrap.cc:654] T 5d82d1279c8f4eea9383cfbe7705aeb8 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
W20240503 02:00:08.288334 6391 consensus_peers.cc:485] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a -> Peer 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079): Couldn't send request to peer 2a01ace80baf4057b6bf1995cf88468b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2d7244b0dc4d419cb5c3f3677acd70c7. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:08.288326 13293 tablet.cc:1722] T 3d3463f1ff2b41a396bb1b039ddf1946 P e1a9251ce1cf4361b8b0ede9fbfc4023: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:08.288911 13293 tablet.cc:1620] T 3d3463f1ff2b41a396bb1b039ddf1946 P e1a9251ce1cf4361b8b0ede9fbfc4023: MemRowSet was empty: no flush needed.
I20240503 02:00:08.289553 12190 raft_consensus.cc:2932] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } } }
I20240503 02:00:08.290027 6720 raft_consensus.cc:2932] T 2d7244b0dc4d419cb5c3f3677acd70c7 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } } }
I20240503 02:00:08.291602 13280 tablet_bootstrap.cc:492] T 5d82d1279c8f4eea9383cfbe7705aeb8 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:08.291744 13280 ts_tablet_manager.cc:1367] T 5d82d1279c8f4eea9383cfbe7705aeb8 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 02:00:08.292320 13280 raft_consensus.cc:348] T 5d82d1279c8f4eea9383cfbe7705aeb8 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:08.292456 13280 raft_consensus.cc:374] T 5d82d1279c8f4eea9383cfbe7705aeb8 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:08.292490 13280 raft_consensus.cc:729] T 5d82d1279c8f4eea9383cfbe7705aeb8 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:08.292610 13280 consensus_queue.cc:260] T 5d82d1279c8f4eea9383cfbe7705aeb8 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:08.292915 13280 ts_tablet_manager.cc:1398] T 5d82d1279c8f4eea9383cfbe7705aeb8 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:08.292763 9047 raft_consensus.cc:2932] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } } }
I20240503 02:00:08.294337 13279 tablet_bootstrap.cc:492] T 5d82d1279c8f4eea9383cfbe7705aeb8 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:08.294441 13279 ts_tablet_manager.cc:1367] T 5d82d1279c8f4eea9383cfbe7705aeb8 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240503 02:00:08.294505 6314 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=954039feb1f843f7adbb2096b370accb] alter complete (version 1)
I20240503 02:00:08.294935 13279 raft_consensus.cc:348] T 5d82d1279c8f4eea9383cfbe7705aeb8 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:08.295053 13279 raft_consensus.cc:374] T 5d82d1279c8f4eea9383cfbe7705aeb8 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:08.295086 13279 raft_consensus.cc:729] T 5d82d1279c8f4eea9383cfbe7705aeb8 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:08.295204 13279 consensus_queue.cc:260] T 5d82d1279c8f4eea9383cfbe7705aeb8 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:08.295488 13279 ts_tablet_manager.cc:1398] T 5d82d1279c8f4eea9383cfbe7705aeb8 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:08.297967 6308 catalog_manager.cc:5484] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: config changed from index 6 to 7, NON_VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:08.349766 10473 raft_consensus.cc:482] T 539e7e4ef5a742c094bd5e4949493ac6 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:08.349911 10473 raft_consensus.cc:504] T 539e7e4ef5a742c094bd5e4949493ac6 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:08.350539 10473 leader_election.cc:288] T 539e7e4ef5a742c094bd5e4949493ac6 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:08.351023 13098 raft_consensus.cc:482] T c70e6cff9fa440219007a0af1c43cdb9 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:08.351133 13098 raft_consensus.cc:504] T c70e6cff9fa440219007a0af1c43cdb9 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:08.351363 6582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "539e7e4ef5a742c094bd5e4949493ac6" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 02:00:08.351534 13098 leader_election.cc:288] T c70e6cff9fa440219007a0af1c43cdb9 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:08.351502 6582 raft_consensus.cc:2454] T 539e7e4ef5a742c094bd5e4949493ac6 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:08.351997 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c70e6cff9fa440219007a0af1c43cdb9" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:08.352062 6850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c70e6cff9fa440219007a0af1c43cdb9" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 02:00:08.352125 8916 raft_consensus.cc:2454] T c70e6cff9fa440219007a0af1c43cdb9 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:08.352195 6850 raft_consensus.cc:2454] T c70e6cff9fa440219007a0af1c43cdb9 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:08.352667 8983 leader_election.cc:302] T 539e7e4ef5a742c094bd5e4949493ac6 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:08.352885 9115 leader_election.cc:302] T c70e6cff9fa440219007a0af1c43cdb9 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:08.352947 10473 raft_consensus.cc:2781] T 539e7e4ef5a742c094bd5e4949493ac6 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:08.353001 10473 raft_consensus.cc:482] T 539e7e4ef5a742c094bd5e4949493ac6 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:08.353039 10473 raft_consensus.cc:3037] T 539e7e4ef5a742c094bd5e4949493ac6 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:08.353114 13098 raft_consensus.cc:2781] T c70e6cff9fa440219007a0af1c43cdb9 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:08.353173 13098 raft_consensus.cc:482] T c70e6cff9fa440219007a0af1c43cdb9 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:08.353216 13098 raft_consensus.cc:3037] T c70e6cff9fa440219007a0af1c43cdb9 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:08.354002 13098 raft_consensus.cc:504] T c70e6cff9fa440219007a0af1c43cdb9 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:08.354048 10473 raft_consensus.cc:504] T 539e7e4ef5a742c094bd5e4949493ac6 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:08.354332 13098 leader_election.cc:288] T c70e6cff9fa440219007a0af1c43cdb9 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Requested vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:08.354468 10473 leader_election.cc:288] T 539e7e4ef5a742c094bd5e4949493ac6 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Requested vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:08.355196 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c70e6cff9fa440219007a0af1c43cdb9" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:08.355271 6850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c70e6cff9fa440219007a0af1c43cdb9" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 02:00:08.355329 8916 raft_consensus.cc:3037] T c70e6cff9fa440219007a0af1c43cdb9 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:08.355402 6850 raft_consensus.cc:3037] T c70e6cff9fa440219007a0af1c43cdb9 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:08.355646 8917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "539e7e4ef5a742c094bd5e4949493ac6" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:08.355742 8917 raft_consensus.cc:2454] T 539e7e4ef5a742c094bd5e4949493ac6 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:08.355923 8915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "539e7e4ef5a742c094bd5e4949493ac6" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:08.356004 8915 raft_consensus.cc:3037] T 539e7e4ef5a742c094bd5e4949493ac6 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:08.356276 6850 raft_consensus.cc:2454] T c70e6cff9fa440219007a0af1c43cdb9 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:08.356353 8916 raft_consensus.cc:2454] T c70e6cff9fa440219007a0af1c43cdb9 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:08.356761 6582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "539e7e4ef5a742c094bd5e4949493ac6" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 02:00:08.356881 6582 raft_consensus.cc:3037] T 539e7e4ef5a742c094bd5e4949493ac6 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:08.356941 8915 raft_consensus.cc:2454] T 539e7e4ef5a742c094bd5e4949493ac6 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:08.357281 9115 leader_election.cc:302] T c70e6cff9fa440219007a0af1c43cdb9 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:08.357383 8984 leader_election.cc:302] T 539e7e4ef5a742c094bd5e4949493ac6 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9942306855404cfd891adf227dd65eb2, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:08.357789 6582 raft_consensus.cc:2454] T 539e7e4ef5a742c094bd5e4949493ac6 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:08.358455 13098 raft_consensus.cc:2781] T c70e6cff9fa440219007a0af1c43cdb9 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:08.358556 13098 raft_consensus.cc:686] T c70e6cff9fa440219007a0af1c43cdb9 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Becoming Leader. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Running, Role: LEADER
I20240503 02:00:08.358700 13098 consensus_queue.cc:237] T c70e6cff9fa440219007a0af1c43cdb9 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:08.360235 10473 raft_consensus.cc:2781] T 539e7e4ef5a742c094bd5e4949493ac6 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:08.360332 10473 raft_consensus.cc:686] T 539e7e4ef5a742c094bd5e4949493ac6 P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Becoming Leader. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Running, Role: LEADER
I20240503 02:00:08.360473 10473 consensus_queue.cc:237] T 539e7e4ef5a742c094bd5e4949493ac6 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:08.361320 6307 catalog_manager.cc:5484] T c70e6cff9fa440219007a0af1c43cdb9 P 2a01ace80baf4057b6bf1995cf88468b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135). New cstate: current_term: 1 leader_uuid: "2a01ace80baf4057b6bf1995cf88468b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:08.363497 6314 catalog_manager.cc:5484] T 539e7e4ef5a742c094bd5e4949493ac6 P 99ead02915954b4fab23b033ad6d546b reported cstate change: term changed from 0 to 1, leader changed from <none> to 99ead02915954b4fab23b033ad6d546b (127.0.99.134). New cstate: current_term: 1 leader_uuid: "99ead02915954b4fab23b033ad6d546b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:08.400465 6307 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:08.407902 12912 tablet.cc:1722] T 3d3463f1ff2b41a396bb1b039ddf1946 P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:08.408001 13293 tablet.cc:1722] T 3d3463f1ff2b41a396bb1b039ddf1946 P e1a9251ce1cf4361b8b0ede9fbfc4023: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:08.408277 12912 tablet.cc:1620] T 3d3463f1ff2b41a396bb1b039ddf1946 P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
I20240503 02:00:08.408275 13293 tablet.cc:1620] T 3d3463f1ff2b41a396bb1b039ddf1946 P e1a9251ce1cf4361b8b0ede9fbfc4023: MemRowSet was empty: no flush needed.
I20240503 02:00:08.409570 13099 tablet.cc:1722] T 3d3463f1ff2b41a396bb1b039ddf1946 P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:08.409900 13099 tablet.cc:1620] T 3d3463f1ff2b41a396bb1b039ddf1946 P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
I20240503 02:00:08.414721 6314 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=954039feb1f843f7adbb2096b370accb] alter complete (version 2)
I20240503 02:00:08.511142 13368 ts_tablet_manager.cc:902] T beeea71f02ae4c89a1af13d3584ef70a P 80fd18ef4d604c86b909d5c008a46b4f: Initiating tablet copy from peer 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:08.511729 13368 tablet_copy_client.cc:308] T beeea71f02ae4c89a1af13d3584ef70a P 80fd18ef4d604c86b909d5c008a46b4f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:46157
I20240503 02:00:08.512219 9068 tablet_copy_service.cc:140] P 99ead02915954b4fab23b033ad6d546b: Received BeginTabletCopySession request for tablet beeea71f02ae4c89a1af13d3584ef70a from peer 80fd18ef4d604c86b909d5c008a46b4f ({username='slave'} at 127.0.99.130:46681)
I20240503 02:00:08.512341 9068 tablet_copy_service.cc:161] P 99ead02915954b4fab23b033ad6d546b: Beginning new tablet copy session on tablet beeea71f02ae4c89a1af13d3584ef70a from peer 80fd18ef4d604c86b909d5c008a46b4f at {username='slave'} at 127.0.99.130:46681: session id = 80fd18ef4d604c86b909d5c008a46b4f-beeea71f02ae4c89a1af13d3584ef70a
I20240503 02:00:08.513113 9068 tablet_copy_source_session.cc:205] T beeea71f02ae4c89a1af13d3584ef70a P 99ead02915954b4fab23b033ad6d546b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:08.514036 13368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet beeea71f02ae4c89a1af13d3584ef70a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:08.516170 13368 tablet_copy_client.cc:783] T beeea71f02ae4c89a1af13d3584ef70a P 80fd18ef4d604c86b909d5c008a46b4f: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:08.516412 13368 tablet_copy_client.cc:647] T beeea71f02ae4c89a1af13d3584ef70a P 80fd18ef4d604c86b909d5c008a46b4f: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:08.517596 13368 tablet_copy_client.cc:523] T beeea71f02ae4c89a1af13d3584ef70a P 80fd18ef4d604c86b909d5c008a46b4f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:08.518308 6308 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:08.519683 13368 tablet_bootstrap.cc:492] T beeea71f02ae4c89a1af13d3584ef70a P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:08.523983 12912 tablet.cc:1722] T 3d3463f1ff2b41a396bb1b039ddf1946 P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:08.524958 13368 tablet_bootstrap.cc:492] T beeea71f02ae4c89a1af13d3584ef70a P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:08.525188 13368 tablet_bootstrap.cc:492] T beeea71f02ae4c89a1af13d3584ef70a P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap complete.
I20240503 02:00:08.525377 13368 ts_tablet_manager.cc:1367] T beeea71f02ae4c89a1af13d3584ef70a P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20240503 02:00:08.525839 13368 raft_consensus.cc:348] T beeea71f02ae4c89a1af13d3584ef70a P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: true } }
I20240503 02:00:08.525975 13368 raft_consensus.cc:729] T beeea71f02ae4c89a1af13d3584ef70a P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: LEARNER
I20240503 02:00:08.526077 13368 consensus_queue.cc:260] T beeea71f02ae4c89a1af13d3584ef70a P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: true } }
I20240503 02:00:08.526155 13099 tablet.cc:1722] T 3d3463f1ff2b41a396bb1b039ddf1946 P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:08.526432 13368 ts_tablet_manager.cc:1398] T beeea71f02ae4c89a1af13d3584ef70a P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 02:00:08.526388 13293 tablet.cc:1722] T 3d3463f1ff2b41a396bb1b039ddf1946 P e1a9251ce1cf4361b8b0ede9fbfc4023: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:08.526922 9068 tablet_copy_service.cc:342] P 99ead02915954b4fab23b033ad6d546b: Request end of tablet copy session 80fd18ef4d604c86b909d5c008a46b4f-beeea71f02ae4c89a1af13d3584ef70a received from {username='slave'} at 127.0.99.130:46681
I20240503 02:00:08.526998 9068 tablet_copy_service.cc:433] P 99ead02915954b4fab23b033ad6d546b: ending tablet copy session 80fd18ef4d604c86b909d5c008a46b4f-beeea71f02ae4c89a1af13d3584ef70a on tablet beeea71f02ae4c89a1af13d3584ef70a with peer 80fd18ef4d604c86b909d5c008a46b4f
I20240503 02:00:08.529847 6314 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=954039feb1f843f7adbb2096b370accb] alter complete (version 3)
I20240503 02:00:08.546157 13374 ts_tablet_manager.cc:902] T b20ff3a6f74f4640afebfd4059b58699 P dec3e4a04d604f7eaaa3f077d081218a: Initiating tablet copy from peer 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:08.546721 13374 tablet_copy_client.cc:308] T b20ff3a6f74f4640afebfd4059b58699 P dec3e4a04d604f7eaaa3f077d081218a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:34079
I20240503 02:00:08.547261 9199 tablet_copy_service.cc:140] P 2a01ace80baf4057b6bf1995cf88468b: Received BeginTabletCopySession request for tablet b20ff3a6f74f4640afebfd4059b58699 from peer dec3e4a04d604f7eaaa3f077d081218a ({username='slave'} at 127.0.99.129:55593)
I20240503 02:00:08.547384 9199 tablet_copy_service.cc:161] P 2a01ace80baf4057b6bf1995cf88468b: Beginning new tablet copy session on tablet b20ff3a6f74f4640afebfd4059b58699 from peer dec3e4a04d604f7eaaa3f077d081218a at {username='slave'} at 127.0.99.129:55593: session id = dec3e4a04d604f7eaaa3f077d081218a-b20ff3a6f74f4640afebfd4059b58699
I20240503 02:00:08.548269 9199 tablet_copy_source_session.cc:205] T b20ff3a6f74f4640afebfd4059b58699 P 2a01ace80baf4057b6bf1995cf88468b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:08.549129 13374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b20ff3a6f74f4640afebfd4059b58699. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:08.551559 13374 tablet_copy_client.cc:783] T b20ff3a6f74f4640afebfd4059b58699 P dec3e4a04d604f7eaaa3f077d081218a: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:08.551798 13374 tablet_copy_client.cc:647] T b20ff3a6f74f4640afebfd4059b58699 P dec3e4a04d604f7eaaa3f077d081218a: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:08.552035 12907 raft_consensus.cc:482] T 5d82d1279c8f4eea9383cfbe7705aeb8 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:08.552131 12907 raft_consensus.cc:504] T 5d82d1279c8f4eea9383cfbe7705aeb8 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:08.552565 12907 leader_election.cc:288] T 5d82d1279c8f4eea9383cfbe7705aeb8 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:08.552881 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d82d1279c8f4eea9383cfbe7705aeb8" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:08.553031 9048 raft_consensus.cc:2454] T 5d82d1279c8f4eea9383cfbe7705aeb8 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:08.553328 9113 leader_election.cc:302] T 5d82d1279c8f4eea9383cfbe7705aeb8 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:08.553329 6582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d82d1279c8f4eea9383cfbe7705aeb8" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 02:00:08.553438 6582 raft_consensus.cc:2454] T 5d82d1279c8f4eea9383cfbe7705aeb8 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:08.553512 12907 raft_consensus.cc:2781] T 5d82d1279c8f4eea9383cfbe7705aeb8 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:08.553557 12907 raft_consensus.cc:482] T 5d82d1279c8f4eea9383cfbe7705aeb8 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:08.553589 12907 raft_consensus.cc:3037] T 5d82d1279c8f4eea9383cfbe7705aeb8 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:08.554591 12907 raft_consensus.cc:504] T 5d82d1279c8f4eea9383cfbe7705aeb8 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:08.554926 12907 leader_election.cc:288] T 5d82d1279c8f4eea9383cfbe7705aeb8 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Requested vote from peers 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:08.555275 6582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d82d1279c8f4eea9383cfbe7705aeb8" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 02:00:08.555400 6582 raft_consensus.cc:3037] T 5d82d1279c8f4eea9383cfbe7705aeb8 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:08.555577 13374 tablet_copy_client.cc:523] T b20ff3a6f74f4640afebfd4059b58699 P dec3e4a04d604f7eaaa3f077d081218a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:08.556500 9039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d82d1279c8f4eea9383cfbe7705aeb8" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:08.556619 9039 raft_consensus.cc:3037] T 5d82d1279c8f4eea9383cfbe7705aeb8 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:08.556622 6582 raft_consensus.cc:2454] T 5d82d1279c8f4eea9383cfbe7705aeb8 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:08.556907 13379 ts_tablet_manager.cc:902] T 96c395771fd44056abacceb6c1acdb9f P c2ac2eb7cc264047b9ab32fe3a4fc025: Initiating tablet copy from peer 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:08.556929 9114 leader_election.cc:302] T 5d82d1279c8f4eea9383cfbe7705aeb8 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 80fd18ef4d604c86b909d5c008a46b4f; no voters:
I20240503 02:00:08.557081 12907 raft_consensus.cc:2781] T 5d82d1279c8f4eea9383cfbe7705aeb8 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:08.557394 12907 raft_consensus.cc:686] T 5d82d1279c8f4eea9383cfbe7705aeb8 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Becoming Leader. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Running, Role: LEADER
I20240503 02:00:08.557427 13379 tablet_copy_client.cc:308] T 96c395771fd44056abacceb6c1acdb9f P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:33211
I20240503 02:00:08.557520 9039 raft_consensus.cc:2454] T 5d82d1279c8f4eea9383cfbe7705aeb8 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:08.557538 12907 consensus_queue.cc:237] T 5d82d1279c8f4eea9383cfbe7705aeb8 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:08.557900 8937 tablet_copy_service.cc:140] P 9942306855404cfd891adf227dd65eb2: Received BeginTabletCopySession request for tablet 96c395771fd44056abacceb6c1acdb9f from peer c2ac2eb7cc264047b9ab32fe3a4fc025 ({username='slave'} at 127.0.99.131:37759)
I20240503 02:00:08.558015 8937 tablet_copy_service.cc:161] P 9942306855404cfd891adf227dd65eb2: Beginning new tablet copy session on tablet 96c395771fd44056abacceb6c1acdb9f from peer c2ac2eb7cc264047b9ab32fe3a4fc025 at {username='slave'} at 127.0.99.131:37759: session id = c2ac2eb7cc264047b9ab32fe3a4fc025-96c395771fd44056abacceb6c1acdb9f
I20240503 02:00:08.558295 13374 tablet_bootstrap.cc:492] T b20ff3a6f74f4640afebfd4059b58699 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:08.558804 8937 tablet_copy_source_session.cc:205] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:08.559722 13379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96c395771fd44056abacceb6c1acdb9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:08.561092 6308 catalog_manager.cc:5484] T 5d82d1279c8f4eea9383cfbe7705aeb8 P 2a01ace80baf4057b6bf1995cf88468b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135). New cstate: current_term: 1 leader_uuid: "2a01ace80baf4057b6bf1995cf88468b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:08.561362 13384 ts_tablet_manager.cc:902] T b2e62c043adf4a8d9024279d786f596c P 9942306855404cfd891adf227dd65eb2: Initiating tablet copy from peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311)
I20240503 02:00:08.561815 13384 tablet_copy_client.cc:308] T b2e62c043adf4a8d9024279d786f596c P 9942306855404cfd891adf227dd65eb2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.136:45311
I20240503 02:00:08.562391 9328 tablet_copy_service.cc:140] P e1a9251ce1cf4361b8b0ede9fbfc4023: Received BeginTabletCopySession request for tablet b2e62c043adf4a8d9024279d786f596c from peer 9942306855404cfd891adf227dd65eb2 ({username='slave'} at 127.0.99.133:60887)
I20240503 02:00:08.562525 9328 tablet_copy_service.cc:161] P e1a9251ce1cf4361b8b0ede9fbfc4023: Beginning new tablet copy session on tablet b2e62c043adf4a8d9024279d786f596c from peer 9942306855404cfd891adf227dd65eb2 at {username='slave'} at 127.0.99.133:60887: session id = 9942306855404cfd891adf227dd65eb2-b2e62c043adf4a8d9024279d786f596c
I20240503 02:00:08.563381 13379 tablet_copy_client.cc:783] T 96c395771fd44056abacceb6c1acdb9f P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:08.563530 13379 tablet_copy_client.cc:647] T 96c395771fd44056abacceb6c1acdb9f P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:08.563602 9328 tablet_copy_source_session.cc:205] T b2e62c043adf4a8d9024279d786f596c P e1a9251ce1cf4361b8b0ede9fbfc4023: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:08.564764 13384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2e62c043adf4a8d9024279d786f596c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:08.566628 13384 tablet_copy_client.cc:783] T b2e62c043adf4a8d9024279d786f596c P 9942306855404cfd891adf227dd65eb2: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:08.566782 13384 tablet_copy_client.cc:647] T b2e62c043adf4a8d9024279d786f596c P 9942306855404cfd891adf227dd65eb2: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:08.567963 13379 tablet_copy_client.cc:523] T 96c395771fd44056abacceb6c1acdb9f P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:08.568297 13384 tablet_copy_client.cc:523] T b2e62c043adf4a8d9024279d786f596c P 9942306855404cfd891adf227dd65eb2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:08.569396 13384 tablet_bootstrap.cc:492] T b2e62c043adf4a8d9024279d786f596c P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:08.572863 13374 tablet_bootstrap.cc:492] T b20ff3a6f74f4640afebfd4059b58699 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:08.573037 13384 tablet_bootstrap.cc:492] T b2e62c043adf4a8d9024279d786f596c P 9942306855404cfd891adf227dd65eb2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:08.573155 13374 tablet_bootstrap.cc:492] T b20ff3a6f74f4640afebfd4059b58699 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap complete.
I20240503 02:00:08.573310 13384 tablet_bootstrap.cc:492] T b2e62c043adf4a8d9024279d786f596c P 9942306855404cfd891adf227dd65eb2: Bootstrap complete.
I20240503 02:00:08.573338 13374 ts_tablet_manager.cc:1367] T b20ff3a6f74f4640afebfd4059b58699 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.001s
I20240503 02:00:08.573454 13384 ts_tablet_manager.cc:1367] T b2e62c043adf4a8d9024279d786f596c P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.004s
I20240503 02:00:08.573853 13384 raft_consensus.cc:348] T b2e62c043adf4a8d9024279d786f596c P 9942306855404cfd891adf227dd65eb2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } }
I20240503 02:00:08.573848 13374 raft_consensus.cc:348] T b20ff3a6f74f4640afebfd4059b58699 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: true } }
I20240503 02:00:08.573979 13384 raft_consensus.cc:729] T b2e62c043adf4a8d9024279d786f596c P 9942306855404cfd891adf227dd65eb2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: LEARNER
I20240503 02:00:08.573979 13374 raft_consensus.cc:729] T b20ff3a6f74f4640afebfd4059b58699 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: LEARNER
I20240503 02:00:08.574075 13384 consensus_queue.cc:260] T b2e62c043adf4a8d9024279d786f596c P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } }
I20240503 02:00:08.574075 13374 consensus_queue.cc:260] T b20ff3a6f74f4640afebfd4059b58699 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: true } }
I20240503 02:00:08.574280 13379 tablet_bootstrap.cc:492] T 96c395771fd44056abacceb6c1acdb9f P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:08.574357 13384 ts_tablet_manager.cc:1398] T b2e62c043adf4a8d9024279d786f596c P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:08.574357 13374 ts_tablet_manager.cc:1398] T b20ff3a6f74f4640afebfd4059b58699 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:08.575116 9199 tablet_copy_service.cc:342] P 2a01ace80baf4057b6bf1995cf88468b: Request end of tablet copy session dec3e4a04d604f7eaaa3f077d081218a-b20ff3a6f74f4640afebfd4059b58699 received from {username='slave'} at 127.0.99.129:55593
I20240503 02:00:08.575206 9199 tablet_copy_service.cc:433] P 2a01ace80baf4057b6bf1995cf88468b: ending tablet copy session dec3e4a04d604f7eaaa3f077d081218a-b20ff3a6f74f4640afebfd4059b58699 on tablet b20ff3a6f74f4640afebfd4059b58699 with peer dec3e4a04d604f7eaaa3f077d081218a
I20240503 02:00:08.575160 9328 tablet_copy_service.cc:342] P e1a9251ce1cf4361b8b0ede9fbfc4023: Request end of tablet copy session 9942306855404cfd891adf227dd65eb2-b2e62c043adf4a8d9024279d786f596c received from {username='slave'} at 127.0.99.133:60887
I20240503 02:00:08.575259 9328 tablet_copy_service.cc:433] P e1a9251ce1cf4361b8b0ede9fbfc4023: ending tablet copy session 9942306855404cfd891adf227dd65eb2-b2e62c043adf4a8d9024279d786f596c on tablet b2e62c043adf4a8d9024279d786f596c with peer 9942306855404cfd891adf227dd65eb2
I20240503 02:00:08.580736 13393 ts_tablet_manager.cc:902] T bd8b9532bd744ecdbe8da66a1225724b P e1a9251ce1cf4361b8b0ede9fbfc4023: Initiating tablet copy from peer 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:08.581018 13379 tablet_bootstrap.cc:492] T 96c395771fd44056abacceb6c1acdb9f P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
F20240503 02:00:08.580938 13393 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-0b659e9d57db4ebf93c9143938970090.queue_time_us, tablet-download-pool-0b659e9d57db4ebf93c9143938970090.queue_time_us) (tablet-download-pool-0b659e9d57db4ebf93c9143938970090.run_cpu_time_us, tablet-download-pool-0b659e9d57db4ebf93c9143938970090.run_cpu_time_us) (tablet-download-pool-0b659e9d57db4ebf93c9143938970090.run_wall_time_us, tablet-download-pool-0b659e9d57db4ebf93c9143938970090.run_wall_time_us) (tablet-download-pool-1fa274c98fc147f29344ed537e2ccf17.queue_time_us, tablet-download-pool-1fa274c98fc147f29344ed537e2ccf17.queue_time_us) (tablet-download-pool-1fa274c98fc147f29344ed537e2ccf17.run_cpu_time_us, tablet-download-pool-1fa274c98fc147f29344ed537e2ccf17.run_cpu_time_us) (tablet-download-pool-1fa274c98fc147f29344ed537e2ccf17.run_wall_time_us, tablet-download-pool-1fa274c98fc147f29344ed537e2ccf17.run_wall_time_us) (tablet-download-pool-1feaf29d8c6b4842829ce66e0f4ec23b.queue_time_us, tablet-download-pool-1feaf29d8c6b4842829ce66e0f4ec23b.queue_time_us) (tablet-download-pool-1feaf29d8c6b4842829ce66e0f4ec23b.run_cpu_time_us, tablet-download-pool-1feaf29d8c6b4842829ce66e0f4ec23b.run_cpu_time_us) (tablet-download-pool-1feaf29d8c6b4842829ce66e0f4ec23b.run_wall_time_us, tablet-download-pool-1feaf29d8c6b4842829ce66e0f4ec23b.run_wall_time_us) (tablet-download-pool-3416b6107e1848f299ac49ccd88ae7c8.queue_time_us, tablet-download-pool-3416b6107e1848f299ac49ccd88ae7c8.queue_time_us) (tablet-download-pool-3416b6107e1848f299ac49ccd88ae7c8.run_cpu_time_us, tablet-download-pool-3416b6107e1848f299ac49ccd88ae7c8.run_cpu_time_us) (tablet-download-pool-3416b6107e1848f299ac49ccd88ae7c8.run_wall_time_us, tablet-download-pool-3416b6107e1848f299ac49ccd88ae7c8.run_wall_time_us) (tablet-download-pool-4385d51d11574192883d01376e2d5620.queue_time_us, tablet-download-pool-4385d51d11574192883d01376e2d5620.queue_time_us) (tablet-download-pool-4385d51d11574192883d01376e2d5620.run_cpu_time_us, tablet-download-pool-4385d51d11574192883d01376e2d5620.run_cpu_time_us) (tablet-download-pool-4385d51d11574192883d01376e2d5620.run_wall_time_us, tablet-download-pool-4385d51d11574192883d01376e2d5620.run_wall_time_us) (tablet-download-pool-46ad540deb624a25b97d3b967fe1f6f4.queue_time_us, tablet-download-pool-46ad540deb624a25b97d3b967fe1f6f4.queue_time_us) (tablet-download-pool-46ad540deb624a25b97d3b967fe1f6f4.run_cpu_time_us, tablet-download-pool-46ad540deb624a25b97d3b967fe1f6f4.run_cpu_time_us) (tablet-download-pool-46ad540deb624a25b97d3b967fe1f6f4.run_wall_time_us, tablet-download-pool-46ad540deb624a25b97d3b967fe1f6f4.run_wall_time_us) (tablet-download-pool-548f43b381f1445c8d3d55705ae5a48a.queue_time_us, tablet-download-pool-548f43b381f1445c8d3d55705ae5a48a.queue_time_us) (tablet-download-pool-548f43b381f1445c8d3d55705ae5a48a.run_cpu_time_us, tablet-download-pool-548f43b381f1445c8d3d55705ae5a48a.run_cpu_time_us) (tablet-download-pool-548f43b381f1445c8d3d55705ae5a48a.run_wall_time_us, tablet-download-pool-548f43b381f1445c8d3d55705ae5a48a.run_wall_time_us) (tablet-download-pool-8246488dec204bf6b09e868d7951ac5a.queue_time_us, tablet-download-pool-8246488dec204bf6b09e868d7951ac5a.queue_time_us) (tablet-download-pool-8246488dec204bf6b09e868d7951ac5a.run_cpu_time_us, tablet-download-pool-8246488dec204bf6b09e868d7951ac5a.run_cpu_time_us) (tablet-download-pool-8246488dec204bf6b09e868d7951ac5a.run_wall_time_us, tablet-download-pool-8246488dec204bf6b09e868d7951ac5a.run_wall_time_us) (tablet-download-pool-915028b3478341d588ed6031e657c259.queue_time_us, tablet-download-pool-915028b3478341d588ed6031e657c259.queue_time_us) (tablet-download-pool-915028b3478341d588ed6031e657c259.run_cpu_time_us, tablet-download-pool-915028b3478341d588ed6031e657c259.run_cpu_time_us) (tablet-download-pool-915028b3478341d588ed6031e657c259.run_wall_time_us, tablet-download-pool-915028b3478341d588ed6031e657c259.run_wall_time_us) (tablet-download-pool-925612b660ec471da5b3c548802ede6e.queue_time_us, tablet-download-pool-925612b660ec471da5b3c548802ede6e.queue_time_us) (tablet-download-pool-925612b660ec471da5b3c548802ede6e.run_cpu_time_us, tablet-download-pool-925612b660ec471da5b3c548802ede6e.run_cpu_time_us) (tablet-download-pool-925612b660ec471da5b3c548802ede6e.run_wall_time_us, tablet-download-pool-925612b660ec471da5b3c548802ede6e.run_wall_time_us) (tablet-download-pool-9f7ade1e2af24e86b6e54e2b5769e4c0.queue_time_us, tablet-download-pool-9f7ade1e2af24e86b6e54e2b5769e4c0.queue_time_us) (tablet-download-pool-9f7ade1e2af24e86b6e54e2b5769e4c0.run_cpu_time_us, tablet-download-pool-9f7ade1e2af24e86b6e54e2b5769e4c0.run_cpu_time_us) (tablet-download-pool-9f7ade1e2af24e86b6e54e2b5769e4c0.run_wall_time_us, tablet-download-pool-9f7ade1e2af24e86b6e54e2b5769e4c0.run_wall_time_us) (tablet-download-pool-bd8b9532bd744ecdbe8da66a1225724b.queue_time_us, tablet-download-pool-bd8b9532bd744ecdbe8da66a1225724b.queue_time_us) (tablet-download-pool-c3485701bf3b48f5ae2e1568afeed133.queue_time_us, tablet-download-pool-c3485701bf3b48f5ae2e1568afeed133.queue_time_us) (tablet-download-pool-c3485701bf3b48f5ae2e1568afeed133.run_cpu_time_us, tablet-download-pool-c3485701bf3b48f5ae2e1568afeed133.run_cpu_time_us) (tablet-download-pool-c3485701bf3b48f5ae2e1568afeed133.run_wall_time_us, tablet-download-pool-c3485701bf3b48f5ae2e1568afeed133.run_wall_time_us) (tablet-download-pool-d0d5dd42db3e4883af7fd16d8813b550.queue_time_us, tablet-download-pool-d0d5dd42db3e4883af7fd16d8813b550.queue_time_us) (tablet-download-pool-d0d5dd42db3e4883af7fd16d8813b550.run_cpu_time_us, tablet-download-pool-d0d5dd42db3e4883af7fd16d8813b550.run_cpu_time_us) (tablet-download-pool-d0d5dd42db3e4883af7fd16d8813b550.run_wall_time_us, tablet-download-pool-d0d5dd42db3e4883af7fd16d8813b550.run_wall_time_us) (tablet-download-pool-d897c21e630d4770832bb10f43778e82.queue_time_us, tablet-download-pool-d897c21e630d4770832bb10f43778e82.queue_time_us) (tablet-download-pool-d897c21e630d4770832bb10f43778e82.run_cpu_time_us, tablet-download-pool-d897c21e630d4770832bb10f43778e82.run_cpu_time_us) (tablet-download-pool-d897c21e630d4770832bb10f43778e82.run_wall_time_us, tablet-download-pool-d897c21e630d4770832bb10f43778e82.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
I20240503 02:00:08.581262 13379 tablet_bootstrap.cc:492] T 96c395771fd44056abacceb6c1acdb9f P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap complete.
*** Check failure stack trace: ***
I20240503 02:00:08.581431 13379 ts_tablet_manager.cc:1367] T 96c395771fd44056abacceb6c1acdb9f P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240503 02:00:08.581858 13379 raft_consensus.cc:348] T 96c395771fd44056abacceb6c1acdb9f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: true } }
I20240503 02:00:08.581971 13379 raft_consensus.cc:729] T 96c395771fd44056abacceb6c1acdb9f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: LEARNER
I20240503 02:00:08.582047 13379 consensus_queue.cc:260] T 96c395771fd44056abacceb6c1acdb9f P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: true } }
I20240503 02:00:08.582326 13379 ts_tablet_manager.cc:1398] T 96c395771fd44056abacceb6c1acdb9f P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:08.584043 8937 tablet_copy_service.cc:342] P 9942306855404cfd891adf227dd65eb2: Request end of tablet copy session c2ac2eb7cc264047b9ab32fe3a4fc025-96c395771fd44056abacceb6c1acdb9f received from {username='slave'} at 127.0.99.131:37759
I20240503 02:00:08.584149 8937 tablet_copy_service.cc:433] P 9942306855404cfd891adf227dd65eb2: ending tablet copy session c2ac2eb7cc264047b9ab32fe3a4fc025-96c395771fd44056abacceb6c1acdb9f on tablet 96c395771fd44056abacceb6c1acdb9f with peer c2ac2eb7cc264047b9ab32fe3a4fc025
@ 0x7fa86e8b7dcd google::LogMessage::Fail() at ??:0
@ 0x7fa86e8bbb93 google::LogMessage::SendToLog() at ??:0
@ 0x7fa86e8b77cc google::LogMessage::Flush() at ??:0
@ 0x7fa86e8b8f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa86fd21044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7fa86fd05f98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7fa86fd03efc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7fa8728b677a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7fa8728bfa45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7fa87294076f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7fa8729544f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7fa87293f226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7fa87294ec0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55dbcbe0975c std::function<>::operator()() at ??:0
@ 0x7fa86fd08e06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fa86fd096ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fa86fd0b03b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55dbcbe0975c std::function<>::operator()() at ??:0
@ 0x7fa86fcf8807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa8705366db start_thread at ??:0
@ 0x7fa86da6e71f clone at ??:0
I20240503 02:00:08.634068 6308 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:08.634243 6308 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:08.637499 6308 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet 3d3463f1ff2b41a396bb1b039ddf1946
I20240503 02:00:08.638360 9158 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d3463f1ff2b41a396bb1b039ddf1946 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:08 UTC) from {username='slave'} at 127.0.0.1:51940
I20240503 02:00:08.638831 8887 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d3463f1ff2b41a396bb1b039ddf1946 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:08 UTC) from {username='slave'} at 127.0.0.1:40296
I20240503 02:00:08.640841 13420 tablet_replica.cc:330] T 3d3463f1ff2b41a396bb1b039ddf1946 P 2a01ace80baf4057b6bf1995cf88468b: stopping tablet replica
I20240503 02:00:08.641079 13420 raft_consensus.cc:2229] T 3d3463f1ff2b41a396bb1b039ddf1946 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:08.641324 13420 raft_consensus.cc:2258] T 3d3463f1ff2b41a396bb1b039ddf1946 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:08.641784 13421 tablet_replica.cc:330] T 3d3463f1ff2b41a396bb1b039ddf1946 P 9942306855404cfd891adf227dd65eb2: stopping tablet replica
I20240503 02:00:08.641922 13421 raft_consensus.cc:2229] T 3d3463f1ff2b41a396bb1b039ddf1946 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:08.642042 13421 raft_consensus.cc:2258] T 3d3463f1ff2b41a396bb1b039ddf1946 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:08.643345 13421 ts_tablet_manager.cc:1875] T 3d3463f1ff2b41a396bb1b039ddf1946 P 9942306855404cfd891adf227dd65eb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:08.643841 13420 ts_tablet_manager.cc:1875] T 3d3463f1ff2b41a396bb1b039ddf1946 P 2a01ace80baf4057b6bf1995cf88468b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:08.646013 13421 ts_tablet_manager.cc:1888] T 3d3463f1ff2b41a396bb1b039ddf1946 P 9942306855404cfd891adf227dd65eb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:08.646095 13421 log.cc:1196] T 3d3463f1ff2b41a396bb1b039ddf1946 P 9942306855404cfd891adf227dd65eb2: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/3d3463f1ff2b41a396bb1b039ddf1946
I20240503 02:00:08.646353 13421 ts_tablet_manager.cc:1909] T 3d3463f1ff2b41a396bb1b039ddf1946 P 9942306855404cfd891adf227dd65eb2: Deleting consensus metadata
I20240503 02:00:08.646441 13379 ts_tablet_manager.cc:902] T 1c721b33983b4c47afd8faa8f155ca84 P c2ac2eb7cc264047b9ab32fe3a4fc025: Initiating tablet copy from peer 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:08.646881 13379 tablet_copy_client.cc:308] T 1c721b33983b4c47afd8faa8f155ca84 P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:33211
I20240503 02:00:08.647336 6303 catalog_manager.cc:4830] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211): tablet 3d3463f1ff2b41a396bb1b039ddf1946 (table rebalancer_dynamic_table_new_name [id=954039feb1f843f7adbb2096b370accb]) successfully deleted
I20240503 02:00:08.648490 8937 tablet_copy_service.cc:140] P 9942306855404cfd891adf227dd65eb2: Received BeginTabletCopySession request for tablet 1c721b33983b4c47afd8faa8f155ca84 from peer c2ac2eb7cc264047b9ab32fe3a4fc025 ({username='slave'} at 127.0.99.131:37759)
I20240503 02:00:08.648591 8937 tablet_copy_service.cc:161] P 9942306855404cfd891adf227dd65eb2: Beginning new tablet copy session on tablet 1c721b33983b4c47afd8faa8f155ca84 from peer c2ac2eb7cc264047b9ab32fe3a4fc025 at {username='slave'} at 127.0.99.131:37759: session id = c2ac2eb7cc264047b9ab32fe3a4fc025-1c721b33983b4c47afd8faa8f155ca84
I20240503 02:00:08.648809 13432 ts_tablet_manager.cc:902] T cea1ea50fa434d31b42451f84f26bc76 P 492448570cc94ed1879feb2ec3b9051d: Initiating tablet copy from peer c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:08.649251 13432 tablet_copy_client.cc:308] T cea1ea50fa434d31b42451f84f26bc76 P 492448570cc94ed1879feb2ec3b9051d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:42655
I20240503 02:00:08.649580 8937 tablet_copy_source_session.cc:205] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:08.649688 6739 tablet_copy_service.cc:140] P c2ac2eb7cc264047b9ab32fe3a4fc025: Received BeginTabletCopySession request for tablet cea1ea50fa434d31b42451f84f26bc76 from peer 492448570cc94ed1879feb2ec3b9051d ({username='slave'} at 127.0.99.132:42269)
I20240503 02:00:08.649775 6739 tablet_copy_service.cc:161] P c2ac2eb7cc264047b9ab32fe3a4fc025: Beginning new tablet copy session on tablet cea1ea50fa434d31b42451f84f26bc76 from peer 492448570cc94ed1879feb2ec3b9051d at {username='slave'} at 127.0.99.132:42269: session id = 492448570cc94ed1879feb2ec3b9051d-cea1ea50fa434d31b42451f84f26bc76
I20240503 02:00:08.650502 13379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c721b33983b4c47afd8faa8f155ca84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:08.650715 6739 tablet_copy_source_session.cc:205] T cea1ea50fa434d31b42451f84f26bc76 P c2ac2eb7cc264047b9ab32fe3a4fc025: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:08.651425 13420 ts_tablet_manager.cc:1888] T 3d3463f1ff2b41a396bb1b039ddf1946 P 2a01ace80baf4057b6bf1995cf88468b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:08.651502 13420 log.cc:1196] T 3d3463f1ff2b41a396bb1b039ddf1946 P 2a01ace80baf4057b6bf1995cf88468b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/3d3463f1ff2b41a396bb1b039ddf1946
I20240503 02:00:08.651710 13432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cea1ea50fa434d31b42451f84f26bc76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:08.651798 13420 ts_tablet_manager.cc:1909] T 3d3463f1ff2b41a396bb1b039ddf1946 P 2a01ace80baf4057b6bf1995cf88468b: Deleting consensus metadata
I20240503 02:00:08.652779 13379 tablet_copy_client.cc:783] T 1c721b33983b4c47afd8faa8f155ca84 P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:08.652930 13379 tablet_copy_client.cc:647] T 1c721b33983b4c47afd8faa8f155ca84 P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:08.654049 6300 catalog_manager.cc:4830] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079): tablet 3d3463f1ff2b41a396bb1b039ddf1946 (table rebalancer_dynamic_table_new_name [id=954039feb1f843f7adbb2096b370accb]) successfully deleted
I20240503 02:00:08.655045 13384 ts_tablet_manager.cc:902] T d73ea6e550a04c04a2695b058e38c566 P 9942306855404cfd891adf227dd65eb2: Initiating tablet copy from peer c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:08.655481 13384 tablet_copy_client.cc:308] T d73ea6e550a04c04a2695b058e38c566 P 9942306855404cfd891adf227dd65eb2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:42655
I20240503 02:00:08.656060 6739 tablet_copy_service.cc:140] P c2ac2eb7cc264047b9ab32fe3a4fc025: Received BeginTabletCopySession request for tablet d73ea6e550a04c04a2695b058e38c566 from peer 9942306855404cfd891adf227dd65eb2 ({username='slave'} at 127.0.99.133:46077)
I20240503 02:00:08.656150 6739 tablet_copy_service.cc:161] P c2ac2eb7cc264047b9ab32fe3a4fc025: Beginning new tablet copy session on tablet d73ea6e550a04c04a2695b058e38c566 from peer 9942306855404cfd891adf227dd65eb2 at {username='slave'} at 127.0.99.133:46077: session id = 9942306855404cfd891adf227dd65eb2-d73ea6e550a04c04a2695b058e38c566
I20240503 02:00:08.657145 6739 tablet_copy_source_session.cc:205] T d73ea6e550a04c04a2695b058e38c566 P c2ac2eb7cc264047b9ab32fe3a4fc025: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:08.657826 13379 tablet_copy_client.cc:523] T 1c721b33983b4c47afd8faa8f155ca84 P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:08.658231 13384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d73ea6e550a04c04a2695b058e38c566. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:08.659382 13379 tablet_bootstrap.cc:492] T 1c721b33983b4c47afd8faa8f155ca84 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:08.659428 13432 tablet_copy_client.cc:783] T cea1ea50fa434d31b42451f84f26bc76 P 492448570cc94ed1879feb2ec3b9051d: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:08.659610 13432 tablet_copy_client.cc:647] T cea1ea50fa434d31b42451f84f26bc76 P 492448570cc94ed1879feb2ec3b9051d: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:08.660444 13384 tablet_copy_client.cc:783] T d73ea6e550a04c04a2695b058e38c566 P 9942306855404cfd891adf227dd65eb2: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:08.660588 13384 tablet_copy_client.cc:647] T d73ea6e550a04c04a2695b058e38c566 P 9942306855404cfd891adf227dd65eb2: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:08.661484 13438 ts_tablet_manager.cc:902] T aa281822047841fab7c58b53d9733e49 P c2ac2eb7cc264047b9ab32fe3a4fc025: Initiating tablet copy from peer 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:08.662595 13432 tablet_copy_client.cc:523] T cea1ea50fa434d31b42451f84f26bc76 P 492448570cc94ed1879feb2ec3b9051d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:08.662869 13438 tablet_copy_client.cc:308] T aa281822047841fab7c58b53d9733e49 P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:46157
I20240503 02:00:08.663470 13384 tablet_copy_client.cc:523] T d73ea6e550a04c04a2695b058e38c566 P 9942306855404cfd891adf227dd65eb2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:08.664213 13432 tablet_bootstrap.cc:492] T cea1ea50fa434d31b42451f84f26bc76 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:08.664863 13384 tablet_bootstrap.cc:492] T d73ea6e550a04c04a2695b058e38c566 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:08.667474 9068 tablet_copy_service.cc:140] P 99ead02915954b4fab23b033ad6d546b: Received BeginTabletCopySession request for tablet aa281822047841fab7c58b53d9733e49 from peer c2ac2eb7cc264047b9ab32fe3a4fc025 ({username='slave'} at 127.0.99.131:52415)
I20240503 02:00:08.667596 9068 tablet_copy_service.cc:161] P 99ead02915954b4fab23b033ad6d546b: Beginning new tablet copy session on tablet aa281822047841fab7c58b53d9733e49 from peer c2ac2eb7cc264047b9ab32fe3a4fc025 at {username='slave'} at 127.0.99.131:52415: session id = c2ac2eb7cc264047b9ab32fe3a4fc025-aa281822047841fab7c58b53d9733e49
I20240503 02:00:08.667934 13379 tablet_bootstrap.cc:492] T 1c721b33983b4c47afd8faa8f155ca84 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:08.668197 13379 tablet_bootstrap.cc:492] T 1c721b33983b4c47afd8faa8f155ca84 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap complete.
I20240503 02:00:08.668367 13379 ts_tablet_manager.cc:1367] T 1c721b33983b4c47afd8faa8f155ca84 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.005s
I20240503 02:00:08.668574 9068 tablet_copy_source_session.cc:205] T aa281822047841fab7c58b53d9733e49 P 99ead02915954b4fab23b033ad6d546b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:08.669102 13379 raft_consensus.cc:348] T 1c721b33983b4c47afd8faa8f155ca84 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: true } }
I20240503 02:00:08.669236 13379 raft_consensus.cc:729] T 1c721b33983b4c47afd8faa8f155ca84 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: LEARNER
I20240503 02:00:08.669339 13379 consensus_queue.cc:260] T 1c721b33983b4c47afd8faa8f155ca84 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: true } }
I20240503 02:00:08.669755 13379 ts_tablet_manager.cc:1398] T 1c721b33983b4c47afd8faa8f155ca84 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:08.671543 13432 tablet_bootstrap.cc:492] T cea1ea50fa434d31b42451f84f26bc76 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:08.671916 13432 tablet_bootstrap.cc:492] T cea1ea50fa434d31b42451f84f26bc76 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap complete.
I20240503 02:00:08.672097 13432 ts_tablet_manager.cc:1367] T cea1ea50fa434d31b42451f84f26bc76 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240503 02:00:08.672691 13432 raft_consensus.cc:348] T cea1ea50fa434d31b42451f84f26bc76 P 492448570cc94ed1879feb2ec3b9051d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { promote: true } }
I20240503 02:00:08.672875 13432 raft_consensus.cc:729] T cea1ea50fa434d31b42451f84f26bc76 P 492448570cc94ed1879feb2ec3b9051d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: LEARNER
I20240503 02:00:08.672986 13432 consensus_queue.cc:260] T cea1ea50fa434d31b42451f84f26bc76 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { promote: true } }
I20240503 02:00:08.673336 13432 ts_tablet_manager.cc:1398] T cea1ea50fa434d31b42451f84f26bc76 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 02:00:08.676052 8937 tablet_copy_service.cc:342] P 9942306855404cfd891adf227dd65eb2: Request end of tablet copy session c2ac2eb7cc264047b9ab32fe3a4fc025-1c721b33983b4c47afd8faa8f155ca84 received from {username='slave'} at 127.0.99.131:37759
I20240503 02:00:08.676153 8937 tablet_copy_service.cc:433] P 9942306855404cfd891adf227dd65eb2: ending tablet copy session c2ac2eb7cc264047b9ab32fe3a4fc025-1c721b33983b4c47afd8faa8f155ca84 on tablet 1c721b33983b4c47afd8faa8f155ca84 with peer c2ac2eb7cc264047b9ab32fe3a4fc025
I20240503 02:00:08.676860 6739 tablet_copy_service.cc:342] P c2ac2eb7cc264047b9ab32fe3a4fc025: Request end of tablet copy session 492448570cc94ed1879feb2ec3b9051d-cea1ea50fa434d31b42451f84f26bc76 received from {username='slave'} at 127.0.99.132:42269
I20240503 02:00:08.676941 6739 tablet_copy_service.cc:433] P c2ac2eb7cc264047b9ab32fe3a4fc025: ending tablet copy session 492448570cc94ed1879feb2ec3b9051d-cea1ea50fa434d31b42451f84f26bc76 on tablet cea1ea50fa434d31b42451f84f26bc76 with peer 492448570cc94ed1879feb2ec3b9051d
I20240503 02:00:08.681908 13438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa281822047841fab7c58b53d9733e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:08.684514 13438 tablet_copy_client.cc:783] T aa281822047841fab7c58b53d9733e49 P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:08.684706 13438 tablet_copy_client.cc:647] T aa281822047841fab7c58b53d9733e49 P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:08.687947 13438 tablet_copy_client.cc:523] T aa281822047841fab7c58b53d9733e49 P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:08.689852 13438 tablet_bootstrap.cc:492] T aa281822047841fab7c58b53d9733e49 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:08.692778 13368 ts_tablet_manager.cc:902] T e52373099af3428497d176e3fcd8ef27 P 80fd18ef4d604c86b909d5c008a46b4f: Initiating tablet copy from peer 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:08.693498 13368 tablet_copy_client.cc:308] T e52373099af3428497d176e3fcd8ef27 P 80fd18ef4d604c86b909d5c008a46b4f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:46157
I20240503 02:00:08.694559 9068 tablet_copy_service.cc:140] P 99ead02915954b4fab23b033ad6d546b: Received BeginTabletCopySession request for tablet e52373099af3428497d176e3fcd8ef27 from peer 80fd18ef4d604c86b909d5c008a46b4f ({username='slave'} at 127.0.99.130:46681)
I20240503 02:00:08.694690 9068 tablet_copy_service.cc:161] P 99ead02915954b4fab23b033ad6d546b: Beginning new tablet copy session on tablet e52373099af3428497d176e3fcd8ef27 from peer 80fd18ef4d604c86b909d5c008a46b4f at {username='slave'} at 127.0.99.130:46681: session id = 80fd18ef4d604c86b909d5c008a46b4f-e52373099af3428497d176e3fcd8ef27
I20240503 02:00:08.695823 9068 tablet_copy_source_session.cc:205] T e52373099af3428497d176e3fcd8ef27 P 99ead02915954b4fab23b033ad6d546b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:08.697028 13368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e52373099af3428497d176e3fcd8ef27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:08.701377 13368 tablet_copy_client.cc:783] T e52373099af3428497d176e3fcd8ef27 P 80fd18ef4d604c86b909d5c008a46b4f: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:08.701594 13368 tablet_copy_client.cc:647] T e52373099af3428497d176e3fcd8ef27 P 80fd18ef4d604c86b909d5c008a46b4f: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:08.703267 13368 tablet_copy_client.cc:523] T e52373099af3428497d176e3fcd8ef27 P 80fd18ef4d604c86b909d5c008a46b4f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:08.705631 13368 tablet_bootstrap.cc:492] T e52373099af3428497d176e3fcd8ef27 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:08.707391 13384 tablet_bootstrap.cc:492] T d73ea6e550a04c04a2695b058e38c566 P 9942306855404cfd891adf227dd65eb2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:08.707659 13384 tablet_bootstrap.cc:492] T d73ea6e550a04c04a2695b058e38c566 P 9942306855404cfd891adf227dd65eb2: Bootstrap complete.
I20240503 02:00:08.707847 13384 ts_tablet_manager.cc:1367] T d73ea6e550a04c04a2695b058e38c566 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.043s user 0.023s sys 0.003s
I20240503 02:00:08.708335 13384 raft_consensus.cc:348] T d73ea6e550a04c04a2695b058e38c566 P 9942306855404cfd891adf227dd65eb2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } }
I20240503 02:00:08.708510 13384 raft_consensus.cc:729] T d73ea6e550a04c04a2695b058e38c566 P 9942306855404cfd891adf227dd65eb2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: LEARNER
I20240503 02:00:08.708616 13384 consensus_queue.cc:260] T d73ea6e550a04c04a2695b058e38c566 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } }
I20240503 02:00:08.709707 13438 tablet_bootstrap.cc:492] T aa281822047841fab7c58b53d9733e49 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:08.709956 13438 tablet_bootstrap.cc:492] T aa281822047841fab7c58b53d9733e49 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap complete.
I20240503 02:00:08.710125 13438 ts_tablet_manager.cc:1367] T aa281822047841fab7c58b53d9733e49 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
I20240503 02:00:08.710986 13438 raft_consensus.cc:348] T aa281822047841fab7c58b53d9733e49 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: true } }
I20240503 02:00:08.711115 13438 raft_consensus.cc:729] T aa281822047841fab7c58b53d9733e49 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: LEARNER
I20240503 02:00:08.711221 13438 consensus_queue.cc:260] T aa281822047841fab7c58b53d9733e49 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: true } }
I20240503 02:00:08.711519 13438 ts_tablet_manager.cc:1398] T aa281822047841fab7c58b53d9733e49 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:08.713073 9068 tablet_copy_service.cc:342] P 99ead02915954b4fab23b033ad6d546b: Request end of tablet copy session c2ac2eb7cc264047b9ab32fe3a4fc025-aa281822047841fab7c58b53d9733e49 received from {username='slave'} at 127.0.99.131:52415
I20240503 02:00:08.713156 9068 tablet_copy_service.cc:433] P 99ead02915954b4fab23b033ad6d546b: ending tablet copy session c2ac2eb7cc264047b9ab32fe3a4fc025-aa281822047841fab7c58b53d9733e49 on tablet aa281822047841fab7c58b53d9733e49 with peer c2ac2eb7cc264047b9ab32fe3a4fc025
I20240503 02:00:08.714386 13384 ts_tablet_manager.cc:1398] T d73ea6e550a04c04a2695b058e38c566 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:00:08.715127 6739 tablet_copy_service.cc:342] P c2ac2eb7cc264047b9ab32fe3a4fc025: Request end of tablet copy session 9942306855404cfd891adf227dd65eb2-d73ea6e550a04c04a2695b058e38c566 received from {username='slave'} at 127.0.99.133:46077
I20240503 02:00:08.715201 6739 tablet_copy_service.cc:433] P c2ac2eb7cc264047b9ab32fe3a4fc025: ending tablet copy session 9942306855404cfd891adf227dd65eb2-d73ea6e550a04c04a2695b058e38c566 on tablet d73ea6e550a04c04a2695b058e38c566 with peer 9942306855404cfd891adf227dd65eb2
I20240503 02:00:08.715673 13368 tablet_bootstrap.cc:492] T e52373099af3428497d176e3fcd8ef27 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:08.715922 13368 tablet_bootstrap.cc:492] T e52373099af3428497d176e3fcd8ef27 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap complete.
I20240503 02:00:08.716101 13368 ts_tablet_manager.cc:1367] T e52373099af3428497d176e3fcd8ef27 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.002s
I20240503 02:00:08.716837 13368 raft_consensus.cc:348] T e52373099af3428497d176e3fcd8ef27 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: true } }
I20240503 02:00:08.716969 13368 raft_consensus.cc:729] T e52373099af3428497d176e3fcd8ef27 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: LEARNER
I20240503 02:00:08.717115 13368 consensus_queue.cc:260] T e52373099af3428497d176e3fcd8ef27 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: true } }
I20240503 02:00:08.718312 13368 ts_tablet_manager.cc:1398] T e52373099af3428497d176e3fcd8ef27 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:00:08.722710 9068 tablet_copy_service.cc:342] P 99ead02915954b4fab23b033ad6d546b: Request end of tablet copy session 80fd18ef4d604c86b909d5c008a46b4f-e52373099af3428497d176e3fcd8ef27 received from {username='slave'} at 127.0.99.130:46681
I20240503 02:00:08.722811 9068 tablet_copy_service.cc:433] P 99ead02915954b4fab23b033ad6d546b: ending tablet copy session 80fd18ef4d604c86b909d5c008a46b4f-e52373099af3428497d176e3fcd8ef27 on tablet e52373099af3428497d176e3fcd8ef27 with peer 80fd18ef4d604c86b909d5c008a46b4f
I20240503 02:00:08.739171 6313 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:00:08.746126 6569 tablet_service.cc:1467] Processing CreateTablet for tablet 7a90fdf10e3c4dd8be7d5e1b54e9a1da (DEFAULT_TABLE table=rebalancer_dynamic_table [id=73d45da493d442e489ee6add747cad2b]), partition=
I20240503 02:00:08.746225 9158 tablet_service.cc:1467] Processing CreateTablet for tablet 7a90fdf10e3c4dd8be7d5e1b54e9a1da (DEFAULT_TABLE table=rebalancer_dynamic_table [id=73d45da493d442e489ee6add747cad2b]), partition=
I20240503 02:00:08.746439 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a90fdf10e3c4dd8be7d5e1b54e9a1da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:08.746470 9158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a90fdf10e3c4dd8be7d5e1b54e9a1da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:08.747480 9028 tablet_service.cc:1467] Processing CreateTablet for tablet 7a90fdf10e3c4dd8be7d5e1b54e9a1da (DEFAULT_TABLE table=rebalancer_dynamic_table [id=73d45da493d442e489ee6add747cad2b]), partition=
I20240503 02:00:08.747778 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a90fdf10e3c4dd8be7d5e1b54e9a1da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:08.748524 13280 tablet_bootstrap.cc:492] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:08.749509 13280 tablet_bootstrap.cc:654] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:08.749797 13279 tablet_bootstrap.cc:492] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:08.749857 13181 tablet_bootstrap.cc:492] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:08.750715 13279 tablet_bootstrap.cc:654] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:08.750777 13181 tablet_bootstrap.cc:654] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:08.751785 13279 tablet_bootstrap.cc:492] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:08.751896 13279 ts_tablet_manager.cc:1367] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:08.752408 13280 tablet_bootstrap.cc:492] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:08.752362 13279 raft_consensus.cc:348] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:08.752501 13280 ts_tablet_manager.cc:1367] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:08.752533 13279 raft_consensus.cc:374] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:08.752568 13279 raft_consensus.cc:729] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:08.752698 13279 consensus_queue.cc:260] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:08.753000 13279 ts_tablet_manager.cc:1398] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240503 02:00:08.752959 13280 raft_consensus.cc:348] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:08.753062 13280 raft_consensus.cc:374] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:08.753095 13280 raft_consensus.cc:729] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:08.753214 13280 consensus_queue.cc:260] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:08.753477 13280 ts_tablet_manager.cc:1398] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:08.755388 12922 consensus_queue.cc:1035] T 539e7e4ef5a742c094bd5e4949493ac6 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:08.755553 13181 tablet_bootstrap.cc:492] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:08.755649 13181 ts_tablet_manager.cc:1367] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 02:00:08.756147 13181 raft_consensus.cc:348] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:08.756254 13181 raft_consensus.cc:374] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:08.756284 13181 raft_consensus.cc:729] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:08.759865 12922 consensus_queue.cc:1035] T 539e7e4ef5a742c094bd5e4949493ac6 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:08.766312 13181 consensus_queue.cc:260] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:08.766957 13181 ts_tablet_manager.cc:1398] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240503 02:00:08.768507 13492 ts_tablet_manager.cc:902] T ac7fa0c3722a461c91fb237fd85794a3 P 2a01ace80baf4057b6bf1995cf88468b: Initiating tablet copy from peer dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 02:00:08.768534 13499 ts_tablet_manager.cc:902] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 2a01ace80baf4057b6bf1995cf88468b: Initiating tablet copy from peer dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 02:00:08.769066 13492 tablet_copy_client.cc:308] T ac7fa0c3722a461c91fb237fd85794a3 P 2a01ace80baf4057b6bf1995cf88468b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:33093
I20240503 02:00:08.769651 6478 tablet_copy_service.cc:140] P dec3e4a04d604f7eaaa3f077d081218a: Received BeginTabletCopySession request for tablet ac7fa0c3722a461c91fb237fd85794a3 from peer 2a01ace80baf4057b6bf1995cf88468b ({username='slave'} at 127.0.99.135:56449)
I20240503 02:00:08.769760 6478 tablet_copy_service.cc:161] P dec3e4a04d604f7eaaa3f077d081218a: Beginning new tablet copy session on tablet ac7fa0c3722a461c91fb237fd85794a3 from peer 2a01ace80baf4057b6bf1995cf88468b at {username='slave'} at 127.0.99.135:56449: session id = 2a01ace80baf4057b6bf1995cf88468b-ac7fa0c3722a461c91fb237fd85794a3
I20240503 02:00:08.769066 13499 tablet_copy_client.cc:308] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 2a01ace80baf4057b6bf1995cf88468b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:33093
I20240503 02:00:08.770449 6477 tablet_copy_service.cc:140] P dec3e4a04d604f7eaaa3f077d081218a: Received BeginTabletCopySession request for tablet 2d7244b0dc4d419cb5c3f3677acd70c7 from peer 2a01ace80baf4057b6bf1995cf88468b ({username='slave'} at 127.0.99.135:56449)
I20240503 02:00:08.770539 6477 tablet_copy_service.cc:161] P dec3e4a04d604f7eaaa3f077d081218a: Beginning new tablet copy session on tablet 2d7244b0dc4d419cb5c3f3677acd70c7 from peer 2a01ace80baf4057b6bf1995cf88468b at {username='slave'} at 127.0.99.135:56449: session id = 2a01ace80baf4057b6bf1995cf88468b-2d7244b0dc4d419cb5c3f3677acd70c7
I20240503 02:00:08.770730 6478 tablet_copy_source_session.cc:205] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:08.771488 6477 tablet_copy_source_session.cc:205] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:08.771605 13492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac7fa0c3722a461c91fb237fd85794a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:08.773108 13499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d7244b0dc4d419cb5c3f3677acd70c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:08.774433 13492 tablet_copy_client.cc:783] T ac7fa0c3722a461c91fb237fd85794a3 P 2a01ace80baf4057b6bf1995cf88468b: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:08.774583 13492 tablet_copy_client.cc:647] T ac7fa0c3722a461c91fb237fd85794a3 P 2a01ace80baf4057b6bf1995cf88468b: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:08.775226 13499 tablet_copy_client.cc:783] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 2a01ace80baf4057b6bf1995cf88468b: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:08.775362 13499 tablet_copy_client.cc:647] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 2a01ace80baf4057b6bf1995cf88468b: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:08.777420 13492 tablet_copy_client.cc:523] T ac7fa0c3722a461c91fb237fd85794a3 P 2a01ace80baf4057b6bf1995cf88468b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:08.778443 13499 tablet_copy_client.cc:523] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 2a01ace80baf4057b6bf1995cf88468b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:08.778909 13492 tablet_bootstrap.cc:492] T ac7fa0c3722a461c91fb237fd85794a3 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:08.779927 13499 tablet_bootstrap.cc:492] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:08.813262 13368 ts_tablet_manager.cc:902] T 9cbb2be1fda74ebe8997123c70629ba3 P 80fd18ef4d604c86b909d5c008a46b4f: Initiating tablet copy from peer dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 02:00:08.814407 13368 tablet_copy_client.cc:308] T 9cbb2be1fda74ebe8997123c70629ba3 P 80fd18ef4d604c86b909d5c008a46b4f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:33093
I20240503 02:00:08.815287 6478 tablet_copy_service.cc:140] P dec3e4a04d604f7eaaa3f077d081218a: Received BeginTabletCopySession request for tablet 9cbb2be1fda74ebe8997123c70629ba3 from peer 80fd18ef4d604c86b909d5c008a46b4f ({username='slave'} at 127.0.99.130:49797)
I20240503 02:00:08.815533 6478 tablet_copy_service.cc:161] P dec3e4a04d604f7eaaa3f077d081218a: Beginning new tablet copy session on tablet 9cbb2be1fda74ebe8997123c70629ba3 from peer 80fd18ef4d604c86b909d5c008a46b4f at {username='slave'} at 127.0.99.130:49797: session id = 80fd18ef4d604c86b909d5c008a46b4f-9cbb2be1fda74ebe8997123c70629ba3
I20240503 02:00:08.817364 6478 tablet_copy_source_session.cc:205] T 9cbb2be1fda74ebe8997123c70629ba3 P dec3e4a04d604f7eaaa3f077d081218a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:08.819790 13368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cbb2be1fda74ebe8997123c70629ba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:08.821419 13492 tablet_bootstrap.cc:492] T ac7fa0c3722a461c91fb237fd85794a3 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:08.822979 13499 tablet_bootstrap.cc:492] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:08.823426 13492 tablet_bootstrap.cc:492] T ac7fa0c3722a461c91fb237fd85794a3 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap complete.
I20240503 02:00:08.823665 13492 ts_tablet_manager.cc:1367] T ac7fa0c3722a461c91fb237fd85794a3 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.045s user 0.029s sys 0.000s
I20240503 02:00:08.824198 13492 raft_consensus.cc:348] T ac7fa0c3722a461c91fb237fd85794a3 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } }
I20240503 02:00:08.824445 13492 raft_consensus.cc:729] T ac7fa0c3722a461c91fb237fd85794a3 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: LEARNER
I20240503 02:00:08.824612 13492 consensus_queue.cc:260] T ac7fa0c3722a461c91fb237fd85794a3 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } }
I20240503 02:00:08.824699 13499 tablet_bootstrap.cc:492] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap complete.
I20240503 02:00:08.824967 13499 ts_tablet_manager.cc:1367] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.045s user 0.019s sys 0.009s
I20240503 02:00:08.825274 13368 tablet_copy_client.cc:783] T 9cbb2be1fda74ebe8997123c70629ba3 P 80fd18ef4d604c86b909d5c008a46b4f: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:08.825421 13492 ts_tablet_manager.cc:1398] T ac7fa0c3722a461c91fb237fd85794a3 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:08.825450 13368 tablet_copy_client.cc:647] T 9cbb2be1fda74ebe8997123c70629ba3 P 80fd18ef4d604c86b909d5c008a46b4f: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:08.825896 13499 raft_consensus.cc:348] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } }
I20240503 02:00:08.825963 6478 tablet_copy_service.cc:342] P dec3e4a04d604f7eaaa3f077d081218a: Request end of tablet copy session 2a01ace80baf4057b6bf1995cf88468b-ac7fa0c3722a461c91fb237fd85794a3 received from {username='slave'} at 127.0.99.135:56449
I20240503 02:00:08.826002 13499 raft_consensus.cc:729] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: LEARNER
I20240503 02:00:08.826035 6478 tablet_copy_service.cc:433] P dec3e4a04d604f7eaaa3f077d081218a: ending tablet copy session 2a01ace80baf4057b6bf1995cf88468b-ac7fa0c3722a461c91fb237fd85794a3 on tablet ac7fa0c3722a461c91fb237fd85794a3 with peer 2a01ace80baf4057b6bf1995cf88468b
I20240503 02:00:08.826093 13499 consensus_queue.cc:260] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: true } }
I20240503 02:00:08.826422 13499 ts_tablet_manager.cc:1398] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:08.826946 6478 tablet_copy_service.cc:342] P dec3e4a04d604f7eaaa3f077d081218a: Request end of tablet copy session 2a01ace80baf4057b6bf1995cf88468b-2d7244b0dc4d419cb5c3f3677acd70c7 received from {username='slave'} at 127.0.99.135:56449
I20240503 02:00:08.827028 6478 tablet_copy_service.cc:433] P dec3e4a04d604f7eaaa3f077d081218a: ending tablet copy session 2a01ace80baf4057b6bf1995cf88468b-2d7244b0dc4d419cb5c3f3677acd70c7 on tablet 2d7244b0dc4d419cb5c3f3677acd70c7 with peer 2a01ace80baf4057b6bf1995cf88468b
I20240503 02:00:08.833557 13368 tablet_copy_client.cc:523] T 9cbb2be1fda74ebe8997123c70629ba3 P 80fd18ef4d604c86b909d5c008a46b4f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:08.835211 13368 tablet_bootstrap.cc:492] T 9cbb2be1fda74ebe8997123c70629ba3 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:08.838656 13368 tablet_bootstrap.cc:492] T 9cbb2be1fda74ebe8997123c70629ba3 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:08.838878 13368 tablet_bootstrap.cc:492] T 9cbb2be1fda74ebe8997123c70629ba3 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap complete.
I20240503 02:00:08.839030 13368 ts_tablet_manager.cc:1367] T 9cbb2be1fda74ebe8997123c70629ba3 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240503 02:00:08.839439 13368 raft_consensus.cc:348] T 9cbb2be1fda74ebe8997123c70629ba3 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: true } }
I20240503 02:00:08.839546 13368 raft_consensus.cc:729] T 9cbb2be1fda74ebe8997123c70629ba3 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: LEARNER
I20240503 02:00:08.839630 13368 consensus_queue.cc:260] T 9cbb2be1fda74ebe8997123c70629ba3 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: true } }
I20240503 02:00:08.839880 13368 ts_tablet_manager.cc:1398] T 9cbb2be1fda74ebe8997123c70629ba3 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:08.840781 6478 tablet_copy_service.cc:342] P dec3e4a04d604f7eaaa3f077d081218a: Request end of tablet copy session 80fd18ef4d604c86b909d5c008a46b4f-9cbb2be1fda74ebe8997123c70629ba3 received from {username='slave'} at 127.0.99.130:49797
I20240503 02:00:08.840869 6478 tablet_copy_service.cc:433] P dec3e4a04d604f7eaaa3f077d081218a: ending tablet copy session 80fd18ef4d604c86b909d5c008a46b4f-9cbb2be1fda74ebe8997123c70629ba3 on tablet 9cbb2be1fda74ebe8997123c70629ba3 with peer 80fd18ef4d604c86b909d5c008a46b4f
I20240503 02:00:08.867733 12907 consensus_queue.cc:1035] T c70e6cff9fa440219007a0af1c43cdb9 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:08.872577 13098 consensus_queue.cc:1035] T c70e6cff9fa440219007a0af1c43cdb9 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:00:08.915776 6654 connection.cc:535] client connection to 127.0.99.136:45311 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:00:08.915958 8982 connection.cc:535] client connection to 127.0.99.136:45311 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:00:08.915968 6654 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:00:08.916054 8982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:00:08.916496 8851 connection.cc:535] client connection to 127.0.99.136:45311 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:00:08.916630 8851 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:00:08.916838 6785 connection.cc:535] client connection to 127.0.99.136:45311 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:00:08.916980 6785 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:00:08.917771 6523 connection.cc:535] client connection to 127.0.99.136:45311 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:00:08.917897 6523 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:00:08.918296 6301 connection.cc:535] client connection to 127.0.99.136:45311 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:00:08.918398 6301 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:00:08.918707 13525 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 02:00:08.916531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.136:45311 (local address 127.0.99.134:50105)
0503 02:00:08.917147 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0503 02:00:08.917167 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0503 02:00:08.917276 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 02:00:08.918163 (+ 887us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":48,"threads_started":1}
W20240503 02:00:08.918820 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:00:08.918936 8982 consensus_peers.cc:485] T aa281822047841fab7c58b53d9733e49 P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:08.919602 6785 consensus_peers.cc:485] T 46ad540deb624a25b97d3b967fe1f6f4 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:08.919689 6785 consensus_peers.cc:485] T 8246488dec204bf6b09e868d7951ac5a P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:08.919736 6785 consensus_peers.cc:485] T c3485701bf3b48f5ae2e1568afeed133 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:08.920523 8982 consensus_peers.cc:485] T 13756a6e44814fe4b09a6b0f087546c3 P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:08.920995 6301 catalog_manager.cc:4573] TS e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3d3463f1ff2b41a396bb1b039ddf1946: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
W20240503 02:00:08.921345 8851 consensus_peers.cc:485] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:08.921420 8851 consensus_peers.cc:485] T 7da8f614a70e4177b70e7c2f31d22a10 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:08.921456 8851 consensus_peers.cc:485] T 067f54011a37432d99157884e70fc601 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:08.921483 13528 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 02:00:08.917355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.136:45311 (local address 127.0.99.131:54921)
0503 02:00:08.920860 (+ 3505us) negotiation.cc:107] Waiting for socket to connect
0503 02:00:08.920922 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":3435,"thread_start_us":67,"threads_started":1}
W20240503 02:00:08.921641 6654 consensus_peers.cc:485] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:08.921694 6654 consensus_peers.cc:485] T ba17d4d3e0754b2798eb95484d37229a P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:08.922113 6523 consensus_peers.cc:485] T 925612b660ec471da5b3c548802ede6e P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:08.922217 6523 consensus_peers.cc:485] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:08.922269 6523 consensus_peers.cc:485] T 4385d51d11574192883d01376e2d5620 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:08.922324 6523 consensus_peers.cc:485] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:08.922365 6523 consensus_peers.cc:485] T d0d5dd42db3e4883af7fd16d8813b550 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:08.924978 6582 raft_consensus.cc:1203] T beeea71f02ae4c89a1af13d3584ef70a P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240503 02:00:08.927146 9113 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
W20240503 02:00:08.928056 9113 consensus_peers.cc:485] T f44b59277c3b47adbd4dd6aff9350af1 P 2a01ace80baf4057b6bf1995cf88468b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:08.947449 6456 raft_consensus.cc:1203] T b20ff3a6f74f4640afebfd4059b58699 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240503 02:00:08.958477 6785 consensus_peers.cc:485] T d897c21e630d4770832bb10f43778e82 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:08.959136 6720 raft_consensus.cc:1203] T 96c395771fd44056abacceb6c1acdb9f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:00:08.970520 13283 raft_consensus.cc:482] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:08.970656 13283 raft_consensus.cc:504] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:08.971105 13283 leader_election.cc:288] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:08.971529 9039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a90fdf10e3c4dd8be7d5e1b54e9a1da" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:08.971693 9039 raft_consensus.cc:2454] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:08.971701 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a90fdf10e3c4dd8be7d5e1b54e9a1da" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:08.971832 9179 raft_consensus.cc:2454] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:08.972149 6522 leader_election.cc:302] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 80fd18ef4d604c86b909d5c008a46b4f; no voters:
I20240503 02:00:08.972337 13283 raft_consensus.cc:2781] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:08.972415 13283 raft_consensus.cc:482] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:08.972448 13283 raft_consensus.cc:3037] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:08.973605 13283 raft_consensus.cc:504] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:08.973984 13283 leader_election.cc:288] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Requested vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:08.974339 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a90fdf10e3c4dd8be7d5e1b54e9a1da" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:08.974476 9179 raft_consensus.cc:3037] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:08.975514 9179 raft_consensus.cc:2454] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:08.976018 6522 leader_election.cc:302] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 80fd18ef4d604c86b909d5c008a46b4f; no voters:
I20240503 02:00:08.976001 9039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a90fdf10e3c4dd8be7d5e1b54e9a1da" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:08.976186 9039 raft_consensus.cc:3037] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:08.976418 13283 raft_consensus.cc:2781] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:08.976500 13283 raft_consensus.cc:686] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Becoming Leader. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Running, Role: LEADER
I20240503 02:00:08.976629 13283 consensus_queue.cc:237] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:08.978034 9039 raft_consensus.cc:2454] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:08.978868 6307 catalog_manager.cc:5484] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } }
W20240503 02:00:08.984606 6785 consensus_peers.cc:485] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:08.990481 8982 consensus_peers.cc:485] T 1a7ee79312ea4bad816264375ca5bb2d P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:09.026587 6720 raft_consensus.cc:1203] T 1c721b33983b4c47afd8faa8f155ca84 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
W20240503 02:00:09.035079 8851 consensus_peers.cc:485] T 5f51fe5f799f4e479b62f5917a2c2606 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:09.062888 9158 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:00:09.065492 8887 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 02:00:09.066459 8851 consensus_peers.cc:485] T f28586e70bd9473fbdc6b9126b666fa9 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:09.066998 9028 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:00:09.072605 10679 consensus_queue.cc:1035] T 5d82d1279c8f4eea9383cfbe7705aeb8 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:09.072821 6831 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:00:09.080992 6436 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:00:09.084062 6700 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:00:09.084510 12907 consensus_queue.cc:1035] T 5d82d1279c8f4eea9383cfbe7705aeb8 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:09.085722 6309 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_extra_table_24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:09.103116 6309 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:09.107295 8887 tablet_service.cc:1467] Processing CreateTablet for tablet be7bbd71b99847068f415cc3d3d49ddd (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=ee48cd7f729e43809c0e360005b8f974]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:09.107640 8887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be7bbd71b99847068f415cc3d3d49ddd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:09.109068 6436 tablet_service.cc:1467] Processing CreateTablet for tablet be7bbd71b99847068f415cc3d3d49ddd (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=ee48cd7f729e43809c0e360005b8f974]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:09.109232 9028 tablet_service.cc:1467] Processing CreateTablet for tablet 7cecefe8f1864583be9d0b9c308a78db (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=ee48cd7f729e43809c0e360005b8f974]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:09.109344 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be7bbd71b99847068f415cc3d3d49ddd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:09.109500 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cecefe8f1864583be9d0b9c308a78db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:09.110389 13606 tablet_bootstrap.cc:492] T be7bbd71b99847068f415cc3d3d49ddd P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:09.111475 13606 tablet_bootstrap.cc:654] T be7bbd71b99847068f415cc3d3d49ddd P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:09.111630 13279 tablet_bootstrap.cc:492] T 7cecefe8f1864583be9d0b9c308a78db P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:09.112665 13279 tablet_bootstrap.cc:654] T 7cecefe8f1864583be9d0b9c308a78db P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:09.112679 9158 tablet_service.cc:1467] Processing CreateTablet for tablet 7cecefe8f1864583be9d0b9c308a78db (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=ee48cd7f729e43809c0e360005b8f974]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:09.112943 9158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cecefe8f1864583be9d0b9c308a78db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:09.113080 9157 tablet_service.cc:1467] Processing CreateTablet for tablet e617d4e31eed42dcb39e03cc40b8e6a6 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=ee48cd7f729e43809c0e360005b8f974]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:09.113318 9157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e617d4e31eed42dcb39e03cc40b8e6a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:09.115540 13181 tablet_bootstrap.cc:492] T 7cecefe8f1864583be9d0b9c308a78db P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:09.115960 6831 tablet_service.cc:1467] Processing CreateTablet for tablet 7cecefe8f1864583be9d0b9c308a78db (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=ee48cd7f729e43809c0e360005b8f974]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:09.116218 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cecefe8f1864583be9d0b9c308a78db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:09.116493 13181 tablet_bootstrap.cc:654] T 7cecefe8f1864583be9d0b9c308a78db P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:09.116672 6699 tablet_service.cc:1467] Processing CreateTablet for tablet e617d4e31eed42dcb39e03cc40b8e6a6 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=ee48cd7f729e43809c0e360005b8f974]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:09.116824 6700 tablet_service.cc:1467] Processing CreateTablet for tablet be7bbd71b99847068f415cc3d3d49ddd (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=ee48cd7f729e43809c0e360005b8f974]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:09.116936 6699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e617d4e31eed42dcb39e03cc40b8e6a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:09.117048 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be7bbd71b99847068f415cc3d3d49ddd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:09.118018 13181 tablet_bootstrap.cc:492] T 7cecefe8f1864583be9d0b9c308a78db P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:09.118115 13181 ts_tablet_manager.cc:1367] T 7cecefe8f1864583be9d0b9c308a78db P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:00:09.118678 13181 raft_consensus.cc:348] T 7cecefe8f1864583be9d0b9c308a78db P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:09.118794 13181 raft_consensus.cc:374] T 7cecefe8f1864583be9d0b9c308a78db P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:09.118826 13181 raft_consensus.cc:729] T 7cecefe8f1864583be9d0b9c308a78db P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:09.118953 13181 consensus_queue.cc:260] T 7cecefe8f1864583be9d0b9c308a78db P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:09.119253 13181 ts_tablet_manager.cc:1398] T 7cecefe8f1864583be9d0b9c308a78db P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240503 02:00:09.119277 8887 tablet_service.cc:1467] Processing CreateTablet for tablet e617d4e31eed42dcb39e03cc40b8e6a6 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=ee48cd7f729e43809c0e360005b8f974]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:09.119429 13181 tablet_bootstrap.cc:492] T e617d4e31eed42dcb39e03cc40b8e6a6 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:09.119534 8887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e617d4e31eed42dcb39e03cc40b8e6a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:09.122236 9039 raft_consensus.cc:1261] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:09.122696 13283 consensus_queue.cc:1035] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:09.126760 9179 raft_consensus.cc:1261] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:09.127535 13283 consensus_queue.cc:1035] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:09.127807 6309 catalog_manager.cc:5576] TS 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157) does not have the latest schema for tablet 7a90fdf10e3c4dd8be7d5e1b54e9a1da (table rebalancer_dynamic_table [id=73d45da493d442e489ee6add747cad2b]). Expected version 1 got 0
W20240503 02:00:09.128683 8982 consensus_peers.cc:485] T beeea71f02ae4c89a1af13d3584ef70a P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:09.130280 13612 tablet_bootstrap.cc:492] T e617d4e31eed42dcb39e03cc40b8e6a6 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:09.130544 6569 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:00:09.135675 6312 catalog_manager.cc:5576] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863) does not have the latest schema for tablet 7a90fdf10e3c4dd8be7d5e1b54e9a1da (table rebalancer_dynamic_table [id=73d45da493d442e489ee6add747cad2b]). Expected version 1 got 0
I20240503 02:00:09.136145 6314 catalog_manager.cc:5576] TS 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157) does not have the latest schema for tablet 7a90fdf10e3c4dd8be7d5e1b54e9a1da (table rebalancer_dynamic_table [id=73d45da493d442e489ee6add747cad2b]). Expected version 1 got 0
I20240503 02:00:09.120352 13181 tablet_bootstrap.cc:654] T e617d4e31eed42dcb39e03cc40b8e6a6 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:09.139051 6582 raft_consensus.cc:1203] T e52373099af3428497d176e3fcd8ef27 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:00:09.140420 13612 tablet_bootstrap.cc:654] T e617d4e31eed42dcb39e03cc40b8e6a6 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:09.142133 13612 tablet_bootstrap.cc:492] T e617d4e31eed42dcb39e03cc40b8e6a6 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:09.142251 13612 ts_tablet_manager.cc:1367] T e617d4e31eed42dcb39e03cc40b8e6a6 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20240503 02:00:09.142776 13612 raft_consensus.cc:348] T e617d4e31eed42dcb39e03cc40b8e6a6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:09.142903 13612 raft_consensus.cc:374] T e617d4e31eed42dcb39e03cc40b8e6a6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:09.142940 13612 raft_consensus.cc:729] T e617d4e31eed42dcb39e03cc40b8e6a6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:09.143074 13612 consensus_queue.cc:260] T e617d4e31eed42dcb39e03cc40b8e6a6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:09.143394 13612 ts_tablet_manager.cc:1398] T e617d4e31eed42dcb39e03cc40b8e6a6 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:09.143843 6309 catalog_manager.cc:5576] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079) does not have the latest schema for tablet 7a90fdf10e3c4dd8be7d5e1b54e9a1da (table rebalancer_dynamic_table [id=73d45da493d442e489ee6add747cad2b]). Expected version 1 got 0
I20240503 02:00:09.144359 13612 tablet_bootstrap.cc:492] T be7bbd71b99847068f415cc3d3d49ddd P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:09.148178 13612 tablet_bootstrap.cc:654] T be7bbd71b99847068f415cc3d3d49ddd P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:09.148519 13616 tablet.cc:1722] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:09.150081 13616 tablet.cc:1620] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 99ead02915954b4fab23b033ad6d546b: MemRowSet was empty: no flush needed.
I20240503 02:00:09.150245 9176 raft_consensus.cc:1203] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 02:00:09.155124 13607 tablet_bootstrap.cc:492] T be7bbd71b99847068f415cc3d3d49ddd P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:09.156265 13607 tablet_bootstrap.cc:654] T be7bbd71b99847068f415cc3d3d49ddd P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:09.157723 13279 tablet_bootstrap.cc:492] T 7cecefe8f1864583be9d0b9c308a78db P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:09.157856 13279 ts_tablet_manager.cc:1367] T 7cecefe8f1864583be9d0b9c308a78db P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.046s user 0.000s sys 0.002s
I20240503 02:00:09.158339 13279 raft_consensus.cc:348] T 7cecefe8f1864583be9d0b9c308a78db P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:09.158455 13279 raft_consensus.cc:374] T 7cecefe8f1864583be9d0b9c308a78db P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:09.158488 13279 raft_consensus.cc:729] T 7cecefe8f1864583be9d0b9c308a78db P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:09.158598 13279 consensus_queue.cc:260] T 7cecefe8f1864583be9d0b9c308a78db P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:09.159039 13279 ts_tablet_manager.cc:1398] T 7cecefe8f1864583be9d0b9c308a78db P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:09.159433 13611 tablet_bootstrap.cc:492] T 7cecefe8f1864583be9d0b9c308a78db P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:09.160411 13611 tablet_bootstrap.cc:654] T 7cecefe8f1864583be9d0b9c308a78db P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:09.161604 13611 tablet_bootstrap.cc:492] T 7cecefe8f1864583be9d0b9c308a78db P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:09.161697 13611 ts_tablet_manager.cc:1367] T 7cecefe8f1864583be9d0b9c308a78db P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:00:09.162115 13611 raft_consensus.cc:348] T 7cecefe8f1864583be9d0b9c308a78db P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:09.162225 13611 raft_consensus.cc:374] T 7cecefe8f1864583be9d0b9c308a78db P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:09.162252 13611 raft_consensus.cc:729] T 7cecefe8f1864583be9d0b9c308a78db P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:09.162371 13611 consensus_queue.cc:260] T 7cecefe8f1864583be9d0b9c308a78db P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:09.162608 13611 ts_tablet_manager.cc:1398] T 7cecefe8f1864583be9d0b9c308a78db P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:09.163249 6720 raft_consensus.cc:1203] T aa281822047841fab7c58b53d9733e49 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:00:09.166579 6314 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=73d45da493d442e489ee6add747cad2b] alter complete (version 1)
I20240503 02:00:09.171067 13607 tablet_bootstrap.cc:492] T be7bbd71b99847068f415cc3d3d49ddd P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:09.171188 13607 ts_tablet_manager.cc:1367] T be7bbd71b99847068f415cc3d3d49ddd P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240503 02:00:09.171684 13607 raft_consensus.cc:348] T be7bbd71b99847068f415cc3d3d49ddd P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:09.171829 13607 raft_consensus.cc:374] T be7bbd71b99847068f415cc3d3d49ddd P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:09.171859 13607 raft_consensus.cc:729] T be7bbd71b99847068f415cc3d3d49ddd P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:09.171965 13607 consensus_queue.cc:260] T be7bbd71b99847068f415cc3d3d49ddd P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:09.172252 13607 ts_tablet_manager.cc:1398] T be7bbd71b99847068f415cc3d3d49ddd P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:09.174239 13615 tablet.cc:1722] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:09.174346 13612 tablet_bootstrap.cc:492] T be7bbd71b99847068f415cc3d3d49ddd P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:09.174453 13612 ts_tablet_manager.cc:1367] T be7bbd71b99847068f415cc3d3d49ddd P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.030s user 0.000s sys 0.003s
I20240503 02:00:09.174495 13181 tablet_bootstrap.cc:492] T e617d4e31eed42dcb39e03cc40b8e6a6 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:09.174588 13181 ts_tablet_manager.cc:1367] T e617d4e31eed42dcb39e03cc40b8e6a6 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.055s user 0.001s sys 0.001s
I20240503 02:00:09.174876 13615 tablet.cc:1620] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 80fd18ef4d604c86b909d5c008a46b4f: MemRowSet was empty: no flush needed.
I20240503 02:00:09.175071 13181 raft_consensus.cc:348] T e617d4e31eed42dcb39e03cc40b8e6a6 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:09.175186 13181 raft_consensus.cc:374] T e617d4e31eed42dcb39e03cc40b8e6a6 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:09.175216 13181 raft_consensus.cc:729] T e617d4e31eed42dcb39e03cc40b8e6a6 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:09.175338 13181 consensus_queue.cc:260] T e617d4e31eed42dcb39e03cc40b8e6a6 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:09.175380 13612 raft_consensus.cc:348] T be7bbd71b99847068f415cc3d3d49ddd P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:09.175482 13612 raft_consensus.cc:374] T be7bbd71b99847068f415cc3d3d49ddd P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:09.175508 13612 raft_consensus.cc:729] T be7bbd71b99847068f415cc3d3d49ddd P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:09.175606 13181 ts_tablet_manager.cc:1398] T e617d4e31eed42dcb39e03cc40b8e6a6 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:09.175647 13612 consensus_queue.cc:260] T be7bbd71b99847068f415cc3d3d49ddd P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:09.175988 13612 ts_tablet_manager.cc:1398] T be7bbd71b99847068f415cc3d3d49ddd P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:09.181272 6850 raft_consensus.cc:1203] T cea1ea50fa434d31b42451f84f26bc76 P 492448570cc94ed1879feb2ec3b9051d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:00:09.184686 6313 catalog_manager.cc:5576] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079) does not have the latest schema for tablet 7a90fdf10e3c4dd8be7d5e1b54e9a1da (table rebalancer_dynamic_table [id=73d45da493d442e489ee6add747cad2b]). Expected version 1 got 0
W20240503 02:00:09.186353 9113 consensus_peers.cc:485] T b1740b877c1e4372a0337d99f5bcab0e P 2a01ace80baf4057b6bf1995cf88468b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:09.189260 13622 tablet.cc:1722] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:09.189879 13622 tablet.cc:1620] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
I20240503 02:00:09.192962 13606 tablet_bootstrap.cc:492] T be7bbd71b99847068f415cc3d3d49ddd P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:09.193097 13606 ts_tablet_manager.cc:1367] T be7bbd71b99847068f415cc3d3d49ddd P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.083s user 0.002s sys 0.000s
I20240503 02:00:09.193598 13606 raft_consensus.cc:348] T be7bbd71b99847068f415cc3d3d49ddd P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:09.193722 13606 raft_consensus.cc:374] T be7bbd71b99847068f415cc3d3d49ddd P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:09.193753 13606 raft_consensus.cc:729] T be7bbd71b99847068f415cc3d3d49ddd P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:09.193872 13606 consensus_queue.cc:260] T be7bbd71b99847068f415cc3d3d49ddd P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:09.194173 13606 ts_tablet_manager.cc:1398] T be7bbd71b99847068f415cc3d3d49ddd P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:09.194347 13606 tablet_bootstrap.cc:492] T e617d4e31eed42dcb39e03cc40b8e6a6 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:09.195298 13606 tablet_bootstrap.cc:654] T e617d4e31eed42dcb39e03cc40b8e6a6 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
W20240503 02:00:09.200538 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:09.202874 13606 tablet_bootstrap.cc:492] T e617d4e31eed42dcb39e03cc40b8e6a6 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:09.203209 13606 ts_tablet_manager.cc:1367] T e617d4e31eed42dcb39e03cc40b8e6a6 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 02:00:09.203761 13606 raft_consensus.cc:348] T e617d4e31eed42dcb39e03cc40b8e6a6 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:09.203956 13606 raft_consensus.cc:374] T e617d4e31eed42dcb39e03cc40b8e6a6 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:09.204032 13606 raft_consensus.cc:729] T e617d4e31eed42dcb39e03cc40b8e6a6 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:09.204417 13606 consensus_queue.cc:260] T e617d4e31eed42dcb39e03cc40b8e6a6 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:09.205839 13606 ts_tablet_manager.cc:1398] T e617d4e31eed42dcb39e03cc40b8e6a6 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
W20240503 02:00:09.213935 6654 consensus_peers.cc:485] T cea1ea50fa434d31b42451f84f26bc76 P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tools/rebalancer_tool-test.cc:1203: Failure
Value of: s.ok()
Actual: false
Expected: true
Runtime error: /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 2
Maximum Replica Count | 44
Average Replica Count | 22.875000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 1
Maximum | 3
Average | 2.857143
stderr: W20240503 01:59:58.536991 9363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:58.537266 9363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:58.540885 9363 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 01:59:59.349107 9363 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240503 01:59:59.626546 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9b77b848294749e09f0854d24da5d9f5: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server 99ead02915954b4fab23b033ad6d546b
W20240503 01:59:59.626806 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 72f29547da0b4a95974018c2782cb359: could not find any suitable replica to move from server 492448570cc94ed1879feb2ec3b9051d to server 99ead02915954b4fab23b033ad6d546b
W20240503 01:59:59.626907 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5ccf0adaf2de4b79bbf38c053646e60e: could not find any suitable replica to move from server dec3e4a04d604f7eaaa3f077d081218a to server e1a9251ce1cf4361b8b0ede9fbfc4023
W20240503 01:59:59.626965 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9b77b848294749e09f0854d24da5d9f5: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server 9942306855404cfd891adf227dd65eb2
W20240503 01:59:59.627007 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 14361c7197694d579466ed5d4f2922f7: could not find any suitable replica to move from server 492448570cc94ed1879feb2ec3b9051d to server 99ead02915954b4fab23b033ad6d546b
W20240503 01:59:59.627096 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 296df7c9486144058745436d9b814a78: could not find any suitable replica to move from server 9942306855404cfd891adf227dd65eb2 to server c2ac2eb7cc264047b9ab32fe3a4fc025
W20240503 01:59:59.627197 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 805c97ce87c74e019deaa2acee6a5329: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server e1a9251ce1cf4361b8b0ede9fbfc4023
W20240503 01:59:59.627235 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e0e5e479de824fb18516cfe56d5892c6: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server 99ead02915954b4fab23b033ad6d546b
W20240503 01:59:59.627310 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table eecab7e70bfa4621832bb99bd9ef066c: could not find any suitable replica to move from server 80fd18ef4d604c86b909d5c008a46b4f to server 9942306855404cfd891adf227dd65eb2
W20240503 01:59:59.627360 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6e24d367bc1d434584d8a7363cf538b6: could not find any suitable replica to move from server 80fd18ef4d604c86b909d5c008a46b4f to server 2a01ace80baf4057b6bf1995cf88468b
W20240503 01:59:59.627408 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7455f5480e88494b83e1ac1d180ca4e6: could not find any suitable replica to move from server 492448570cc94ed1879feb2ec3b9051d to server 99ead02915954b4fab23b033ad6d546b
W20240503 01:59:59.627471 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d732c32bc9004748904a109f59db08fd: could not find any suitable replica to move from server dec3e4a04d604f7eaaa3f077d081218a to server 2a01ace80baf4057b6bf1995cf88468b
W20240503 01:59:59.627532 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 029e2476b8c44a8d8cdb98005eb84a4b: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server e1a9251ce1cf4361b8b0ede9fbfc4023
W20240503 01:59:59.627566 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c65135643094446abc8eba15be23cd99: could not find any suitable replica to move from server 492448570cc94ed1879feb2ec3b9051d to server 99ead02915954b4fab23b033ad6d546b
W20240503 01:59:59.627588 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 65ed012903ed4deabf9e7b0dfe803452: could not find any suitable replica to move from server dec3e4a04d604f7eaaa3f077d081218a to server 9942306855404cfd891adf227dd65eb2
W20240503 01:59:59.627617 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4e740b0167ed439291b74d77653ec386: could not find any suitable replica to move from server 80fd18ef4d604c86b909d5c008a46b4f to server e1a9251ce1cf4361b8b0ede9fbfc4023
W20240503 01:59:59.627645 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d93ca01e08fd40d886555f16d21192fc: could not find any suitable replica to move from server 492448570cc94ed1879feb2ec3b9051d to server 99ead02915954b4fab23b033ad6d546b
W20240503 01:59:59.627671 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 250f0720cafa4425b45a0360ebe04a03: could not find any suitable replica to move from server 80fd18ef4d604c86b909d5c008a46b4f to server e1a9251ce1cf4361b8b0ede9fbfc4023
W20240503 01:59:59.627705 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 043b44e4ca1745038882551026afda53: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server 9942306855404cfd891adf227dd65eb2
W20240503 01:59:59.627792 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d930c636ed124c0eb64e1d6ab7c8d106: could not find any suitable replica to move from server 492448570cc94ed1879feb2ec3b9051d to server 2a01ace80baf4057b6bf1995cf88468b
W20240503 01:59:59.627842 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8b571be2b4d14eab84dc1bc1e5b8d53a: could not find any suitable replica to move from server 80fd18ef4d604c86b909d5c008a46b4f to server 9942306855404cfd891adf227dd65eb2
W20240503 01:59:59.627890 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6e24d367bc1d434584d8a7363cf538b6: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server 9942306855404cfd891adf227dd65eb2
W20240503 01:59:59.627974 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7455f5480e88494b83e1ac1d180ca4e6: could not find any suitable replica to move from server 80fd18ef4d604c86b909d5c008a46b4f to server e1a9251ce1cf4361b8b0ede9fbfc4023
W20240503 01:59:59.628016 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5ccf0adaf2de4b79bbf38c053646e60e: could not find any suitable replica to move from server 492448570cc94ed1879feb2ec3b9051d to server 2a01ace80baf4057b6bf1995cf88468b
W20240503 01:59:59.628063 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d732c32bc9004748904a109f59db08fd: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server 9942306855404cfd891adf227dd65eb2
W20240503 01:59:59.628108 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 029e2476b8c44a8d8cdb98005eb84a4b: could not find any suitable replica to move from server dec3e4a04d604f7eaaa3f077d081218a to server 2a01ace80baf4057b6bf1995cf88468b
I20240503 01:59:59.647352 9363 rebalancer_tool.cc:1074] tablet 1fa274c98fc147f29344ed537e2ccf17: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> 'e1a9251ce1cf4361b8b0ede9fbfc4023' move scheduled
I20240503 01:59:59.663408 9363 rebalancer_tool.cc:1074] tablet ceca04391cd446ae932b3485d958237a: '80fd18ef4d604c86b909d5c008a46b4f' -> '9942306855404cfd891adf227dd65eb2' move scheduled
I20240503 01:59:59.688750 9363 rebalancer_tool.cc:1074] tablet 1e445a3cc928429caf9e3c6193a63010: '492448570cc94ed1879feb2ec3b9051d' -> '2a01ace80baf4057b6bf1995cf88468b' move scheduled
I20240503 01:59:59.709553 9363 rebalancer_tool.cc:1074] tablet d82268925a934c559dfcf51a84f03d2e: 'dec3e4a04d604f7eaaa3f077d081218a' -> '99ead02915954b4fab23b033ad6d546b' move scheduled
I20240503 01:59:59.725863 9363 rebalancer_tool.cc:1074] tablet 3416b6107e1848f299ac49ccd88ae7c8: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> 'e1a9251ce1cf4361b8b0ede9fbfc4023' move scheduled
I20240503 01:59:59.742455 9363 rebalancer_tool.cc:1074] tablet 5f176537159143e8a770d65ba63ad62e: '80fd18ef4d604c86b909d5c008a46b4f' -> '9942306855404cfd891adf227dd65eb2' move scheduled
I20240503 01:59:59.762373 9363 rebalancer_tool.cc:1074] tablet 1feaf29d8c6b4842829ce66e0f4ec23b: '492448570cc94ed1879feb2ec3b9051d' -> '2a01ace80baf4057b6bf1995cf88468b' move scheduled
I20240503 01:59:59.804647 9363 rebalancer_tool.cc:1074] tablet 3cf186a0901f420390c8fda54316c5af: 'dec3e4a04d604f7eaaa3f077d081218a' -> '99ead02915954b4fab23b033ad6d546b' move scheduled
I20240503 01:59:59.840487 9363 rebalancer_tool.cc:1074] tablet 46ad540deb624a25b97d3b967fe1f6f4: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> 'e1a9251ce1cf4361b8b0ede9fbfc4023' move scheduled
I20240503 01:59:59.857347 9363 rebalancer_tool.cc:1074] tablet 7a44b6d65bfa43e291b06b0dfd38fab3: '80fd18ef4d604c86b909d5c008a46b4f' -> '9942306855404cfd891adf227dd65eb2' move scheduled
I20240503 01:59:59.875558 9363 rebalancer_tool.cc:1074] tablet 485d1d89a8b94505969d663be92124cd: '492448570cc94ed1879feb2ec3b9051d' -> '99ead02915954b4fab23b033ad6d546b' move scheduled
I20240503 01:59:59.892912 9363 rebalancer_tool.cc:1074] tablet 90ef3584830240acb0d8aad14ae851a6: 'dec3e4a04d604f7eaaa3f077d081218a' -> '2a01ace80baf4057b6bf1995cf88468b' move scheduled
I20240503 01:59:59.931536 9363 rebalancer_tool.cc:1074] tablet 915028b3478341d588ed6031e657c259: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> '9942306855404cfd891adf227dd65eb2' move scheduled
I20240503 01:59:59.953246 9363 rebalancer_tool.cc:1074] tablet 548f43b381f1445c8d3d55705ae5a48a: '80fd18ef4d604c86b909d5c008a46b4f' -> 'e1a9251ce1cf4361b8b0ede9fbfc4023' move scheduled
I20240503 01:59:59.969899 9363 rebalancer_tool.cc:1074] tablet 74ebf5d0fb9a42e6a7ec743a98960e13: '492448570cc94ed1879feb2ec3b9051d' -> '99ead02915954b4fab23b033ad6d546b' move scheduled
I20240503 01:59:59.988325 9363 rebalancer_tool.cc:1074] tablet 8a76d3e2d4264687aa2972c179341b5e: 'dec3e4a04d604f7eaaa3f077d081218a' -> '2a01ace80baf4057b6bf1995cf88468b' move scheduled
I20240503 02:00:00.004880 9363 rebalancer_tool.cc:1074] tablet 2b1d700d09534fa3adcf51bb3c97c8f5: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> '9942306855404cfd891adf227dd65eb2' move scheduled
I20240503 02:00:00.005923 9363 rebalancer_tool.cc:1084] tablet 396a2a8083d240529a2db48519f6139e: '80fd18ef4d604c86b909d5c008a46b4f' -> 'e1a9251ce1cf4361b8b0ede9fbfc4023' move ignored: Not found: 396a2a8083d240529a2db48519f6139e: tablet not found
I20240503 02:00:00.006011 9363 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 02:00:00.296618 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9b77b848294749e09f0854d24da5d9f5: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server 99ead02915954b4fab23b033ad6d546b
W20240503 02:00:00.296752 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 54f1891be3f54f19821aad5d1dfbd85d: could not find any suitable replica to move from server 9942306855404cfd891adf227dd65eb2 to server 80fd18ef4d604c86b909d5c008a46b4f
W20240503 02:00:00.297022 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6e24d367bc1d434584d8a7363cf538b6: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server 2a01ace80baf4057b6bf1995cf88468b
W20240503 02:00:00.297132 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 029e2476b8c44a8d8cdb98005eb84a4b: could not find any suitable replica to move from server dec3e4a04d604f7eaaa3f077d081218a to server e1a9251ce1cf4361b8b0ede9fbfc4023
W20240503 02:00:00.297273 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c65135643094446abc8eba15be23cd99: could not find any suitable replica to move from server 492448570cc94ed1879feb2ec3b9051d to server 9942306855404cfd891adf227dd65eb2
W20240503 02:00:00.297374 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table eecab7e70bfa4621832bb99bd9ef066c: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server 2a01ace80baf4057b6bf1995cf88468b
W20240503 02:00:00.297405 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 54f1891be3f54f19821aad5d1dfbd85d: could not find any suitable replica to move from server dec3e4a04d604f7eaaa3f077d081218a to server c2ac2eb7cc264047b9ab32fe3a4fc025
W20240503 02:00:00.297479 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 805c97ce87c74e019deaa2acee6a5329: could not find any suitable replica to move from server 492448570cc94ed1879feb2ec3b9051d to server 2a01ace80baf4057b6bf1995cf88468b
W20240503 02:00:00.297538 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d732c32bc9004748904a109f59db08fd: could not find any suitable replica to move from server 80fd18ef4d604c86b909d5c008a46b4f to server 99ead02915954b4fab23b033ad6d546b
W20240503 02:00:00.297577 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 14361c7197694d579466ed5d4f2922f7: could not find any suitable replica to move from server 492448570cc94ed1879feb2ec3b9051d to server e1a9251ce1cf4361b8b0ede9fbfc4023
W20240503 02:00:00.297641 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 250f0720cafa4425b45a0360ebe04a03: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server 99ead02915954b4fab23b033ad6d546b
W20240503 02:00:00.297773 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9b77b848294749e09f0854d24da5d9f5: could not find any suitable replica to move from server 492448570cc94ed1879feb2ec3b9051d to server e1a9251ce1cf4361b8b0ede9fbfc4023
W20240503 02:00:00.297820 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 54f1891be3f54f19821aad5d1dfbd85d: could not find any suitable replica to move from server 9942306855404cfd891adf227dd65eb2 to server 492448570cc94ed1879feb2ec3b9051d
W20240503 02:00:00.297855 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 043b44e4ca1745038882551026afda53: could not find any suitable replica to move from server 492448570cc94ed1879feb2ec3b9051d to server 9942306855404cfd891adf227dd65eb2
W20240503 02:00:00.297888 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4e740b0167ed439291b74d77653ec386: could not find any suitable replica to move from server dec3e4a04d604f7eaaa3f077d081218a to server 9942306855404cfd891adf227dd65eb2
W20240503 02:00:00.297932 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7455f5480e88494b83e1ac1d180ca4e6: could not find any suitable replica to move from server dec3e4a04d604f7eaaa3f077d081218a to server e1a9251ce1cf4361b8b0ede9fbfc4023
W20240503 02:00:00.298027 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5ccf0adaf2de4b79bbf38c053646e60e: could not find any suitable replica to move from server 492448570cc94ed1879feb2ec3b9051d to server 99ead02915954b4fab23b033ad6d546b
W20240503 02:00:00.298069 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 65ed012903ed4deabf9e7b0dfe803452: could not find any suitable replica to move from server 492448570cc94ed1879feb2ec3b9051d to server e1a9251ce1cf4361b8b0ede9fbfc4023
W20240503 02:00:00.298117 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d93ca01e08fd40d886555f16d21192fc: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server 99ead02915954b4fab23b033ad6d546b
W20240503 02:00:00.298151 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e0e5e479de824fb18516cfe56d5892c6: could not find any suitable replica to move from server dec3e4a04d604f7eaaa3f077d081218a to server 2a01ace80baf4057b6bf1995cf88468b
W20240503 02:00:00.298183 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 72f29547da0b4a95974018c2782cb359: could not find any suitable replica to move from server 492448570cc94ed1879feb2ec3b9051d to server 2a01ace80baf4057b6bf1995cf88468b
W20240503 02:00:00.298244 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6aaa25ca43754a899cf8589522ff71da: could not find any suitable replica to move from server 80fd18ef4d604c86b909d5c008a46b4f to server 9942306855404cfd891adf227dd65eb2
W20240503 02:00:00.298282 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 029e2476b8c44a8d8cdb98005eb84a4b: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server 2a01ace80baf4057b6bf1995cf88468b
W20240503 02:00:00.298319 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c65135643094446abc8eba15be23cd99: could not find any suitable replica to move from server dec3e4a04d604f7eaaa3f077d081218a to server e1a9251ce1cf4361b8b0ede9fbfc4023
W20240503 02:00:00.298359 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6e24d367bc1d434584d8a7363cf538b6: could not find any suitable replica to move from server 80fd18ef4d604c86b909d5c008a46b4f to server 9942306855404cfd891adf227dd65eb2
W20240503 02:00:00.298390 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9b77b848294749e09f0854d24da5d9f5: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server 99ead02915954b4fab23b033ad6d546b
W20240503 02:00:00.298420 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d732c32bc9004748904a109f59db08fd: could not find any suitable replica to move from server dec3e4a04d604f7eaaa3f077d081218a to server 2a01ace80baf4057b6bf1995cf88468b
W20240503 02:00:00.298451 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d93ca01e08fd40d886555f16d21192fc: could not find any suitable replica to move from server 492448570cc94ed1879feb2ec3b9051d to server e1a9251ce1cf4361b8b0ede9fbfc4023
I20240503 02:00:00.394961 9363 rebalancer_tool.cc:1074] tablet 0b659e9d57db4ebf93c9143938970090: '80fd18ef4d604c86b909d5c008a46b4f' -> 'e1a9251ce1cf4361b8b0ede9fbfc4023' move scheduled
I20240503 02:00:00.493460 9363 rebalancer_tool.cc:1074] tablet 7a4a910a8b9d4134adebb52a73dfae1d: '492448570cc94ed1879feb2ec3b9051d' -> '99ead02915954b4fab23b033ad6d546b' move scheduled
I20240503 02:00:00.537493 9363 rebalancer_tool.cc:1074] tablet 7f637097ad06465cb93c6ff6d97c1d17: 'dec3e4a04d604f7eaaa3f077d081218a' -> '2a01ace80baf4057b6bf1995cf88468b' move scheduled
I20240503 02:00:00.627055 9363 rebalancer_tool.cc:1123] tablet 5f176537159143e8a770d65ba63ad62e: '80fd18ef4d604c86b909d5c008a46b4f' -> '9942306855404cfd891adf227dd65eb2' move is abandoned: Not found: 5f176537159143e8a770d65ba63ad62e: tablet not found
I20240503 02:00:00.754491 9363 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 02:00:01.011598 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9b77b848294749e09f0854d24da5d9f5: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server e1a9251ce1cf4361b8b0ede9fbfc4023
W20240503 02:00:01.011711 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c1969b96b61041869ddd81db8c84bf44: could not find any suitable replica to move from server e1a9251ce1cf4361b8b0ede9fbfc4023 to server dec3e4a04d604f7eaaa3f077d081218a
W20240503 02:00:01.011828 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6e24d367bc1d434584d8a7363cf538b6: could not find any suitable replica to move from server 80fd18ef4d604c86b909d5c008a46b4f to server 2a01ace80baf4057b6bf1995cf88468b
W20240503 02:00:01.011883 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7455f5480e88494b83e1ac1d180ca4e6: could not find any suitable replica to move from server dec3e4a04d604f7eaaa3f077d081218a to server e1a9251ce1cf4361b8b0ede9fbfc4023
W20240503 02:00:01.012025 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 029e2476b8c44a8d8cdb98005eb84a4b: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server e1a9251ce1cf4361b8b0ede9fbfc4023
W20240503 02:00:01.012079 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c65135643094446abc8eba15be23cd99: could not find any suitable replica to move from server 492448570cc94ed1879feb2ec3b9051d to server 9942306855404cfd891adf227dd65eb2
W20240503 02:00:01.012203 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c1969b96b61041869ddd81db8c84bf44: could not find any suitable replica to move from server 99ead02915954b4fab23b033ad6d546b to server c2ac2eb7cc264047b9ab32fe3a4fc025
W20240503 02:00:01.012253 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d732c32bc9004748904a109f59db08fd: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server 9942306855404cfd891adf227dd65eb2
W20240503 02:00:01.012416 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d93ca01e08fd40d886555f16d21192fc: could not find any suitable replica to move from server 492448570cc94ed1879feb2ec3b9051d to server 99ead02915954b4fab23b033ad6d546b
W20240503 02:00:01.012503 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e0e5e479de824fb18516cfe56d5892c6: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server 9942306855404cfd891adf227dd65eb2
W20240503 02:00:01.012589 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4e740b0167ed439291b74d77653ec386: could not find any suitable replica to move from server 80fd18ef4d604c86b909d5c008a46b4f to server e1a9251ce1cf4361b8b0ede9fbfc4023
W20240503 02:00:01.012656 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 043b44e4ca1745038882551026afda53: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server 2a01ace80baf4057b6bf1995cf88468b
W20240503 02:00:01.012701 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9b77b848294749e09f0854d24da5d9f5: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server 99ead02915954b4fab23b033ad6d546b
W20240503 02:00:01.012743 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 805c97ce87c74e019deaa2acee6a5329: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server 9942306855404cfd891adf227dd65eb2
W20240503 02:00:01.012779 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c1969b96b61041869ddd81db8c84bf44: could not find any suitable replica to move from server e1a9251ce1cf4361b8b0ede9fbfc4023 to server 80fd18ef4d604c86b909d5c008a46b4f
W20240503 02:00:01.012840 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5ccf0adaf2de4b79bbf38c053646e60e: could not find any suitable replica to move from server dec3e4a04d604f7eaaa3f077d081218a to server 9942306855404cfd891adf227dd65eb2
W20240503 02:00:01.012883 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 14361c7197694d579466ed5d4f2922f7: could not find any suitable replica to move from server dec3e4a04d604f7eaaa3f077d081218a to server 2a01ace80baf4057b6bf1995cf88468b
W20240503 02:00:01.012914 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 250f0720cafa4425b45a0360ebe04a03: could not find any suitable replica to move from server 80fd18ef4d604c86b909d5c008a46b4f to server e1a9251ce1cf4361b8b0ede9fbfc4023
W20240503 02:00:01.012943 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table eecab7e70bfa4621832bb99bd9ef066c: could not find any suitable replica to move from server 80fd18ef4d604c86b909d5c008a46b4f to server 2a01ace80baf4057b6bf1995cf88468b
W20240503 02:00:01.012979 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e0e5e479de824fb18516cfe56d5892c6: could not find any suitable replica to move from server 492448570cc94ed1879feb2ec3b9051d to server 99ead02915954b4fab23b033ad6d546b
W20240503 02:00:01.013017 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 72f29547da0b4a95974018c2782cb359: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server 2a01ace80baf4057b6bf1995cf88468b
W20240503 02:00:01.013072 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3295265fc41646e7942b5cd0d2d8ab20: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server e1a9251ce1cf4361b8b0ede9fbfc4023
W20240503 02:00:01.013108 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 029e2476b8c44a8d8cdb98005eb84a4b: could not find any suitable replica to move from server dec3e4a04d604f7eaaa3f077d081218a to server 99ead02915954b4fab23b033ad6d546b
W20240503 02:00:01.013141 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6e24d367bc1d434584d8a7363cf538b6: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server e1a9251ce1cf4361b8b0ede9fbfc4023
W20240503 02:00:01.013175 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7455f5480e88494b83e1ac1d180ca4e6: could not find any suitable replica to move from server 80fd18ef4d604c86b909d5c008a46b4f to server 9942306855404cfd891adf227dd65eb2
W20240503 02:00:01.013204 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4e740b0167ed439291b74d77653ec386: could not find any suitable replica to move from server dec3e4a04d604f7eaaa3f077d081218a to server e1a9251ce1cf4361b8b0ede9fbfc4023
W20240503 02:00:01.013238 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 805c97ce87c74e019deaa2acee6a5329: could not find any suitable replica to move from server 492448570cc94ed1879feb2ec3b9051d to server 99ead02915954b4fab23b033ad6d546b
I20240503 02:00:01.034834 9363 rebalancer_tool.cc:1074] tablet 0099654461ca41efa4323fc9f5adab4a: '9942306855404cfd891adf227dd65eb2' -> '80fd18ef4d604c86b909d5c008a46b4f' move scheduled
I20240503 02:00:01.050669 9363 rebalancer_tool.cc:1123] tablet 7f637097ad06465cb93c6ff6d97c1d17: 'dec3e4a04d604f7eaaa3f077d081218a' -> '2a01ace80baf4057b6bf1995cf88468b' move is abandoned: Not found: 7f637097ad06465cb93c6ff6d97c1d17: tablet not found
I20240503 02:00:01.106911 9363 rebalancer_tool.cc:1136] tablet 1fa274c98fc147f29344ed537e2ccf17: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> 'e1a9251ce1cf4361b8b0ede9fbfc4023' move completed: OK
I20240503 02:00:01.142973 9363 rebalancer_tool.cc:1123] tablet 548f43b381f1445c8d3d55705ae5a48a: '80fd18ef4d604c86b909d5c008a46b4f' -> 'e1a9251ce1cf4361b8b0ede9fbfc4023' move is abandoned: Not found: 548f43b381f1445c8d3d55705ae5a48a: tablet not found
I20240503 02:00:01.424549 9363 rebalancer_tool.cc:1074] tablet 9f7ade1e2af24e86b6e54e2b5769e4c0: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> 'e1a9251ce1cf4361b8b0ede9fbfc4023' move scheduled
I20240503 02:00:01.572912 9363 rebalancer_tool.cc:1136] tablet 2b1d700d09534fa3adcf51bb3c97c8f5: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> '9942306855404cfd891adf227dd65eb2' move completed: OK
I20240503 02:00:01.591578 9363 rebalancer_tool.cc:1136] tablet 3416b6107e1848f299ac49ccd88ae7c8: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> 'e1a9251ce1cf4361b8b0ede9fbfc4023' move completed: OK
I20240503 02:00:01.604578 9363 rebalancer_tool.cc:1136] tablet d82268925a934c559dfcf51a84f03d2e: 'dec3e4a04d604f7eaaa3f077d081218a' -> '99ead02915954b4fab23b033ad6d546b' move completed: OK
I20240503 02:00:01.615350 9363 rebalancer_tool.cc:1136] tablet 1e445a3cc928429caf9e3c6193a63010: '492448570cc94ed1879feb2ec3b9051d' -> '2a01ace80baf4057b6bf1995cf88468b' move completed: OK
I20240503 02:00:01.623342 9363 rebalancer_tool.cc:1136] tablet ceca04391cd446ae932b3485d958237a: '80fd18ef4d604c86b909d5c008a46b4f' -> '9942306855404cfd891adf227dd65eb2' move completed: OK
I20240503 02:00:01.630910 9363 rebalancer_tool.cc:1136] tablet 90ef3584830240acb0d8aad14ae851a6: 'dec3e4a04d604f7eaaa3f077d081218a' -> '2a01ace80baf4057b6bf1995cf88468b' move completed: OK
I20240503 02:00:01.637506 9363 rebalancer_tool.cc:1136] tablet 485d1d89a8b94505969d663be92124cd: '492448570cc94ed1879feb2ec3b9051d' -> '99ead02915954b4fab23b033ad6d546b' move completed: OK
I20240503 02:00:01.643836 9363 rebalancer_tool.cc:1136] tablet 1feaf29d8c6b4842829ce66e0f4ec23b: '492448570cc94ed1879feb2ec3b9051d' -> '2a01ace80baf4057b6bf1995cf88468b' move completed: OK
I20240503 02:00:01.652536 9363 rebalancer_tool.cc:1136] tablet 915028b3478341d588ed6031e657c259: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> '9942306855404cfd891adf227dd65eb2' move completed: OK
I20240503 02:00:01.658653 9363 rebalancer_tool.cc:1136] tablet 3cf186a0901f420390c8fda54316c5af: 'dec3e4a04d604f7eaaa3f077d081218a' -> '99ead02915954b4fab23b033ad6d546b' move completed: OK
I20240503 02:00:01.664966 9363 rebalancer_tool.cc:1136] tablet 7a44b6d65bfa43e291b06b0dfd38fab3: '80fd18ef4d604c86b909d5c008a46b4f' -> '9942306855404cfd891adf227dd65eb2' move completed: OK
I20240503 02:00:01.671490 9363 rebalancer_tool.cc:1136] tablet 46ad540deb624a25b97d3b967fe1f6f4: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> 'e1a9251ce1cf4361b8b0ede9fbfc4023' move completed: OK
I20240503 02:00:01.680900 9363 rebalancer_tool.cc:1136] tablet 74ebf5d0fb9a42e6a7ec743a98960e13: '492448570cc94ed1879feb2ec3b9051d' -> '99ead02915954b4fab23b033ad6d546b' move completed: OK
I20240503 02:00:01.718674 9363 rebalancer_tool.cc:1074] tablet 7df4c083550046e883e5e61d3d4e1572: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> '9942306855404cfd891adf227dd65eb2' move scheduled
I20240503 02:00:01.739480 9363 rebalancer_tool.cc:1074] tablet 34120f4948504cbabcd28e0c4f08f9f8: '492448570cc94ed1879feb2ec3b9051d' -> '99ead02915954b4fab23b033ad6d546b' move scheduled
I20240503 02:00:01.756099 9363 rebalancer_tool.cc:1074] tablet 2fc834abc6a94c638d6202b03320a2d2: 'dec3e4a04d604f7eaaa3f077d081218a' -> '2a01ace80baf4057b6bf1995cf88468b' move scheduled
I20240503 02:00:01.757140 9363 rebalancer_tool.cc:1084] tablet ce8bfc835792444e8703e73252b1f638: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> '9942306855404cfd891adf227dd65eb2' move ignored: Not found: ce8bfc835792444e8703e73252b1f638: tablet not found
I20240503 02:00:01.757217 9363 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 02:00:02.025200 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7455f5480e88494b83e1ac1d180ca4e6: could not find any suitable replica to move from server dec3e4a04d604f7eaaa3f077d081218a to server e1a9251ce1cf4361b8b0ede9fbfc4023
W20240503 02:00:02.025489 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0392495bb8a743099943f2bba5f15608: could not find any suitable replica to move from server 9942306855404cfd891adf227dd65eb2 to server c2ac2eb7cc264047b9ab32fe3a4fc025
W20240503 02:00:02.025547 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 805c97ce87c74e019deaa2acee6a5329: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server 9942306855404cfd891adf227dd65eb2
W20240503 02:00:02.025590 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 043b44e4ca1745038882551026afda53: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server 2a01ace80baf4057b6bf1995cf88468b
W20240503 02:00:02.025723 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e0e5e479de824fb18516cfe56d5892c6: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server 9942306855404cfd891adf227dd65eb2
W20240503 02:00:02.025764 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 54f1891be3f54f19821aad5d1dfbd85d: could not find any suitable replica to move from server 2a01ace80baf4057b6bf1995cf88468b to server 492448570cc94ed1879feb2ec3b9051d
W20240503 02:00:02.025794 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table eecab7e70bfa4621832bb99bd9ef066c: could not find any suitable replica to move from server 80fd18ef4d604c86b909d5c008a46b4f to server 99ead02915954b4fab23b033ad6d546b
W20240503 02:00:02.025825 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4e740b0167ed439291b74d77653ec386: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server 2a01ace80baf4057b6bf1995cf88468b
W20240503 02:00:02.025856 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d732c32bc9004748904a109f59db08fd: could not find any suitable replica to move from server 80fd18ef4d604c86b909d5c008a46b4f to server 9942306855404cfd891adf227dd65eb2
W20240503 02:00:02.025925 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c1969b96b61041869ddd81db8c84bf44: could not find any suitable replica to move from server 99ead02915954b4fab23b033ad6d546b to server dec3e4a04d604f7eaaa3f077d081218a
W20240503 02:00:02.025974 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5ccf0adaf2de4b79bbf38c053646e60e: could not find any suitable replica to move from server dec3e4a04d604f7eaaa3f077d081218a to server 2a01ace80baf4057b6bf1995cf88468b
W20240503 02:00:02.026050 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0392495bb8a743099943f2bba5f15608: could not find any suitable replica to move from server 80fd18ef4d604c86b909d5c008a46b4f to server 492448570cc94ed1879feb2ec3b9051d
W20240503 02:00:02.026119 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 54f1891be3f54f19821aad5d1dfbd85d: could not find any suitable replica to move from server dec3e4a04d604f7eaaa3f077d081218a to server 9942306855404cfd891adf227dd65eb2
I20240503 02:00:02.045171 9363 rebalancer_tool.cc:1074] tablet 4c13642db7fc40a5a934aa5e3c2b08fc: '80fd18ef4d604c86b909d5c008a46b4f' -> '2a01ace80baf4057b6bf1995cf88468b' move scheduled
I20240503 02:00:02.061486 9363 rebalancer_tool.cc:1074] tablet 925612b660ec471da5b3c548802ede6e: '9942306855404cfd891adf227dd65eb2' -> 'e1a9251ce1cf4361b8b0ede9fbfc4023' move scheduled
I20240503 02:00:02.081604 9363 rebalancer_tool.cc:1074] tablet 90ef3584830240acb0d8aad14ae851a6: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> '99ead02915954b4fab23b033ad6d546b' move scheduled
I20240503 02:00:02.096154 9363 rebalancer_tool.cc:1074] tablet 46ad540deb624a25b97d3b967fe1f6f4: 'dec3e4a04d604f7eaaa3f077d081218a' -> '9942306855404cfd891adf227dd65eb2' move scheduled
I20240503 02:00:02.177619 9363 rebalancer_tool.cc:1074] tablet 3cf186a0901f420390c8fda54316c5af: '492448570cc94ed1879feb2ec3b9051d' -> '2a01ace80baf4057b6bf1995cf88468b' move scheduled
I20240503 02:00:02.198985 9363 rebalancer_tool.cc:1074] tablet c3485701bf3b48f5ae2e1568afeed133: '80fd18ef4d604c86b909d5c008a46b4f' -> 'e1a9251ce1cf4361b8b0ede9fbfc4023' move scheduled
I20240503 02:00:02.216543 9363 rebalancer_tool.cc:1074] tablet af00a839bb5e402791f87288493efef6: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> '99ead02915954b4fab23b033ad6d546b' move scheduled
I20240503 02:00:02.237360 9363 rebalancer_tool.cc:1074] tablet 1e445a3cc928429caf9e3c6193a63010: 'dec3e4a04d604f7eaaa3f077d081218a' -> '9942306855404cfd891adf227dd65eb2' move scheduled
I20240503 02:00:02.284821 9363 rebalancer_tool.cc:1074] tablet 915028b3478341d588ed6031e657c259: '492448570cc94ed1879feb2ec3b9051d' -> 'e1a9251ce1cf4361b8b0ede9fbfc4023' move scheduled
I20240503 02:00:02.306334 9363 rebalancer_tool.cc:1074] tablet f28586e70bd9473fbdc6b9126b666fa9: '2a01ace80baf4057b6bf1995cf88468b' -> '80fd18ef4d604c86b909d5c008a46b4f' move scheduled
I20240503 02:00:02.323058 9363 rebalancer_tool.cc:1074] tablet ac1632cc5eb94554b1faa7eb715706ef: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> '99ead02915954b4fab23b033ad6d546b' move scheduled
I20240503 02:00:02.420534 9363 rebalancer_tool.cc:1123] tablet 34120f4948504cbabcd28e0c4f08f9f8: '492448570cc94ed1879feb2ec3b9051d' -> '99ead02915954b4fab23b033ad6d546b' move is abandoned: Not found: 34120f4948504cbabcd28e0c4f08f9f8: tablet not found
I20240503 02:00:02.463819 9363 rebalancer_tool.cc:1136] tablet 0b659e9d57db4ebf93c9143938970090: '80fd18ef4d604c86b909d5c008a46b4f' -> 'e1a9251ce1cf4361b8b0ede9fbfc4023' move completed: OK
I20240503 02:00:02.471427 9363 rebalancer_tool.cc:1136] tablet 7a4a910a8b9d4134adebb52a73dfae1d: '492448570cc94ed1879feb2ec3b9051d' -> '99ead02915954b4fab23b033ad6d546b' move completed: OK
I20240503 02:00:02.511894 9363 rebalancer_tool.cc:1074] tablet 2d7244b0dc4d419cb5c3f3677acd70c7: '492448570cc94ed1879feb2ec3b9051d' -> '99ead02915954b4fab23b033ad6d546b' move scheduled
I20240503 02:00:02.530963 9363 rebalancer_tool.cc:1074] tablet 4385d51d11574192883d01376e2d5620: 'dec3e4a04d604f7eaaa3f077d081218a' -> 'e1a9251ce1cf4361b8b0ede9fbfc4023' move scheduled
I20240503 02:00:02.672992 9363 rebalancer_tool.cc:1136] tablet 0099654461ca41efa4323fc9f5adab4a: '9942306855404cfd891adf227dd65eb2' -> '80fd18ef4d604c86b909d5c008a46b4f' move completed: OK
I20240503 02:00:02.692983 9363 rebalancer_tool.cc:1084] tablet 7b392142a4d948e7a36b49314ae15129: '80fd18ef4d604c86b909d5c008a46b4f' -> '9942306855404cfd891adf227dd65eb2' move ignored: Not found: 7b392142a4d948e7a36b49314ae15129: tablet not found
I20240503 02:00:02.693099 9363 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 02:00:03.064880 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4e740b0167ed439291b74d77653ec386: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server 2a01ace80baf4057b6bf1995cf88468b
W20240503 02:00:03.064970 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d732c32bc9004748904a109f59db08fd: could not find any suitable replica to move from server 80fd18ef4d604c86b909d5c008a46b4f to server 9942306855404cfd891adf227dd65eb2
W20240503 02:00:03.065078 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 805c97ce87c74e019deaa2acee6a5329: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server 9942306855404cfd891adf227dd65eb2
W20240503 02:00:03.065155 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5ccf0adaf2de4b79bbf38c053646e60e: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server 2a01ace80baf4057b6bf1995cf88468b
W20240503 02:00:03.065469 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0392495bb8a743099943f2bba5f15608: could not find any suitable replica to move from server 9942306855404cfd891adf227dd65eb2 to server c2ac2eb7cc264047b9ab32fe3a4fc025
W20240503 02:00:03.065640 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 58ca374f38374792b2e80195b318352c: could not find any suitable replica to move from server dec3e4a04d604f7eaaa3f077d081218a to server e1a9251ce1cf4361b8b0ede9fbfc4023
I20240503 02:00:03.098309 9363 rebalancer_tool.cc:1074] tablet cb873018448549ccbe3d14b784e47d26: '80fd18ef4d604c86b909d5c008a46b4f' -> '492448570cc94ed1879feb2ec3b9051d' move scheduled
I20240503 02:00:03.118934 9363 rebalancer_tool.cc:1074] tablet f5ed01740019467d8c1d55cc2fa18ca3: '99ead02915954b4fab23b033ad6d546b' -> '80fd18ef4d604c86b909d5c008a46b4f' move scheduled
I20240503 02:00:03.163864 9363 rebalancer_tool.cc:1074] tablet 3b189fba0d6c461694dda002a3c5eb2f: '492448570cc94ed1879feb2ec3b9051d' -> '9942306855404cfd891adf227dd65eb2' move scheduled
I20240503 02:00:03.297506 9363 rebalancer_tool.cc:1136] tablet 9f7ade1e2af24e86b6e54e2b5769e4c0: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> 'e1a9251ce1cf4361b8b0ede9fbfc4023' move completed: OK
I20240503 02:00:03.311035 9363 rebalancer_tool.cc:1136] tablet 7df4c083550046e883e5e61d3d4e1572: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> '9942306855404cfd891adf227dd65eb2' move completed: OK
I20240503 02:00:03.340684 9363 rebalancer_tool.cc:1136] tablet 8a76d3e2d4264687aa2972c179341b5e: 'dec3e4a04d604f7eaaa3f077d081218a' -> '2a01ace80baf4057b6bf1995cf88468b' move completed: OK
I20240503 02:00:03.355592 9363 rebalancer_tool.cc:1074] tablet d0d5dd42db3e4883af7fd16d8813b550: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> 'e1a9251ce1cf4361b8b0ede9fbfc4023' move scheduled
I20240503 02:00:03.383069 9363 rebalancer_tool.cc:1074] tablet 485d1d89a8b94505969d663be92124cd: 'dec3e4a04d604f7eaaa3f077d081218a' -> '9942306855404cfd891adf227dd65eb2' move scheduled
I20240503 02:00:03.530129 9363 rebalancer_tool.cc:1136] tablet 925612b660ec471da5b3c548802ede6e: '9942306855404cfd891adf227dd65eb2' -> 'e1a9251ce1cf4361b8b0ede9fbfc4023' move completed: OK
I20240503 02:00:03.545480 9363 rebalancer_tool.cc:1123] tablet 4c13642db7fc40a5a934aa5e3c2b08fc: '80fd18ef4d604c86b909d5c008a46b4f' -> '2a01ace80baf4057b6bf1995cf88468b' move is abandoned: Not found: 4c13642db7fc40a5a934aa5e3c2b08fc: tablet not found
I20240503 02:00:03.553505 9363 rebalancer_tool.cc:1136] tablet 2fc834abc6a94c638d6202b03320a2d2: 'dec3e4a04d604f7eaaa3f077d081218a' -> '2a01ace80baf4057b6bf1995cf88468b' move completed: OK
I20240503 02:00:03.594683 9363 rebalancer_tool.cc:1074] tablet 96c395771fd44056abacceb6c1acdb9f: '2a01ace80baf4057b6bf1995cf88468b' -> 'dec3e4a04d604f7eaaa3f077d081218a' move scheduled
I20240503 02:00:03.595628 9363 rebalancer_tool.cc:1084] tablet 0b659e9d57db4ebf93c9143938970090: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> '9942306855404cfd891adf227dd65eb2' move ignored: Not found: 0b659e9d57db4ebf93c9143938970090: tablet not found
I20240503 02:00:03.595695 9363 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 02:00:03.943215 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5ccf0adaf2de4b79bbf38c053646e60e: could not find any suitable replica to move from server dec3e4a04d604f7eaaa3f077d081218a to server 2a01ace80baf4057b6bf1995cf88468b
W20240503 02:00:03.943372 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 805c97ce87c74e019deaa2acee6a5329: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server 9942306855404cfd891adf227dd65eb2
W20240503 02:00:03.943480 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0392495bb8a743099943f2bba5f15608: could not find any suitable replica to move from server 99ead02915954b4fab23b033ad6d546b to server c2ac2eb7cc264047b9ab32fe3a4fc025
W20240503 02:00:03.944453 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 404345ac1896484ca34a8c78c6770131: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server 492448570cc94ed1879feb2ec3b9051d
W20240503 02:00:03.944505 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 72f29547da0b4a95974018c2782cb359: could not find any suitable replica to move from server 80fd18ef4d604c86b909d5c008a46b4f to server 2a01ace80baf4057b6bf1995cf88468b
W20240503 02:00:03.944571 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e43866993a7646be8fc624a6f2150d34: could not find any suitable replica to move from server 9942306855404cfd891adf227dd65eb2 to server c2ac2eb7cc264047b9ab32fe3a4fc025
W20240503 02:00:03.944643 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5ccf0adaf2de4b79bbf38c053646e60e: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server e1a9251ce1cf4361b8b0ede9fbfc4023
W20240503 02:00:03.944693 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4e740b0167ed439291b74d77653ec386: could not find any suitable replica to move from server c2ac2eb7cc264047b9ab32fe3a4fc025 to server dec3e4a04d604f7eaaa3f077d081218a
I20240503 02:00:03.976511 9363 rebalancer_tool.cc:1074] tablet 7df4c083550046e883e5e61d3d4e1572: '9942306855404cfd891adf227dd65eb2' -> '2a01ace80baf4057b6bf1995cf88468b' move scheduled
I20240503 02:00:04.011806 9363 rebalancer_tool.cc:1074] tablet 51b82ef8e4cc474387f6902f2e526ac0: 'e1a9251ce1cf4361b8b0ede9fbfc4023' -> 'c2ac2eb7cc264047b9ab32fe3a4fc025' move scheduled
I20240503 02:00:04.052577 9363 rebalancer_tool.cc:1074] tablet 1fa274c98fc147f29344ed537e2ccf17: '80fd18ef4d604c86b909d5c008a46b4f' -> '2a01ace80baf4057b6bf1995cf88468b' move scheduled
I20240503 02:00:04.121596 9363 rebalancer_tool.cc:1123] tablet af00a839bb5e402791f87288493efef6: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> '99ead02915954b4fab23b033ad6d546b' move is abandoned: Not found: af00a839bb5e402791f87288493efef6: tablet not found
I20240503 02:00:04.148805 9363 rebalancer_tool.cc:1136] tablet 915028b3478341d588ed6031e657c259: '492448570cc94ed1879feb2ec3b9051d' -> 'e1a9251ce1cf4361b8b0ede9fbfc4023' move completed: OK
I20240503 02:00:04.160297 9363 rebalancer_tool.cc:1136] tablet 3cf186a0901f420390c8fda54316c5af: '492448570cc94ed1879feb2ec3b9051d' -> '2a01ace80baf4057b6bf1995cf88468b' move completed: OK
I20240503 02:00:04.185410 9363 rebalancer_tool.cc:1136] tablet 90ef3584830240acb0d8aad14ae851a6: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> '99ead02915954b4fab23b033ad6d546b' move completed: OK
I20240503 02:00:04.216631 9363 rebalancer_tool.cc:1136] tablet 1e445a3cc928429caf9e3c6193a63010: 'dec3e4a04d604f7eaaa3f077d081218a' -> '9942306855404cfd891adf227dd65eb2' move completed: OK
I20240503 02:00:04.226469 9363 rebalancer_tool.cc:1136] tablet 2d7244b0dc4d419cb5c3f3677acd70c7: '492448570cc94ed1879feb2ec3b9051d' -> '99ead02915954b4fab23b033ad6d546b' move completed: OK
I20240503 02:00:04.241597 9363 rebalancer_tool.cc:1136] tablet f28586e70bd9473fbdc6b9126b666fa9: '2a01ace80baf4057b6bf1995cf88468b' -> '80fd18ef4d604c86b909d5c008a46b4f' move completed: OK
I20240503 02:00:04.263391 9363 rebalancer_tool.cc:1136] tablet ac1632cc5eb94554b1faa7eb715706ef: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> '99ead02915954b4fab23b033ad6d546b' move completed: OK
I20240503 02:00:04.271962 9363 rebalancer_tool.cc:1136] tablet 4385d51d11574192883d01376e2d5620: 'dec3e4a04d604f7eaaa3f077d081218a' -> 'e1a9251ce1cf4361b8b0ede9fbfc4023' move completed: OK
I20240503 02:00:04.293192 9363 rebalancer_tool.cc:1074] tablet 3dc213593e4048c99ccbe10161ff8e7f: '492448570cc94ed1879feb2ec3b9051d' -> '99ead02915954b4fab23b033ad6d546b' move scheduled
I20240503 02:00:04.308706 9363 rebalancer_tool.cc:1074] tablet 0ee05e70ccbb496ca3cff08dc38112e8: '2a01ace80baf4057b6bf1995cf88468b' -> 'dec3e4a04d604f7eaaa3f077d081218a' move scheduled
I20240503 02:00:04.325034 9363 rebalancer_tool.cc:1074] tablet d897c21e630d4770832bb10f43778e82: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> 'e1a9251ce1cf4361b8b0ede9fbfc4023' move scheduled
I20240503 02:00:04.341096 9363 rebalancer_tool.cc:1074] tablet 3416b6107e1848f299ac49ccd88ae7c8: '492448570cc94ed1879feb2ec3b9051d' -> '9942306855404cfd891adf227dd65eb2' move scheduled
I20240503 02:00:04.356585 9363 rebalancer_tool.cc:1074] tablet 2e967e377984468d90799041c29882bd: '80fd18ef4d604c86b909d5c008a46b4f' -> '99ead02915954b4fab23b033ad6d546b' move scheduled
I20240503 02:00:04.396317 9363 rebalancer_tool.cc:1074] tablet 067f54011a37432d99157884e70fc601: '2a01ace80baf4057b6bf1995cf88468b' -> '492448570cc94ed1879feb2ec3b9051d' move scheduled
I20240503 02:00:04.416498 9363 rebalancer_tool.cc:1074] tablet 9c8a53980b00421d8c2bd3ae73f0c8cc: 'dec3e4a04d604f7eaaa3f077d081218a' -> 'c2ac2eb7cc264047b9ab32fe3a4fc025' move scheduled
I20240503 02:00:04.962145 9363 rebalancer_tool.cc:1136] tablet d0d5dd42db3e4883af7fd16d8813b550: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> 'e1a9251ce1cf4361b8b0ede9fbfc4023' move completed: OK
I20240503 02:00:05.020974 9363 rebalancer_tool.cc:1136] tablet 3b189fba0d6c461694dda002a3c5eb2f: '492448570cc94ed1879feb2ec3b9051d' -> '9942306855404cfd891adf227dd65eb2' move completed: OK
I20240503 02:00:05.074697 9363 rebalancer_tool.cc:1136] tablet f5ed01740019467d8c1d55cc2fa18ca3: '99ead02915954b4fab23b033ad6d546b' -> '80fd18ef4d604c86b909d5c008a46b4f' move completed: OK
I20240503 02:00:05.120497 9363 rebalancer_tool.cc:1074] tablet 8246488dec204bf6b09e868d7951ac5a: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> 'e1a9251ce1cf4361b8b0ede9fbfc4023' move scheduled
I20240503 02:00:05.203215 9363 rebalancer_tool.cc:1136] tablet 485d1d89a8b94505969d663be92124cd: 'dec3e4a04d604f7eaaa3f077d081218a' -> '9942306855404cfd891adf227dd65eb2' move completed: OK
I20240503 02:00:05.256238 9363 rebalancer_tool.cc:1136] tablet 96c395771fd44056abacceb6c1acdb9f: '2a01ace80baf4057b6bf1995cf88468b' -> 'dec3e4a04d604f7eaaa3f077d081218a' move completed: OK
I20240503 02:00:05.270459 9363 rebalancer_tool.cc:1074] tablet b91a933fb66b4075acb40c5b92ef4824: 'dec3e4a04d604f7eaaa3f077d081218a' -> '9942306855404cfd891adf227dd65eb2' move scheduled
I20240503 02:00:05.300743 9363 rebalancer_tool.cc:1074] tablet 6a82dcc9be5b4c93b5fff03f789350ab: 'e1a9251ce1cf4361b8b0ede9fbfc4023' -> 'dec3e4a04d604f7eaaa3f077d081218a' move scheduled
I20240503 02:00:05.422181 9363 rebalancer_tool.cc:1136] tablet 46ad540deb624a25b97d3b967fe1f6f4: 'dec3e4a04d604f7eaaa3f077d081218a' -> '9942306855404cfd891adf227dd65eb2' move completed: OK
I20240503 02:00:05.518419 9363 rebalancer_tool.cc:1136] tablet 51b82ef8e4cc474387f6902f2e526ac0: 'e1a9251ce1cf4361b8b0ede9fbfc4023' -> 'c2ac2eb7cc264047b9ab32fe3a4fc025' move completed: OK
I20240503 02:00:05.628218 9363 rebalancer_tool.cc:1074] tablet ac7fa0c3722a461c91fb237fd85794a3: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> '99ead02915954b4fab23b033ad6d546b' move scheduled
I20240503 02:00:05.662603 9363 rebalancer_tool.cc:1074] tablet 1feaf29d8c6b4842829ce66e0f4ec23b: 'dec3e4a04d604f7eaaa3f077d081218a' -> 'e1a9251ce1cf4361b8b0ede9fbfc4023' move scheduled
I20240503 02:00:05.811722 9363 rebalancer_tool.cc:1136] tablet 3416b6107e1848f299ac49ccd88ae7c8: '492448570cc94ed1879feb2ec3b9051d' -> '9942306855404cfd891adf227dd65eb2' move completed: OK
I20240503 02:00:05.851882 9363 rebalancer_tool.cc:1136] tablet 3dc213593e4048c99ccbe10161ff8e7f: '492448570cc94ed1879feb2ec3b9051d' -> '99ead02915954b4fab23b033ad6d546b' move completed: OK
I20240503 02:00:05.864341 9363 rebalancer_tool.cc:1136] tablet 1fa274c98fc147f29344ed537e2ccf17: '80fd18ef4d604c86b909d5c008a46b4f' -> '2a01ace80baf4057b6bf1995cf88468b' move completed: OK
I20240503 02:00:05.884183 9363 rebalancer_tool.cc:1136] tablet c3485701bf3b48f5ae2e1568afeed133: '80fd18ef4d604c86b909d5c008a46b4f' -> 'e1a9251ce1cf4361b8b0ede9fbfc4023' move completed: OK
I20240503 02:00:05.890595 9363 rebalancer_tool.cc:1136] tablet 9c8a53980b00421d8c2bd3ae73f0c8cc: 'dec3e4a04d604f7eaaa3f077d081218a' -> 'c2ac2eb7cc264047b9ab32fe3a4fc025' move completed: OK
I20240503 02:00:05.900506 9363 rebalancer_tool.cc:1136] tablet 7df4c083550046e883e5e61d3d4e1572: '9942306855404cfd891adf227dd65eb2' -> '2a01ace80baf4057b6bf1995cf88468b' move completed: OK
I20240503 02:00:05.935667 9363 rebalancer_tool.cc:1074] tablet 51c0137930464635be1e6a97f5e9d952: '9942306855404cfd891adf227dd65eb2' -> 'c2ac2eb7cc264047b9ab32fe3a4fc025' move scheduled
I20240503 02:00:05.961105 9363 rebalancer_tool.cc:1074] tablet 1c721b33983b4c47afd8faa8f155ca84: 'e1a9251ce1cf4361b8b0ede9fbfc4023' -> 'dec3e4a04d604f7eaaa3f077d081218a' move scheduled
I20240503 02:00:06.015184 9363 rebalancer_tool.cc:1136] tablet 2e967e377984468d90799041c29882bd: '80fd18ef4d604c86b909d5c008a46b4f' -> '99ead02915954b4fab23b033ad6d546b' move completed: OK
I20240503 02:00:06.152053 9363 rebalancer_tool.cc:1136] tablet 067f54011a37432d99157884e70fc601: '2a01ace80baf4057b6bf1995cf88468b' -> '492448570cc94ed1879feb2ec3b9051d' move completed: OK
I20240503 02:00:06.525871 9363 rebalancer_tool.cc:1136] tablet 8246488dec204bf6b09e868d7951ac5a: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> 'e1a9251ce1cf4361b8b0ede9fbfc4023' move completed: OK
I20240503 02:00:06.564509 9363 rebalancer_tool.cc:1136] tablet cb873018448549ccbe3d14b784e47d26: '80fd18ef4d604c86b909d5c008a46b4f' -> '492448570cc94ed1879feb2ec3b9051d' move completed: OK
I20240503 02:00:06.880039 9363 rebalancer_tool.cc:1136] tablet b91a933fb66b4075acb40c5b92ef4824: 'dec3e4a04d604f7eaaa3f077d081218a' -> '9942306855404cfd891adf227dd65eb2' move completed: OK
I20240503 02:00:06.917583 9363 rebalancer_tool.cc:1136] tablet d897c21e630d4770832bb10f43778e82: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> 'e1a9251ce1cf4361b8b0ede9fbfc4023' move completed: OK
I20240503 02:00:06.973349 9363 rebalancer_tool.cc:1136] tablet 6a82dcc9be5b4c93b5fff03f789350ab: 'e1a9251ce1cf4361b8b0ede9fbfc4023' -> 'dec3e4a04d604f7eaaa3f077d081218a' move completed: OK
I20240503 02:00:07.037731 9363 rebalancer_tool.cc:1136] tablet 1feaf29d8c6b4842829ce66e0f4ec23b: 'dec3e4a04d604f7eaaa3f077d081218a' -> 'e1a9251ce1cf4361b8b0ede9fbfc4023' move completed: OK
I20240503 02:00:07.070804 9363 rebalancer_tool.cc:1136] tablet ac7fa0c3722a461c91fb237fd85794a3: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> '99ead02915954b4fab23b033ad6d546b' move completed: OK
I20240503 02:00:07.297852 9363 rebalancer_tool.cc:1136] tablet 51c0137930464635be1e6a97f5e9d952: '9942306855404cfd891adf227dd65eb2' -> 'c2ac2eb7cc264047b9ab32fe3a4fc025' move completed: OK
I20240503 02:00:07.539635 9363 rebalancer_tool.cc:1136] tablet 0ee05e70ccbb496ca3cff08dc38112e8: '2a01ace80baf4057b6bf1995cf88468b' -> 'dec3e4a04d604f7eaaa3f077d081218a' move completed: OK
I20240503 02:00:07.772747 9363 rebalancer_tool.cc:1136] tablet 1c721b33983b4c47afd8faa8f155ca84: 'e1a9251ce1cf4361b8b0ede9fbfc4023' -> 'dec3e4a04d604f7eaaa3f077d081218a' move completed: OK
W20240503 02:00:07.984575 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table dd81a75146044b9d9fe1ec1f86a7aebd: could not find any suitable replica to move from server 9942306855404cfd891adf227dd65eb2 to server 80fd18ef4d604c86b909d5c008a46b4f
W20240503 02:00:07.984735 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table dd81a75146044b9d9fe1ec1f86a7aebd: could not find any suitable replica to move from server e1a9251ce1cf4361b8b0ede9fbfc4023 to server 492448570cc94ed1879feb2ec3b9051d
W20240503 02:00:07.984961 9363 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table dd81a75146044b9d9fe1ec1f86a7aebd: could not find any suitable replica to move from server 9942306855404cfd891adf227dd65eb2 to server 99ead02915954b4fab23b033ad6d546b
I20240503 02:00:08.009354 9363 rebalancer_tool.cc:1074] tablet beeea71f02ae4c89a1af13d3584ef70a: '2a01ace80baf4057b6bf1995cf88468b' -> '80fd18ef4d604c86b909d5c008a46b4f' move scheduled
I20240503 02:00:08.039716 9363 rebalancer_tool.cc:1074] tablet 96c395771fd44056abacceb6c1acdb9f: '99ead02915954b4fab23b033ad6d546b' -> 'c2ac2eb7cc264047b9ab32fe3a4fc025' move scheduled
I20240503 02:00:08.061971 9363 rebalancer_tool.cc:1074] tablet b2e62c043adf4a8d9024279d786f596c: '2a01ace80baf4057b6bf1995cf88468b' -> '9942306855404cfd891adf227dd65eb2' move scheduled
I20240503 02:00:08.081597 9363 rebalancer_tool.cc:1074] tablet cea1ea50fa434d31b42451f84f26bc76: '99ead02915954b4fab23b033ad6d546b' -> '492448570cc94ed1879feb2ec3b9051d' move scheduled
I20240503 02:00:08.098927 9363 rebalancer_tool.cc:1074] tablet b20ff3a6f74f4640afebfd4059b58699: '99ead02915954b4fab23b033ad6d546b' -> 'dec3e4a04d604f7eaaa3f077d081218a' move scheduled
I20240503 02:00:08.113816 9363 rebalancer_tool.cc:1074] tablet aa281822047841fab7c58b53d9733e49: 'e1a9251ce1cf4361b8b0ede9fbfc4023' -> 'c2ac2eb7cc264047b9ab32fe3a4fc025' move scheduled
I20240503 02:00:08.132983 9363 rebalancer_tool.cc:1074] tablet ac7fa0c3722a461c91fb237fd85794a3: '80fd18ef4d604c86b909d5c008a46b4f' -> '2a01ace80baf4057b6bf1995cf88468b' move scheduled
I20240503 02:00:08.150460 9363 rebalancer_tool.cc:1074] tablet 1c721b33983b4c47afd8faa8f155ca84: '9942306855404cfd891adf227dd65eb2' -> 'c2ac2eb7cc264047b9ab32fe3a4fc025' move scheduled
I20240503 02:00:08.186134 9363 rebalancer_tool.cc:1074] tablet bd8b9532bd744ecdbe8da66a1225724b: '99ead02915954b4fab23b033ad6d546b' -> 'e1a9251ce1cf4361b8b0ede9fbfc4023' move scheduled
I20240503 02:00:08.207420 9363 rebalancer_tool.cc:1074] tablet e52373099af3428497d176e3fcd8ef27: '2a01ace80baf4057b6bf1995cf88468b' -> '80fd18ef4d604c86b909d5c008a46b4f' move scheduled
I20240503 02:00:08.225680 9363 rebalancer_tool.cc:1074] tablet d73ea6e550a04c04a2695b058e38c566: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> '9942306855404cfd891adf227dd65eb2' move scheduled
I20240503 02:00:08.243218 9363 rebalancer_tool.cc:1074] tablet 9cbb2be1fda74ebe8997123c70629ba3: '99ead02915954b4fab23b033ad6d546b' -> '80fd18ef4d604c86b909d5c008a46b4f' move scheduled
I20240503 02:00:08.296453 9363 rebalancer_tool.cc:1074] tablet 2d7244b0dc4d419cb5c3f3677acd70c7: 'c2ac2eb7cc264047b9ab32fe3a4fc025' -> '2a01ace80baf4057b6bf1995cf88468b' move scheduled
W20240503 02:00:08.918651 13514 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 02:00:08.788636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.136:45311 (local address 127.0.0.1:45750)
0503 02:00:08.792921 (+ 4285us) negotiation.cc:107] Waiting for socket to connect
0503 02:00:08.792939 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0503 02:00:08.793046 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 02:00:08.917818 (+124772us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":4179,"thread_start_us":122,"threads_started":1}
I20240503 02:00:08.919864 9363 rebalancer_tool.cc:1123] tablet b2e62c043adf4a8d9024279d786f596c: '2a01ace80baf4057b6bf1995cf88468b' -> '9942306855404cfd891adf227dd65eb2' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:00:08.941910 9363 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): unacceptable health status UNAVAILABLE
I20240503 02:00:09.242514 9179 raft_consensus.cc:1203] T ac7fa0c3722a461c91fb237fd85794a3 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 02:00:09.251286 13626 raft_consensus.cc:482] T be7bbd71b99847068f415cc3d3d49ddd P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:09.251439 13626 raft_consensus.cc:504] T be7bbd71b99847068f415cc3d3d49ddd P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:09.251957 13626 leader_election.cc:288] T be7bbd71b99847068f415cc3d3d49ddd P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:09.252473 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be7bbd71b99847068f415cc3d3d49ddd" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:09.252631 8916 raft_consensus.cc:2454] T be7bbd71b99847068f415cc3d3d49ddd P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:09.252621 6720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be7bbd71b99847068f415cc3d3d49ddd" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 02:00:09.252772 6720 raft_consensus.cc:2454] T be7bbd71b99847068f415cc3d3d49ddd P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:09.252967 6394 leader_election.cc:302] T be7bbd71b99847068f415cc3d3d49ddd P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9942306855404cfd891adf227dd65eb2, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:09.253165 13626 raft_consensus.cc:2781] T be7bbd71b99847068f415cc3d3d49ddd P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:09.253255 13626 raft_consensus.cc:482] T be7bbd71b99847068f415cc3d3d49ddd P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:09.253315 13626 raft_consensus.cc:3037] T be7bbd71b99847068f415cc3d3d49ddd P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:09.254390 13626 raft_consensus.cc:504] T be7bbd71b99847068f415cc3d3d49ddd P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:09.254801 13626 leader_election.cc:288] T be7bbd71b99847068f415cc3d3d49ddd P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Requested vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:09.255148 6720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be7bbd71b99847068f415cc3d3d49ddd" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 02:00:09.255177 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be7bbd71b99847068f415cc3d3d49ddd" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:09.255293 6720 raft_consensus.cc:3037] T be7bbd71b99847068f415cc3d3d49ddd P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:09.255293 8916 raft_consensus.cc:3037] T be7bbd71b99847068f415cc3d3d49ddd P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:09.256351 6720 raft_consensus.cc:2454] T be7bbd71b99847068f415cc3d3d49ddd P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:09.256352 8916 raft_consensus.cc:2454] T be7bbd71b99847068f415cc3d3d49ddd P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:09.256716 6394 leader_election.cc:302] T be7bbd71b99847068f415cc3d3d49ddd P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2ac2eb7cc264047b9ab32fe3a4fc025, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:09.257027 13626 raft_consensus.cc:2781] T be7bbd71b99847068f415cc3d3d49ddd P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:09.257144 13626 raft_consensus.cc:686] T be7bbd71b99847068f415cc3d3d49ddd P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Becoming Leader. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Running, Role: LEADER
I20240503 02:00:09.257329 13626 consensus_queue.cc:237] T be7bbd71b99847068f415cc3d3d49ddd P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:09.259914 6308 catalog_manager.cc:5484] T be7bbd71b99847068f415cc3d3d49ddd P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: term changed from 0 to 1, leader changed from <none> to dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129). New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:09.275573 6309 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:09.275993 8916 raft_consensus.cc:1203] T d73ea6e550a04c04a2695b058e38c566 P 9942306855404cfd891adf227dd65eb2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:00:09.281991 13615 tablet.cc:1722] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:09.282349 13615 tablet.cc:1620] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 80fd18ef4d604c86b909d5c008a46b4f: MemRowSet was empty: no flush needed.
I20240503 02:00:09.282691 13622 tablet.cc:1722] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:09.282961 13622 tablet.cc:1620] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
I20240503 02:00:09.284221 13616 tablet.cc:1722] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:09.284572 13616 tablet.cc:1620] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 99ead02915954b4fab23b033ad6d546b: MemRowSet was empty: no flush needed.
I20240503 02:00:09.289158 6308 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=73d45da493d442e489ee6add747cad2b] alter complete (version 2)
I20240503 02:00:09.334766 13623 raft_consensus.cc:482] T 7cecefe8f1864583be9d0b9c308a78db P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:09.334932 13623 raft_consensus.cc:504] T 7cecefe8f1864583be9d0b9c308a78db P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:09.335323 13623 leader_election.cc:288] T 7cecefe8f1864583be9d0b9c308a78db P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:09.335752 9039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cecefe8f1864583be9d0b9c308a78db" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:09.335739 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cecefe8f1864583be9d0b9c308a78db" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:09.335882 9179 raft_consensus.cc:2454] T 7cecefe8f1864583be9d0b9c308a78db P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 02:00:09.335887 9039 raft_consensus.cc:2454] T 7cecefe8f1864583be9d0b9c308a78db P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 02:00:09.336164 6784 leader_election.cc:302] T 7cecefe8f1864583be9d0b9c308a78db P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 492448570cc94ed1879feb2ec3b9051d; no voters:
I20240503 02:00:09.336354 13623 raft_consensus.cc:2781] T 7cecefe8f1864583be9d0b9c308a78db P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:09.336467 13623 raft_consensus.cc:482] T 7cecefe8f1864583be9d0b9c308a78db P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:09.336503 13623 raft_consensus.cc:3037] T 7cecefe8f1864583be9d0b9c308a78db P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:09.337455 13623 raft_consensus.cc:504] T 7cecefe8f1864583be9d0b9c308a78db P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:09.338119 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cecefe8f1864583be9d0b9c308a78db" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:09.338254 9179 raft_consensus.cc:3037] T 7cecefe8f1864583be9d0b9c308a78db P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:09.339171 9179 raft_consensus.cc:2454] T 7cecefe8f1864583be9d0b9c308a78db P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 02:00:09.339438 13623 leader_election.cc:288] T 7cecefe8f1864583be9d0b9c308a78db P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Requested vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:09.339490 6784 leader_election.cc:302] T 7cecefe8f1864583be9d0b9c308a78db P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 492448570cc94ed1879feb2ec3b9051d; no voters:
I20240503 02:00:09.339615 13623 raft_consensus.cc:2781] T 7cecefe8f1864583be9d0b9c308a78db P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:09.339692 13623 raft_consensus.cc:686] T 7cecefe8f1864583be9d0b9c308a78db P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Becoming Leader. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Running, Role: LEADER
I20240503 02:00:09.339834 13623 consensus_queue.cc:237] T 7cecefe8f1864583be9d0b9c308a78db P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:09.339967 9039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cecefe8f1864583be9d0b9c308a78db" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:09.340085 9039 raft_consensus.cc:3037] T 7cecefe8f1864583be9d0b9c308a78db P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:09.341111 9039 raft_consensus.cc:2454] T 7cecefe8f1864583be9d0b9c308a78db P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 02:00:09.342141 6309 catalog_manager.cc:5484] T 7cecefe8f1864583be9d0b9c308a78db P 492448570cc94ed1879feb2ec3b9051d reported cstate change: term changed from 0 to 1, leader changed from <none> to 492448570cc94ed1879feb2ec3b9051d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:09.374761 13102 raft_consensus.cc:1050] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2: attempting to promote NON_VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 to VOTER
I20240503 02:00:09.375222 13102 consensus_queue.cc:237] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } }
I20240503 02:00:09.377110 6456 raft_consensus.cc:1261] T 96c395771fd44056abacceb6c1acdb9f P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:09.377116 9039 raft_consensus.cc:1261] T 96c395771fd44056abacceb6c1acdb9f P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:09.377161 6720 raft_consensus.cc:1261] T 96c395771fd44056abacceb6c1acdb9f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEARNER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:09.377544 13634 consensus_queue.cc:1035] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:09.377786 13559 consensus_queue.cc:1035] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:09.378131 13018 consensus_queue.cc:1035] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:09.380782 13559 raft_consensus.cc:2932] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } } }
I20240503 02:00:09.382921 6720 raft_consensus.cc:2932] T 96c395771fd44056abacceb6c1acdb9f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } } }
I20240503 02:00:09.382957 6456 raft_consensus.cc:2932] T 96c395771fd44056abacceb6c1acdb9f P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } } }
I20240503 02:00:09.383731 6309 catalog_manager.cc:5484] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2 reported cstate change: config changed from index 5 to 6, c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9942306855404cfd891adf227dd65eb2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:09.389151 8916 consensus_queue.cc:237] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } }
I20240503 02:00:09.390475 6720 raft_consensus.cc:1261] T 96c395771fd44056abacceb6c1acdb9f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:09.390434 6456 raft_consensus.cc:1261] T 96c395771fd44056abacceb6c1acdb9f P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:09.390852 13634 consensus_queue.cc:1035] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:09.391017 13018 consensus_queue.cc:1035] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:09.392221 6589 raft_consensus.cc:1203] T 9cbb2be1fda74ebe8997123c70629ba3 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:00:09.392834 13102 raft_consensus.cc:2932] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } } }
I20240503 02:00:09.393219 6456 raft_consensus.cc:2932] T 96c395771fd44056abacceb6c1acdb9f P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } } }
I20240503 02:00:09.393646 6308 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:09.395148 6719 raft_consensus.cc:2932] T 96c395771fd44056abacceb6c1acdb9f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } } }
I20240503 02:00:09.395861 6303 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 96c395771fd44056abacceb6c1acdb9f with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:09.396762 6309 catalog_manager.cc:5484] T 96c395771fd44056abacceb6c1acdb9f P 9942306855404cfd891adf227dd65eb2 reported cstate change: config changed from index 6 to 7, VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "9942306855404cfd891adf227dd65eb2" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:09.400933 9028 tablet_service.cc:1514] Processing DeleteTablet for tablet 96c395771fd44056abacceb6c1acdb9f with delete_type TABLET_DATA_TOMBSTONED (TS 99ead02915954b4fab23b033ad6d546b not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:52584
I20240503 02:00:09.402688 13638 tablet_replica.cc:330] T 96c395771fd44056abacceb6c1acdb9f P 99ead02915954b4fab23b033ad6d546b: stopping tablet replica
I20240503 02:00:09.402710 13615 tablet.cc:1722] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:09.402840 13638 raft_consensus.cc:2229] T 96c395771fd44056abacceb6c1acdb9f P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:09.402974 13638 pending_rounds.cc:62] T 96c395771fd44056abacceb6c1acdb9f P 99ead02915954b4fab23b033ad6d546b: Trying to abort 1 pending ops.
I20240503 02:00:09.403020 13638 pending_rounds.cc:68] T 96c395771fd44056abacceb6c1acdb9f P 99ead02915954b4fab23b033ad6d546b: Aborting op as it isn't in flight: id { term: 1 index: 6 } timestamp: 7023417791999979520 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "96c395771fd44056abacceb6c1acdb9f" old_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: true } } } new_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } } } }
I20240503 02:00:09.403198 13638 raft_consensus.cc:2894] T 96c395771fd44056abacceb6c1acdb9f P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Aborting config change with OpId 1.6: Aborted: Op aborted
I20240503 02:00:09.403496 13622 tablet.cc:1722] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:09.403645 13638 raft_consensus.cc:2866] T 96c395771fd44056abacceb6c1acdb9f P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 02:00:09.403712 13638 raft_consensus.cc:2258] T 96c395771fd44056abacceb6c1acdb9f P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:09.404333 13638 ts_tablet_manager.cc:1875] T 96c395771fd44056abacceb6c1acdb9f P 99ead02915954b4fab23b033ad6d546b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:09.408407 13638 ts_tablet_manager.cc:1888] T 96c395771fd44056abacceb6c1acdb9f P 99ead02915954b4fab23b033ad6d546b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:00:09.408499 13638 log.cc:1196] T 96c395771fd44056abacceb6c1acdb9f P 99ead02915954b4fab23b033ad6d546b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/96c395771fd44056abacceb6c1acdb9f
I20240503 02:00:09.408371 6308 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=73d45da493d442e489ee6add747cad2b] alter complete (version 3)
I20240503 02:00:09.409436 6301 catalog_manager.cc:4830] TS 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157): tablet 96c395771fd44056abacceb6c1acdb9f (table rebalancer_extra_table_6 [id=0392495bb8a743099943f2bba5f15608]) successfully deleted
I20240503 02:00:09.411470 13616 tablet.cc:1722] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:09.464993 13496 raft_consensus.cc:482] T e617d4e31eed42dcb39e03cc40b8e6a6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:09.465134 13496 raft_consensus.cc:504] T e617d4e31eed42dcb39e03cc40b8e6a6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:09.465555 13496 leader_election.cc:288] T e617d4e31eed42dcb39e03cc40b8e6a6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:09.465992 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e617d4e31eed42dcb39e03cc40b8e6a6" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:09.466059 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e617d4e31eed42dcb39e03cc40b8e6a6" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:09.466147 9179 raft_consensus.cc:2454] T e617d4e31eed42dcb39e03cc40b8e6a6 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:09.466204 8916 raft_consensus.cc:2454] T e617d4e31eed42dcb39e03cc40b8e6a6 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:09.466459 6653 leader_election.cc:302] T e617d4e31eed42dcb39e03cc40b8e6a6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:09.466605 13496 raft_consensus.cc:2781] T e617d4e31eed42dcb39e03cc40b8e6a6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:09.466665 13496 raft_consensus.cc:482] T e617d4e31eed42dcb39e03cc40b8e6a6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:09.466703 13496 raft_consensus.cc:3037] T e617d4e31eed42dcb39e03cc40b8e6a6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:09.467612 13496 raft_consensus.cc:504] T e617d4e31eed42dcb39e03cc40b8e6a6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:09.467952 13496 leader_election.cc:288] T e617d4e31eed42dcb39e03cc40b8e6a6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Requested vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:09.468405 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e617d4e31eed42dcb39e03cc40b8e6a6" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:09.468408 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e617d4e31eed42dcb39e03cc40b8e6a6" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:09.468533 8916 raft_consensus.cc:3037] T e617d4e31eed42dcb39e03cc40b8e6a6 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:09.468533 9179 raft_consensus.cc:3037] T e617d4e31eed42dcb39e03cc40b8e6a6 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:09.469285 9179 raft_consensus.cc:2454] T e617d4e31eed42dcb39e03cc40b8e6a6 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:09.469295 8916 raft_consensus.cc:2454] T e617d4e31eed42dcb39e03cc40b8e6a6 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:09.469573 6653 leader_election.cc:302] T e617d4e31eed42dcb39e03cc40b8e6a6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:09.469740 13496 raft_consensus.cc:2781] T e617d4e31eed42dcb39e03cc40b8e6a6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:09.469826 13496 raft_consensus.cc:686] T e617d4e31eed42dcb39e03cc40b8e6a6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Becoming Leader. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Running, Role: LEADER
I20240503 02:00:09.469974 13496 consensus_queue.cc:237] T e617d4e31eed42dcb39e03cc40b8e6a6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:09.472131 6313 catalog_manager.cc:5484] T e617d4e31eed42dcb39e03cc40b8e6a6 P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:09.509586 12922 raft_consensus.cc:1050] T beeea71f02ae4c89a1af13d3584ef70a P 99ead02915954b4fab23b033ad6d546b: attempting to promote NON_VOTER 80fd18ef4d604c86b909d5c008a46b4f to VOTER
I20240503 02:00:09.510079 12922 consensus_queue.cc:237] T beeea71f02ae4c89a1af13d3584ef70a P 99ead02915954b4fab23b033ad6d546b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } }
I20240503 02:00:09.510715 6308 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:09.510838 6308 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:09.511658 9179 raft_consensus.cc:1261] T beeea71f02ae4c89a1af13d3584ef70a P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer 99ead02915954b4fab23b033ad6d546b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:09.512012 6589 raft_consensus.cc:1261] T beeea71f02ae4c89a1af13d3584ef70a P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEARNER]: Refusing update from remote peer 99ead02915954b4fab23b033ad6d546b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:09.512326 13631 consensus_queue.cc:1035] T beeea71f02ae4c89a1af13d3584ef70a P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:09.512580 10473 consensus_queue.cc:1035] T beeea71f02ae4c89a1af13d3584ef70a P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240503 02:00:09.513459 8982 consensus_peers.cc:485] T beeea71f02ae4c89a1af13d3584ef70a P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:09.513489 6308 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet 7a90fdf10e3c4dd8be7d5e1b54e9a1da
I20240503 02:00:09.514161 9028 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a90fdf10e3c4dd8be7d5e1b54e9a1da with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:09 UTC) from {username='slave'} at 127.0.0.1:52584
I20240503 02:00:09.514338 6569 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a90fdf10e3c4dd8be7d5e1b54e9a1da with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:09 UTC) from {username='slave'} at 127.0.0.1:59846
I20240503 02:00:09.514392 9157 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a90fdf10e3c4dd8be7d5e1b54e9a1da with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:09 UTC) from {username='slave'} at 127.0.0.1:51940
I20240503 02:00:09.514489 13638 tablet_replica.cc:330] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 99ead02915954b4fab23b033ad6d546b: stopping tablet replica
I20240503 02:00:09.514778 13644 tablet_replica.cc:330] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 2a01ace80baf4057b6bf1995cf88468b: stopping tablet replica
I20240503 02:00:09.514755 13643 tablet_replica.cc:330] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 02:00:09.515065 13643 raft_consensus.cc:2229] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:09.515204 13644 raft_consensus.cc:2229] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:09.515136 13495 raft_consensus.cc:2932] T beeea71f02ae4c89a1af13d3584ef70a P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:09.515329 13643 raft_consensus.cc:2258] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:09.515347 13644 raft_consensus.cc:2258] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:09.515498 6589 raft_consensus.cc:2932] T beeea71f02ae4c89a1af13d3584ef70a P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:09.515923 13643 ts_tablet_manager.cc:1875] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:09.514896 13638 raft_consensus.cc:2229] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:09.516291 13638 raft_consensus.cc:2258] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:09.516770 13644 ts_tablet_manager.cc:1875] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 2a01ace80baf4057b6bf1995cf88468b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:09.517235 13638 ts_tablet_manager.cc:1875] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 99ead02915954b4fab23b033ad6d546b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:09.517185 9179 raft_consensus.cc:2932] T beeea71f02ae4c89a1af13d3584ef70a P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:09.518862 6308 catalog_manager.cc:5484] T beeea71f02ae4c89a1af13d3584ef70a P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: config changed from index 2 to 3, 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "99ead02915954b4fab23b033ad6d546b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:09.520248 13644 ts_tablet_manager.cc:1888] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 2a01ace80baf4057b6bf1995cf88468b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:09.520327 13644 log.cc:1196] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 2a01ace80baf4057b6bf1995cf88468b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/7a90fdf10e3c4dd8be7d5e1b54e9a1da
I20240503 02:00:09.520591 13644 ts_tablet_manager.cc:1909] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 2a01ace80baf4057b6bf1995cf88468b: Deleting consensus metadata
I20240503 02:00:09.521564 6300 catalog_manager.cc:4830] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079): tablet 7a90fdf10e3c4dd8be7d5e1b54e9a1da (table rebalancer_dynamic_table_new_name [id=73d45da493d442e489ee6add747cad2b]) successfully deleted
I20240503 02:00:09.521803 13643 ts_tablet_manager.cc:1888] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:09.521893 13643 log.cc:1196] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/7a90fdf10e3c4dd8be7d5e1b54e9a1da
I20240503 02:00:09.522207 13643 ts_tablet_manager.cc:1909] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 80fd18ef4d604c86b909d5c008a46b4f: Deleting consensus metadata
I20240503 02:00:09.523046 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet 7a90fdf10e3c4dd8be7d5e1b54e9a1da (table rebalancer_dynamic_table_new_name [id=73d45da493d442e489ee6add747cad2b]) successfully deleted
I20240503 02:00:09.523798 13638 ts_tablet_manager.cc:1888] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 99ead02915954b4fab23b033ad6d546b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:09.523873 13638 log.cc:1196] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 99ead02915954b4fab23b033ad6d546b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/7a90fdf10e3c4dd8be7d5e1b54e9a1da
I20240503 02:00:09.524065 13638 ts_tablet_manager.cc:1909] T 7a90fdf10e3c4dd8be7d5e1b54e9a1da P 99ead02915954b4fab23b033ad6d546b: Deleting consensus metadata
I20240503 02:00:09.524916 6301 catalog_manager.cc:4830] TS 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157): tablet 7a90fdf10e3c4dd8be7d5e1b54e9a1da (table rebalancer_dynamic_table_new_name [id=73d45da493d442e489ee6add747cad2b]) successfully deleted
I20240503 02:00:09.546100 13631 raft_consensus.cc:1050] T aa281822047841fab7c58b53d9733e49 P 99ead02915954b4fab23b033ad6d546b: attempting to promote NON_VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 to VOTER
I20240503 02:00:09.546546 13631 consensus_queue.cc:237] T aa281822047841fab7c58b53d9733e49 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } }
I20240503 02:00:09.548002 6719 raft_consensus.cc:1261] T aa281822047841fab7c58b53d9733e49 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEARNER]: Refusing update from remote peer 99ead02915954b4fab23b033ad6d546b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:09.548475 13631 consensus_queue.cc:1035] T aa281822047841fab7c58b53d9733e49 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240503 02:00:09.550274 8982 consensus_peers.cc:485] T aa281822047841fab7c58b53d9733e49 P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:09.550779 9179 raft_consensus.cc:1261] T aa281822047841fab7c58b53d9733e49 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer 99ead02915954b4fab23b033ad6d546b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:09.551157 10473 consensus_queue.cc:1035] T aa281822047841fab7c58b53d9733e49 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:09.552815 13634 raft_consensus.cc:1050] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2: attempting to promote NON_VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 to VOTER
I20240503 02:00:09.553207 13634 consensus_queue.cc:237] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } }
I20240503 02:00:09.554414 6719 raft_consensus.cc:1261] T 1c721b33983b4c47afd8faa8f155ca84 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEARNER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:09.554920 6850 raft_consensus.cc:1261] T 1c721b33983b4c47afd8faa8f155ca84 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:09.554921 6456 raft_consensus.cc:1261] T 1c721b33983b4c47afd8faa8f155ca84 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:09.555334 13634 consensus_queue.cc:1035] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:09.555555 13018 consensus_queue.cc:1035] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:09.555740 13102 consensus_queue.cc:1035] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:09.558002 13559 raft_consensus.cc:2932] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } } }
I20240503 02:00:09.559485 6456 raft_consensus.cc:2932] T 1c721b33983b4c47afd8faa8f155ca84 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } } }
I20240503 02:00:09.561419 6313 catalog_manager.cc:5484] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2 reported cstate change: config changed from index 5 to 6, c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9942306855404cfd891adf227dd65eb2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:09.561410 6850 raft_consensus.cc:2932] T 1c721b33983b4c47afd8faa8f155ca84 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } } }
I20240503 02:00:09.561570 6719 raft_consensus.cc:2932] T 1c721b33983b4c47afd8faa8f155ca84 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { replace: true } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } } }
I20240503 02:00:09.562937 12907 raft_consensus.cc:482] T b2e62c043adf4a8d9024279d786f596c P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e1a9251ce1cf4361b8b0ede9fbfc4023)
I20240503 02:00:09.563046 12907 raft_consensus.cc:504] T b2e62c043adf4a8d9024279d786f596c P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } }
I20240503 02:00:09.563472 12907 leader_election.cc:288] T b2e62c043adf4a8d9024279d786f596c P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311)
I20240503 02:00:09.566058 6850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2e62c043adf4a8d9024279d786f596c" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
W20240503 02:00:09.566674 9113 leader_election.cc:334] T b2e62c043adf4a8d9024279d786f596c P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:09.566754 9113 leader_election.cc:302] T b2e62c043adf4a8d9024279d786f596c P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b; no voters: 492448570cc94ed1879feb2ec3b9051d, e1a9251ce1cf4361b8b0ede9fbfc4023
I20240503 02:00:09.567063 12907 raft_consensus.cc:2726] T b2e62c043adf4a8d9024279d786f596c P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:09.569295 13631 raft_consensus.cc:2932] T aa281822047841fab7c58b53d9733e49 P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } } }
I20240503 02:00:09.569757 9179 raft_consensus.cc:2932] T aa281822047841fab7c58b53d9733e49 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } } }
I20240503 02:00:09.571523 6719 raft_consensus.cc:2932] T aa281822047841fab7c58b53d9733e49 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { replace: true } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } } }
I20240503 02:00:09.572793 6308 catalog_manager.cc:5484] T aa281822047841fab7c58b53d9733e49 P 99ead02915954b4fab23b033ad6d546b reported cstate change: config changed from index 2 to 3, c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "99ead02915954b4fab23b033ad6d546b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:09.574023 12907 raft_consensus.cc:1050] T b20ff3a6f74f4640afebfd4059b58699 P 2a01ace80baf4057b6bf1995cf88468b: attempting to promote NON_VOTER dec3e4a04d604f7eaaa3f077d081218a to VOTER
I20240503 02:00:09.574445 12907 consensus_queue.cc:237] T b20ff3a6f74f4640afebfd4059b58699 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } }
I20240503 02:00:09.575690 6456 raft_consensus.cc:1261] T b20ff3a6f74f4640afebfd4059b58699 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEARNER]: Refusing update from remote peer 2a01ace80baf4057b6bf1995cf88468b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:09.576154 6719 raft_consensus.cc:1261] T b20ff3a6f74f4640afebfd4059b58699 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer 2a01ace80baf4057b6bf1995cf88468b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:09.576613 13629 consensus_queue.cc:1035] T b20ff3a6f74f4640afebfd4059b58699 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:09.576911 13098 consensus_queue.cc:1035] T b20ff3a6f74f4640afebfd4059b58699 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:09.577023 9039 raft_consensus.cc:1261] T b20ff3a6f74f4640afebfd4059b58699 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer 2a01ace80baf4057b6bf1995cf88468b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:09.577605 13098 consensus_queue.cc:1035] T b20ff3a6f74f4640afebfd4059b58699 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:09.578176 9039 consensus_queue.cc:237] T aa281822047841fab7c58b53d9733e49 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } }
I20240503 02:00:09.579339 9179 raft_consensus.cc:1261] T aa281822047841fab7c58b53d9733e49 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer 99ead02915954b4fab23b033ad6d546b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:09.579340 6720 raft_consensus.cc:1261] T aa281822047841fab7c58b53d9733e49 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer 99ead02915954b4fab23b033ad6d546b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:09.579718 10473 consensus_queue.cc:1035] T aa281822047841fab7c58b53d9733e49 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:09.579897 13098 raft_consensus.cc:2932] T b20ff3a6f74f4640afebfd4059b58699 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } }
I20240503 02:00:09.580035 12922 consensus_queue.cc:1035] T aa281822047841fab7c58b53d9733e49 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:09.581183 10473 raft_consensus.cc:2932] T aa281822047841fab7c58b53d9733e49 P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } } }
I20240503 02:00:09.581611 9179 raft_consensus.cc:2932] T aa281822047841fab7c58b53d9733e49 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } } }
I20240503 02:00:09.582527 6720 raft_consensus.cc:2932] T b20ff3a6f74f4640afebfd4059b58699 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } }
I20240503 02:00:09.582901 6301 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet aa281822047841fab7c58b53d9733e49 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:09.583675 6313 catalog_manager.cc:5484] T b20ff3a6f74f4640afebfd4059b58699 P 2a01ace80baf4057b6bf1995cf88468b reported cstate change: config changed from index 2 to 3, dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2a01ace80baf4057b6bf1995cf88468b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:09.585357 6309 catalog_manager.cc:5484] T aa281822047841fab7c58b53d9733e49 P 99ead02915954b4fab23b033ad6d546b reported cstate change: config changed from index 3 to 4, VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "99ead02915954b4fab23b033ad6d546b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:09.585660 6456 raft_consensus.cc:2932] T b20ff3a6f74f4640afebfd4059b58699 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } }
I20240503 02:00:09.587291 9179 consensus_queue.cc:237] T b20ff3a6f74f4640afebfd4059b58699 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } }
I20240503 02:00:09.587875 9048 raft_consensus.cc:2932] T b20ff3a6f74f4640afebfd4059b58699 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } }
I20240503 02:00:09.588675 6719 raft_consensus.cc:1261] T b20ff3a6f74f4640afebfd4059b58699 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer 2a01ace80baf4057b6bf1995cf88468b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:09.589674 6456 raft_consensus.cc:1261] T b20ff3a6f74f4640afebfd4059b58699 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 2a01ace80baf4057b6bf1995cf88468b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:09.589784 12907 consensus_queue.cc:1035] T b20ff3a6f74f4640afebfd4059b58699 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:09.590021 13629 consensus_queue.cc:1035] T b20ff3a6f74f4640afebfd4059b58699 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:00:09.591463 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:09.592409 6720 raft_consensus.cc:2932] T aa281822047841fab7c58b53d9733e49 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { promote: false } } }
I20240503 02:00:09.595741 13629 raft_consensus.cc:2932] T b20ff3a6f74f4640afebfd4059b58699 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } }
I20240503 02:00:09.595817 6456 raft_consensus.cc:2932] T b20ff3a6f74f4640afebfd4059b58699 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } }
I20240503 02:00:09.596071 6719 raft_consensus.cc:2932] T b20ff3a6f74f4640afebfd4059b58699 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } }
I20240503 02:00:09.597739 6300 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b20ff3a6f74f4640afebfd4059b58699 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:09.598503 6308 catalog_manager.cc:5484] T b20ff3a6f74f4640afebfd4059b58699 P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: config changed from index 3 to 4, VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "2a01ace80baf4057b6bf1995cf88468b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } attrs { promote: false } } }
I20240503 02:00:09.600836 9028 tablet_service.cc:1514] Processing DeleteTablet for tablet b20ff3a6f74f4640afebfd4059b58699 with delete_type TABLET_DATA_TOMBSTONED (TS 99ead02915954b4fab23b033ad6d546b not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52584
I20240503 02:00:09.601137 13638 tablet_replica.cc:330] T b20ff3a6f74f4640afebfd4059b58699 P 99ead02915954b4fab23b033ad6d546b: stopping tablet replica
I20240503 02:00:09.601275 13638 raft_consensus.cc:2229] T b20ff3a6f74f4640afebfd4059b58699 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:09.601419 13638 raft_consensus.cc:2258] T b20ff3a6f74f4640afebfd4059b58699 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:09.602030 13638 ts_tablet_manager.cc:1875] T b20ff3a6f74f4640afebfd4059b58699 P 99ead02915954b4fab23b033ad6d546b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:09.604246 13638 ts_tablet_manager.cc:1888] T b20ff3a6f74f4640afebfd4059b58699 P 99ead02915954b4fab23b033ad6d546b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:00:09.604318 13638 log.cc:1196] T b20ff3a6f74f4640afebfd4059b58699 P 99ead02915954b4fab23b033ad6d546b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/b20ff3a6f74f4640afebfd4059b58699
I20240503 02:00:09.604844 6301 catalog_manager.cc:4830] TS 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157): tablet b20ff3a6f74f4640afebfd4059b58699 (table rebalancer_extra_table_18 [id=e035403de1c6423e98f3bb44dcb47f8a]) successfully deleted
I20240503 02:00:09.610656 13298 raft_consensus.cc:1050] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a: attempting to promote NON_VOTER 2a01ace80baf4057b6bf1995cf88468b to VOTER
I20240503 02:00:09.611199 13298 consensus_queue.cc:237] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } }
I20240503 02:00:09.612859 6719 raft_consensus.cc:1261] T 2d7244b0dc4d419cb5c3f3677acd70c7 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:09.613085 9176 raft_consensus.cc:1261] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEARNER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:09.613117 9048 raft_consensus.cc:1261] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:09.613189 13495 raft_consensus.cc:1050] T e52373099af3428497d176e3fcd8ef27 P 99ead02915954b4fab23b033ad6d546b: attempting to promote NON_VOTER 80fd18ef4d604c86b909d5c008a46b4f to VOTER
I20240503 02:00:09.613540 13626 consensus_queue.cc:1035] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:09.613600 13495 consensus_queue.cc:237] T e52373099af3428497d176e3fcd8ef27 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } }
I20240503 02:00:09.613925 13626 consensus_queue.cc:1035] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:09.614063 13657 consensus_queue.cc:1035] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:09.614945 6589 raft_consensus.cc:1261] T e52373099af3428497d176e3fcd8ef27 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEARNER]: Refusing update from remote peer 99ead02915954b4fab23b033ad6d546b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:09.615249 6720 raft_consensus.cc:1261] T e52373099af3428497d176e3fcd8ef27 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer 99ead02915954b4fab23b033ad6d546b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:09.615514 12922 consensus_queue.cc:1035] T e52373099af3428497d176e3fcd8ef27 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:09.615778 10473 consensus_queue.cc:1035] T e52373099af3428497d176e3fcd8ef27 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:09.616504 9178 raft_consensus.cc:1261] T e52373099af3428497d176e3fcd8ef27 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer 99ead02915954b4fab23b033ad6d546b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:09.616525 6308 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:00:09.618618 12922 consensus_queue.cc:1035] T e52373099af3428497d176e3fcd8ef27 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:09.619658 13298 raft_consensus.cc:2932] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:09.620023 10473 raft_consensus.cc:2932] T e52373099af3428497d176e3fcd8ef27 P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:09.621601 9048 raft_consensus.cc:2932] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:09.622074 9179 raft_consensus.cc:2932] T e52373099af3428497d176e3fcd8ef27 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:09.623287 6589 raft_consensus.cc:2932] T e52373099af3428497d176e3fcd8ef27 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:09.623648 9178 raft_consensus.cc:2932] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:09.624460 6309 catalog_manager.cc:5484] T e52373099af3428497d176e3fcd8ef27 P 99ead02915954b4fab23b033ad6d546b reported cstate change: config changed from index 2 to 3, 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "99ead02915954b4fab23b033ad6d546b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:09.625151 6308 catalog_manager.cc:5484] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: config changed from index 7 to 8, 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:09.625960 6719 raft_consensus.cc:2932] T e52373099af3428497d176e3fcd8ef27 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:09.627414 6436 tablet_service.cc:1467] Processing CreateTablet for tablet ad245987cb6c4d0680a1597edc14801b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b3c4040b69424425906498cc28fc97b6]), partition=
I20240503 02:00:09.627642 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad245987cb6c4d0680a1597edc14801b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:09.629142 9157 tablet_service.cc:1467] Processing CreateTablet for tablet ad245987cb6c4d0680a1597edc14801b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b3c4040b69424425906498cc28fc97b6]), partition=
I20240503 02:00:09.629567 9157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad245987cb6c4d0680a1597edc14801b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:09.629766 13607 tablet_bootstrap.cc:492] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:09.626006 6720 raft_consensus.cc:2932] T 2d7244b0dc4d419cb5c3f3677acd70c7 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:09.630642 13607 tablet_bootstrap.cc:654] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:09.631814 8887 tablet_service.cc:1467] Processing CreateTablet for tablet ad245987cb6c4d0680a1597edc14801b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b3c4040b69424425906498cc28fc97b6]), partition=
I20240503 02:00:09.632054 8887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad245987cb6c4d0680a1597edc14801b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:09.632516 9048 consensus_queue.cc:237] T e52373099af3428497d176e3fcd8ef27 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } }
I20240503 02:00:09.633471 6456 consensus_queue.cc:237] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } }
I20240503 02:00:09.633898 6720 raft_consensus.cc:1261] T e52373099af3428497d176e3fcd8ef27 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer 99ead02915954b4fab23b033ad6d546b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:09.634280 12922 consensus_queue.cc:1035] T e52373099af3428497d176e3fcd8ef27 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:09.637229 12922 raft_consensus.cc:2932] T e52373099af3428497d176e3fcd8ef27 P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:09.637585 6720 raft_consensus.cc:2932] T e52373099af3428497d176e3fcd8ef27 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:09.638069 13181 tablet_bootstrap.cc:492] T ad245987cb6c4d0680a1597edc14801b P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:09.638882 13181 tablet_bootstrap.cc:654] T ad245987cb6c4d0680a1597edc14801b P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:09.638936 6301 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e52373099af3428497d176e3fcd8ef27 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:09.639323 6589 raft_consensus.cc:1261] T e52373099af3428497d176e3fcd8ef27 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer 99ead02915954b4fab23b033ad6d546b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:09.639662 9048 raft_consensus.cc:1261] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:09.640041 13495 consensus_queue.cc:1035] T e52373099af3428497d176e3fcd8ef27 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:09.640480 13606 tablet_bootstrap.cc:492] T ad245987cb6c4d0680a1597edc14801b P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:09.640480 13657 consensus_queue.cc:1035] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:09.641371 13606 tablet_bootstrap.cc:654] T ad245987cb6c4d0680a1597edc14801b P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:09.642168 6589 raft_consensus.cc:2932] T e52373099af3428497d176e3fcd8ef27 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:09.642671 13606 tablet_bootstrap.cc:492] T ad245987cb6c4d0680a1597edc14801b P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:09.642760 13606 ts_tablet_manager.cc:1367] T ad245987cb6c4d0680a1597edc14801b P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:09.642987 9178 raft_consensus.cc:1261] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:09.643198 13606 raft_consensus.cc:348] T ad245987cb6c4d0680a1597edc14801b P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:09.643299 13606 raft_consensus.cc:374] T ad245987cb6c4d0680a1597edc14801b P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:09.643330 13606 raft_consensus.cc:729] T ad245987cb6c4d0680a1597edc14801b P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:09.643443 13606 consensus_queue.cc:260] T ad245987cb6c4d0680a1597edc14801b P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:09.643743 13606 ts_tablet_manager.cc:1398] T ad245987cb6c4d0680a1597edc14801b P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:09.643850 13657 consensus_queue.cc:1035] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:09.643841 6313 catalog_manager.cc:5484] T e52373099af3428497d176e3fcd8ef27 P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: config changed from index 3 to 4, VOTER 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "99ead02915954b4fab23b033ad6d546b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:09.644470 12190 raft_consensus.cc:2932] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:09.646615 6300 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2d7244b0dc4d419cb5c3f3677acd70c7 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:09.647231 9048 raft_consensus.cc:2932] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:09.647420 6312 catalog_manager.cc:5484] T 2d7244b0dc4d419cb5c3f3677acd70c7 P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: config changed from index 8 to 9, VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:09.647907 13607 tablet_bootstrap.cc:492] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:09.648000 13607 ts_tablet_manager.cc:1367] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20240503 02:00:09.648541 13607 raft_consensus.cc:348] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:09.648646 13607 raft_consensus.cc:374] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:09.648676 13607 raft_consensus.cc:729] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:09.648787 13607 consensus_queue.cc:260] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:09.649057 13607 ts_tablet_manager.cc:1398] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:09.650694 6700 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d7244b0dc4d419cb5c3f3677acd70c7 with delete_type TABLET_DATA_TOMBSTONED (TS c2ac2eb7cc264047b9ab32fe3a4fc025 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:40336
I20240503 02:00:09.651727 13626 consensus_queue.cc:1035] T be7bbd71b99847068f415cc3d3d49ddd P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:09.652858 9157 tablet_service.cc:1514] Processing DeleteTablet for tablet e52373099af3428497d176e3fcd8ef27 with delete_type TABLET_DATA_TOMBSTONED (TS 2a01ace80baf4057b6bf1995cf88468b not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:51940
I20240503 02:00:09.653249 13644 tablet_replica.cc:330] T e52373099af3428497d176e3fcd8ef27 P 2a01ace80baf4057b6bf1995cf88468b: stopping tablet replica
I20240503 02:00:09.653386 13644 raft_consensus.cc:2229] T e52373099af3428497d176e3fcd8ef27 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:09.653515 13644 raft_consensus.cc:2258] T e52373099af3428497d176e3fcd8ef27 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:09.654845 13644 ts_tablet_manager.cc:1875] T e52373099af3428497d176e3fcd8ef27 P 2a01ace80baf4057b6bf1995cf88468b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:09.658095 13298 consensus_queue.cc:1035] T be7bbd71b99847068f415cc3d3d49ddd P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:09.662212 13644 ts_tablet_manager.cc:1888] T e52373099af3428497d176e3fcd8ef27 P 2a01ace80baf4057b6bf1995cf88468b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:00:09.662319 13644 log.cc:1196] T e52373099af3428497d176e3fcd8ef27 P 2a01ace80baf4057b6bf1995cf88468b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/e52373099af3428497d176e3fcd8ef27
I20240503 02:00:09.663182 6300 catalog_manager.cc:4830] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079): tablet e52373099af3428497d176e3fcd8ef27 (table rebalancer_extra_table_12 [id=2d54fb96d5ab4898ac53ffc16334115d]) successfully deleted
I20240503 02:00:09.663509 9178 raft_consensus.cc:2932] T 2d7244b0dc4d419cb5c3f3677acd70c7 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:09.665685 13657 raft_consensus.cc:482] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:09.665809 13657 raft_consensus.cc:504] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:09.666196 13657 leader_election.cc:288] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:09.666808 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad245987cb6c4d0680a1597edc14801b" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:09.666954 8916 raft_consensus.cc:2454] T ad245987cb6c4d0680a1597edc14801b P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:09.667150 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad245987cb6c4d0680a1597edc14801b" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:09.667404 6394 leader_election.cc:302] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9942306855404cfd891adf227dd65eb2, dec3e4a04d604f7eaaa3f077d081218a; no voters:
W20240503 02:00:09.667517 6391 leader_election.cc:341] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:00:09.667567 13626 raft_consensus.cc:2781] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:09.667604 13626 raft_consensus.cc:482] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:09.667649 13626 raft_consensus.cc:3037] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:09.668336 13181 tablet_bootstrap.cc:492] T ad245987cb6c4d0680a1597edc14801b P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:09.668463 13181 ts_tablet_manager.cc:1367] T ad245987cb6c4d0680a1597edc14801b P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.030s user 0.001s sys 0.001s
I20240503 02:00:09.668839 13626 raft_consensus.cc:504] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:09.668933 13181 raft_consensus.cc:348] T ad245987cb6c4d0680a1597edc14801b P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:09.669034 13181 raft_consensus.cc:374] T ad245987cb6c4d0680a1597edc14801b P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:09.669072 13181 raft_consensus.cc:729] T ad245987cb6c4d0680a1597edc14801b P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:09.669219 13181 consensus_queue.cc:260] T ad245987cb6c4d0680a1597edc14801b P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:09.669533 13181 ts_tablet_manager.cc:1398] T ad245987cb6c4d0680a1597edc14801b P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:09.669570 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad245987cb6c4d0680a1597edc14801b" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:09.669662 9178 raft_consensus.cc:3037] T ad245987cb6c4d0680a1597edc14801b P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:09.669881 13669 tablet_replica.cc:330] T 2d7244b0dc4d419cb5c3f3677acd70c7 P c2ac2eb7cc264047b9ab32fe3a4fc025: stopping tablet replica
I20240503 02:00:09.670009 13669 raft_consensus.cc:2229] T 2d7244b0dc4d419cb5c3f3677acd70c7 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:09.670226 13669 raft_consensus.cc:2258] T 2d7244b0dc4d419cb5c3f3677acd70c7 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:09.670641 9178 raft_consensus.cc:2454] T ad245987cb6c4d0680a1597edc14801b P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:09.670820 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad245987cb6c4d0680a1597edc14801b" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:09.670946 8916 raft_consensus.cc:3037] T ad245987cb6c4d0680a1597edc14801b P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:09.670957 6391 leader_election.cc:302] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:09.671244 13657 raft_consensus.cc:2781] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:09.671329 13657 raft_consensus.cc:686] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Becoming Leader. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Running, Role: LEADER
I20240503 02:00:09.671459 13657 consensus_queue.cc:237] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:09.671882 13626 leader_election.cc:288] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Requested vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:09.673456 13669 ts_tablet_manager.cc:1875] T 2d7244b0dc4d419cb5c3f3677acd70c7 P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:09.674970 8916 raft_consensus.cc:2454] T ad245987cb6c4d0680a1597edc14801b P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:09.677033 6309 catalog_manager.cc:5484] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: term changed from 0 to 1, leader changed from <none> to dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129). New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:09.677515 13669 ts_tablet_manager.cc:1888] T 2d7244b0dc4d419cb5c3f3677acd70c7 P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 02:00:09.677587 13669 log.cc:1196] T 2d7244b0dc4d419cb5c3f3677acd70c7 P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/2d7244b0dc4d419cb5c3f3677acd70c7
I20240503 02:00:09.678321 6303 catalog_manager.cc:4830] TS c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655): tablet 2d7244b0dc4d419cb5c3f3677acd70c7 (table rebalance_test_table_17 [id=5ccf0adaf2de4b79bbf38c053646e60e]) successfully deleted
I20240503 02:00:09.733016 13496 raft_consensus.cc:1050] T cea1ea50fa434d31b42451f84f26bc76 P c2ac2eb7cc264047b9ab32fe3a4fc025: attempting to promote NON_VOTER 492448570cc94ed1879feb2ec3b9051d to VOTER
I20240503 02:00:09.733390 13496 consensus_queue.cc:237] T cea1ea50fa434d31b42451f84f26bc76 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { promote: false } }
I20240503 02:00:09.734695 12872 raft_consensus.cc:1050] T d73ea6e550a04c04a2695b058e38c566 P c2ac2eb7cc264047b9ab32fe3a4fc025: attempting to promote NON_VOTER 9942306855404cfd891adf227dd65eb2 to VOTER
I20240503 02:00:09.735095 12872 consensus_queue.cc:237] T d73ea6e550a04c04a2695b058e38c566 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } }
I20240503 02:00:09.735167 9048 raft_consensus.cc:1261] T cea1ea50fa434d31b42451f84f26bc76 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:09.735536 13496 consensus_queue.cc:1035] T cea1ea50fa434d31b42451f84f26bc76 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240503 02:00:09.735992 6654 consensus_peers.cc:485] T cea1ea50fa434d31b42451f84f26bc76 P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:09.736331 8916 raft_consensus.cc:1261] T d73ea6e550a04c04a2695b058e38c566 P 9942306855404cfd891adf227dd65eb2 [term 1 LEARNER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:09.736527 6850 raft_consensus.cc:1261] T cea1ea50fa434d31b42451f84f26bc76 P 492448570cc94ed1879feb2ec3b9051d [term 1 LEARNER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:09.736552 6456 raft_consensus.cc:1261] T d73ea6e550a04c04a2695b058e38c566 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:09.736770 12872 consensus_queue.cc:1035] T d73ea6e550a04c04a2695b058e38c566 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:09.737046 13625 consensus_queue.cc:1035] T cea1ea50fa434d31b42451f84f26bc76 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:09.737082 12872 consensus_queue.cc:1035] T d73ea6e550a04c04a2695b058e38c566 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:09.738212 6589 raft_consensus.cc:1261] T d73ea6e550a04c04a2695b058e38c566 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:09.739635 13679 consensus_queue.cc:1035] T d73ea6e550a04c04a2695b058e38c566 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:09.739941 13625 raft_consensus.cc:2932] T cea1ea50fa434d31b42451f84f26bc76 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { promote: false } } }
I20240503 02:00:09.741212 13056 consensus_queue.cc:1035] T 7cecefe8f1864583be9d0b9c308a78db P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:09.741969 13625 raft_consensus.cc:2932] T d73ea6e550a04c04a2695b058e38c566 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:09.742405 6589 raft_consensus.cc:2932] T d73ea6e550a04c04a2695b058e38c566 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:09.743615 6314 catalog_manager.cc:5484] T cea1ea50fa434d31b42451f84f26bc76 P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: config changed from index 2 to 3, 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:09.745079 9048 raft_consensus.cc:2932] T cea1ea50fa434d31b42451f84f26bc76 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { promote: false } } }
I20240503 02:00:09.746120 8916 raft_consensus.cc:2932] T d73ea6e550a04c04a2695b058e38c566 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:09.747133 6850 raft_consensus.cc:2932] T cea1ea50fa434d31b42451f84f26bc76 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 492448570cc94ed1879feb2ec3b9051d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } attrs { promote: false } } }
I20240503 02:00:09.749919 6309 catalog_manager.cc:5484] T d73ea6e550a04c04a2695b058e38c566 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: config changed from index 4 to 5, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:09.750176 6456 raft_consensus.cc:2932] T d73ea6e550a04c04a2695b058e38c566 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:09.751888 13056 consensus_queue.cc:1035] T 7cecefe8f1864583be9d0b9c308a78db P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:09.782696 6308 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:09.786841 9178 raft_consensus.cc:1261] T ad245987cb6c4d0680a1597edc14801b P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:09.786985 8916 raft_consensus.cc:1261] T ad245987cb6c4d0680a1597edc14801b P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:09.787315 13298 consensus_queue.cc:1035] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:09.787483 13657 consensus_queue.cc:1035] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:09.790390 6308 catalog_manager.cc:5576] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211) does not have the latest schema for tablet ad245987cb6c4d0680a1597edc14801b (table rebalancer_dynamic_table [id=b3c4040b69424425906498cc28fc97b6]). Expected version 1 got 0
I20240503 02:00:09.790912 13690 tablet.cc:1722] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:09.791142 6314 catalog_manager.cc:5576] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079) does not have the latest schema for tablet ad245987cb6c4d0680a1597edc14801b (table rebalancer_dynamic_table [id=b3c4040b69424425906498cc28fc97b6]). Expected version 1 got 0
I20240503 02:00:09.791280 13690 tablet.cc:1620] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a: MemRowSet was empty: no flush needed.
I20240503 02:00:09.792474 13622 tablet.cc:1722] T ad245987cb6c4d0680a1597edc14801b P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:09.792804 13622 tablet.cc:1620] T ad245987cb6c4d0680a1597edc14801b P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
I20240503 02:00:09.795266 6313 catalog_manager.cc:5576] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211) does not have the latest schema for tablet ad245987cb6c4d0680a1597edc14801b (table rebalancer_dynamic_table [id=b3c4040b69424425906498cc28fc97b6]). Expected version 1 got 0
I20240503 02:00:09.795581 6314 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=b3c4040b69424425906498cc28fc97b6] alter complete (version 1)
I20240503 02:00:09.800688 13691 tablet.cc:1722] T ad245987cb6c4d0680a1597edc14801b P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:09.801137 13691 tablet.cc:1620] T ad245987cb6c4d0680a1597edc14801b P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
I20240503 02:00:09.852532 13625 consensus_queue.cc:1035] T e617d4e31eed42dcb39e03cc40b8e6a6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:09.855805 13679 consensus_queue.cc:1035] T e617d4e31eed42dcb39e03cc40b8e6a6 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:09.857746 13657 raft_consensus.cc:1050] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a: attempting to promote NON_VOTER 2a01ace80baf4057b6bf1995cf88468b to VOTER
I20240503 02:00:09.858209 13657 consensus_queue.cc:237] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } }
I20240503 02:00:09.859539 9178 raft_consensus.cc:1261] T ac7fa0c3722a461c91fb237fd85794a3 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEARNER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:09.859875 13626 consensus_queue.cc:1035] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:09.860325 9039 raft_consensus.cc:1261] T ac7fa0c3722a461c91fb237fd85794a3 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:09.860365 6589 raft_consensus.cc:1261] T ac7fa0c3722a461c91fb237fd85794a3 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:00:09.860754 13626 consensus_queue.cc:1035] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:09.860952 13657 consensus_queue.cc:1035] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:09.862726 12190 raft_consensus.cc:2932] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:09.863317 6589 raft_consensus.cc:2932] T ac7fa0c3722a461c91fb237fd85794a3 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:09.863909 13298 raft_consensus.cc:1050] T 9cbb2be1fda74ebe8997123c70629ba3 P dec3e4a04d604f7eaaa3f077d081218a: attempting to promote NON_VOTER 80fd18ef4d604c86b909d5c008a46b4f to VOTER
I20240503 02:00:09.864260 13298 consensus_queue.cc:237] T 9cbb2be1fda74ebe8997123c70629ba3 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } }
I20240503 02:00:09.865931 9048 raft_consensus.cc:1261] T 9cbb2be1fda74ebe8997123c70629ba3 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:09.865885 9039 raft_consensus.cc:2932] T ac7fa0c3722a461c91fb237fd85794a3 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:09.866461 13657 consensus_queue.cc:1035] T 9cbb2be1fda74ebe8997123c70629ba3 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:09.866444 6309 catalog_manager.cc:5484] T ac7fa0c3722a461c91fb237fd85794a3 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: config changed from index 7 to 8, 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:09.867554 8916 raft_consensus.cc:1261] T 9cbb2be1fda74ebe8997123c70629ba3 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:09.867388 9178 raft_consensus.cc:2932] T ac7fa0c3722a461c91fb237fd85794a3 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:09.869542 12190 consensus_queue.cc:1035] T 9cbb2be1fda74ebe8997123c70629ba3 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:09.870330 6589 raft_consensus.cc:1261] T 9cbb2be1fda74ebe8997123c70629ba3 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEARNER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:09.871026 13626 consensus_queue.cc:1035] T 9cbb2be1fda74ebe8997123c70629ba3 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:09.872356 6456 consensus_queue.cc:237] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } }
I20240503 02:00:09.873651 9039 raft_consensus.cc:1261] T ac7fa0c3722a461c91fb237fd85794a3 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:09.873839 9178 raft_consensus.cc:1261] T ac7fa0c3722a461c91fb237fd85794a3 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:00:09.874102 12190 raft_consensus.cc:2932] T 9cbb2be1fda74ebe8997123c70629ba3 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:09.874325 13626 consensus_queue.cc:1035] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:09.874459 6589 raft_consensus.cc:2932] T 9cbb2be1fda74ebe8997123c70629ba3 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:09.874635 13626 consensus_queue.cc:1035] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:00:09.876227 8916 raft_consensus.cc:2932] T 9cbb2be1fda74ebe8997123c70629ba3 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:09.876636 13298 raft_consensus.cc:2932] T ac7fa0c3722a461c91fb237fd85794a3 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:09.878655 9039 raft_consensus.cc:2932] T ac7fa0c3722a461c91fb237fd85794a3 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:09.878845 9178 raft_consensus.cc:2932] T ac7fa0c3722a461c91fb237fd85794a3 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:09.879130 6309 catalog_manager.cc:5484] T 9cbb2be1fda74ebe8997123c70629ba3 P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: config changed from index 2 to 3, 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:09.881665 6314 catalog_manager.cc:5484] T ac7fa0c3722a461c91fb237fd85794a3 P 99ead02915954b4fab23b033ad6d546b reported cstate change: config changed from index 8 to 9, VOTER 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { promote: false } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { promote: false } } }
I20240503 02:00:09.883559 6300 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ac7fa0c3722a461c91fb237fd85794a3 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:09.885469 6456 consensus_queue.cc:237] T 9cbb2be1fda74ebe8997123c70629ba3 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } }
I20240503 02:00:09.885434 9048 raft_consensus.cc:2932] T 9cbb2be1fda74ebe8997123c70629ba3 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } attrs { replace: true } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:09.887266 8916 raft_consensus.cc:1261] T 9cbb2be1fda74ebe8997123c70629ba3 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:09.887324 6569 tablet_service.cc:1514] Processing DeleteTablet for tablet ac7fa0c3722a461c91fb237fd85794a3 with delete_type TABLET_DATA_TOMBSTONED (TS 80fd18ef4d604c86b909d5c008a46b4f not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:59846
I20240503 02:00:09.887599 13643 tablet_replica.cc:330] T ac7fa0c3722a461c91fb237fd85794a3 P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 02:00:09.887748 13643 raft_consensus.cc:2229] T ac7fa0c3722a461c91fb237fd85794a3 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:09.887884 6589 raft_consensus.cc:1261] T 9cbb2be1fda74ebe8997123c70629ba3 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:09.887938 13643 raft_consensus.cc:2258] T ac7fa0c3722a461c91fb237fd85794a3 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:09.888227 12190 consensus_queue.cc:1035] T 9cbb2be1fda74ebe8997123c70629ba3 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:09.888465 13298 consensus_queue.cc:1035] T 9cbb2be1fda74ebe8997123c70629ba3 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:09.888988 13643 ts_tablet_manager.cc:1875] T ac7fa0c3722a461c91fb237fd85794a3 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:09.890179 12190 raft_consensus.cc:2932] T 9cbb2be1fda74ebe8997123c70629ba3 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:09.890650 8916 raft_consensus.cc:2932] T 9cbb2be1fda74ebe8997123c70629ba3 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:09.891294 13643 ts_tablet_manager.cc:1888] T ac7fa0c3722a461c91fb237fd85794a3 P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 02:00:09.891356 13643 log.cc:1196] T ac7fa0c3722a461c91fb237fd85794a3 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/ac7fa0c3722a461c91fb237fd85794a3
I20240503 02:00:09.891466 6300 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9cbb2be1fda74ebe8997123c70629ba3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:09.891847 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet ac7fa0c3722a461c91fb237fd85794a3 (table rebalance_test_table_3 [id=72f29547da0b4a95974018c2782cb359]) successfully deleted
I20240503 02:00:09.894189 6582 raft_consensus.cc:2932] T 9cbb2be1fda74ebe8997123c70629ba3 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:09.894291 6312 catalog_manager.cc:5484] T 9cbb2be1fda74ebe8997123c70629ba3 P 9942306855404cfd891adf227dd65eb2 reported cstate change: config changed from index 3 to 4, VOTER 99ead02915954b4fab23b033ad6d546b (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } attrs { promote: false } } }
I20240503 02:00:09.896818 9028 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cbb2be1fda74ebe8997123c70629ba3 with delete_type TABLET_DATA_TOMBSTONED (TS 99ead02915954b4fab23b033ad6d546b not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52584
I20240503 02:00:09.897073 13638 tablet_replica.cc:330] T 9cbb2be1fda74ebe8997123c70629ba3 P 99ead02915954b4fab23b033ad6d546b: stopping tablet replica
I20240503 02:00:09.897208 13638 raft_consensus.cc:2229] T 9cbb2be1fda74ebe8997123c70629ba3 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:09.897346 13638 raft_consensus.cc:2258] T 9cbb2be1fda74ebe8997123c70629ba3 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:09.897900 13638 ts_tablet_manager.cc:1875] T 9cbb2be1fda74ebe8997123c70629ba3 P 99ead02915954b4fab23b033ad6d546b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:09.899960 13638 ts_tablet_manager.cc:1888] T 9cbb2be1fda74ebe8997123c70629ba3 P 99ead02915954b4fab23b033ad6d546b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:00:09.900036 13638 log.cc:1196] T 9cbb2be1fda74ebe8997123c70629ba3 P 99ead02915954b4fab23b033ad6d546b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/9cbb2be1fda74ebe8997123c70629ba3
I20240503 02:00:09.899950 6312 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:09.900573 6301 catalog_manager.cc:4830] TS 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157): tablet 9cbb2be1fda74ebe8997123c70629ba3 (table rebalancer_extra_table_16 [id=9b52669a75314ef88150c5df1cac862c]) successfully deleted
I20240503 02:00:09.905081 13690 tablet.cc:1722] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:09.905452 13690 tablet.cc:1620] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a: MemRowSet was empty: no flush needed.
I20240503 02:00:09.905565 13622 tablet.cc:1722] T ad245987cb6c4d0680a1597edc14801b P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:09.905848 13622 tablet.cc:1620] T ad245987cb6c4d0680a1597edc14801b P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
I20240503 02:00:09.905910 13691 tablet.cc:1722] T ad245987cb6c4d0680a1597edc14801b P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:09.906210 13691 tablet.cc:1620] T ad245987cb6c4d0680a1597edc14801b P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
I20240503 02:00:09.911020 6308 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=b3c4040b69424425906498cc28fc97b6] alter complete (version 2)
W20240503 02:00:09.960605 6301 catalog_manager.cc:4573] TS e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet aa281822047841fab7c58b53d9733e49: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:10.013221 6314 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:10.016950 6313 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_extra_table_26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:10.018888 13690 tablet.cc:1722] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:10.019968 13691 tablet.cc:1722] T ad245987cb6c4d0680a1597edc14801b P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:10.023487 13622 tablet.cc:1722] T ad245987cb6c4d0680a1597edc14801b P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:10.024416 6831 tablet_service.cc:1467] Processing CreateTablet for tablet 493053c0766648c3bfeaf2490732bbb8 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=e1b413681beb485981324abe8f697565]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:10.024721 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 493053c0766648c3bfeaf2490732bbb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:10.024698 8887 tablet_service.cc:1467] Processing CreateTablet for tablet 493053c0766648c3bfeaf2490732bbb8 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=e1b413681beb485981324abe8f697565]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:10.024946 8887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 493053c0766648c3bfeaf2490732bbb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:10.026382 8897 tablet_service.cc:1467] Processing CreateTablet for tablet 56824b553788471c9a2fcd6ec82d507e (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=e1b413681beb485981324abe8f697565]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:10.026628 8897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56824b553788471c9a2fcd6ec82d507e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:10.026926 8880 tablet_service.cc:1467] Processing CreateTablet for tablet 71b7ef5ff3974a1f836bef781c42d94e (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=e1b413681beb485981324abe8f697565]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:10.026999 13702 tablet_bootstrap.cc:492] T 493053c0766648c3bfeaf2490732bbb8 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:10.027153 8880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71b7ef5ff3974a1f836bef781c42d94e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:10.027235 6569 tablet_service.cc:1467] Processing CreateTablet for tablet 71b7ef5ff3974a1f836bef781c42d94e (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=e1b413681beb485981324abe8f697565]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:10.027491 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71b7ef5ff3974a1f836bef781c42d94e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:10.027993 13702 tablet_bootstrap.cc:654] T 493053c0766648c3bfeaf2490732bbb8 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:10.029736 13704 tablet_bootstrap.cc:492] T 71b7ef5ff3974a1f836bef781c42d94e P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:10.029728 9028 tablet_service.cc:1467] Processing CreateTablet for tablet 493053c0766648c3bfeaf2490732bbb8 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=e1b413681beb485981324abe8f697565]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:10.029992 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 493053c0766648c3bfeaf2490732bbb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:10.029992 9025 tablet_service.cc:1467] Processing CreateTablet for tablet 56824b553788471c9a2fcd6ec82d507e (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=e1b413681beb485981324abe8f697565]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:10.030205 9025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56824b553788471c9a2fcd6ec82d507e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:10.031004 13704 tablet_bootstrap.cc:654] T 71b7ef5ff3974a1f836bef781c42d94e P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:10.032581 13706 tablet_bootstrap.cc:492] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:10.032658 13606 tablet_bootstrap.cc:492] T 493053c0766648c3bfeaf2490732bbb8 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:10.033810 13606 tablet_bootstrap.cc:654] T 493053c0766648c3bfeaf2490732bbb8 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:10.033944 13706 tablet_bootstrap.cc:654] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:10.035709 9157 tablet_service.cc:1467] Processing CreateTablet for tablet 71b7ef5ff3974a1f836bef781c42d94e (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=e1b413681beb485981324abe8f697565]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:10.036016 9157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71b7ef5ff3974a1f836bef781c42d94e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:10.038244 13702 tablet_bootstrap.cc:492] T 493053c0766648c3bfeaf2490732bbb8 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:10.038362 13702 ts_tablet_manager.cc:1367] T 493053c0766648c3bfeaf2490732bbb8 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 02:00:10.038568 13181 tablet_bootstrap.cc:492] T 71b7ef5ff3974a1f836bef781c42d94e P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:10.038867 13702 raft_consensus.cc:348] T 493053c0766648c3bfeaf2490732bbb8 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:10.038976 13702 raft_consensus.cc:374] T 493053c0766648c3bfeaf2490732bbb8 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:10.038992 13706 tablet_bootstrap.cc:492] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:10.039008 13702 raft_consensus.cc:729] T 493053c0766648c3bfeaf2490732bbb8 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:10.039134 13706 ts_tablet_manager.cc:1367] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:00:10.039135 13702 consensus_queue.cc:260] T 493053c0766648c3bfeaf2490732bbb8 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:10.039403 13702 ts_tablet_manager.cc:1398] T 493053c0766648c3bfeaf2490732bbb8 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:10.039499 13181 tablet_bootstrap.cc:654] T 71b7ef5ff3974a1f836bef781c42d94e P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:10.039618 13706 raft_consensus.cc:348] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:10.039738 13706 raft_consensus.cc:374] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:10.039767 13706 raft_consensus.cc:729] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:10.039904 13706 consensus_queue.cc:260] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:10.040151 6308 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=b3c4040b69424425906498cc28fc97b6] alter complete (version 3)
I20240503 02:00:10.040283 13706 ts_tablet_manager.cc:1398] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:10.040580 13706 tablet_bootstrap.cc:492] T 493053c0766648c3bfeaf2490732bbb8 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:10.041554 13706 tablet_bootstrap.cc:654] T 493053c0766648c3bfeaf2490732bbb8 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:10.043586 13181 tablet_bootstrap.cc:492] T 71b7ef5ff3974a1f836bef781c42d94e P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:10.043689 13181 ts_tablet_manager.cc:1367] T 71b7ef5ff3974a1f836bef781c42d94e P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240503 02:00:10.044131 13181 raft_consensus.cc:348] T 71b7ef5ff3974a1f836bef781c42d94e P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:10.044246 13181 raft_consensus.cc:374] T 71b7ef5ff3974a1f836bef781c42d94e P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:10.044279 13181 raft_consensus.cc:729] T 71b7ef5ff3974a1f836bef781c42d94e P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:10.044440 13181 consensus_queue.cc:260] T 71b7ef5ff3974a1f836bef781c42d94e P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:10.044729 13181 ts_tablet_manager.cc:1398] T 71b7ef5ff3974a1f836bef781c42d94e P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:10.046111 13704 tablet_bootstrap.cc:492] T 71b7ef5ff3974a1f836bef781c42d94e P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:10.046227 13704 ts_tablet_manager.cc:1367] T 71b7ef5ff3974a1f836bef781c42d94e P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240503 02:00:10.046710 13704 raft_consensus.cc:348] T 71b7ef5ff3974a1f836bef781c42d94e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:10.046834 13704 raft_consensus.cc:374] T 71b7ef5ff3974a1f836bef781c42d94e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:10.046860 13704 raft_consensus.cc:729] T 71b7ef5ff3974a1f836bef781c42d94e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:10.046978 13704 consensus_queue.cc:260] T 71b7ef5ff3974a1f836bef781c42d94e P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:10.048264 13704 ts_tablet_manager.cc:1398] T 71b7ef5ff3974a1f836bef781c42d94e P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:10.049235 13606 tablet_bootstrap.cc:492] T 493053c0766648c3bfeaf2490732bbb8 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:10.049384 13606 ts_tablet_manager.cc:1367] T 493053c0766648c3bfeaf2490732bbb8 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20240503 02:00:10.049919 13606 raft_consensus.cc:348] T 493053c0766648c3bfeaf2490732bbb8 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:10.050052 13606 raft_consensus.cc:374] T 493053c0766648c3bfeaf2490732bbb8 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:10.050082 13606 raft_consensus.cc:729] T 493053c0766648c3bfeaf2490732bbb8 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:10.050225 13606 consensus_queue.cc:260] T 493053c0766648c3bfeaf2490732bbb8 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:10.050570 13606 ts_tablet_manager.cc:1398] T 493053c0766648c3bfeaf2490732bbb8 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:10.050812 13606 tablet_bootstrap.cc:492] T 56824b553788471c9a2fcd6ec82d507e P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:10.051869 13606 tablet_bootstrap.cc:654] T 56824b553788471c9a2fcd6ec82d507e P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:10.053107 13706 tablet_bootstrap.cc:492] T 493053c0766648c3bfeaf2490732bbb8 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:10.053237 13706 ts_tablet_manager.cc:1367] T 493053c0766648c3bfeaf2490732bbb8 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 02:00:10.053689 13706 raft_consensus.cc:348] T 493053c0766648c3bfeaf2490732bbb8 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:10.053839 13706 raft_consensus.cc:374] T 493053c0766648c3bfeaf2490732bbb8 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:10.053881 13706 raft_consensus.cc:729] T 493053c0766648c3bfeaf2490732bbb8 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:10.054019 13706 consensus_queue.cc:260] T 493053c0766648c3bfeaf2490732bbb8 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:10.055207 13606 tablet_bootstrap.cc:492] T 56824b553788471c9a2fcd6ec82d507e P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:10.055308 13606 ts_tablet_manager.cc:1367] T 56824b553788471c9a2fcd6ec82d507e P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240503 02:00:10.055392 13706 ts_tablet_manager.cc:1398] T 493053c0766648c3bfeaf2490732bbb8 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:10.055790 13606 raft_consensus.cc:348] T 56824b553788471c9a2fcd6ec82d507e P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:10.055895 13606 raft_consensus.cc:374] T 56824b553788471c9a2fcd6ec82d507e P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:10.055919 13606 raft_consensus.cc:729] T 56824b553788471c9a2fcd6ec82d507e P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:10.056028 13606 consensus_queue.cc:260] T 56824b553788471c9a2fcd6ec82d507e P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:10.056269 13606 ts_tablet_manager.cc:1398] T 56824b553788471c9a2fcd6ec82d507e P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:10.056429 13606 tablet_bootstrap.cc:492] T 71b7ef5ff3974a1f836bef781c42d94e P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:10.057359 13606 tablet_bootstrap.cc:654] T 71b7ef5ff3974a1f836bef781c42d94e P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:10.061242 13606 tablet_bootstrap.cc:492] T 71b7ef5ff3974a1f836bef781c42d94e P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:10.063570 13606 ts_tablet_manager.cc:1367] T 71b7ef5ff3974a1f836bef781c42d94e P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.002s
I20240503 02:00:10.064807 13606 raft_consensus.cc:348] T 71b7ef5ff3974a1f836bef781c42d94e P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:10.064940 13606 raft_consensus.cc:374] T 71b7ef5ff3974a1f836bef781c42d94e P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:10.064968 13606 raft_consensus.cc:729] T 71b7ef5ff3974a1f836bef781c42d94e P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:10.065181 13606 consensus_queue.cc:260] T 71b7ef5ff3974a1f836bef781c42d94e P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:10.065466 13606 ts_tablet_manager.cc:1398] T 71b7ef5ff3974a1f836bef781c42d94e P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:10.076464 13495 raft_consensus.cc:482] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:10.076620 13495 raft_consensus.cc:504] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:10.077193 13495 leader_election.cc:288] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:10.078372 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56824b553788471c9a2fcd6ec82d507e" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:10.078526 8916 raft_consensus.cc:2454] T 56824b553788471c9a2fcd6ec82d507e P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
W20240503 02:00:10.078688 8982 leader_election.cc:334] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:10.078887 8984 leader_election.cc:302] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9942306855404cfd891adf227dd65eb2, 99ead02915954b4fab23b033ad6d546b; no voters: e1a9251ce1cf4361b8b0ede9fbfc4023
I20240503 02:00:10.079043 13495 raft_consensus.cc:2781] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:10.079090 13495 raft_consensus.cc:482] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:10.079140 13495 raft_consensus.cc:3037] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:10.080264 13495 raft_consensus.cc:504] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:10.080663 13495 leader_election.cc:288] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Requested vote from peers e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:10.081104 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56824b553788471c9a2fcd6ec82d507e" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:10.081223 8916 raft_consensus.cc:3037] T 56824b553788471c9a2fcd6ec82d507e P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:00:10.081871 8982 leader_election.cc:334] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:10.082250 8916 raft_consensus.cc:2454] T 56824b553788471c9a2fcd6ec82d507e P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:10.082667 8984 leader_election.cc:302] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9942306855404cfd891adf227dd65eb2, 99ead02915954b4fab23b033ad6d546b; no voters: e1a9251ce1cf4361b8b0ede9fbfc4023
I20240503 02:00:10.083447 13495 raft_consensus.cc:2781] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:10.083592 13495 raft_consensus.cc:686] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Becoming Leader. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Running, Role: LEADER
I20240503 02:00:10.083827 13495 consensus_queue.cc:237] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:10.084511 12331 raft_consensus.cc:482] T 71b7ef5ff3974a1f836bef781c42d94e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:10.084681 12331 raft_consensus.cc:504] T 71b7ef5ff3974a1f836bef781c42d94e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:10.085116 12331 leader_election.cc:288] T 71b7ef5ff3974a1f836bef781c42d94e P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:10.085501 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71b7ef5ff3974a1f836bef781c42d94e" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:10.085709 8916 raft_consensus.cc:2454] T 71b7ef5ff3974a1f836bef781c42d94e P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:10.086042 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71b7ef5ff3974a1f836bef781c42d94e" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:10.086241 9178 raft_consensus.cc:2454] T 71b7ef5ff3974a1f836bef781c42d94e P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:10.086449 6525 leader_election.cc:302] T 71b7ef5ff3974a1f836bef781c42d94e P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:10.086620 12331 raft_consensus.cc:2781] T 71b7ef5ff3974a1f836bef781c42d94e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:10.086668 12331 raft_consensus.cc:482] T 71b7ef5ff3974a1f836bef781c42d94e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:10.086701 12331 raft_consensus.cc:3037] T 71b7ef5ff3974a1f836bef781c42d94e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:10.087699 12331 raft_consensus.cc:504] T 71b7ef5ff3974a1f836bef781c42d94e P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:10.088099 12331 leader_election.cc:288] T 71b7ef5ff3974a1f836bef781c42d94e P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Requested vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:10.088519 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71b7ef5ff3974a1f836bef781c42d94e" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:10.088428 6309 catalog_manager.cc:5484] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b reported cstate change: term changed from 0 to 1, leader changed from <none> to 99ead02915954b4fab23b033ad6d546b (127.0.99.134). New cstate: current_term: 1 leader_uuid: "99ead02915954b4fab23b033ad6d546b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:10.088641 9178 raft_consensus.cc:3037] T 71b7ef5ff3974a1f836bef781c42d94e P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:10.089677 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71b7ef5ff3974a1f836bef781c42d94e" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:10.089843 8916 raft_consensus.cc:3037] T 71b7ef5ff3974a1f836bef781c42d94e P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:10.089856 9178 raft_consensus.cc:2454] T 71b7ef5ff3974a1f836bef781c42d94e P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:10.090813 6522 leader_election.cc:302] T 71b7ef5ff3974a1f836bef781c42d94e P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 80fd18ef4d604c86b909d5c008a46b4f; no voters:
I20240503 02:00:10.090917 8916 raft_consensus.cc:2454] T 71b7ef5ff3974a1f836bef781c42d94e P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:10.091032 12331 raft_consensus.cc:2781] T 71b7ef5ff3974a1f836bef781c42d94e P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:10.091145 12331 raft_consensus.cc:686] T 71b7ef5ff3974a1f836bef781c42d94e P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Becoming Leader. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Running, Role: LEADER
I20240503 02:00:10.091286 12331 consensus_queue.cc:237] T 71b7ef5ff3974a1f836bef781c42d94e P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:10.093873 6309 catalog_manager.cc:5484] T 71b7ef5ff3974a1f836bef781c42d94e P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:10.120187 13630 raft_consensus.cc:482] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e1a9251ce1cf4361b8b0ede9fbfc4023)
I20240503 02:00:10.120340 13630 raft_consensus.cc:504] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } }
I20240503 02:00:10.120893 13630 leader_election.cc:288] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311)
I20240503 02:00:10.121506 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2e62c043adf4a8d9024279d786f596c" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:10.121693 9178 raft_consensus.cc:2454] T b2e62c043adf4a8d9024279d786f596c P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 02:00:10.121994 6784 leader_election.cc:302] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 492448570cc94ed1879feb2ec3b9051d; no voters:
W20240503 02:00:10.122108 6785 leader_election.cc:334] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:10.122167 13630 raft_consensus.cc:2781] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:00:10.122220 13630 raft_consensus.cc:482] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Starting leader election (detected failure of leader e1a9251ce1cf4361b8b0ede9fbfc4023)
I20240503 02:00:10.122256 13630 raft_consensus.cc:3037] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:10.123152 13630 raft_consensus.cc:504] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } }
I20240503 02:00:10.123504 13630 leader_election.cc:288] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 2 election: Requested vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311)
I20240503 02:00:10.123987 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2e62c043adf4a8d9024279d786f596c" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:10.124130 9178 raft_consensus.cc:3037] T b2e62c043adf4a8d9024279d786f596c P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:00:10.124656 6785 leader_election.cc:334] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:10.125118 9178 raft_consensus.cc:2454] T b2e62c043adf4a8d9024279d786f596c P 2a01ace80baf4057b6bf1995cf88468b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 2.
I20240503 02:00:10.125422 6784 leader_election.cc:302] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 492448570cc94ed1879feb2ec3b9051d; no voters: e1a9251ce1cf4361b8b0ede9fbfc4023
I20240503 02:00:10.125684 13630 raft_consensus.cc:2781] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:10.125790 13630 raft_consensus.cc:686] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d [term 2 LEADER]: Becoming Leader. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Running, Role: LEADER
I20240503 02:00:10.125953 13630 consensus_queue.cc:237] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } }
I20240503 02:00:10.128043 6309 catalog_manager.cc:5484] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d reported cstate change: term changed from 1 to 2, leader changed from e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136) to 492448570cc94ed1879feb2ec3b9051d (127.0.99.132). New cstate: current_term: 2 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:10.148238 6309 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:10.148444 6309 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:10.150705 6309 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet ad245987cb6c4d0680a1597edc14801b
I20240503 02:00:10.151315 8880 tablet_service.cc:1514] Processing DeleteTablet for tablet ad245987cb6c4d0680a1597edc14801b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:10 UTC) from {username='slave'} at 127.0.0.1:40296
I20240503 02:00:10.151346 9157 tablet_service.cc:1514] Processing DeleteTablet for tablet ad245987cb6c4d0680a1597edc14801b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:10 UTC) from {username='slave'} at 127.0.0.1:51940
I20240503 02:00:10.151544 13644 tablet_replica.cc:330] T ad245987cb6c4d0680a1597edc14801b P 2a01ace80baf4057b6bf1995cf88468b: stopping tablet replica
I20240503 02:00:10.151625 6436 tablet_service.cc:1514] Processing DeleteTablet for tablet ad245987cb6c4d0680a1597edc14801b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:10 UTC) from {username='slave'} at 127.0.0.1:51338
I20240503 02:00:10.151669 13717 tablet_replica.cc:330] T ad245987cb6c4d0680a1597edc14801b P 9942306855404cfd891adf227dd65eb2: stopping tablet replica
I20240503 02:00:10.151726 13644 raft_consensus.cc:2229] T ad245987cb6c4d0680a1597edc14801b P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:10.151809 13717 raft_consensus.cc:2229] T ad245987cb6c4d0680a1597edc14801b P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:10.151854 13644 raft_consensus.cc:2258] T ad245987cb6c4d0680a1597edc14801b P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:10.151911 13717 raft_consensus.cc:2258] T ad245987cb6c4d0680a1597edc14801b P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:10.151919 13718 tablet_replica.cc:330] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a: stopping tablet replica
I20240503 02:00:10.152058 13718 raft_consensus.cc:2229] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:10.152269 13718 raft_consensus.cc:2258] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:10.152567 13644 ts_tablet_manager.cc:1875] T ad245987cb6c4d0680a1597edc14801b P 2a01ace80baf4057b6bf1995cf88468b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:10.152590 13717 ts_tablet_manager.cc:1875] T ad245987cb6c4d0680a1597edc14801b P 9942306855404cfd891adf227dd65eb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:10.152786 13718 ts_tablet_manager.cc:1875] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:10.154978 13644 ts_tablet_manager.cc:1888] T ad245987cb6c4d0680a1597edc14801b P 2a01ace80baf4057b6bf1995cf88468b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:10.155256 13644 log.cc:1196] T ad245987cb6c4d0680a1597edc14801b P 2a01ace80baf4057b6bf1995cf88468b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/ad245987cb6c4d0680a1597edc14801b
I20240503 02:00:10.155680 13644 ts_tablet_manager.cc:1909] T ad245987cb6c4d0680a1597edc14801b P 2a01ace80baf4057b6bf1995cf88468b: Deleting consensus metadata
I20240503 02:00:10.160938 6300 catalog_manager.cc:4830] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079): tablet ad245987cb6c4d0680a1597edc14801b (table rebalancer_dynamic_table_new_name [id=b3c4040b69424425906498cc28fc97b6]) successfully deleted
I20240503 02:00:10.161034 13717 ts_tablet_manager.cc:1888] T ad245987cb6c4d0680a1597edc14801b P 9942306855404cfd891adf227dd65eb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:10.161123 13717 log.cc:1196] T ad245987cb6c4d0680a1597edc14801b P 9942306855404cfd891adf227dd65eb2: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/ad245987cb6c4d0680a1597edc14801b
I20240503 02:00:10.161418 13717 ts_tablet_manager.cc:1909] T ad245987cb6c4d0680a1597edc14801b P 9942306855404cfd891adf227dd65eb2: Deleting consensus metadata
I20240503 02:00:10.161842 13718 ts_tablet_manager.cc:1888] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:10.161931 13718 log.cc:1196] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/ad245987cb6c4d0680a1597edc14801b
I20240503 02:00:10.162309 13718 ts_tablet_manager.cc:1909] T ad245987cb6c4d0680a1597edc14801b P dec3e4a04d604f7eaaa3f077d081218a: Deleting consensus metadata
I20240503 02:00:10.162274 6303 catalog_manager.cc:4830] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211): tablet ad245987cb6c4d0680a1597edc14801b (table rebalancer_dynamic_table_new_name [id=b3c4040b69424425906498cc28fc97b6]) successfully deleted
I20240503 02:00:10.162976 6300 catalog_manager.cc:4830] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): tablet ad245987cb6c4d0680a1597edc14801b (table rebalancer_dynamic_table_new_name [id=b3c4040b69424425906498cc28fc97b6]) successfully deleted
I20240503 02:00:10.186240 13630 raft_consensus.cc:482] T 493053c0766648c3bfeaf2490732bbb8 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:10.186375 13630 raft_consensus.cc:504] T 493053c0766648c3bfeaf2490732bbb8 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:10.186815 13630 leader_election.cc:288] T 493053c0766648c3bfeaf2490732bbb8 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:10.187189 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "493053c0766648c3bfeaf2490732bbb8" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:10.187328 8916 raft_consensus.cc:2454] T 493053c0766648c3bfeaf2490732bbb8 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 02:00:10.187585 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "493053c0766648c3bfeaf2490732bbb8" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:10.187604 6787 leader_election.cc:302] T 493053c0766648c3bfeaf2490732bbb8 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:10.187705 9048 raft_consensus.cc:2454] T 493053c0766648c3bfeaf2490732bbb8 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 02:00:10.187800 13630 raft_consensus.cc:2781] T 493053c0766648c3bfeaf2490732bbb8 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:10.187850 13630 raft_consensus.cc:482] T 493053c0766648c3bfeaf2490732bbb8 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:10.187886 13630 raft_consensus.cc:3037] T 493053c0766648c3bfeaf2490732bbb8 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:10.188781 13630 raft_consensus.cc:504] T 493053c0766648c3bfeaf2490732bbb8 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:10.189106 13630 leader_election.cc:288] T 493053c0766648c3bfeaf2490732bbb8 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Requested vote from peers 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:10.189462 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "493053c0766648c3bfeaf2490732bbb8" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:10.189455 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "493053c0766648c3bfeaf2490732bbb8" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:10.189580 9048 raft_consensus.cc:3037] T 493053c0766648c3bfeaf2490732bbb8 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:10.189585 8916 raft_consensus.cc:3037] T 493053c0766648c3bfeaf2490732bbb8 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:10.190550 9048 raft_consensus.cc:2454] T 493053c0766648c3bfeaf2490732bbb8 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 02:00:10.190560 8916 raft_consensus.cc:2454] T 493053c0766648c3bfeaf2490732bbb8 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 02:00:10.190886 6787 leader_election.cc:302] T 493053c0766648c3bfeaf2490732bbb8 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:10.191059 13630 raft_consensus.cc:2781] T 493053c0766648c3bfeaf2490732bbb8 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:10.191147 13630 raft_consensus.cc:686] T 493053c0766648c3bfeaf2490732bbb8 P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Becoming Leader. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Running, Role: LEADER
I20240503 02:00:10.191291 13630 consensus_queue.cc:237] T 493053c0766648c3bfeaf2490732bbb8 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:10.193512 6309 catalog_manager.cc:5484] T 493053c0766648c3bfeaf2490732bbb8 P 492448570cc94ed1879feb2ec3b9051d reported cstate change: term changed from 0 to 1, leader changed from <none> to 492448570cc94ed1879feb2ec3b9051d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } }
W20240503 02:00:10.205881 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:10.252988 6309 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:00:10.258814 6436 tablet_service.cc:1467] Processing CreateTablet for tablet cc19d71f40fb44f1801f1b49dcc76fe0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ee25cc5084614e039daac0216839e837]), partition=
I20240503 02:00:10.258814 9157 tablet_service.cc:1467] Processing CreateTablet for tablet cc19d71f40fb44f1801f1b49dcc76fe0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ee25cc5084614e039daac0216839e837]), partition=
I20240503 02:00:10.259102 9157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc19d71f40fb44f1801f1b49dcc76fe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:10.259104 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc19d71f40fb44f1801f1b49dcc76fe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:10.261520 13181 tablet_bootstrap.cc:492] T cc19d71f40fb44f1801f1b49dcc76fe0 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:10.261698 13721 tablet_bootstrap.cc:492] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:10.262490 13721 tablet_bootstrap.cc:654] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:10.262490 13181 tablet_bootstrap.cc:654] T cc19d71f40fb44f1801f1b49dcc76fe0 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:10.263716 13181 tablet_bootstrap.cc:492] T cc19d71f40fb44f1801f1b49dcc76fe0 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:10.263808 13181 ts_tablet_manager.cc:1367] T cc19d71f40fb44f1801f1b49dcc76fe0 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:00:10.264320 13721 tablet_bootstrap.cc:492] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:10.264287 13181 raft_consensus.cc:348] T cc19d71f40fb44f1801f1b49dcc76fe0 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:10.264428 13721 ts_tablet_manager.cc:1367] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:10.264428 13181 raft_consensus.cc:374] T cc19d71f40fb44f1801f1b49dcc76fe0 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:10.264461 13181 raft_consensus.cc:729] T cc19d71f40fb44f1801f1b49dcc76fe0 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:10.264590 13181 consensus_queue.cc:260] T cc19d71f40fb44f1801f1b49dcc76fe0 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:10.264871 13721 raft_consensus.cc:348] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:10.264966 13721 raft_consensus.cc:374] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:10.264993 13721 raft_consensus.cc:729] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:10.265141 13721 consensus_queue.cc:260] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:10.265409 13721 ts_tablet_manager.cc:1398] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:10.265633 13181 ts_tablet_manager.cc:1398] T cc19d71f40fb44f1801f1b49dcc76fe0 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:10.484159 13298 raft_consensus.cc:482] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:10.484308 13298 raft_consensus.cc:504] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:10.484764 13298 leader_election.cc:288] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:10.485242 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc19d71f40fb44f1801f1b49dcc76fe0" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:10.485386 9178 raft_consensus.cc:2454] T cc19d71f40fb44f1801f1b49dcc76fe0 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:10.485790 6391 leader_election.cc:302] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, dec3e4a04d604f7eaaa3f077d081218a; no voters:
W20240503 02:00:10.485857 6392 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:10.485967 13298 raft_consensus.cc:2781] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:10.486020 13298 raft_consensus.cc:482] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:10.486045 13298 raft_consensus.cc:3037] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:00:10.486725 6392 leader_election.cc:334] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:10.487021 13298 raft_consensus.cc:504] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:10.487346 13298 leader_election.cc:288] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Requested vote from peers e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:10.487886 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc19d71f40fb44f1801f1b49dcc76fe0" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:10.488032 9178 raft_consensus.cc:3037] T cc19d71f40fb44f1801f1b49dcc76fe0 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:00:10.488512 6392 leader_election.cc:334] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:10.489089 9178 raft_consensus.cc:2454] T cc19d71f40fb44f1801f1b49dcc76fe0 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:10.489388 6391 leader_election.cc:302] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, dec3e4a04d604f7eaaa3f077d081218a; no voters: e1a9251ce1cf4361b8b0ede9fbfc4023
I20240503 02:00:10.489557 13298 raft_consensus.cc:2781] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:10.489661 13298 raft_consensus.cc:686] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Becoming Leader. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Running, Role: LEADER
I20240503 02:00:10.489805 13298 consensus_queue.cc:237] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:10.491941 6308 catalog_manager.cc:5484] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: term changed from 0 to 1, leader changed from <none> to dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129). New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:10.551856 12331 consensus_queue.cc:1035] T 71b7ef5ff3974a1f836bef781c42d94e P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:10.556882 12331 consensus_queue.cc:1035] T 71b7ef5ff3974a1f836bef781c42d94e P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:10.595242 6309 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:10.599769 9178 raft_consensus.cc:1261] T cc19d71f40fb44f1801f1b49dcc76fe0 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:00:10.600461 6392 consensus_peers.cc:485] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:10.600556 13298 consensus_queue.cc:1035] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:10.603451 13737 tablet.cc:1722] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:10.603521 6309 catalog_manager.cc:5576] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079) does not have the latest schema for tablet cc19d71f40fb44f1801f1b49dcc76fe0 (table rebalancer_dynamic_table [id=ee25cc5084614e039daac0216839e837]). Expected version 1 got 0
I20240503 02:00:10.603969 13737 tablet.cc:1620] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a: MemRowSet was empty: no flush needed.
I20240503 02:00:10.605180 13738 tablet.cc:1722] T cc19d71f40fb44f1801f1b49dcc76fe0 P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:10.605459 13738 tablet.cc:1620] T cc19d71f40fb44f1801f1b49dcc76fe0 P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
I20240503 02:00:10.605732 6314 catalog_manager.cc:5576] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093) does not have the latest schema for tablet cc19d71f40fb44f1801f1b49dcc76fe0 (table rebalancer_dynamic_table [id=ee25cc5084614e039daac0216839e837]). Expected version 1 got 0
I20240503 02:00:10.607450 6314 catalog_manager.cc:5576] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079) does not have the latest schema for tablet cc19d71f40fb44f1801f1b49dcc76fe0 (table rebalancer_dynamic_table [id=ee25cc5084614e039daac0216839e837]). Expected version 1 got 0
I20240503 02:00:10.608047 13720 consensus_queue.cc:1035] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:00:10.611481 8982 consensus_peers.cc:485] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:10.613070 6314 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=ee25cc5084614e039daac0216839e837] alter complete (version 1)
W20240503 02:00:10.625428 6785 consensus_peers.cc:485] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:10.687713 13056 consensus_queue.cc:1035] T 493053c0766648c3bfeaf2490732bbb8 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:10.692517 13056 consensus_queue.cc:1035] T 493053c0766648c3bfeaf2490732bbb8 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:10.704962 9178 raft_consensus.cc:1261] T b2e62c043adf4a8d9024279d786f596c P 2a01ace80baf4057b6bf1995cf88468b [term 2 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 02:00:10.705418 13056 consensus_queue.cc:1035] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:10.707919 8916 raft_consensus.cc:3037] T b2e62c043adf4a8d9024279d786f596c P 9942306855404cfd891adf227dd65eb2 [term 1 LEARNER]: Advancing to term 2
I20240503 02:00:10.709193 8916 raft_consensus.cc:1261] T b2e62c043adf4a8d9024279d786f596c P 9942306855404cfd891adf227dd65eb2 [term 2 LEARNER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 02:00:10.709611 13630 consensus_queue.cc:1035] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:10.711612 13630 raft_consensus.cc:1050] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d: attempting to promote NON_VOTER 9942306855404cfd891adf227dd65eb2 to VOTER
I20240503 02:00:10.712083 13630 consensus_queue.cc:237] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } }
I20240503 02:00:10.713764 8916 raft_consensus.cc:1261] T b2e62c043adf4a8d9024279d786f596c P 9942306855404cfd891adf227dd65eb2 [term 2 LEARNER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:00:10.714210 9178 raft_consensus.cc:1261] T b2e62c043adf4a8d9024279d786f596c P 2a01ace80baf4057b6bf1995cf88468b [term 2 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:00:10.714187 13746 consensus_queue.cc:1035] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:00:10.714603 6785 consensus_peers.cc:485] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:10.714663 13724 consensus_queue.cc:1035] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:10.715966 6314 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:10.716081 6309 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_extra_table_28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:10.716315 13724 raft_consensus.cc:2932] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:10.718605 8916 raft_consensus.cc:2932] T b2e62c043adf4a8d9024279d786f596c P 9942306855404cfd891adf227dd65eb2 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:10.720443 6314 catalog_manager.cc:5484] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d reported cstate change: config changed from index 2 to 4, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:10.723513 13737 tablet.cc:1722] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:10.723886 13737 tablet.cc:1620] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a: MemRowSet was empty: no flush needed.
I20240503 02:00:10.723801 9179 raft_consensus.cc:2932] T b2e62c043adf4a8d9024279d786f596c P 2a01ace80baf4057b6bf1995cf88468b [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 9942306855404cfd891adf227dd65eb2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } attrs { replace: true } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } attrs { promote: false } } }
I20240503 02:00:10.725363 13738 tablet.cc:1722] T cc19d71f40fb44f1801f1b49dcc76fe0 P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:10.725716 13738 tablet.cc:1620] T cc19d71f40fb44f1801f1b49dcc76fe0 P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
I20240503 02:00:10.732615 9028 tablet_service.cc:1467] Processing CreateTablet for tablet 99bc944f874248b5a778971f7a2aaa0c (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=56c5cfdd298a4554989547d44cd96854]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:10.732949 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99bc944f874248b5a778971f7a2aaa0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:10.733107 6700 tablet_service.cc:1467] Processing CreateTablet for tablet 8dfe23ff4f2a456384690fa9e48e51c1 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=56c5cfdd298a4554989547d44cd96854]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:10.733386 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8dfe23ff4f2a456384690fa9e48e51c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:10.734169 9158 tablet_service.cc:1467] Processing CreateTablet for tablet 2dfc375064e24f2ea5bba0ae88535f87 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=56c5cfdd298a4554989547d44cd96854]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:10.734169 9157 tablet_service.cc:1467] Processing CreateTablet for tablet 99bc944f874248b5a778971f7a2aaa0c (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=56c5cfdd298a4554989547d44cd96854]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:10.734876 9158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dfc375064e24f2ea5bba0ae88535f87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:10.735560 13749 tablet_bootstrap.cc:492] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:10.735805 13750 tablet_bootstrap.cc:492] T 8dfe23ff4f2a456384690fa9e48e51c1 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:10.736335 6569 tablet_service.cc:1467] Processing CreateTablet for tablet 8dfe23ff4f2a456384690fa9e48e51c1 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=56c5cfdd298a4554989547d44cd96854]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:10.736529 13749 tablet_bootstrap.cc:654] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:10.736629 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8dfe23ff4f2a456384690fa9e48e51c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:10.737087 13181 tablet_bootstrap.cc:492] T 2dfc375064e24f2ea5bba0ae88535f87 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:10.737998 13181 tablet_bootstrap.cc:654] T 2dfc375064e24f2ea5bba0ae88535f87 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:10.738637 9157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99bc944f874248b5a778971f7a2aaa0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:10.738983 8880 tablet_service.cc:1467] Processing CreateTablet for tablet 2dfc375064e24f2ea5bba0ae88535f87 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=56c5cfdd298a4554989547d44cd96854]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:10.739102 13181 tablet_bootstrap.cc:492] T 2dfc375064e24f2ea5bba0ae88535f87 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:10.739200 13181 ts_tablet_manager.cc:1367] T 2dfc375064e24f2ea5bba0ae88535f87 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:10.739244 8880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dfc375064e24f2ea5bba0ae88535f87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:10.739683 13181 raft_consensus.cc:348] T 2dfc375064e24f2ea5bba0ae88535f87 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:10.739784 13181 raft_consensus.cc:374] T 2dfc375064e24f2ea5bba0ae88535f87 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:10.739812 13181 raft_consensus.cc:729] T 2dfc375064e24f2ea5bba0ae88535f87 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:10.739915 13181 consensus_queue.cc:260] T 2dfc375064e24f2ea5bba0ae88535f87 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:10.740206 13181 ts_tablet_manager.cc:1398] T 2dfc375064e24f2ea5bba0ae88535f87 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:10.740758 9028 tablet_service.cc:1467] Processing CreateTablet for tablet 2dfc375064e24f2ea5bba0ae88535f87 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=56c5cfdd298a4554989547d44cd96854]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:10.741009 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dfc375064e24f2ea5bba0ae88535f87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:10.741443 13754 tablet_bootstrap.cc:492] T 2dfc375064e24f2ea5bba0ae88535f87 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:10.742385 13754 tablet_bootstrap.cc:654] T 2dfc375064e24f2ea5bba0ae88535f87 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
W20240503 02:00:10.744119 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:10.744519 13750 tablet_bootstrap.cc:654] T 8dfe23ff4f2a456384690fa9e48e51c1 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:10.745004 13754 tablet_bootstrap.cc:492] T 2dfc375064e24f2ea5bba0ae88535f87 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:10.745101 13754 ts_tablet_manager.cc:1367] T 2dfc375064e24f2ea5bba0ae88535f87 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240503 02:00:10.745566 13754 raft_consensus.cc:348] T 2dfc375064e24f2ea5bba0ae88535f87 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:10.745677 13754 raft_consensus.cc:374] T 2dfc375064e24f2ea5bba0ae88535f87 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:10.745707 13754 raft_consensus.cc:729] T 2dfc375064e24f2ea5bba0ae88535f87 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:10.745824 13754 consensus_queue.cc:260] T 2dfc375064e24f2ea5bba0ae88535f87 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:10.746116 13754 ts_tablet_manager.cc:1398] T 2dfc375064e24f2ea5bba0ae88535f87 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:10.746665 13750 tablet_bootstrap.cc:492] T 8dfe23ff4f2a456384690fa9e48e51c1 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:10.746752 13750 ts_tablet_manager.cc:1367] T 8dfe23ff4f2a456384690fa9e48e51c1 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.001s
I20240503 02:00:10.747174 13181 tablet_bootstrap.cc:492] T 99bc944f874248b5a778971f7a2aaa0c P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:10.747378 6309 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=ee25cc5084614e039daac0216839e837] alter complete (version 2)
I20240503 02:00:10.747596 13750 raft_consensus.cc:348] T 8dfe23ff4f2a456384690fa9e48e51c1 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:10.747720 13750 raft_consensus.cc:374] T 8dfe23ff4f2a456384690fa9e48e51c1 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:10.747750 13750 raft_consensus.cc:729] T 8dfe23ff4f2a456384690fa9e48e51c1 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:10.747900 13750 consensus_queue.cc:260] T 8dfe23ff4f2a456384690fa9e48e51c1 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:10.748453 13750 ts_tablet_manager.cc:1398] T 8dfe23ff4f2a456384690fa9e48e51c1 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:00:10.748667 13181 tablet_bootstrap.cc:654] T 99bc944f874248b5a778971f7a2aaa0c P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:10.749444 13753 tablet_bootstrap.cc:492] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:10.750490 13753 tablet_bootstrap.cc:654] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:10.751243 13181 tablet_bootstrap.cc:492] T 99bc944f874248b5a778971f7a2aaa0c P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:10.751338 13181 ts_tablet_manager.cc:1367] T 99bc944f874248b5a778971f7a2aaa0c P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20240503 02:00:10.751823 13181 raft_consensus.cc:348] T 99bc944f874248b5a778971f7a2aaa0c P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } }
I20240503 02:00:10.751932 13181 raft_consensus.cc:374] T 99bc944f874248b5a778971f7a2aaa0c P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:10.751957 13181 raft_consensus.cc:729] T 99bc944f874248b5a778971f7a2aaa0c P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:10.752080 13181 consensus_queue.cc:260] T 99bc944f874248b5a778971f7a2aaa0c P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } }
I20240503 02:00:10.752152 13749 tablet_bootstrap.cc:492] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:10.752290 13749 ts_tablet_manager.cc:1367] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.002s
I20240503 02:00:10.752336 13181 ts_tablet_manager.cc:1398] T 99bc944f874248b5a778971f7a2aaa0c P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:10.752864 13749 raft_consensus.cc:348] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } }
I20240503 02:00:10.752967 13749 raft_consensus.cc:374] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:10.752996 13749 raft_consensus.cc:729] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:10.753087 13753 tablet_bootstrap.cc:492] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:10.753132 13749 consensus_queue.cc:260] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } }
I20240503 02:00:10.753214 13753 ts_tablet_manager.cc:1367] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:10.753682 13753 raft_consensus.cc:348] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:10.753811 13753 raft_consensus.cc:374] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:10.753863 13753 raft_consensus.cc:729] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:10.754160 13753 consensus_queue.cc:260] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:10.754837 13749 ts_tablet_manager.cc:1398] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 02:00:10.755088 13753 ts_tablet_manager.cc:1398] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:10.755451 13749 tablet_bootstrap.cc:492] T 2dfc375064e24f2ea5bba0ae88535f87 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:10.757295 13749 tablet_bootstrap.cc:654] T 2dfc375064e24f2ea5bba0ae88535f87 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:10.759279 13749 tablet_bootstrap.cc:492] T 2dfc375064e24f2ea5bba0ae88535f87 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:10.759385 13749 ts_tablet_manager.cc:1367] T 2dfc375064e24f2ea5bba0ae88535f87 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:10.759884 13749 raft_consensus.cc:348] T 2dfc375064e24f2ea5bba0ae88535f87 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:10.760015 13749 raft_consensus.cc:374] T 2dfc375064e24f2ea5bba0ae88535f87 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:10.760046 13749 raft_consensus.cc:729] T 2dfc375064e24f2ea5bba0ae88535f87 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:10.760165 13749 consensus_queue.cc:260] T 2dfc375064e24f2ea5bba0ae88535f87 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:10.761327 13749 ts_tablet_manager.cc:1398] T 2dfc375064e24f2ea5bba0ae88535f87 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20240503 02:00:10.841645 6523 consensus_peers.cc:485] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:00:10.850346 6314 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:10.855607 13737 tablet.cc:1722] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:10.855991 13738 tablet.cc:1722] T cc19d71f40fb44f1801f1b49dcc76fe0 P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:10.860743 6314 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=ee25cc5084614e039daac0216839e837] alter complete (version 3)
I20240503 02:00:10.877092 13720 raft_consensus.cc:482] T 2dfc375064e24f2ea5bba0ae88535f87 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:10.877233 13720 raft_consensus.cc:504] T 2dfc375064e24f2ea5bba0ae88535f87 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:10.877626 13720 leader_election.cc:288] T 2dfc375064e24f2ea5bba0ae88535f87 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:10.878077 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dfc375064e24f2ea5bba0ae88535f87" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:10.878113 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dfc375064e24f2ea5bba0ae88535f87" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:10.878235 9179 raft_consensus.cc:2454] T 2dfc375064e24f2ea5bba0ae88535f87 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:10.878250 8916 raft_consensus.cc:2454] T 2dfc375064e24f2ea5bba0ae88535f87 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:10.878579 8984 leader_election.cc:302] T 2dfc375064e24f2ea5bba0ae88535f87 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9942306855404cfd891adf227dd65eb2, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:10.878762 13720 raft_consensus.cc:2781] T 2dfc375064e24f2ea5bba0ae88535f87 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:10.878819 13720 raft_consensus.cc:482] T 2dfc375064e24f2ea5bba0ae88535f87 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:10.878857 13720 raft_consensus.cc:3037] T 2dfc375064e24f2ea5bba0ae88535f87 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:10.879793 13720 raft_consensus.cc:504] T 2dfc375064e24f2ea5bba0ae88535f87 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:10.880168 13720 leader_election.cc:288] T 2dfc375064e24f2ea5bba0ae88535f87 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Requested vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:10.880549 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dfc375064e24f2ea5bba0ae88535f87" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:10.880563 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dfc375064e24f2ea5bba0ae88535f87" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:10.880681 9179 raft_consensus.cc:3037] T 2dfc375064e24f2ea5bba0ae88535f87 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:10.880681 8916 raft_consensus.cc:3037] T 2dfc375064e24f2ea5bba0ae88535f87 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:10.881676 9179 raft_consensus.cc:2454] T 2dfc375064e24f2ea5bba0ae88535f87 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:10.881676 8916 raft_consensus.cc:2454] T 2dfc375064e24f2ea5bba0ae88535f87 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:10.882017 8981 leader_election.cc:302] T 2dfc375064e24f2ea5bba0ae88535f87 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:10.882186 13720 raft_consensus.cc:2781] T 2dfc375064e24f2ea5bba0ae88535f87 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:10.882305 13720 raft_consensus.cc:686] T 2dfc375064e24f2ea5bba0ae88535f87 P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Becoming Leader. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Running, Role: LEADER
I20240503 02:00:10.882483 13720 consensus_queue.cc:237] T 2dfc375064e24f2ea5bba0ae88535f87 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:10.885130 6312 catalog_manager.cc:5484] T 2dfc375064e24f2ea5bba0ae88535f87 P 99ead02915954b4fab23b033ad6d546b reported cstate change: term changed from 0 to 1, leader changed from <none> to 99ead02915954b4fab23b033ad6d546b (127.0.99.134). New cstate: current_term: 1 leader_uuid: "99ead02915954b4fab23b033ad6d546b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:10.963135 6312 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:10.963304 6312 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:10.965520 6312 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet cc19d71f40fb44f1801f1b49dcc76fe0
I20240503 02:00:10.966337 9157 tablet_service.cc:1514] Processing DeleteTablet for tablet cc19d71f40fb44f1801f1b49dcc76fe0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:10 UTC) from {username='slave'} at 127.0.0.1:51940
W20240503 02:00:10.966892 6301 catalog_manager.cc:4573] TS e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cc19d71f40fb44f1801f1b49dcc76fe0: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:10.966920 6436 tablet_service.cc:1514] Processing DeleteTablet for tablet cc19d71f40fb44f1801f1b49dcc76fe0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:10 UTC) from {username='slave'} at 127.0.0.1:51338
I20240503 02:00:10.967237 13764 tablet_replica.cc:330] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a: stopping tablet replica
I20240503 02:00:10.967386 13764 raft_consensus.cc:2229] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:10.967625 13764 raft_consensus.cc:2258] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:10.968178 13764 ts_tablet_manager.cc:1875] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:10.970297 13764 ts_tablet_manager.cc:1888] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:10.970382 13764 log.cc:1196] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/cc19d71f40fb44f1801f1b49dcc76fe0
I20240503 02:00:10.970675 13764 ts_tablet_manager.cc:1909] T cc19d71f40fb44f1801f1b49dcc76fe0 P dec3e4a04d604f7eaaa3f077d081218a: Deleting consensus metadata
I20240503 02:00:10.971352 6300 catalog_manager.cc:4830] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): tablet cc19d71f40fb44f1801f1b49dcc76fe0 (table rebalancer_dynamic_table_new_name [id=ee25cc5084614e039daac0216839e837]) successfully deleted
I20240503 02:00:10.976523 13763 tablet_replica.cc:330] T cc19d71f40fb44f1801f1b49dcc76fe0 P 2a01ace80baf4057b6bf1995cf88468b: stopping tablet replica
I20240503 02:00:10.976711 13763 raft_consensus.cc:2229] T cc19d71f40fb44f1801f1b49dcc76fe0 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:10.976862 13763 raft_consensus.cc:2258] T cc19d71f40fb44f1801f1b49dcc76fe0 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:10.977480 13763 ts_tablet_manager.cc:1875] T cc19d71f40fb44f1801f1b49dcc76fe0 P 2a01ace80baf4057b6bf1995cf88468b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:10.979625 13763 ts_tablet_manager.cc:1888] T cc19d71f40fb44f1801f1b49dcc76fe0 P 2a01ace80baf4057b6bf1995cf88468b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:10.979699 13763 log.cc:1196] T cc19d71f40fb44f1801f1b49dcc76fe0 P 2a01ace80baf4057b6bf1995cf88468b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/cc19d71f40fb44f1801f1b49dcc76fe0
I20240503 02:00:10.979907 13763 ts_tablet_manager.cc:1909] T cc19d71f40fb44f1801f1b49dcc76fe0 P 2a01ace80baf4057b6bf1995cf88468b: Deleting consensus metadata
I20240503 02:00:10.981001 6300 catalog_manager.cc:4830] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079): tablet cc19d71f40fb44f1801f1b49dcc76fe0 (table rebalancer_dynamic_table_new_name [id=ee25cc5084614e039daac0216839e837]) successfully deleted
I20240503 02:00:11.067387 6312 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:00:11.072795 9157 tablet_service.cc:1467] Processing CreateTablet for tablet 286073252b944a0898aca512e6f3177a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=03276591ff8a400eb2636d7ca6dc2353]), partition=
I20240503 02:00:11.073014 6831 tablet_service.cc:1467] Processing CreateTablet for tablet 286073252b944a0898aca512e6f3177a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=03276591ff8a400eb2636d7ca6dc2353]), partition=
I20240503 02:00:11.073086 9157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 286073252b944a0898aca512e6f3177a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:11.073241 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 286073252b944a0898aca512e6f3177a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:11.075351 13181 tablet_bootstrap.cc:492] T 286073252b944a0898aca512e6f3177a P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:11.075671 8880 tablet_service.cc:1467] Processing CreateTablet for tablet 286073252b944a0898aca512e6f3177a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=03276591ff8a400eb2636d7ca6dc2353]), partition=
I20240503 02:00:11.075847 13765 tablet_bootstrap.cc:492] T 286073252b944a0898aca512e6f3177a P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:11.076026 8880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 286073252b944a0898aca512e6f3177a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:11.076481 13181 tablet_bootstrap.cc:654] T 286073252b944a0898aca512e6f3177a P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:11.076779 13765 tablet_bootstrap.cc:654] T 286073252b944a0898aca512e6f3177a P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:11.078059 13181 tablet_bootstrap.cc:492] T 286073252b944a0898aca512e6f3177a P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:11.078142 13181 ts_tablet_manager.cc:1367] T 286073252b944a0898aca512e6f3177a P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 02:00:11.078202 13754 tablet_bootstrap.cc:492] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:11.078936 13181 raft_consensus.cc:348] T 286073252b944a0898aca512e6f3177a P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:11.079057 13181 raft_consensus.cc:374] T 286073252b944a0898aca512e6f3177a P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:11.079089 13181 raft_consensus.cc:729] T 286073252b944a0898aca512e6f3177a P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:11.079092 13754 tablet_bootstrap.cc:654] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:11.079210 13181 consensus_queue.cc:260] T 286073252b944a0898aca512e6f3177a P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:11.079480 13181 ts_tablet_manager.cc:1398] T 286073252b944a0898aca512e6f3177a P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:11.080930 13754 tablet_bootstrap.cc:492] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:11.081033 13754 ts_tablet_manager.cc:1367] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:00:11.081552 13754 raft_consensus.cc:348] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:11.081660 13754 raft_consensus.cc:374] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:11.081689 13754 raft_consensus.cc:729] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:11.081810 13754 consensus_queue.cc:260] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:11.082091 13754 ts_tablet_manager.cc:1398] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:11.083158 13765 tablet_bootstrap.cc:492] T 286073252b944a0898aca512e6f3177a P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:11.083317 13765 ts_tablet_manager.cc:1367] T 286073252b944a0898aca512e6f3177a P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 02:00:11.083912 13765 raft_consensus.cc:348] T 286073252b944a0898aca512e6f3177a P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:11.084110 13765 raft_consensus.cc:374] T 286073252b944a0898aca512e6f3177a P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:11.084185 13765 raft_consensus.cc:729] T 286073252b944a0898aca512e6f3177a P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:11.084365 13765 consensus_queue.cc:260] T 286073252b944a0898aca512e6f3177a P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:11.084769 13765 ts_tablet_manager.cc:1398] T 286073252b944a0898aca512e6f3177a P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20240503 02:00:11.090085 8851 consensus_peers.cc:485] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:11.096484 8851 consensus_peers.cc:485] T 067f54011a37432d99157884e70fc601 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:00:11.134701 12331 raft_consensus.cc:482] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:11.134850 12331 raft_consensus.cc:504] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:11.135337 12331 leader_election.cc:288] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311)
I20240503 02:00:11.135766 6719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dfe23ff4f2a456384690fa9e48e51c1" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 02:00:11.135916 6719 raft_consensus.cc:2454] T 8dfe23ff4f2a456384690fa9e48e51c1 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:11.136233 6525 leader_election.cc:302] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
W20240503 02:00:11.136406 6523 leader_election.cc:334] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:11.136413 12331 raft_consensus.cc:2781] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:11.136493 12331 raft_consensus.cc:482] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:11.136538 12331 raft_consensus.cc:3037] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:11.137398 12331 raft_consensus.cc:504] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:11.137753 12331 leader_election.cc:288] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Requested vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311)
I20240503 02:00:11.138114 6719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dfe23ff4f2a456384690fa9e48e51c1" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 02:00:11.138242 6719 raft_consensus.cc:3037] T 8dfe23ff4f2a456384690fa9e48e51c1 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:11.139035 6719 raft_consensus.cc:2454] T 8dfe23ff4f2a456384690fa9e48e51c1 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:11.139335 6525 leader_election.cc:302] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:11.139501 12331 raft_consensus.cc:2781] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:11.139570 12331 raft_consensus.cc:686] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Becoming Leader. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Running, Role: LEADER
I20240503 02:00:11.139707 12331 consensus_queue.cc:237] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
W20240503 02:00:11.139734 6523 leader_election.cc:334] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:11.142150 6312 catalog_manager.cc:5484] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } }
W20240503 02:00:11.160152 6654 consensus_peers.cc:485] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:11.165657 6523 consensus_peers.cc:485] T 4385d51d11574192883d01376e2d5620 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:11.169885 6785 consensus_peers.cc:485] T c3485701bf3b48f5ae2e1568afeed133 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:00:11.190601 12922 raft_consensus.cc:482] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:11.190730 12922 raft_consensus.cc:504] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } }
I20240503 02:00:11.191128 12922 leader_election.cc:288] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311)
I20240503 02:00:11.191573 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99bc944f874248b5a778971f7a2aaa0c" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:11.191746 9179 raft_consensus.cc:2454] T 99bc944f874248b5a778971f7a2aaa0c P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:11.192066 8981 leader_election.cc:302] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 99ead02915954b4fab23b033ad6d546b; no voters:
W20240503 02:00:11.192206 8982 leader_election.cc:334] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:11.192258 12922 raft_consensus.cc:2781] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:11.192296 12922 raft_consensus.cc:482] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:11.192318 12922 raft_consensus.cc:3037] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:11.193259 12922 raft_consensus.cc:504] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } }
I20240503 02:00:11.193600 12922 leader_election.cc:288] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Requested vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311)
I20240503 02:00:11.193924 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99bc944f874248b5a778971f7a2aaa0c" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:11.194053 9179 raft_consensus.cc:3037] T 99bc944f874248b5a778971f7a2aaa0c P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:00:11.194561 8982 leader_election.cc:334] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:11.194974 9179 raft_consensus.cc:2454] T 99bc944f874248b5a778971f7a2aaa0c P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:11.195246 8981 leader_election.cc:302] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 99ead02915954b4fab23b033ad6d546b; no voters: e1a9251ce1cf4361b8b0ede9fbfc4023
I20240503 02:00:11.195391 12922 raft_consensus.cc:2781] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:11.195475 12922 raft_consensus.cc:686] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Becoming Leader. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Running, Role: LEADER
I20240503 02:00:11.195614 12922 consensus_queue.cc:237] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } }
I20240503 02:00:11.213079 6312 catalog_manager.cc:5484] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b reported cstate change: term changed from 0 to 1, leader changed from <none> to 99ead02915954b4fab23b033ad6d546b (127.0.99.134). New cstate: current_term: 1 leader_uuid: "99ead02915954b4fab23b033ad6d546b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1a9251ce1cf4361b8b0ede9fbfc4023" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45311 } health_report { overall_health: UNKNOWN } } }
W20240503 02:00:11.232805 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
W20240503 02:00:11.260202 6523 consensus_peers.cc:485] T 925612b660ec471da5b3c548802ede6e P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:11.262396 6785 consensus_peers.cc:485] T 46ad540deb624a25b97d3b967fe1f6f4 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:00:11.269199 12922 consensus_queue.cc:1035] T 2dfc375064e24f2ea5bba0ae88535f87 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:11.273387 12922 consensus_queue.cc:1035] T 2dfc375064e24f2ea5bba0ae88535f87 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:00:11.281561 6654 consensus_peers.cc:485] T ba17d4d3e0754b2798eb95484d37229a P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:00:11.283671 13102 raft_consensus.cc:482] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:11.283797 13102 raft_consensus.cc:504] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:11.284194 13102 leader_election.cc:288] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:11.284698 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286073252b944a0898aca512e6f3177a" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:11.284698 6850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286073252b944a0898aca512e6f3177a" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 02:00:11.284849 6850 raft_consensus.cc:2454] T 286073252b944a0898aca512e6f3177a P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 0.
I20240503 02:00:11.284849 9179 raft_consensus.cc:2454] T 286073252b944a0898aca512e6f3177a P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 0.
I20240503 02:00:11.285157 8853 leader_election.cc:302] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:11.285391 13102 raft_consensus.cc:2781] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:11.285442 13102 raft_consensus.cc:482] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:11.285476 13102 raft_consensus.cc:3037] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:11.286321 13102 raft_consensus.cc:504] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:11.286695 13102 leader_election.cc:288] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 election: Requested vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:11.287073 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286073252b944a0898aca512e6f3177a" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:11.287232 9179 raft_consensus.cc:3037] T 286073252b944a0898aca512e6f3177a P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:11.287263 6850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286073252b944a0898aca512e6f3177a" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 02:00:11.287386 6850 raft_consensus.cc:3037] T 286073252b944a0898aca512e6f3177a P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:11.288122 9179 raft_consensus.cc:2454] T 286073252b944a0898aca512e6f3177a P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 1.
I20240503 02:00:11.288127 6850 raft_consensus.cc:2454] T 286073252b944a0898aca512e6f3177a P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 1.
I20240503 02:00:11.288434 8853 leader_election.cc:302] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:11.288610 13102 raft_consensus.cc:2781] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:11.288691 13102 raft_consensus.cc:686] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2 [term 1 LEADER]: Becoming Leader. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Running, Role: LEADER
I20240503 02:00:11.288861 13102 consensus_queue.cc:237] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:11.290953 6312 catalog_manager.cc:5484] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9942306855404cfd891adf227dd65eb2 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "9942306855404cfd891adf227dd65eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: HEALTHY } } }
W20240503 02:00:11.297222 6523 consensus_peers.cc:485] T d0d5dd42db3e4883af7fd16d8813b550 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:11.357892 6523 consensus_peers.cc:485] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:11.376219 8851 consensus_peers.cc:485] T 7da8f614a70e4177b70e7c2f31d22a10 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:11.379478 8982 consensus_peers.cc:485] T 13756a6e44814fe4b09a6b0f087546c3 P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:11.401546 6785 consensus_peers.cc:485] T 8246488dec204bf6b09e868d7951ac5a P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:00:11.414672 6312 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:11.418668 6850 raft_consensus.cc:1261] T 286073252b944a0898aca512e6f3177a P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:11.419095 9179 raft_consensus.cc:1261] T 286073252b944a0898aca512e6f3177a P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:11.419153 13102 consensus_queue.cc:1035] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:11.419526 13102 consensus_queue.cc:1035] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:11.422124 13781 tablet.cc:1722] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:11.422173 6312 catalog_manager.cc:5576] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719) does not have the latest schema for tablet 286073252b944a0898aca512e6f3177a (table rebalancer_dynamic_table [id=03276591ff8a400eb2636d7ca6dc2353]). Expected version 1 got 0
I20240503 02:00:11.422575 13781 tablet.cc:1620] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
I20240503 02:00:11.424854 6314 catalog_manager.cc:5576] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079) does not have the latest schema for tablet 286073252b944a0898aca512e6f3177a (table rebalancer_dynamic_table [id=03276591ff8a400eb2636d7ca6dc2353]). Expected version 1 got 0
I20240503 02:00:11.425967 13783 tablet.cc:1722] T 286073252b944a0898aca512e6f3177a P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:11.427362 13784 tablet.cc:1722] T 286073252b944a0898aca512e6f3177a P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:11.427781 13784 tablet.cc:1620] T 286073252b944a0898aca512e6f3177a P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
I20240503 02:00:11.427866 6309 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=03276591ff8a400eb2636d7ca6dc2353] alter complete (version 1)
I20240503 02:00:11.428416 6314 catalog_manager.cc:5576] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079) does not have the latest schema for tablet 286073252b944a0898aca512e6f3177a (table rebalancer_dynamic_table [id=03276591ff8a400eb2636d7ca6dc2353]). Expected version 1 got 0
I20240503 02:00:11.430562 13783 tablet.cc:1620] T 286073252b944a0898aca512e6f3177a P 492448570cc94ed1879feb2ec3b9051d: MemRowSet was empty: no flush needed.
W20240503 02:00:11.440701 6785 consensus_peers.cc:485] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:00:11.533891 6314 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:11.539758 13781 tablet.cc:1722] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:11.539884 13784 tablet.cc:1722] T 286073252b944a0898aca512e6f3177a P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:11.540084 13783 tablet.cc:1722] T 286073252b944a0898aca512e6f3177a P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:11.540148 13781 tablet.cc:1620] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
I20240503 02:00:11.540194 13784 tablet.cc:1620] T 286073252b944a0898aca512e6f3177a P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
I20240503 02:00:11.540349 13783 tablet.cc:1620] T 286073252b944a0898aca512e6f3177a P 492448570cc94ed1879feb2ec3b9051d: MemRowSet was empty: no flush needed.
I20240503 02:00:11.546074 6309 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=03276591ff8a400eb2636d7ca6dc2353] alter complete (version 2)
W20240503 02:00:11.563577 9113 consensus_peers.cc:485] T f44b59277c3b47adbd4dd6aff9350af1 P 2a01ace80baf4057b6bf1995cf88468b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:11.573029 8851 consensus_peers.cc:485] T f28586e70bd9473fbdc6b9126b666fa9 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:11.574934 9113 consensus_peers.cc:485] T b1740b877c1e4372a0337d99f5bcab0e P 2a01ace80baf4057b6bf1995cf88468b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:11.581688 8851 consensus_peers.cc:485] T 5f51fe5f799f4e479b62f5917a2c2606 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:11.605531 6785 consensus_peers.cc:485] T d897c21e630d4770832bb10f43778e82 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:11.638689 8982 consensus_peers.cc:485] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:11.648957 6312 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:11.655812 13781 tablet.cc:1722] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:11.656965 13783 tablet.cc:1722] T 286073252b944a0898aca512e6f3177a P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:11.657963 13784 tablet.cc:1722] T 286073252b944a0898aca512e6f3177a P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:11.664798 6309 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=03276591ff8a400eb2636d7ca6dc2353] alter complete (version 3)
W20240503 02:00:11.682772 8982 consensus_peers.cc:485] T 1a7ee79312ea4bad816264375ca5bb2d P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:00:11.685608 12922 consensus_queue.cc:1035] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:11.700259 12331 consensus_queue.cc:1035] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:00:11.704149 6523 consensus_peers.cc:485] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:11.764353 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:11.766753 6312 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:11.766929 6312 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:11.769258 6312 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet 286073252b944a0898aca512e6f3177a
I20240503 02:00:11.770069 8880 tablet_service.cc:1514] Processing DeleteTablet for tablet 286073252b944a0898aca512e6f3177a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:11 UTC) from {username='slave'} at 127.0.0.1:40296
I20240503 02:00:11.770135 6831 tablet_service.cc:1514] Processing DeleteTablet for tablet 286073252b944a0898aca512e6f3177a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:11 UTC) from {username='slave'} at 127.0.0.1:52812
I20240503 02:00:11.770488 9157 tablet_service.cc:1514] Processing DeleteTablet for tablet 286073252b944a0898aca512e6f3177a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:11 UTC) from {username='slave'} at 127.0.0.1:51940
I20240503 02:00:11.770776 13794 tablet_replica.cc:330] T 286073252b944a0898aca512e6f3177a P 2a01ace80baf4057b6bf1995cf88468b: stopping tablet replica
I20240503 02:00:11.770934 13794 raft_consensus.cc:2229] T 286073252b944a0898aca512e6f3177a P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:11.771085 13794 raft_consensus.cc:2258] T 286073252b944a0898aca512e6f3177a P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:11.780067 13793 tablet_replica.cc:330] T 286073252b944a0898aca512e6f3177a P 492448570cc94ed1879feb2ec3b9051d: stopping tablet replica
I20240503 02:00:11.780248 13793 raft_consensus.cc:2229] T 286073252b944a0898aca512e6f3177a P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:11.780431 13793 raft_consensus.cc:2258] T 286073252b944a0898aca512e6f3177a P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:11.780689 13792 tablet_replica.cc:330] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2: stopping tablet replica
I20240503 02:00:11.780840 13792 raft_consensus.cc:2229] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:11.781067 13792 raft_consensus.cc:2258] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:11.781605 13793 ts_tablet_manager.cc:1875] T 286073252b944a0898aca512e6f3177a P 492448570cc94ed1879feb2ec3b9051d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:11.781730 13792 ts_tablet_manager.cc:1875] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:11.783849 13794 ts_tablet_manager.cc:1875] T 286073252b944a0898aca512e6f3177a P 2a01ace80baf4057b6bf1995cf88468b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:11.784215 13793 ts_tablet_manager.cc:1888] T 286073252b944a0898aca512e6f3177a P 492448570cc94ed1879feb2ec3b9051d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:11.784298 13793 log.cc:1196] T 286073252b944a0898aca512e6f3177a P 492448570cc94ed1879feb2ec3b9051d: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/286073252b944a0898aca512e6f3177a
I20240503 02:00:11.784346 13792 ts_tablet_manager.cc:1888] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:11.784484 13792 log.cc:1196] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/286073252b944a0898aca512e6f3177a
I20240503 02:00:11.784710 13793 ts_tablet_manager.cc:1909] T 286073252b944a0898aca512e6f3177a P 492448570cc94ed1879feb2ec3b9051d: Deleting consensus metadata
I20240503 02:00:11.784770 13792 ts_tablet_manager.cc:1909] T 286073252b944a0898aca512e6f3177a P 9942306855404cfd891adf227dd65eb2: Deleting consensus metadata
I20240503 02:00:11.785611 6303 catalog_manager.cc:4830] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211): tablet 286073252b944a0898aca512e6f3177a (table rebalancer_dynamic_table_new_name [id=03276591ff8a400eb2636d7ca6dc2353]) successfully deleted
I20240503 02:00:11.785799 6303 catalog_manager.cc:4830] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719): tablet 286073252b944a0898aca512e6f3177a (table rebalancer_dynamic_table_new_name [id=03276591ff8a400eb2636d7ca6dc2353]) successfully deleted
I20240503 02:00:11.786248 13794 ts_tablet_manager.cc:1888] T 286073252b944a0898aca512e6f3177a P 2a01ace80baf4057b6bf1995cf88468b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:11.786321 13794 log.cc:1196] T 286073252b944a0898aca512e6f3177a P 2a01ace80baf4057b6bf1995cf88468b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/286073252b944a0898aca512e6f3177a
I20240503 02:00:11.786530 13794 ts_tablet_manager.cc:1909] T 286073252b944a0898aca512e6f3177a P 2a01ace80baf4057b6bf1995cf88468b: Deleting consensus metadata
I20240503 02:00:11.787319 6300 catalog_manager.cc:4830] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079): tablet 286073252b944a0898aca512e6f3177a (table rebalancer_dynamic_table_new_name [id=03276591ff8a400eb2636d7ca6dc2353]) successfully deleted
I20240503 02:00:11.795964 6312 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_extra_table_30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:11.802423 8880 tablet_service.cc:1467] Processing CreateTablet for tablet 5625259b3ac54b4ea639e2823902aee5 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=21fd6878ffc94acaa25ab621ead27223]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:11.802762 8880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5625259b3ac54b4ea639e2823902aee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:11.802850 8897 tablet_service.cc:1467] Processing CreateTablet for tablet e8466d03df4c4ed8bf571bffd936f515 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=21fd6878ffc94acaa25ab621ead27223]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:11.803103 8897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8466d03df4c4ed8bf571bffd936f515. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:11.803051 9028 tablet_service.cc:1467] Processing CreateTablet for tablet d244f21f8c6a4ab296e9a08f50d9c990 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=21fd6878ffc94acaa25ab621ead27223]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:11.803304 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d244f21f8c6a4ab296e9a08f50d9c990. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:11.805178 6700 tablet_service.cc:1467] Processing CreateTablet for tablet d244f21f8c6a4ab296e9a08f50d9c990 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=21fd6878ffc94acaa25ab621ead27223]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:11.805491 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d244f21f8c6a4ab296e9a08f50d9c990. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:11.805554 6438 tablet_service.cc:1467] Processing CreateTablet for tablet e8466d03df4c4ed8bf571bffd936f515 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=21fd6878ffc94acaa25ab621ead27223]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:11.805809 6438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8466d03df4c4ed8bf571bffd936f515. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:11.805805 13796 tablet_bootstrap.cc:492] T d244f21f8c6a4ab296e9a08f50d9c990 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:11.807004 13796 tablet_bootstrap.cc:654] T d244f21f8c6a4ab296e9a08f50d9c990 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:11.807544 13797 tablet_bootstrap.cc:492] T e8466d03df4c4ed8bf571bffd936f515 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:11.808056 6436 tablet_service.cc:1467] Processing CreateTablet for tablet 5625259b3ac54b4ea639e2823902aee5 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=21fd6878ffc94acaa25ab621ead27223]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:11.808821 13798 tablet_bootstrap.cc:492] T d244f21f8c6a4ab296e9a08f50d9c990 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:11.808835 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5625259b3ac54b4ea639e2823902aee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:11.808877 9158 tablet_service.cc:1467] Processing CreateTablet for tablet e8466d03df4c4ed8bf571bffd936f515 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=21fd6878ffc94acaa25ab621ead27223]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:11.809146 9158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8466d03df4c4ed8bf571bffd936f515. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:11.809842 13798 tablet_bootstrap.cc:654] T d244f21f8c6a4ab296e9a08f50d9c990 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:11.810004 13797 tablet_bootstrap.cc:654] T e8466d03df4c4ed8bf571bffd936f515 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:11.811108 13797 tablet_bootstrap.cc:492] T e8466d03df4c4ed8bf571bffd936f515 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:11.811244 13797 ts_tablet_manager.cc:1367] T e8466d03df4c4ed8bf571bffd936f515 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:11.811424 9159 tablet_service.cc:1467] Processing CreateTablet for tablet d244f21f8c6a4ab296e9a08f50d9c990 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=21fd6878ffc94acaa25ab621ead27223]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:11.811515 13795 tablet_bootstrap.cc:492] T 5625259b3ac54b4ea639e2823902aee5 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:11.811698 9159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d244f21f8c6a4ab296e9a08f50d9c990. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:11.811733 13797 raft_consensus.cc:348] T e8466d03df4c4ed8bf571bffd936f515 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:11.811841 13797 raft_consensus.cc:374] T e8466d03df4c4ed8bf571bffd936f515 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:11.811872 13797 raft_consensus.cc:729] T e8466d03df4c4ed8bf571bffd936f515 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:11.811988 13797 consensus_queue.cc:260] T e8466d03df4c4ed8bf571bffd936f515 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:11.812785 13797 ts_tablet_manager.cc:1398] T e8466d03df4c4ed8bf571bffd936f515 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:11.812813 13795 tablet_bootstrap.cc:654] T 5625259b3ac54b4ea639e2823902aee5 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:11.812983 13797 tablet_bootstrap.cc:492] T 5625259b3ac54b4ea639e2823902aee5 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:11.814216 13798 tablet_bootstrap.cc:492] T d244f21f8c6a4ab296e9a08f50d9c990 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:11.814304 13798 ts_tablet_manager.cc:1367] T d244f21f8c6a4ab296e9a08f50d9c990 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:00:11.814435 13797 tablet_bootstrap.cc:654] T 5625259b3ac54b4ea639e2823902aee5 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:11.814769 13798 raft_consensus.cc:348] T d244f21f8c6a4ab296e9a08f50d9c990 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:11.814880 13798 raft_consensus.cc:374] T d244f21f8c6a4ab296e9a08f50d9c990 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:11.814911 13798 raft_consensus.cc:729] T d244f21f8c6a4ab296e9a08f50d9c990 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:11.815037 13798 consensus_queue.cc:260] T d244f21f8c6a4ab296e9a08f50d9c990 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:11.815313 13798 ts_tablet_manager.cc:1398] T d244f21f8c6a4ab296e9a08f50d9c990 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:11.815407 9157 tablet_service.cc:1467] Processing CreateTablet for tablet 5625259b3ac54b4ea639e2823902aee5 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=21fd6878ffc94acaa25ab621ead27223]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:11.815791 9157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5625259b3ac54b4ea639e2823902aee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:11.821483 13801 tablet_bootstrap.cc:492] T e8466d03df4c4ed8bf571bffd936f515 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:11.822060 13796 tablet_bootstrap.cc:492] T d244f21f8c6a4ab296e9a08f50d9c990 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:11.822187 13796 ts_tablet_manager.cc:1367] T d244f21f8c6a4ab296e9a08f50d9c990 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240503 02:00:11.822767 13796 raft_consensus.cc:348] T d244f21f8c6a4ab296e9a08f50d9c990 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:11.822914 13796 raft_consensus.cc:374] T d244f21f8c6a4ab296e9a08f50d9c990 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:11.822948 13796 raft_consensus.cc:729] T d244f21f8c6a4ab296e9a08f50d9c990 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:11.823083 13796 consensus_queue.cc:260] T d244f21f8c6a4ab296e9a08f50d9c990 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:11.823418 13796 ts_tablet_manager.cc:1398] T d244f21f8c6a4ab296e9a08f50d9c990 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 02:00:11.824713 13801 tablet_bootstrap.cc:654] T e8466d03df4c4ed8bf571bffd936f515 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:11.826112 13795 tablet_bootstrap.cc:492] T 5625259b3ac54b4ea639e2823902aee5 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:11.826234 13795 ts_tablet_manager.cc:1367] T 5625259b3ac54b4ea639e2823902aee5 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240503 02:00:11.826395 13801 tablet_bootstrap.cc:492] T e8466d03df4c4ed8bf571bffd936f515 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:11.826489 13801 ts_tablet_manager.cc:1367] T e8466d03df4c4ed8bf571bffd936f515 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:00:11.826766 13795 raft_consensus.cc:348] T 5625259b3ac54b4ea639e2823902aee5 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:11.826900 13795 raft_consensus.cc:374] T 5625259b3ac54b4ea639e2823902aee5 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:11.827067 13795 raft_consensus.cc:729] T 5625259b3ac54b4ea639e2823902aee5 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:11.827299 13795 consensus_queue.cc:260] T 5625259b3ac54b4ea639e2823902aee5 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:11.827822 13801 raft_consensus.cc:348] T e8466d03df4c4ed8bf571bffd936f515 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:11.828044 13801 raft_consensus.cc:374] T e8466d03df4c4ed8bf571bffd936f515 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:11.828326 13801 raft_consensus.cc:729] T e8466d03df4c4ed8bf571bffd936f515 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:11.828617 13801 consensus_queue.cc:260] T e8466d03df4c4ed8bf571bffd936f515 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:11.828743 13795 ts_tablet_manager.cc:1398] T 5625259b3ac54b4ea639e2823902aee5 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:11.829104 13795 tablet_bootstrap.cc:492] T e8466d03df4c4ed8bf571bffd936f515 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:11.829104 13801 ts_tablet_manager.cc:1398] T e8466d03df4c4ed8bf571bffd936f515 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:11.829298 13801 tablet_bootstrap.cc:492] T 5625259b3ac54b4ea639e2823902aee5 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:11.830183 13795 tablet_bootstrap.cc:654] T e8466d03df4c4ed8bf571bffd936f515 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:11.830724 13801 tablet_bootstrap.cc:654] T 5625259b3ac54b4ea639e2823902aee5 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:11.831463 13797 tablet_bootstrap.cc:492] T 5625259b3ac54b4ea639e2823902aee5 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:11.831657 13797 ts_tablet_manager.cc:1367] T 5625259b3ac54b4ea639e2823902aee5 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240503 02:00:11.832895 13801 tablet_bootstrap.cc:492] T 5625259b3ac54b4ea639e2823902aee5 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:11.833369 13801 ts_tablet_manager.cc:1367] T 5625259b3ac54b4ea639e2823902aee5 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:11.834008 13801 raft_consensus.cc:348] T 5625259b3ac54b4ea639e2823902aee5 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:11.834194 13801 raft_consensus.cc:374] T 5625259b3ac54b4ea639e2823902aee5 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:11.834265 13801 raft_consensus.cc:729] T 5625259b3ac54b4ea639e2823902aee5 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:11.834463 13801 consensus_queue.cc:260] T 5625259b3ac54b4ea639e2823902aee5 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:11.834667 13797 raft_consensus.cc:348] T 5625259b3ac54b4ea639e2823902aee5 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:11.834851 13797 raft_consensus.cc:374] T 5625259b3ac54b4ea639e2823902aee5 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:11.834905 13801 ts_tablet_manager.cc:1398] T 5625259b3ac54b4ea639e2823902aee5 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:11.835143 13797 raft_consensus.cc:729] T 5625259b3ac54b4ea639e2823902aee5 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:11.835242 13801 tablet_bootstrap.cc:492] T d244f21f8c6a4ab296e9a08f50d9c990 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:11.836436 13801 tablet_bootstrap.cc:654] T d244f21f8c6a4ab296e9a08f50d9c990 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:11.836997 13795 tablet_bootstrap.cc:492] T e8466d03df4c4ed8bf571bffd936f515 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:11.837111 13795 ts_tablet_manager.cc:1367] T e8466d03df4c4ed8bf571bffd936f515 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 02:00:11.837015 13797 consensus_queue.cc:260] T 5625259b3ac54b4ea639e2823902aee5 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:11.837430 13797 ts_tablet_manager.cc:1398] T 5625259b3ac54b4ea639e2823902aee5 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:00:11.837610 13795 raft_consensus.cc:348] T e8466d03df4c4ed8bf571bffd936f515 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:11.837725 13795 raft_consensus.cc:374] T e8466d03df4c4ed8bf571bffd936f515 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:11.837754 13795 raft_consensus.cc:729] T e8466d03df4c4ed8bf571bffd936f515 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:11.837884 13795 consensus_queue.cc:260] T e8466d03df4c4ed8bf571bffd936f515 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:11.838153 13795 ts_tablet_manager.cc:1398] T e8466d03df4c4ed8bf571bffd936f515 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:11.841984 13801 tablet_bootstrap.cc:492] T d244f21f8c6a4ab296e9a08f50d9c990 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:11.842078 13801 ts_tablet_manager.cc:1367] T d244f21f8c6a4ab296e9a08f50d9c990 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:00:11.842489 13801 raft_consensus.cc:348] T d244f21f8c6a4ab296e9a08f50d9c990 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:11.842589 13801 raft_consensus.cc:374] T d244f21f8c6a4ab296e9a08f50d9c990 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:11.842618 13801 raft_consensus.cc:729] T d244f21f8c6a4ab296e9a08f50d9c990 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:11.842716 13801 consensus_queue.cc:260] T d244f21f8c6a4ab296e9a08f50d9c990 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:11.842984 13801 ts_tablet_manager.cc:1398] T d244f21f8c6a4ab296e9a08f50d9c990 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:11.871536 6314 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:00:11.877476 9028 tablet_service.cc:1467] Processing CreateTablet for tablet 141d454209d3476c93071c291e16a44c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=55bf5f42f2d9482bad4ceb69fb30c3e5]), partition=
I20240503 02:00:11.877791 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 141d454209d3476c93071c291e16a44c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:11.878105 8897 tablet_service.cc:1467] Processing CreateTablet for tablet 141d454209d3476c93071c291e16a44c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=55bf5f42f2d9482bad4ceb69fb30c3e5]), partition=
I20240503 02:00:11.878317 8897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 141d454209d3476c93071c291e16a44c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:11.880143 13796 tablet_bootstrap.cc:492] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:11.880362 6569 tablet_service.cc:1467] Processing CreateTablet for tablet 141d454209d3476c93071c291e16a44c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=55bf5f42f2d9482bad4ceb69fb30c3e5]), partition=
I20240503 02:00:11.880625 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 141d454209d3476c93071c291e16a44c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:11.880838 13795 tablet_bootstrap.cc:492] T 141d454209d3476c93071c291e16a44c P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:11.881297 13796 tablet_bootstrap.cc:654] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:11.881884 13795 tablet_bootstrap.cc:654] T 141d454209d3476c93071c291e16a44c P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:11.882944 13795 tablet_bootstrap.cc:492] T 141d454209d3476c93071c291e16a44c P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:11.882972 13811 tablet_bootstrap.cc:492] T 141d454209d3476c93071c291e16a44c P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:11.883139 13795 ts_tablet_manager.cc:1367] T 141d454209d3476c93071c291e16a44c P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:11.883206 13796 tablet_bootstrap.cc:492] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:11.883296 13796 ts_tablet_manager.cc:1367] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:11.883728 13796 raft_consensus.cc:348] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:11.883708 13795 raft_consensus.cc:348] T 141d454209d3476c93071c291e16a44c P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:11.883834 13796 raft_consensus.cc:374] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:11.883863 13811 tablet_bootstrap.cc:654] T 141d454209d3476c93071c291e16a44c P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:11.883867 13795 raft_consensus.cc:374] T 141d454209d3476c93071c291e16a44c P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:11.883884 13796 raft_consensus.cc:729] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:11.883908 13795 raft_consensus.cc:729] T 141d454209d3476c93071c291e16a44c P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:11.884013 13796 consensus_queue.cc:260] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:11.884101 13795 consensus_queue.cc:260] T 141d454209d3476c93071c291e16a44c P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:11.884308 13796 ts_tablet_manager.cc:1398] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:11.885303 13795 ts_tablet_manager.cc:1398] T 141d454209d3476c93071c291e16a44c P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:11.886150 13811 tablet_bootstrap.cc:492] T 141d454209d3476c93071c291e16a44c P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:11.886284 13811 ts_tablet_manager.cc:1367] T 141d454209d3476c93071c291e16a44c P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:00:11.886790 13811 raft_consensus.cc:348] T 141d454209d3476c93071c291e16a44c P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:11.886935 13811 raft_consensus.cc:374] T 141d454209d3476c93071c291e16a44c P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:11.886981 13811 raft_consensus.cc:729] T 141d454209d3476c93071c291e16a44c P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:11.887141 13811 consensus_queue.cc:260] T 141d454209d3476c93071c291e16a44c P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:11.887445 13811 ts_tablet_manager.cc:1398] T 141d454209d3476c93071c291e16a44c P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:11.921594 13791 raft_consensus.cc:482] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:11.921730 13791 raft_consensus.cc:504] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:11.922191 13791 leader_election.cc:288] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:11.922555 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141d454209d3476c93071c291e16a44c" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 02:00:11.922657 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141d454209d3476c93071c291e16a44c" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:11.922724 6589 raft_consensus.cc:2454] T 141d454209d3476c93071c291e16a44c P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:11.922789 8916 raft_consensus.cc:2454] T 141d454209d3476c93071c291e16a44c P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:11.923044 8983 leader_election.cc:302] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:11.923254 13791 raft_consensus.cc:2781] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:11.923305 13791 raft_consensus.cc:482] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:11.923339 13791 raft_consensus.cc:3037] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:11.924515 13791 raft_consensus.cc:504] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:11.924994 13791 leader_election.cc:288] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Requested vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:11.925323 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141d454209d3476c93071c291e16a44c" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:11.925323 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141d454209d3476c93071c291e16a44c" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 02:00:11.925451 8916 raft_consensus.cc:3037] T 141d454209d3476c93071c291e16a44c P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:11.925451 6589 raft_consensus.cc:3037] T 141d454209d3476c93071c291e16a44c P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:11.926462 8916 raft_consensus.cc:2454] T 141d454209d3476c93071c291e16a44c P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:11.926462 6589 raft_consensus.cc:2454] T 141d454209d3476c93071c291e16a44c P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:11.926774 8984 leader_election.cc:302] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9942306855404cfd891adf227dd65eb2, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:11.926991 13791 raft_consensus.cc:2781] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:11.927093 13791 raft_consensus.cc:686] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Becoming Leader. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Running, Role: LEADER
I20240503 02:00:11.927240 13791 consensus_queue.cc:237] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:11.929544 6314 catalog_manager.cc:5484] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b reported cstate change: term changed from 0 to 1, leader changed from <none> to 99ead02915954b4fab23b033ad6d546b (127.0.99.134). New cstate: current_term: 1 leader_uuid: "99ead02915954b4fab23b033ad6d546b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } }
W20240503 02:00:11.960851 8982 consensus_peers.cc:485] T beeea71f02ae4c89a1af13d3584ef70a P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:00:11.998099 13776 raft_consensus.cc:482] T e8466d03df4c4ed8bf571bffd936f515 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:11.998248 13776 raft_consensus.cc:504] T e8466d03df4c4ed8bf571bffd936f515 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:11.998700 13776 leader_election.cc:288] T e8466d03df4c4ed8bf571bffd936f515 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:11.999089 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8466d03df4c4ed8bf571bffd936f515" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:11.999143 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8466d03df4c4ed8bf571bffd936f515" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 02:00:11.999228 8916 raft_consensus.cc:2454] T e8466d03df4c4ed8bf571bffd936f515 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:11.999325 6456 raft_consensus.cc:2454] T e8466d03df4c4ed8bf571bffd936f515 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:11.999482 9115 leader_election.cc:302] T e8466d03df4c4ed8bf571bffd936f515 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:11.999677 13776 raft_consensus.cc:2781] T e8466d03df4c4ed8bf571bffd936f515 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:11.999729 13776 raft_consensus.cc:482] T e8466d03df4c4ed8bf571bffd936f515 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:11.999764 13776 raft_consensus.cc:3037] T e8466d03df4c4ed8bf571bffd936f515 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:12.000902 13776 raft_consensus.cc:504] T e8466d03df4c4ed8bf571bffd936f515 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:12.001307 13776 leader_election.cc:288] T e8466d03df4c4ed8bf571bffd936f515 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Requested vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:12.001657 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8466d03df4c4ed8bf571bffd936f515" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 02:00:12.001690 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8466d03df4c4ed8bf571bffd936f515" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:12.001778 6456 raft_consensus.cc:3037] T e8466d03df4c4ed8bf571bffd936f515 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:12.001822 8916 raft_consensus.cc:3037] T e8466d03df4c4ed8bf571bffd936f515 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:12.002611 6456 raft_consensus.cc:2454] T e8466d03df4c4ed8bf571bffd936f515 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:12.002918 8916 raft_consensus.cc:2454] T e8466d03df4c4ed8bf571bffd936f515 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:12.003052 9112 leader_election.cc:302] T e8466d03df4c4ed8bf571bffd936f515 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:12.003583 13776 raft_consensus.cc:2781] T e8466d03df4c4ed8bf571bffd936f515 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:12.003854 13776 raft_consensus.cc:686] T e8466d03df4c4ed8bf571bffd936f515 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Becoming Leader. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Running, Role: LEADER
I20240503 02:00:12.004048 13776 consensus_queue.cc:237] T e8466d03df4c4ed8bf571bffd936f515 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:12.006366 6314 catalog_manager.cc:5484] T e8466d03df4c4ed8bf571bffd936f515 P 2a01ace80baf4057b6bf1995cf88468b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135). New cstate: current_term: 1 leader_uuid: "2a01ace80baf4057b6bf1995cf88468b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:12.034338 6314 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:12.038699 8916 raft_consensus.cc:1261] T 141d454209d3476c93071c291e16a44c P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Refusing update from remote peer 99ead02915954b4fab23b033ad6d546b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:12.038699 6589 raft_consensus.cc:1261] T 141d454209d3476c93071c291e16a44c P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer 99ead02915954b4fab23b033ad6d546b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:12.039188 12922 raft_consensus.cc:482] T d244f21f8c6a4ab296e9a08f50d9c990 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:12.039301 12922 raft_consensus.cc:504] T d244f21f8c6a4ab296e9a08f50d9c990 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:12.039295 13791 consensus_queue.cc:1035] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:12.039481 13720 consensus_queue.cc:1035] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:12.039754 12922 leader_election.cc:288] T d244f21f8c6a4ab296e9a08f50d9c990 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:12.040277 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d244f21f8c6a4ab296e9a08f50d9c990" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:12.040441 9179 raft_consensus.cc:2454] T d244f21f8c6a4ab296e9a08f50d9c990 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:12.040699 8981 leader_election.cc:302] T d244f21f8c6a4ab296e9a08f50d9c990 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:12.040855 13791 raft_consensus.cc:2781] T d244f21f8c6a4ab296e9a08f50d9c990 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:12.040939 13791 raft_consensus.cc:482] T d244f21f8c6a4ab296e9a08f50d9c990 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:12.040983 13791 raft_consensus.cc:3037] T d244f21f8c6a4ab296e9a08f50d9c990 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:12.041100 6719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d244f21f8c6a4ab296e9a08f50d9c990" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 02:00:12.041245 6719 raft_consensus.cc:2454] T d244f21f8c6a4ab296e9a08f50d9c990 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:12.041859 13791 raft_consensus.cc:504] T d244f21f8c6a4ab296e9a08f50d9c990 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:12.042222 13791 leader_election.cc:288] T d244f21f8c6a4ab296e9a08f50d9c990 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Requested vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:12.042531 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d244f21f8c6a4ab296e9a08f50d9c990" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:12.042646 9179 raft_consensus.cc:3037] T d244f21f8c6a4ab296e9a08f50d9c990 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:12.042820 6719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d244f21f8c6a4ab296e9a08f50d9c990" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 02:00:12.042930 6719 raft_consensus.cc:3037] T d244f21f8c6a4ab296e9a08f50d9c990 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:12.043416 6314 catalog_manager.cc:5576] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211) does not have the latest schema for tablet 141d454209d3476c93071c291e16a44c (table rebalancer_dynamic_table [id=55bf5f42f2d9482bad4ceb69fb30c3e5]). Expected version 1 got 0
I20240503 02:00:12.043895 9179 raft_consensus.cc:2454] T d244f21f8c6a4ab296e9a08f50d9c990 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:12.043895 6719 raft_consensus.cc:2454] T d244f21f8c6a4ab296e9a08f50d9c990 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:12.044214 8981 leader_election.cc:302] T d244f21f8c6a4ab296e9a08f50d9c990 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:12.044363 13791 raft_consensus.cc:2781] T d244f21f8c6a4ab296e9a08f50d9c990 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:12.044476 13791 raft_consensus.cc:686] T d244f21f8c6a4ab296e9a08f50d9c990 P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Becoming Leader. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Running, Role: LEADER
I20240503 02:00:12.044631 13791 consensus_queue.cc:237] T d244f21f8c6a4ab296e9a08f50d9c990 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:12.046669 6314 catalog_manager.cc:5576] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863) does not have the latest schema for tablet 141d454209d3476c93071c291e16a44c (table rebalancer_dynamic_table [id=55bf5f42f2d9482bad4ceb69fb30c3e5]). Expected version 1 got 0
I20240503 02:00:12.047564 6313 catalog_manager.cc:5484] T d244f21f8c6a4ab296e9a08f50d9c990 P 99ead02915954b4fab23b033ad6d546b reported cstate change: term changed from 0 to 1, leader changed from <none> to 99ead02915954b4fab23b033ad6d546b (127.0.99.134). New cstate: current_term: 1 leader_uuid: "99ead02915954b4fab23b033ad6d546b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:12.047950 13820 tablet.cc:1722] T 141d454209d3476c93071c291e16a44c P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:12.048202 13781 tablet.cc:1722] T 141d454209d3476c93071c291e16a44c P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:12.048343 13820 tablet.cc:1620] T 141d454209d3476c93071c291e16a44c P 80fd18ef4d604c86b909d5c008a46b4f: MemRowSet was empty: no flush needed.
I20240503 02:00:12.048538 13781 tablet.cc:1620] T 141d454209d3476c93071c291e16a44c P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
I20240503 02:00:12.046614 13819 tablet.cc:1722] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:12.053874 6314 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=55bf5f42f2d9482bad4ceb69fb30c3e5] alter complete (version 1)
I20240503 02:00:12.054417 13819 tablet.cc:1620] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b: MemRowSet was empty: no flush needed.
I20240503 02:00:12.157114 6309 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:12.164134 13819 tablet.cc:1722] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:12.164857 13819 tablet.cc:1620] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b: MemRowSet was empty: no flush needed.
I20240503 02:00:12.164999 13781 tablet.cc:1722] T 141d454209d3476c93071c291e16a44c P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:12.165298 13781 tablet.cc:1620] T 141d454209d3476c93071c291e16a44c P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
I20240503 02:00:12.167824 13814 raft_consensus.cc:482] T 5625259b3ac54b4ea639e2823902aee5 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:12.168004 13814 raft_consensus.cc:504] T 5625259b3ac54b4ea639e2823902aee5 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:12.168642 13814 leader_election.cc:288] T 5625259b3ac54b4ea639e2823902aee5 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 02:00:12.169306 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5625259b3ac54b4ea639e2823902aee5" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 02:00:12.169440 6456 raft_consensus.cc:2454] T 5625259b3ac54b4ea639e2823902aee5 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:12.169471 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5625259b3ac54b4ea639e2823902aee5" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:12.169612 8916 raft_consensus.cc:2454] T 5625259b3ac54b4ea639e2823902aee5 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:12.169592 13820 tablet.cc:1722] T 141d454209d3476c93071c291e16a44c P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:12.169867 9115 leader_election.cc:302] T 5625259b3ac54b4ea639e2823902aee5 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:12.169909 13820 tablet.cc:1620] T 141d454209d3476c93071c291e16a44c P 80fd18ef4d604c86b909d5c008a46b4f: MemRowSet was empty: no flush needed.
I20240503 02:00:12.170428 13814 raft_consensus.cc:2781] T 5625259b3ac54b4ea639e2823902aee5 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:12.170534 13814 raft_consensus.cc:482] T 5625259b3ac54b4ea639e2823902aee5 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:12.170621 13814 raft_consensus.cc:3037] T 5625259b3ac54b4ea639e2823902aee5 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:12.173409 13814 raft_consensus.cc:504] T 5625259b3ac54b4ea639e2823902aee5 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:12.173636 6314 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=55bf5f42f2d9482bad4ceb69fb30c3e5] alter complete (version 2)
I20240503 02:00:12.175136 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5625259b3ac54b4ea639e2823902aee5" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:12.175469 8916 raft_consensus.cc:3037] T 5625259b3ac54b4ea639e2823902aee5 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:12.176617 8916 raft_consensus.cc:2454] T 5625259b3ac54b4ea639e2823902aee5 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:12.176645 13814 leader_election.cc:288] T 5625259b3ac54b4ea639e2823902aee5 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Requested vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 02:00:12.176918 9115 leader_election.cc:302] T 5625259b3ac54b4ea639e2823902aee5 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:12.177119 13814 raft_consensus.cc:2781] T 5625259b3ac54b4ea639e2823902aee5 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:12.177079 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5625259b3ac54b4ea639e2823902aee5" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 02:00:12.177217 6456 raft_consensus.cc:3037] T 5625259b3ac54b4ea639e2823902aee5 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:12.177210 13814 raft_consensus.cc:686] T 5625259b3ac54b4ea639e2823902aee5 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Becoming Leader. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Running, Role: LEADER
I20240503 02:00:12.177359 13814 consensus_queue.cc:237] T 5625259b3ac54b4ea639e2823902aee5 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:12.178584 6456 raft_consensus.cc:2454] T 5625259b3ac54b4ea639e2823902aee5 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:12.179916 6309 catalog_manager.cc:5484] T 5625259b3ac54b4ea639e2823902aee5 P 2a01ace80baf4057b6bf1995cf88468b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135). New cstate: current_term: 1 leader_uuid: "2a01ace80baf4057b6bf1995cf88468b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } }
W20240503 02:00:12.205469 6301 catalog_manager.cc:4573] TS e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cc19d71f40fb44f1801f1b49dcc76fe0: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
W20240503 02:00:12.264587 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:12.276485 6309 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:12.282011 13819 tablet.cc:1722] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:12.282416 13781 tablet.cc:1722] T 141d454209d3476c93071c291e16a44c P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:12.282420 13820 tablet.cc:1722] T 141d454209d3476c93071c291e16a44c P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:12.287559 6313 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=55bf5f42f2d9482bad4ceb69fb30c3e5] alter complete (version 3)
W20240503 02:00:12.309191 6654 consensus_peers.cc:485] T cea1ea50fa434d31b42451f84f26bc76 P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:00:12.390223 6309 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:12.390383 6309 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:12.392812 6309 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet 141d454209d3476c93071c291e16a44c
I20240503 02:00:12.393450 6569 tablet_service.cc:1514] Processing DeleteTablet for tablet 141d454209d3476c93071c291e16a44c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:12 UTC) from {username='slave'} at 127.0.0.1:59846
I20240503 02:00:12.393529 8897 tablet_service.cc:1514] Processing DeleteTablet for tablet 141d454209d3476c93071c291e16a44c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:12 UTC) from {username='slave'} at 127.0.0.1:40296
I20240503 02:00:12.393770 9028 tablet_service.cc:1514] Processing DeleteTablet for tablet 141d454209d3476c93071c291e16a44c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:12 UTC) from {username='slave'} at 127.0.0.1:52584
I20240503 02:00:12.393791 13822 tablet_replica.cc:330] T 141d454209d3476c93071c291e16a44c P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 02:00:12.393955 13822 raft_consensus.cc:2229] T 141d454209d3476c93071c291e16a44c P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:12.393966 13823 tablet_replica.cc:330] T 141d454209d3476c93071c291e16a44c P 9942306855404cfd891adf227dd65eb2: stopping tablet replica
I20240503 02:00:12.394045 13824 tablet_replica.cc:330] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b: stopping tablet replica
I20240503 02:00:12.394081 13822 raft_consensus.cc:2258] T 141d454209d3476c93071c291e16a44c P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:12.394104 13823 raft_consensus.cc:2229] T 141d454209d3476c93071c291e16a44c P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:12.394167 13824 raft_consensus.cc:2229] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:12.394274 13823 raft_consensus.cc:2258] T 141d454209d3476c93071c291e16a44c P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:12.394410 13824 raft_consensus.cc:2258] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:12.394670 13822 ts_tablet_manager.cc:1875] T 141d454209d3476c93071c291e16a44c P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:12.394879 13823 ts_tablet_manager.cc:1875] T 141d454209d3476c93071c291e16a44c P 9942306855404cfd891adf227dd65eb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:12.394940 13824 ts_tablet_manager.cc:1875] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:12.397526 13824 ts_tablet_manager.cc:1888] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:12.397658 13824 log.cc:1196] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/141d454209d3476c93071c291e16a44c
I20240503 02:00:12.397799 13822 ts_tablet_manager.cc:1888] T 141d454209d3476c93071c291e16a44c P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:12.397917 13822 log.cc:1196] T 141d454209d3476c93071c291e16a44c P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/141d454209d3476c93071c291e16a44c
I20240503 02:00:12.398124 13823 ts_tablet_manager.cc:1888] T 141d454209d3476c93071c291e16a44c P 9942306855404cfd891adf227dd65eb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:12.398108 13824 ts_tablet_manager.cc:1909] T 141d454209d3476c93071c291e16a44c P 99ead02915954b4fab23b033ad6d546b: Deleting consensus metadata
I20240503 02:00:12.398195 13823 log.cc:1196] T 141d454209d3476c93071c291e16a44c P 9942306855404cfd891adf227dd65eb2: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/141d454209d3476c93071c291e16a44c
I20240503 02:00:12.398458 13823 ts_tablet_manager.cc:1909] T 141d454209d3476c93071c291e16a44c P 9942306855404cfd891adf227dd65eb2: Deleting consensus metadata
I20240503 02:00:12.398512 13822 ts_tablet_manager.cc:1909] T 141d454209d3476c93071c291e16a44c P 80fd18ef4d604c86b909d5c008a46b4f: Deleting consensus metadata
I20240503 02:00:12.399319 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet 141d454209d3476c93071c291e16a44c (table rebalancer_dynamic_table_new_name [id=55bf5f42f2d9482bad4ceb69fb30c3e5]) successfully deleted
I20240503 02:00:12.399319 6303 catalog_manager.cc:4830] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211): tablet 141d454209d3476c93071c291e16a44c (table rebalancer_dynamic_table_new_name [id=55bf5f42f2d9482bad4ceb69fb30c3e5]) successfully deleted
I20240503 02:00:12.399724 6301 catalog_manager.cc:4830] TS 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157): tablet 141d454209d3476c93071c291e16a44c (table rebalancer_dynamic_table_new_name [id=55bf5f42f2d9482bad4ceb69fb30c3e5]) successfully deleted
I20240503 02:00:12.429766 13814 consensus_queue.cc:1035] T e8466d03df4c4ed8bf571bffd936f515 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:12.433298 13814 consensus_queue.cc:1035] T e8466d03df4c4ed8bf571bffd936f515 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:12.494925 6314 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:00:12.500411 8897 tablet_service.cc:1467] Processing CreateTablet for tablet 7b69d44bbbfc408c8d46f54d21f8a427 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=271a62be2be84ada961794cdfcb28433]), partition=
I20240503 02:00:12.500403 6831 tablet_service.cc:1467] Processing CreateTablet for tablet 7b69d44bbbfc408c8d46f54d21f8a427 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=271a62be2be84ada961794cdfcb28433]), partition=
I20240503 02:00:12.500412 6569 tablet_service.cc:1467] Processing CreateTablet for tablet 7b69d44bbbfc408c8d46f54d21f8a427 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=271a62be2be84ada961794cdfcb28433]), partition=
I20240503 02:00:12.500699 8897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b69d44bbbfc408c8d46f54d21f8a427. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:12.500697 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b69d44bbbfc408c8d46f54d21f8a427. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:12.500697 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b69d44bbbfc408c8d46f54d21f8a427. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:12.502714 13830 tablet_bootstrap.cc:492] T 7b69d44bbbfc408c8d46f54d21f8a427 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:12.503940 13830 tablet_bootstrap.cc:654] T 7b69d44bbbfc408c8d46f54d21f8a427 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:12.504058 13831 tablet_bootstrap.cc:492] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:12.504271 13832 tablet_bootstrap.cc:492] T 7b69d44bbbfc408c8d46f54d21f8a427 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:12.505255 13832 tablet_bootstrap.cc:654] T 7b69d44bbbfc408c8d46f54d21f8a427 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:12.505255 13831 tablet_bootstrap.cc:654] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:12.505967 13830 tablet_bootstrap.cc:492] T 7b69d44bbbfc408c8d46f54d21f8a427 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:12.506084 13830 ts_tablet_manager.cc:1367] T 7b69d44bbbfc408c8d46f54d21f8a427 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:12.506438 13831 tablet_bootstrap.cc:492] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:12.506552 13831 ts_tablet_manager.cc:1367] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:12.506592 13832 tablet_bootstrap.cc:492] T 7b69d44bbbfc408c8d46f54d21f8a427 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:12.506554 13830 raft_consensus.cc:348] T 7b69d44bbbfc408c8d46f54d21f8a427 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:12.506680 13830 raft_consensus.cc:374] T 7b69d44bbbfc408c8d46f54d21f8a427 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:12.506698 13832 ts_tablet_manager.cc:1367] T 7b69d44bbbfc408c8d46f54d21f8a427 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:12.506717 13830 raft_consensus.cc:729] T 7b69d44bbbfc408c8d46f54d21f8a427 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:12.506842 13830 consensus_queue.cc:260] T 7b69d44bbbfc408c8d46f54d21f8a427 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:12.507048 13831 raft_consensus.cc:348] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:12.507135 13830 ts_tablet_manager.cc:1398] T 7b69d44bbbfc408c8d46f54d21f8a427 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:12.507222 13831 raft_consensus.cc:374] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:12.507254 13832 raft_consensus.cc:348] T 7b69d44bbbfc408c8d46f54d21f8a427 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:12.507315 13831 raft_consensus.cc:729] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:12.507388 13832 raft_consensus.cc:374] T 7b69d44bbbfc408c8d46f54d21f8a427 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:12.507448 13832 raft_consensus.cc:729] T 7b69d44bbbfc408c8d46f54d21f8a427 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:12.507498 13831 consensus_queue.cc:260] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:12.507580 13832 consensus_queue.cc:260] T 7b69d44bbbfc408c8d46f54d21f8a427 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:12.507787 13831 ts_tablet_manager.cc:1398] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:12.510342 13832 ts_tablet_manager.cc:1398] T 7b69d44bbbfc408c8d46f54d21f8a427 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 02:00:12.562429 13814 consensus_queue.cc:1035] T 5625259b3ac54b4ea639e2823902aee5 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:12.567183 13814 consensus_queue.cc:1035] T 5625259b3ac54b4ea639e2823902aee5 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:12.568960 13791 consensus_queue.cc:1035] T d244f21f8c6a4ab296e9a08f50d9c990 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:12.572906 13720 consensus_queue.cc:1035] T d244f21f8c6a4ab296e9a08f50d9c990 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:12.691711 12331 raft_consensus.cc:482] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:12.691861 12331 raft_consensus.cc:504] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:12.692270 12331 leader_election.cc:288] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:12.692777 6850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b69d44bbbfc408c8d46f54d21f8a427" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 02:00:12.692775 8915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b69d44bbbfc408c8d46f54d21f8a427" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:12.692920 6850 raft_consensus.cc:2454] T 7b69d44bbbfc408c8d46f54d21f8a427 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:12.692929 8915 raft_consensus.cc:2454] T 7b69d44bbbfc408c8d46f54d21f8a427 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:12.693221 6525 leader_election.cc:302] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:12.693400 12331 raft_consensus.cc:2781] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:12.693485 12331 raft_consensus.cc:482] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:12.693536 12331 raft_consensus.cc:3037] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:12.694661 12331 raft_consensus.cc:504] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:12.695031 12331 leader_election.cc:288] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Requested vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:12.695436 8915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b69d44bbbfc408c8d46f54d21f8a427" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:12.695466 6850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b69d44bbbfc408c8d46f54d21f8a427" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 02:00:12.695554 8915 raft_consensus.cc:3037] T 7b69d44bbbfc408c8d46f54d21f8a427 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:12.695593 6850 raft_consensus.cc:3037] T 7b69d44bbbfc408c8d46f54d21f8a427 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:12.696717 8915 raft_consensus.cc:2454] T 7b69d44bbbfc408c8d46f54d21f8a427 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:12.696743 6850 raft_consensus.cc:2454] T 7b69d44bbbfc408c8d46f54d21f8a427 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:12.697089 6525 leader_election.cc:302] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:12.697294 12331 raft_consensus.cc:2781] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:12.697386 12331 raft_consensus.cc:686] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Becoming Leader. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Running, Role: LEADER
I20240503 02:00:12.697691 12331 consensus_queue.cc:237] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:12.699725 6312 catalog_manager.cc:5484] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:12.767403 6312 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_extra_table_32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:12.774739 6436 tablet_service.cc:1467] Processing CreateTablet for tablet 71c44cb6883a495797823588f7c2b4a3 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=89c19ab2e5c145068b45e07be7e5ca99]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:12.775092 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71c44cb6883a495797823588f7c2b4a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:12.775086 9025 tablet_service.cc:1467] Processing CreateTablet for tablet ccf7a4ad8b1d44efa6548772c271d497 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=89c19ab2e5c145068b45e07be7e5ca99]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:12.775326 9025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccf7a4ad8b1d44efa6548772c271d497. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:12.776718 9028 tablet_service.cc:1467] Processing CreateTablet for tablet 9ba716e7e1a04275be92428ccbd651b8 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=89c19ab2e5c145068b45e07be7e5ca99]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:12.776993 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ba716e7e1a04275be92428ccbd651b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:12.779417 9159 tablet_service.cc:1467] Processing CreateTablet for tablet 71c44cb6883a495797823588f7c2b4a3 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=89c19ab2e5c145068b45e07be7e5ca99]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:12.779706 9159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71c44cb6883a495797823588f7c2b4a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:12.781716 13846 tablet_bootstrap.cc:492] T 71c44cb6883a495797823588f7c2b4a3 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:12.781967 9157 tablet_service.cc:1467] Processing CreateTablet for tablet 9ba716e7e1a04275be92428ccbd651b8 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=89c19ab2e5c145068b45e07be7e5ca99]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:12.782224 9157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ba716e7e1a04275be92428ccbd651b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:12.782974 13846 tablet_bootstrap.cc:654] T 71c44cb6883a495797823588f7c2b4a3 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:12.783753 13844 tablet_bootstrap.cc:492] T ccf7a4ad8b1d44efa6548772c271d497 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
W20240503 02:00:12.783754 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:12.784508 13846 tablet_bootstrap.cc:492] T 71c44cb6883a495797823588f7c2b4a3 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:12.784740 9158 tablet_service.cc:1467] Processing CreateTablet for tablet ccf7a4ad8b1d44efa6548772c271d497 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=89c19ab2e5c145068b45e07be7e5ca99]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:12.784799 13846 ts_tablet_manager.cc:1367] T 71c44cb6883a495797823588f7c2b4a3 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:12.785027 9158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccf7a4ad8b1d44efa6548772c271d497. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:12.785339 13846 raft_consensus.cc:348] T 71c44cb6883a495797823588f7c2b4a3 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:12.785436 13846 raft_consensus.cc:374] T 71c44cb6883a495797823588f7c2b4a3 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:12.785461 13846 raft_consensus.cc:729] T 71c44cb6883a495797823588f7c2b4a3 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:12.785578 13846 consensus_queue.cc:260] T 71c44cb6883a495797823588f7c2b4a3 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:12.785874 13846 ts_tablet_manager.cc:1398] T 71c44cb6883a495797823588f7c2b4a3 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:12.786052 13846 tablet_bootstrap.cc:492] T 9ba716e7e1a04275be92428ccbd651b8 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:12.786608 13844 tablet_bootstrap.cc:654] T ccf7a4ad8b1d44efa6548772c271d497 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:12.786952 13846 tablet_bootstrap.cc:654] T 9ba716e7e1a04275be92428ccbd651b8 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:12.787817 13844 tablet_bootstrap.cc:492] T ccf7a4ad8b1d44efa6548772c271d497 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:12.787922 13844 ts_tablet_manager.cc:1367] T ccf7a4ad8b1d44efa6548772c271d497 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:12.788347 13844 raft_consensus.cc:348] T ccf7a4ad8b1d44efa6548772c271d497 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:12.788509 13844 raft_consensus.cc:374] T ccf7a4ad8b1d44efa6548772c271d497 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:12.788549 13844 raft_consensus.cc:729] T ccf7a4ad8b1d44efa6548772c271d497 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:12.788686 13844 consensus_queue.cc:260] T ccf7a4ad8b1d44efa6548772c271d497 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:12.788990 13844 ts_tablet_manager.cc:1398] T ccf7a4ad8b1d44efa6548772c271d497 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:12.789163 13844 tablet_bootstrap.cc:492] T 9ba716e7e1a04275be92428ccbd651b8 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:12.789165 6831 tablet_service.cc:1467] Processing CreateTablet for tablet 71c44cb6883a495797823588f7c2b4a3 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=89c19ab2e5c145068b45e07be7e5ca99]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:12.789180 8897 tablet_service.cc:1467] Processing CreateTablet for tablet 9ba716e7e1a04275be92428ccbd651b8 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=89c19ab2e5c145068b45e07be7e5ca99]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:12.789448 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71c44cb6883a495797823588f7c2b4a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:12.789458 8897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ba716e7e1a04275be92428ccbd651b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:12.790076 13844 tablet_bootstrap.cc:654] T 9ba716e7e1a04275be92428ccbd651b8 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:12.791591 13832 tablet_bootstrap.cc:492] T 71c44cb6883a495797823588f7c2b4a3 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:12.792481 13830 tablet_bootstrap.cc:492] T 9ba716e7e1a04275be92428ccbd651b8 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:12.792615 13832 tablet_bootstrap.cc:654] T 71c44cb6883a495797823588f7c2b4a3 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:12.793370 13830 tablet_bootstrap.cc:654] T 9ba716e7e1a04275be92428ccbd651b8 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:12.793613 13844 tablet_bootstrap.cc:492] T 9ba716e7e1a04275be92428ccbd651b8 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:12.793613 13832 tablet_bootstrap.cc:492] T 71c44cb6883a495797823588f7c2b4a3 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:12.793692 13844 ts_tablet_manager.cc:1367] T 9ba716e7e1a04275be92428ccbd651b8 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 02:00:12.793692 13832 ts_tablet_manager.cc:1367] T 71c44cb6883a495797823588f7c2b4a3 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:12.794147 13832 raft_consensus.cc:348] T 71c44cb6883a495797823588f7c2b4a3 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:12.794147 13844 raft_consensus.cc:348] T 9ba716e7e1a04275be92428ccbd651b8 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:12.794253 13832 raft_consensus.cc:374] T 71c44cb6883a495797823588f7c2b4a3 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:12.794261 13844 raft_consensus.cc:374] T 9ba716e7e1a04275be92428ccbd651b8 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:12.794284 13832 raft_consensus.cc:729] T 71c44cb6883a495797823588f7c2b4a3 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:12.794291 13844 raft_consensus.cc:729] T 9ba716e7e1a04275be92428ccbd651b8 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:12.794411 13832 consensus_queue.cc:260] T 71c44cb6883a495797823588f7c2b4a3 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:12.794421 13844 consensus_queue.cc:260] T 9ba716e7e1a04275be92428ccbd651b8 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:12.794695 13844 ts_tablet_manager.cc:1398] T 9ba716e7e1a04275be92428ccbd651b8 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:12.794703 13832 ts_tablet_manager.cc:1398] T 71c44cb6883a495797823588f7c2b4a3 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:12.797761 13845 tablet_bootstrap.cc:492] T 71c44cb6883a495797823588f7c2b4a3 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:12.800113 8897 tablet_service.cc:1467] Processing CreateTablet for tablet ccf7a4ad8b1d44efa6548772c271d497 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=89c19ab2e5c145068b45e07be7e5ca99]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:12.800455 8897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccf7a4ad8b1d44efa6548772c271d497. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:12.801244 13845 tablet_bootstrap.cc:654] T 71c44cb6883a495797823588f7c2b4a3 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:12.802744 13846 tablet_bootstrap.cc:492] T 9ba716e7e1a04275be92428ccbd651b8 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:12.802953 13846 ts_tablet_manager.cc:1367] T 9ba716e7e1a04275be92428ccbd651b8 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240503 02:00:12.803028 13845 tablet_bootstrap.cc:492] T 71c44cb6883a495797823588f7c2b4a3 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:12.803117 13845 ts_tablet_manager.cc:1367] T 71c44cb6883a495797823588f7c2b4a3 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 02:00:12.803587 13845 raft_consensus.cc:348] T 71c44cb6883a495797823588f7c2b4a3 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:12.803586 13846 raft_consensus.cc:348] T 9ba716e7e1a04275be92428ccbd651b8 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:12.803710 13845 raft_consensus.cc:374] T 71c44cb6883a495797823588f7c2b4a3 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:12.803747 13845 raft_consensus.cc:729] T 71c44cb6883a495797823588f7c2b4a3 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:12.803843 13846 raft_consensus.cc:374] T 9ba716e7e1a04275be92428ccbd651b8 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:12.803917 13846 raft_consensus.cc:729] T 9ba716e7e1a04275be92428ccbd651b8 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:12.803882 13845 consensus_queue.cc:260] T 71c44cb6883a495797823588f7c2b4a3 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:12.804433 13846 consensus_queue.cc:260] T 9ba716e7e1a04275be92428ccbd651b8 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:12.804781 13830 tablet_bootstrap.cc:492] T 9ba716e7e1a04275be92428ccbd651b8 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:12.804891 13830 ts_tablet_manager.cc:1367] T 9ba716e7e1a04275be92428ccbd651b8 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20240503 02:00:12.805114 13845 ts_tablet_manager.cc:1398] T 71c44cb6883a495797823588f7c2b4a3 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:12.805397 13830 raft_consensus.cc:348] T 9ba716e7e1a04275be92428ccbd651b8 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:12.805514 13830 raft_consensus.cc:374] T 9ba716e7e1a04275be92428ccbd651b8 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:12.805523 13846 ts_tablet_manager.cc:1398] T 9ba716e7e1a04275be92428ccbd651b8 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:12.805542 13830 raft_consensus.cc:729] T 9ba716e7e1a04275be92428ccbd651b8 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:12.805703 13846 tablet_bootstrap.cc:492] T ccf7a4ad8b1d44efa6548772c271d497 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:12.805747 13830 consensus_queue.cc:260] T 9ba716e7e1a04275be92428ccbd651b8 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:12.806300 13830 ts_tablet_manager.cc:1398] T 9ba716e7e1a04275be92428ccbd651b8 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:12.806607 13830 tablet_bootstrap.cc:492] T ccf7a4ad8b1d44efa6548772c271d497 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:12.806682 13846 tablet_bootstrap.cc:654] T ccf7a4ad8b1d44efa6548772c271d497 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:12.807884 13830 tablet_bootstrap.cc:654] T ccf7a4ad8b1d44efa6548772c271d497 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:12.812252 13846 tablet_bootstrap.cc:492] T ccf7a4ad8b1d44efa6548772c271d497 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:12.812325 13830 tablet_bootstrap.cc:492] T ccf7a4ad8b1d44efa6548772c271d497 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:12.812482 13846 ts_tablet_manager.cc:1367] T ccf7a4ad8b1d44efa6548772c271d497 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:00:12.812494 13830 ts_tablet_manager.cc:1367] T ccf7a4ad8b1d44efa6548772c271d497 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20240503 02:00:12.812966 13846 raft_consensus.cc:348] T ccf7a4ad8b1d44efa6548772c271d497 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:12.812989 13830 raft_consensus.cc:348] T ccf7a4ad8b1d44efa6548772c271d497 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:12.813076 13846 raft_consensus.cc:374] T ccf7a4ad8b1d44efa6548772c271d497 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:12.813105 13846 raft_consensus.cc:729] T ccf7a4ad8b1d44efa6548772c271d497 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:12.813103 13830 raft_consensus.cc:374] T ccf7a4ad8b1d44efa6548772c271d497 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:12.813138 13830 raft_consensus.cc:729] T ccf7a4ad8b1d44efa6548772c271d497 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:12.813222 13846 consensus_queue.cc:260] T ccf7a4ad8b1d44efa6548772c271d497 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:12.813257 13830 consensus_queue.cc:260] T ccf7a4ad8b1d44efa6548772c271d497 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:12.813508 13846 ts_tablet_manager.cc:1398] T ccf7a4ad8b1d44efa6548772c271d497 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:12.813608 13830 ts_tablet_manager.cc:1398] T ccf7a4ad8b1d44efa6548772c271d497 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:12.834424 13102 raft_consensus.cc:482] T ccf7a4ad8b1d44efa6548772c271d497 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:12.834561 13102 raft_consensus.cc:504] T ccf7a4ad8b1d44efa6548772c271d497 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:12.834980 13102 leader_election.cc:288] T ccf7a4ad8b1d44efa6548772c271d497 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:12.835331 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccf7a4ad8b1d44efa6548772c271d497" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:12.835331 9039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccf7a4ad8b1d44efa6548772c271d497" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:12.835465 9039 raft_consensus.cc:2454] T ccf7a4ad8b1d44efa6548772c271d497 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 0.
I20240503 02:00:12.835464 9179 raft_consensus.cc:2454] T ccf7a4ad8b1d44efa6548772c271d497 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 0.
I20240503 02:00:12.835896 8850 leader_election.cc:302] T ccf7a4ad8b1d44efa6548772c271d497 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:12.836093 13102 raft_consensus.cc:2781] T ccf7a4ad8b1d44efa6548772c271d497 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:12.836150 13102 raft_consensus.cc:482] T ccf7a4ad8b1d44efa6548772c271d497 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:12.836189 13102 raft_consensus.cc:3037] T ccf7a4ad8b1d44efa6548772c271d497 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:12.837275 13102 raft_consensus.cc:504] T ccf7a4ad8b1d44efa6548772c271d497 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:12.837646 13102 leader_election.cc:288] T ccf7a4ad8b1d44efa6548772c271d497 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 election: Requested vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:12.837951 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccf7a4ad8b1d44efa6548772c271d497" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:12.838060 9179 raft_consensus.cc:3037] T ccf7a4ad8b1d44efa6548772c271d497 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:12.838238 9039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccf7a4ad8b1d44efa6548772c271d497" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:12.838430 9039 raft_consensus.cc:3037] T ccf7a4ad8b1d44efa6548772c271d497 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:12.839102 9179 raft_consensus.cc:2454] T ccf7a4ad8b1d44efa6548772c271d497 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 1.
I20240503 02:00:12.839286 9039 raft_consensus.cc:2454] T ccf7a4ad8b1d44efa6548772c271d497 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 1.
I20240503 02:00:12.839527 8850 leader_election.cc:302] T ccf7a4ad8b1d44efa6548772c271d497 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:12.839697 13102 raft_consensus.cc:2781] T ccf7a4ad8b1d44efa6548772c271d497 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:12.839849 13102 raft_consensus.cc:686] T ccf7a4ad8b1d44efa6548772c271d497 P 9942306855404cfd891adf227dd65eb2 [term 1 LEADER]: Becoming Leader. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Running, Role: LEADER
I20240503 02:00:12.840013 13102 consensus_queue.cc:237] T ccf7a4ad8b1d44efa6548772c271d497 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:12.842272 6309 catalog_manager.cc:5484] T ccf7a4ad8b1d44efa6548772c271d497 P 9942306855404cfd891adf227dd65eb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9942306855404cfd891adf227dd65eb2 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "9942306855404cfd891adf227dd65eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:12.844110 6313 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:12.849185 8915 raft_consensus.cc:1261] T 7b69d44bbbfc408c8d46f54d21f8a427 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:12.849179 6850 raft_consensus.cc:1261] T 7b69d44bbbfc408c8d46f54d21f8a427 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:12.849578 13836 consensus_queue.cc:1035] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:12.849745 12331 consensus_queue.cc:1035] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:12.852850 6313 catalog_manager.cc:5576] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719) does not have the latest schema for tablet 7b69d44bbbfc408c8d46f54d21f8a427 (table rebalancer_dynamic_table [id=271a62be2be84ada961794cdfcb28433]). Expected version 1 got 0
I20240503 02:00:12.853242 6309 catalog_manager.cc:5576] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211) does not have the latest schema for tablet 7b69d44bbbfc408c8d46f54d21f8a427 (table rebalancer_dynamic_table [id=271a62be2be84ada961794cdfcb28433]). Expected version 1 got 0
I20240503 02:00:12.854534 13863 tablet.cc:1722] T 7b69d44bbbfc408c8d46f54d21f8a427 P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:12.855180 13863 tablet.cc:1620] T 7b69d44bbbfc408c8d46f54d21f8a427 P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
I20240503 02:00:12.855885 13862 tablet.cc:1722] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:12.856161 6312 catalog_manager.cc:5576] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211) does not have the latest schema for tablet 7b69d44bbbfc408c8d46f54d21f8a427 (table rebalancer_dynamic_table [id=271a62be2be84ada961794cdfcb28433]). Expected version 1 got 0
I20240503 02:00:12.856441 13862 tablet.cc:1620] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f: MemRowSet was empty: no flush needed.
I20240503 02:00:12.856650 6308 catalog_manager.cc:5576] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719) does not have the latest schema for tablet 7b69d44bbbfc408c8d46f54d21f8a427 (table rebalancer_dynamic_table [id=271a62be2be84ada961794cdfcb28433]). Expected version 1 got 0
I20240503 02:00:12.857821 6313 catalog_manager.cc:5576] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863) does not have the latest schema for tablet 7b69d44bbbfc408c8d46f54d21f8a427 (table rebalancer_dynamic_table [id=271a62be2be84ada961794cdfcb28433]). Expected version 1 got 0
I20240503 02:00:12.858912 13864 tablet.cc:1722] T 7b69d44bbbfc408c8d46f54d21f8a427 P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:12.859302 13864 tablet.cc:1620] T 7b69d44bbbfc408c8d46f54d21f8a427 P 492448570cc94ed1879feb2ec3b9051d: MemRowSet was empty: no flush needed.
I20240503 02:00:12.862383 6313 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=271a62be2be84ada961794cdfcb28433] alter complete (version 1)
I20240503 02:00:12.865540 13814 raft_consensus.cc:482] T 71c44cb6883a495797823588f7c2b4a3 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:12.865689 13814 raft_consensus.cc:504] T 71c44cb6883a495797823588f7c2b4a3 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:12.866145 13814 leader_election.cc:288] T 71c44cb6883a495797823588f7c2b4a3 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:12.866674 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71c44cb6883a495797823588f7c2b4a3" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 02:00:12.866724 6850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71c44cb6883a495797823588f7c2b4a3" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 02:00:12.866820 6456 raft_consensus.cc:2454] T 71c44cb6883a495797823588f7c2b4a3 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:12.866888 6850 raft_consensus.cc:2454] T 71c44cb6883a495797823588f7c2b4a3 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:12.867170 9115 leader_election.cc:302] T 71c44cb6883a495797823588f7c2b4a3 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 492448570cc94ed1879feb2ec3b9051d; no voters:
I20240503 02:00:12.867393 13814 raft_consensus.cc:2781] T 71c44cb6883a495797823588f7c2b4a3 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:12.867447 13814 raft_consensus.cc:482] T 71c44cb6883a495797823588f7c2b4a3 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:12.867493 13814 raft_consensus.cc:3037] T 71c44cb6883a495797823588f7c2b4a3 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:12.868476 13814 raft_consensus.cc:504] T 71c44cb6883a495797823588f7c2b4a3 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:12.868842 13814 leader_election.cc:288] T 71c44cb6883a495797823588f7c2b4a3 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Requested vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:12.869197 6850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71c44cb6883a495797823588f7c2b4a3" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 02:00:12.869217 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71c44cb6883a495797823588f7c2b4a3" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 02:00:12.869307 6850 raft_consensus.cc:3037] T 71c44cb6883a495797823588f7c2b4a3 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:12.869319 6456 raft_consensus.cc:3037] T 71c44cb6883a495797823588f7c2b4a3 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:12.870095 6850 raft_consensus.cc:2454] T 71c44cb6883a495797823588f7c2b4a3 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:12.870115 6456 raft_consensus.cc:2454] T 71c44cb6883a495797823588f7c2b4a3 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:12.870416 9112 leader_election.cc:302] T 71c44cb6883a495797823588f7c2b4a3 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:12.870602 13814 raft_consensus.cc:2781] T 71c44cb6883a495797823588f7c2b4a3 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:12.870702 13814 raft_consensus.cc:686] T 71c44cb6883a495797823588f7c2b4a3 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Becoming Leader. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Running, Role: LEADER
I20240503 02:00:12.870839 13814 consensus_queue.cc:237] T 71c44cb6883a495797823588f7c2b4a3 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:12.873275 6308 catalog_manager.cc:5484] T 71c44cb6883a495797823588f7c2b4a3 P 2a01ace80baf4057b6bf1995cf88468b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135). New cstate: current_term: 1 leader_uuid: "2a01ace80baf4057b6bf1995cf88468b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:12.964897 6308 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:12.971239 13862 tablet.cc:1722] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:12.971530 13864 tablet.cc:1722] T 7b69d44bbbfc408c8d46f54d21f8a427 P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:12.971648 13862 tablet.cc:1620] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f: MemRowSet was empty: no flush needed.
I20240503 02:00:12.971812 13864 tablet.cc:1620] T 7b69d44bbbfc408c8d46f54d21f8a427 P 492448570cc94ed1879feb2ec3b9051d: MemRowSet was empty: no flush needed.
I20240503 02:00:12.973825 13863 tablet.cc:1722] T 7b69d44bbbfc408c8d46f54d21f8a427 P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:12.975112 13863 tablet.cc:1620] T 7b69d44bbbfc408c8d46f54d21f8a427 P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
I20240503 02:00:12.979889 6308 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=271a62be2be84ada961794cdfcb28433] alter complete (version 2)
I20240503 02:00:13.020646 13786 raft_consensus.cc:482] T 9ba716e7e1a04275be92428ccbd651b8 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:13.020782 13786 raft_consensus.cc:504] T 9ba716e7e1a04275be92428ccbd651b8 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:13.021239 13786 leader_election.cc:288] T 9ba716e7e1a04275be92428ccbd651b8 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:13.021615 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ba716e7e1a04275be92428ccbd651b8" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:13.021667 9039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ba716e7e1a04275be92428ccbd651b8" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:13.021776 9039 raft_consensus.cc:2454] T 9ba716e7e1a04275be92428ccbd651b8 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 0.
I20240503 02:00:13.021773 9178 raft_consensus.cc:2454] T 9ba716e7e1a04275be92428ccbd651b8 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 0.
I20240503 02:00:13.022079 8850 leader_election.cc:302] T 9ba716e7e1a04275be92428ccbd651b8 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:13.022466 13786 raft_consensus.cc:2781] T 9ba716e7e1a04275be92428ccbd651b8 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:13.022529 13786 raft_consensus.cc:482] T 9ba716e7e1a04275be92428ccbd651b8 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:13.022573 13786 raft_consensus.cc:3037] T 9ba716e7e1a04275be92428ccbd651b8 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:13.023808 13786 raft_consensus.cc:504] T 9ba716e7e1a04275be92428ccbd651b8 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:13.024166 13786 leader_election.cc:288] T 9ba716e7e1a04275be92428ccbd651b8 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 election: Requested vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:13.024549 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ba716e7e1a04275be92428ccbd651b8" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:13.024580 9039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ba716e7e1a04275be92428ccbd651b8" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:13.024679 9178 raft_consensus.cc:3037] T 9ba716e7e1a04275be92428ccbd651b8 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:13.024699 9039 raft_consensus.cc:3037] T 9ba716e7e1a04275be92428ccbd651b8 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:13.025497 9178 raft_consensus.cc:2454] T 9ba716e7e1a04275be92428ccbd651b8 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 1.
I20240503 02:00:13.025833 9039 raft_consensus.cc:2454] T 9ba716e7e1a04275be92428ccbd651b8 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 1.
I20240503 02:00:13.026515 8851 leader_election.cc:302] T 9ba716e7e1a04275be92428ccbd651b8 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9942306855404cfd891adf227dd65eb2, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:13.026701 13786 raft_consensus.cc:2781] T 9ba716e7e1a04275be92428ccbd651b8 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:13.026772 13786 raft_consensus.cc:686] T 9ba716e7e1a04275be92428ccbd651b8 P 9942306855404cfd891adf227dd65eb2 [term 1 LEADER]: Becoming Leader. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Running, Role: LEADER
I20240503 02:00:13.026901 13786 consensus_queue.cc:237] T 9ba716e7e1a04275be92428ccbd651b8 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:13.029790 6308 catalog_manager.cc:5484] T 9ba716e7e1a04275be92428ccbd651b8 P 9942306855404cfd891adf227dd65eb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9942306855404cfd891adf227dd65eb2 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "9942306855404cfd891adf227dd65eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:13.081164 6308 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:13.086980 13862 tablet.cc:1722] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:13.087666 13864 tablet.cc:1722] T 7b69d44bbbfc408c8d46f54d21f8a427 P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:13.088770 13863 tablet.cc:1722] T 7b69d44bbbfc408c8d46f54d21f8a427 P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:13.093823 6308 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=271a62be2be84ada961794cdfcb28433] alter complete (version 3)
I20240503 02:00:13.195286 6309 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:13.195598 6309 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:13.197645 6309 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet 7b69d44bbbfc408c8d46f54d21f8a427
I20240503 02:00:13.198364 6831 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b69d44bbbfc408c8d46f54d21f8a427 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:13 UTC) from {username='slave'} at 127.0.0.1:52812
I20240503 02:00:13.198518 6569 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b69d44bbbfc408c8d46f54d21f8a427 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:13 UTC) from {username='slave'} at 127.0.0.1:59846
I20240503 02:00:13.198676 8897 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b69d44bbbfc408c8d46f54d21f8a427 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:13 UTC) from {username='slave'} at 127.0.0.1:40296
I20240503 02:00:13.198825 13871 tablet_replica.cc:330] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 02:00:13.198941 13870 tablet_replica.cc:330] T 7b69d44bbbfc408c8d46f54d21f8a427 P 492448570cc94ed1879feb2ec3b9051d: stopping tablet replica
I20240503 02:00:13.198974 13871 raft_consensus.cc:2229] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:13.199097 13870 raft_consensus.cc:2229] T 7b69d44bbbfc408c8d46f54d21f8a427 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:13.199190 13871 raft_consensus.cc:2258] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:13.199226 13870 raft_consensus.cc:2258] T 7b69d44bbbfc408c8d46f54d21f8a427 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:13.199735 13871 ts_tablet_manager.cc:1875] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:13.199829 13870 ts_tablet_manager.cc:1875] T 7b69d44bbbfc408c8d46f54d21f8a427 P 492448570cc94ed1879feb2ec3b9051d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:13.200588 13872 tablet_replica.cc:330] T 7b69d44bbbfc408c8d46f54d21f8a427 P 9942306855404cfd891adf227dd65eb2: stopping tablet replica
I20240503 02:00:13.200803 13872 raft_consensus.cc:2229] T 7b69d44bbbfc408c8d46f54d21f8a427 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:13.201001 13872 raft_consensus.cc:2258] T 7b69d44bbbfc408c8d46f54d21f8a427 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:13.201647 13872 ts_tablet_manager.cc:1875] T 7b69d44bbbfc408c8d46f54d21f8a427 P 9942306855404cfd891adf227dd65eb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:13.202155 13871 ts_tablet_manager.cc:1888] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:13.202164 13870 ts_tablet_manager.cc:1888] T 7b69d44bbbfc408c8d46f54d21f8a427 P 492448570cc94ed1879feb2ec3b9051d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:13.202231 13870 log.cc:1196] T 7b69d44bbbfc408c8d46f54d21f8a427 P 492448570cc94ed1879feb2ec3b9051d: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/7b69d44bbbfc408c8d46f54d21f8a427
I20240503 02:00:13.202301 13871 log.cc:1196] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/7b69d44bbbfc408c8d46f54d21f8a427
I20240503 02:00:13.202648 13870 ts_tablet_manager.cc:1909] T 7b69d44bbbfc408c8d46f54d21f8a427 P 492448570cc94ed1879feb2ec3b9051d: Deleting consensus metadata
I20240503 02:00:13.202747 13871 ts_tablet_manager.cc:1909] T 7b69d44bbbfc408c8d46f54d21f8a427 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting consensus metadata
I20240503 02:00:13.203570 6303 catalog_manager.cc:4830] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719): tablet 7b69d44bbbfc408c8d46f54d21f8a427 (table rebalancer_dynamic_table_new_name [id=271a62be2be84ada961794cdfcb28433]) successfully deleted
I20240503 02:00:13.203570 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet 7b69d44bbbfc408c8d46f54d21f8a427 (table rebalancer_dynamic_table_new_name [id=271a62be2be84ada961794cdfcb28433]) successfully deleted
I20240503 02:00:13.204069 13872 ts_tablet_manager.cc:1888] T 7b69d44bbbfc408c8d46f54d21f8a427 P 9942306855404cfd891adf227dd65eb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:13.204135 13872 log.cc:1196] T 7b69d44bbbfc408c8d46f54d21f8a427 P 9942306855404cfd891adf227dd65eb2: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/7b69d44bbbfc408c8d46f54d21f8a427
I20240503 02:00:13.204339 13872 ts_tablet_manager.cc:1909] T 7b69d44bbbfc408c8d46f54d21f8a427 P 9942306855404cfd891adf227dd65eb2: Deleting consensus metadata
I20240503 02:00:13.205103 6303 catalog_manager.cc:4830] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211): tablet 7b69d44bbbfc408c8d46f54d21f8a427 (table rebalancer_dynamic_table_new_name [id=271a62be2be84ada961794cdfcb28433]) successfully deleted
W20240503 02:00:13.243845 6301 catalog_manager.cc:4573] TS e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cc19d71f40fb44f1801f1b49dcc76fe0: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:13.273573 13776 consensus_queue.cc:1035] T 71c44cb6883a495797823588f7c2b4a3 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:13.278100 13814 consensus_queue.cc:1035] T 71c44cb6883a495797823588f7c2b4a3 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:13.279635 13786 consensus_queue.cc:1035] T ccf7a4ad8b1d44efa6548772c271d497 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:13.283886 13878 consensus_queue.cc:1035] T ccf7a4ad8b1d44efa6548772c271d497 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20240503 02:00:13.289288 8982 consensus_peers.cc:485] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:00:13.299371 6309 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:00:13.304796 8897 tablet_service.cc:1467] Processing CreateTablet for tablet 7b37a8876d0349f4a6885868f63ee6ce (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2e477496b2bb4a5d8f1af5475a756447]), partition=
I20240503 02:00:13.304885 6569 tablet_service.cc:1467] Processing CreateTablet for tablet 7b37a8876d0349f4a6885868f63ee6ce (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2e477496b2bb4a5d8f1af5475a756447]), partition=
I20240503 02:00:13.305109 8897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b37a8876d0349f4a6885868f63ee6ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:13.305145 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b37a8876d0349f4a6885868f63ee6ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:13.307025 13830 tablet_bootstrap.cc:492] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:13.307142 13881 tablet_bootstrap.cc:492] T 7b37a8876d0349f4a6885868f63ee6ce P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:13.307292 9028 tablet_service.cc:1467] Processing CreateTablet for tablet 7b37a8876d0349f4a6885868f63ee6ce (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2e477496b2bb4a5d8f1af5475a756447]), partition=
I20240503 02:00:13.307557 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b37a8876d0349f4a6885868f63ee6ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:13.308048 13881 tablet_bootstrap.cc:654] T 7b37a8876d0349f4a6885868f63ee6ce P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:13.308162 13830 tablet_bootstrap.cc:654] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:13.309885 13830 tablet_bootstrap.cc:492] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:13.310011 13830 ts_tablet_manager.cc:1367] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:13.310052 13884 tablet_bootstrap.cc:492] T 7b37a8876d0349f4a6885868f63ee6ce P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:13.310499 13830 raft_consensus.cc:348] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:13.310606 13830 raft_consensus.cc:374] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:13.310612 13881 tablet_bootstrap.cc:492] T 7b37a8876d0349f4a6885868f63ee6ce P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:13.310626 13830 raft_consensus.cc:729] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:13.310703 13881 ts_tablet_manager.cc:1367] T 7b37a8876d0349f4a6885868f63ee6ce P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 02:00:13.310741 13830 consensus_queue.cc:260] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:13.311003 13830 ts_tablet_manager.cc:1398] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:13.311141 13881 raft_consensus.cc:348] T 7b37a8876d0349f4a6885868f63ee6ce P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:13.311240 13881 raft_consensus.cc:374] T 7b37a8876d0349f4a6885868f63ee6ce P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:13.311267 13881 raft_consensus.cc:729] T 7b37a8876d0349f4a6885868f63ee6ce P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:13.311398 13881 consensus_queue.cc:260] T 7b37a8876d0349f4a6885868f63ee6ce P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:13.311691 13881 ts_tablet_manager.cc:1398] T 7b37a8876d0349f4a6885868f63ee6ce P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:13.313949 13884 tablet_bootstrap.cc:654] T 7b37a8876d0349f4a6885868f63ee6ce P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:13.315470 13884 tablet_bootstrap.cc:492] T 7b37a8876d0349f4a6885868f63ee6ce P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:13.315608 13884 ts_tablet_manager.cc:1367] T 7b37a8876d0349f4a6885868f63ee6ce P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:00:13.316128 13884 raft_consensus.cc:348] T 7b37a8876d0349f4a6885868f63ee6ce P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:13.316258 13884 raft_consensus.cc:374] T 7b37a8876d0349f4a6885868f63ee6ce P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:13.316304 13884 raft_consensus.cc:729] T 7b37a8876d0349f4a6885868f63ee6ce P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:13.316457 13884 consensus_queue.cc:260] T 7b37a8876d0349f4a6885868f63ee6ce P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:13.316720 13884 ts_tablet_manager.cc:1398] T 7b37a8876d0349f4a6885868f63ee6ce P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20240503 02:00:13.374754 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:13.375975 13878 raft_consensus.cc:482] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:13.376111 13878 raft_consensus.cc:504] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:13.376549 13878 leader_election.cc:288] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:13.376981 6582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b37a8876d0349f4a6885868f63ee6ce" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 02:00:13.376983 9039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b37a8876d0349f4a6885868f63ee6ce" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:13.377117 6582 raft_consensus.cc:2454] T 7b37a8876d0349f4a6885868f63ee6ce P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 0.
I20240503 02:00:13.377127 9039 raft_consensus.cc:2454] T 7b37a8876d0349f4a6885868f63ee6ce P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 0.
I20240503 02:00:13.377398 8851 leader_election.cc:302] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9942306855404cfd891adf227dd65eb2, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:13.377553 13878 raft_consensus.cc:2781] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:13.377611 13878 raft_consensus.cc:482] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:13.377655 13878 raft_consensus.cc:3037] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:13.378937 13878 raft_consensus.cc:504] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:13.379333 13878 leader_election.cc:288] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 election: Requested vote from peers 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:13.379854 6582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b37a8876d0349f4a6885868f63ee6ce" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 02:00:13.379866 9039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b37a8876d0349f4a6885868f63ee6ce" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:13.380010 6582 raft_consensus.cc:3037] T 7b37a8876d0349f4a6885868f63ee6ce P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:13.380023 9039 raft_consensus.cc:3037] T 7b37a8876d0349f4a6885868f63ee6ce P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:13.381127 6582 raft_consensus.cc:2454] T 7b37a8876d0349f4a6885868f63ee6ce P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 1.
I20240503 02:00:13.381121 9039 raft_consensus.cc:2454] T 7b37a8876d0349f4a6885868f63ee6ce P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 1.
I20240503 02:00:13.381450 8852 leader_election.cc:302] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:13.381655 13878 raft_consensus.cc:2781] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:13.381752 13878 raft_consensus.cc:686] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2 [term 1 LEADER]: Becoming Leader. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Running, Role: LEADER
I20240503 02:00:13.381927 13878 consensus_queue.cc:237] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:13.384775 6308 catalog_manager.cc:5484] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9942306855404cfd891adf227dd65eb2 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "9942306855404cfd891adf227dd65eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } }
W20240503 02:00:13.500560 6785 consensus_peers.cc:485] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:00:13.500980 13878 consensus_queue.cc:1035] T 9ba716e7e1a04275be92428ccbd651b8 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:00:13.503530 8851 consensus_peers.cc:485] T 067f54011a37432d99157884e70fc601 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:00:13.506847 6313 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:13.508611 13878 consensus_queue.cc:1035] T 9ba716e7e1a04275be92428ccbd651b8 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:13.512068 6582 raft_consensus.cc:1261] T 7b37a8876d0349f4a6885868f63ee6ce P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:13.512101 9039 raft_consensus.cc:1261] T 7b37a8876d0349f4a6885868f63ee6ce P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer 9942306855404cfd891adf227dd65eb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:13.512571 13102 consensus_queue.cc:1035] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:13.512976 13102 consensus_queue.cc:1035] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:13.515846 6313 catalog_manager.cc:5576] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863) does not have the latest schema for tablet 7b37a8876d0349f4a6885868f63ee6ce (table rebalancer_dynamic_table [id=2e477496b2bb4a5d8f1af5475a756447]). Expected version 1 got 0
I20240503 02:00:13.516033 13863 tablet.cc:1722] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:13.516414 13863 tablet.cc:1620] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
I20240503 02:00:13.516746 13862 tablet.cc:1722] T 7b37a8876d0349f4a6885868f63ee6ce P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:13.517066 13862 tablet.cc:1620] T 7b37a8876d0349f4a6885868f63ee6ce P 80fd18ef4d604c86b909d5c008a46b4f: MemRowSet was empty: no flush needed.
I20240503 02:00:13.520326 6313 catalog_manager.cc:5576] TS 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157) does not have the latest schema for tablet 7b37a8876d0349f4a6885868f63ee6ce (table rebalancer_dynamic_table [id=2e477496b2bb4a5d8f1af5475a756447]). Expected version 1 got 0
I20240503 02:00:13.521487 13892 tablet.cc:1722] T 7b37a8876d0349f4a6885868f63ee6ce P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:13.522586 13892 tablet.cc:1620] T 7b37a8876d0349f4a6885868f63ee6ce P 99ead02915954b4fab23b033ad6d546b: MemRowSet was empty: no flush needed.
I20240503 02:00:13.523502 6308 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=2e477496b2bb4a5d8f1af5475a756447] alter complete (version 1)
W20240503 02:00:13.526813 6523 consensus_peers.cc:485] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:00:13.535724 6309 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_extra_table_34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:13.542019 6831 tablet_service.cc:1467] Processing CreateTablet for tablet 849308d0b7e744f7946c892dc444f1f0 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=dc0ffce47c184e1f816da7595fee0912]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:13.542029 6569 tablet_service.cc:1467] Processing CreateTablet for tablet 849308d0b7e744f7946c892dc444f1f0 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=dc0ffce47c184e1f816da7595fee0912]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:13.542160 9028 tablet_service.cc:1467] Processing CreateTablet for tablet 849308d0b7e744f7946c892dc444f1f0 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=dc0ffce47c184e1f816da7595fee0912]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:13.542363 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 849308d0b7e744f7946c892dc444f1f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:13.542371 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 849308d0b7e744f7946c892dc444f1f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:13.542397 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 849308d0b7e744f7946c892dc444f1f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:13.544625 13881 tablet_bootstrap.cc:492] T 849308d0b7e744f7946c892dc444f1f0 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:13.544690 13893 tablet_bootstrap.cc:492] T 849308d0b7e744f7946c892dc444f1f0 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:13.545730 13881 tablet_bootstrap.cc:654] T 849308d0b7e744f7946c892dc444f1f0 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:13.545647 9157 tablet_service.cc:1467] Processing CreateTablet for tablet 603eb0789cfb44ea930f447cd2190e6a (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=dc0ffce47c184e1f816da7595fee0912]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:13.545751 13893 tablet_bootstrap.cc:654] T 849308d0b7e744f7946c892dc444f1f0 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:13.545647 9158 tablet_service.cc:1467] Processing CreateTablet for tablet e1770e6ac12d420eadb2be38ba010bba (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=dc0ffce47c184e1f816da7595fee0912]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:13.545956 9157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 603eb0789cfb44ea930f447cd2190e6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:13.546144 9158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1770e6ac12d420eadb2be38ba010bba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:13.546437 13884 tablet_bootstrap.cc:492] T 849308d0b7e744f7946c892dc444f1f0 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:13.547458 13884 tablet_bootstrap.cc:654] T 849308d0b7e744f7946c892dc444f1f0 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:13.548269 13897 tablet_bootstrap.cc:492] T 603eb0789cfb44ea930f447cd2190e6a P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:13.549579 6700 tablet_service.cc:1467] Processing CreateTablet for tablet 603eb0789cfb44ea930f447cd2190e6a (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=dc0ffce47c184e1f816da7595fee0912]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:13.549579 6831 tablet_service.cc:1467] Processing CreateTablet for tablet 603eb0789cfb44ea930f447cd2190e6a (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=dc0ffce47c184e1f816da7595fee0912]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:13.549702 13897 tablet_bootstrap.cc:654] T 603eb0789cfb44ea930f447cd2190e6a P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:13.549865 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 603eb0789cfb44ea930f447cd2190e6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:13.549865 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 603eb0789cfb44ea930f447cd2190e6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:13.549579 6699 tablet_service.cc:1467] Processing CreateTablet for tablet e1770e6ac12d420eadb2be38ba010bba (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=dc0ffce47c184e1f816da7595fee0912]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:13.553565 6699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1770e6ac12d420eadb2be38ba010bba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:13.554760 13881 tablet_bootstrap.cc:492] T 849308d0b7e744f7946c892dc444f1f0 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:13.554855 13881 ts_tablet_manager.cc:1367] T 849308d0b7e744f7946c892dc444f1f0 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.002s
I20240503 02:00:13.555330 13881 raft_consensus.cc:348] T 849308d0b7e744f7946c892dc444f1f0 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:13.555454 13881 raft_consensus.cc:374] T 849308d0b7e744f7946c892dc444f1f0 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:13.555483 13881 raft_consensus.cc:729] T 849308d0b7e744f7946c892dc444f1f0 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:13.555624 13881 consensus_queue.cc:260] T 849308d0b7e744f7946c892dc444f1f0 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:13.555898 13881 ts_tablet_manager.cc:1398] T 849308d0b7e744f7946c892dc444f1f0 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:13.556310 6436 tablet_service.cc:1467] Processing CreateTablet for tablet e1770e6ac12d420eadb2be38ba010bba (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=dc0ffce47c184e1f816da7595fee0912]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:13.556640 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1770e6ac12d420eadb2be38ba010bba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:13.556741 13897 tablet_bootstrap.cc:492] T 603eb0789cfb44ea930f447cd2190e6a P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:13.556842 13897 ts_tablet_manager.cc:1367] T 603eb0789cfb44ea930f447cd2190e6a P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 02:00:13.557314 13893 tablet_bootstrap.cc:492] T 849308d0b7e744f7946c892dc444f1f0 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:13.557307 13897 raft_consensus.cc:348] T 603eb0789cfb44ea930f447cd2190e6a P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:13.557405 13893 ts_tablet_manager.cc:1367] T 849308d0b7e744f7946c892dc444f1f0 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 02:00:13.557417 13897 raft_consensus.cc:374] T 603eb0789cfb44ea930f447cd2190e6a P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:13.557448 13897 raft_consensus.cc:729] T 603eb0789cfb44ea930f447cd2190e6a P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:13.557571 13897 consensus_queue.cc:260] T 603eb0789cfb44ea930f447cd2190e6a P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:13.557863 13897 ts_tablet_manager.cc:1398] T 603eb0789cfb44ea930f447cd2190e6a P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:13.557929 13893 raft_consensus.cc:348] T 849308d0b7e744f7946c892dc444f1f0 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:13.558036 13893 raft_consensus.cc:374] T 849308d0b7e744f7946c892dc444f1f0 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:13.558046 13897 tablet_bootstrap.cc:492] T e1770e6ac12d420eadb2be38ba010bba P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:13.558068 13893 raft_consensus.cc:729] T 849308d0b7e744f7946c892dc444f1f0 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:13.558187 13893 consensus_queue.cc:260] T 849308d0b7e744f7946c892dc444f1f0 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:13.558446 13893 ts_tablet_manager.cc:1398] T 849308d0b7e744f7946c892dc444f1f0 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:13.558624 13893 tablet_bootstrap.cc:492] T 603eb0789cfb44ea930f447cd2190e6a P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:13.560034 13893 tablet_bootstrap.cc:654] T 603eb0789cfb44ea930f447cd2190e6a P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:13.562052 13899 tablet_bootstrap.cc:492] T 603eb0789cfb44ea930f447cd2190e6a P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:13.563341 13901 tablet_bootstrap.cc:492] T e1770e6ac12d420eadb2be38ba010bba P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:13.564112 13884 tablet_bootstrap.cc:492] T 849308d0b7e744f7946c892dc444f1f0 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:13.564234 13884 ts_tablet_manager.cc:1367] T 849308d0b7e744f7946c892dc444f1f0 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.002s
I20240503 02:00:13.564785 13884 raft_consensus.cc:348] T 849308d0b7e744f7946c892dc444f1f0 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:13.564904 13884 raft_consensus.cc:374] T 849308d0b7e744f7946c892dc444f1f0 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:13.564934 13884 raft_consensus.cc:729] T 849308d0b7e744f7946c892dc444f1f0 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:13.565060 13884 consensus_queue.cc:260] T 849308d0b7e744f7946c892dc444f1f0 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:13.565362 13884 ts_tablet_manager.cc:1398] T 849308d0b7e744f7946c892dc444f1f0 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:13.565876 13901 tablet_bootstrap.cc:654] T e1770e6ac12d420eadb2be38ba010bba P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
W20240503 02:00:13.566018 6785 consensus_peers.cc:485] T c3485701bf3b48f5ae2e1568afeed133 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:00:13.566108 13899 tablet_bootstrap.cc:654] T 603eb0789cfb44ea930f447cd2190e6a P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:13.567034 13897 tablet_bootstrap.cc:654] T e1770e6ac12d420eadb2be38ba010bba P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:13.568221 13899 tablet_bootstrap.cc:492] T 603eb0789cfb44ea930f447cd2190e6a P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:13.568323 13899 ts_tablet_manager.cc:1367] T 603eb0789cfb44ea930f447cd2190e6a P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:00:13.568818 13899 raft_consensus.cc:348] T 603eb0789cfb44ea930f447cd2190e6a P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:13.568933 13899 raft_consensus.cc:374] T 603eb0789cfb44ea930f447cd2190e6a P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:13.568959 13899 raft_consensus.cc:729] T 603eb0789cfb44ea930f447cd2190e6a P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:13.569075 13899 consensus_queue.cc:260] T 603eb0789cfb44ea930f447cd2190e6a P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:13.569348 13899 ts_tablet_manager.cc:1398] T 603eb0789cfb44ea930f447cd2190e6a P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:13.569892 13899 tablet_bootstrap.cc:492] T e1770e6ac12d420eadb2be38ba010bba P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:13.570876 13899 tablet_bootstrap.cc:654] T e1770e6ac12d420eadb2be38ba010bba P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:13.572850 13901 tablet_bootstrap.cc:492] T e1770e6ac12d420eadb2be38ba010bba P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:13.572973 13901 ts_tablet_manager.cc:1367] T e1770e6ac12d420eadb2be38ba010bba P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 02:00:13.573482 13901 raft_consensus.cc:348] T e1770e6ac12d420eadb2be38ba010bba P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:13.573606 13901 raft_consensus.cc:374] T e1770e6ac12d420eadb2be38ba010bba P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:13.573638 13901 raft_consensus.cc:729] T e1770e6ac12d420eadb2be38ba010bba P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:13.573644 13899 tablet_bootstrap.cc:492] T e1770e6ac12d420eadb2be38ba010bba P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:13.573757 13901 consensus_queue.cc:260] T e1770e6ac12d420eadb2be38ba010bba P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:13.574038 13901 ts_tablet_manager.cc:1398] T e1770e6ac12d420eadb2be38ba010bba P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:13.572211 13897 tablet_bootstrap.cc:492] T e1770e6ac12d420eadb2be38ba010bba P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:13.577260 13897 ts_tablet_manager.cc:1367] T e1770e6ac12d420eadb2be38ba010bba P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
W20240503 02:00:13.577387 6654 consensus_peers.cc:485] T ba17d4d3e0754b2798eb95484d37229a P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:00:13.573724 13899 ts_tablet_manager.cc:1367] T e1770e6ac12d420eadb2be38ba010bba P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 02:00:13.578169 13899 raft_consensus.cc:348] T e1770e6ac12d420eadb2be38ba010bba P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:13.578787 13897 raft_consensus.cc:348] T e1770e6ac12d420eadb2be38ba010bba P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:13.578902 13897 raft_consensus.cc:374] T e1770e6ac12d420eadb2be38ba010bba P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:13.578982 13897 raft_consensus.cc:729] T e1770e6ac12d420eadb2be38ba010bba P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:13.579144 13897 consensus_queue.cc:260] T e1770e6ac12d420eadb2be38ba010bba P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:13.579201 13893 tablet_bootstrap.cc:492] T 603eb0789cfb44ea930f447cd2190e6a P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:13.579281 13899 raft_consensus.cc:374] T e1770e6ac12d420eadb2be38ba010bba P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:13.579303 13893 ts_tablet_manager.cc:1367] T 603eb0789cfb44ea930f447cd2190e6a P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.000s
I20240503 02:00:13.579373 13899 raft_consensus.cc:729] T e1770e6ac12d420eadb2be38ba010bba P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:13.579535 13899 consensus_queue.cc:260] T e1770e6ac12d420eadb2be38ba010bba P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:13.579756 13893 raft_consensus.cc:348] T 603eb0789cfb44ea930f447cd2190e6a P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:13.579891 13893 raft_consensus.cc:374] T 603eb0789cfb44ea930f447cd2190e6a P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:13.579924 13893 raft_consensus.cc:729] T 603eb0789cfb44ea930f447cd2190e6a P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:13.580046 13893 consensus_queue.cc:260] T 603eb0789cfb44ea930f447cd2190e6a P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:13.580356 13893 ts_tablet_manager.cc:1398] T 603eb0789cfb44ea930f447cd2190e6a P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:13.582379 13897 ts_tablet_manager.cc:1398] T e1770e6ac12d420eadb2be38ba010bba P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 02:00:13.582769 13899 ts_tablet_manager.cc:1398] T e1770e6ac12d420eadb2be38ba010bba P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
W20240503 02:00:13.600319 6785 consensus_peers.cc:485] T 46ad540deb624a25b97d3b967fe1f6f4 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:13.610780 8851 consensus_peers.cc:485] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:00:13.622576 13814 raft_consensus.cc:482] T 603eb0789cfb44ea930f447cd2190e6a P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:13.622720 13814 raft_consensus.cc:504] T 603eb0789cfb44ea930f447cd2190e6a P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:13.623135 13814 leader_election.cc:288] T 603eb0789cfb44ea930f447cd2190e6a P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:13.623431 13887 raft_consensus.cc:482] T 849308d0b7e744f7946c892dc444f1f0 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:13.623544 13887 raft_consensus.cc:504] T 849308d0b7e744f7946c892dc444f1f0 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:13.623644 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "603eb0789cfb44ea930f447cd2190e6a" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 02:00:13.623668 6720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "603eb0789cfb44ea930f447cd2190e6a" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 02:00:13.623788 6720 raft_consensus.cc:2454] T 603eb0789cfb44ea930f447cd2190e6a P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:13.623808 6849 raft_consensus.cc:2454] T 603eb0789cfb44ea930f447cd2190e6a P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:13.623978 13887 leader_election.cc:288] T 849308d0b7e744f7946c892dc444f1f0 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:13.624224 9115 leader_election.cc:302] T 603eb0789cfb44ea930f447cd2190e6a P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 492448570cc94ed1879feb2ec3b9051d; no voters:
I20240503 02:00:13.624402 13814 raft_consensus.cc:2781] T 603eb0789cfb44ea930f447cd2190e6a P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:13.624363 6582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "849308d0b7e744f7946c892dc444f1f0" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 02:00:13.624341 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "849308d0b7e744f7946c892dc444f1f0" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:13.624454 13814 raft_consensus.cc:482] T 603eb0789cfb44ea930f447cd2190e6a P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:13.624487 13814 raft_consensus.cc:3037] T 603eb0789cfb44ea930f447cd2190e6a P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:13.624514 9048 raft_consensus.cc:2454] T 849308d0b7e744f7946c892dc444f1f0 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 02:00:13.624514 6582 raft_consensus.cc:2454] T 849308d0b7e744f7946c892dc444f1f0 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 02:00:13.624765 6785 leader_election.cc:302] T 849308d0b7e744f7946c892dc444f1f0 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:13.624917 13887 raft_consensus.cc:2781] T 849308d0b7e744f7946c892dc444f1f0 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:13.624969 13887 raft_consensus.cc:482] T 849308d0b7e744f7946c892dc444f1f0 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:13.625005 13887 raft_consensus.cc:3037] T 849308d0b7e744f7946c892dc444f1f0 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:13.625568 13814 raft_consensus.cc:504] T 603eb0789cfb44ea930f447cd2190e6a P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:13.625808 13887 raft_consensus.cc:504] T 849308d0b7e744f7946c892dc444f1f0 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:13.625900 13814 leader_election.cc:288] T 603eb0789cfb44ea930f447cd2190e6a P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Requested vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:13.626132 13887 leader_election.cc:288] T 849308d0b7e744f7946c892dc444f1f0 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Requested vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:13.626227 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "603eb0789cfb44ea930f447cd2190e6a" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 02:00:13.626328 6849 raft_consensus.cc:3037] T 603eb0789cfb44ea930f447cd2190e6a P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:13.626302 6720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "603eb0789cfb44ea930f447cd2190e6a" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 02:00:13.626417 6720 raft_consensus.cc:3037] T 603eb0789cfb44ea930f447cd2190e6a P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:13.626526 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "849308d0b7e744f7946c892dc444f1f0" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:13.626624 9048 raft_consensus.cc:3037] T 849308d0b7e744f7946c892dc444f1f0 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:13.627351 6849 raft_consensus.cc:2454] T 603eb0789cfb44ea930f447cd2190e6a P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:13.627351 6720 raft_consensus.cc:2454] T 603eb0789cfb44ea930f447cd2190e6a P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:13.627398 9048 raft_consensus.cc:2454] T 849308d0b7e744f7946c892dc444f1f0 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 02:00:13.627710 6785 leader_election.cc:302] T 849308d0b7e744f7946c892dc444f1f0 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:13.627817 6582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "849308d0b7e744f7946c892dc444f1f0" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 02:00:13.627875 13887 raft_consensus.cc:2781] T 849308d0b7e744f7946c892dc444f1f0 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:13.627939 13887 raft_consensus.cc:686] T 849308d0b7e744f7946c892dc444f1f0 P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Becoming Leader. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Running, Role: LEADER
I20240503 02:00:13.627919 9115 leader_election.cc:302] T 603eb0789cfb44ea930f447cd2190e6a P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 492448570cc94ed1879feb2ec3b9051d; no voters:
I20240503 02:00:13.627951 6582 raft_consensus.cc:3037] T 849308d0b7e744f7946c892dc444f1f0 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:13.628072 13887 consensus_queue.cc:237] T 849308d0b7e744f7946c892dc444f1f0 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:13.628304 13814 raft_consensus.cc:2781] T 603eb0789cfb44ea930f447cd2190e6a P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:13.628409 13814 raft_consensus.cc:686] T 603eb0789cfb44ea930f447cd2190e6a P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Becoming Leader. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Running, Role: LEADER
I20240503 02:00:13.628540 13814 consensus_queue.cc:237] T 603eb0789cfb44ea930f447cd2190e6a P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:13.630204 6582 raft_consensus.cc:2454] T 849308d0b7e744f7946c892dc444f1f0 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 02:00:13.630582 6309 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:13.630722 6313 catalog_manager.cc:5484] T 849308d0b7e744f7946c892dc444f1f0 P 492448570cc94ed1879feb2ec3b9051d reported cstate change: term changed from 0 to 1, leader changed from <none> to 492448570cc94ed1879feb2ec3b9051d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:13.630909 6308 catalog_manager.cc:5484] T 603eb0789cfb44ea930f447cd2190e6a P 2a01ace80baf4057b6bf1995cf88468b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135). New cstate: current_term: 1 leader_uuid: "2a01ace80baf4057b6bf1995cf88468b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:13.637346 13863 tablet.cc:1722] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:13.637733 13863 tablet.cc:1620] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
I20240503 02:00:13.637759 13862 tablet.cc:1722] T 7b37a8876d0349f4a6885868f63ee6ce P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:13.638075 13862 tablet.cc:1620] T 7b37a8876d0349f4a6885868f63ee6ce P 80fd18ef4d604c86b909d5c008a46b4f: MemRowSet was empty: no flush needed.
I20240503 02:00:13.638712 13892 tablet.cc:1722] T 7b37a8876d0349f4a6885868f63ee6ce P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:13.639004 13892 tablet.cc:1620] T 7b37a8876d0349f4a6885868f63ee6ce P 99ead02915954b4fab23b033ad6d546b: MemRowSet was empty: no flush needed.
I20240503 02:00:13.645073 6313 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=2e477496b2bb4a5d8f1af5475a756447] alter complete (version 2)
W20240503 02:00:13.692667 6523 consensus_peers.cc:485] T d0d5dd42db3e4883af7fd16d8813b550 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:00:13.699349 13755 raft_consensus.cc:482] T e1770e6ac12d420eadb2be38ba010bba P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:13.699498 13755 raft_consensus.cc:504] T e1770e6ac12d420eadb2be38ba010bba P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:13.699930 13755 leader_election.cc:288] T e1770e6ac12d420eadb2be38ba010bba P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:13.700323 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1770e6ac12d420eadb2be38ba010bba" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 02:00:13.700506 6456 raft_consensus.cc:2454] T e1770e6ac12d420eadb2be38ba010bba P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:13.700542 6720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1770e6ac12d420eadb2be38ba010bba" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 02:00:13.700666 6720 raft_consensus.cc:2454] T e1770e6ac12d420eadb2be38ba010bba P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:13.700891 9112 leader_election.cc:302] T e1770e6ac12d420eadb2be38ba010bba P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:13.701084 13755 raft_consensus.cc:2781] T e1770e6ac12d420eadb2be38ba010bba P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:13.701141 13755 raft_consensus.cc:482] T e1770e6ac12d420eadb2be38ba010bba P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:13.701169 13755 raft_consensus.cc:3037] T e1770e6ac12d420eadb2be38ba010bba P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:13.702076 13755 raft_consensus.cc:504] T e1770e6ac12d420eadb2be38ba010bba P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:13.702448 13755 leader_election.cc:288] T e1770e6ac12d420eadb2be38ba010bba P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Requested vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:13.702960 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1770e6ac12d420eadb2be38ba010bba" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 02:00:13.702962 6720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1770e6ac12d420eadb2be38ba010bba" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 02:00:13.703078 6720 raft_consensus.cc:3037] T e1770e6ac12d420eadb2be38ba010bba P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:13.703084 6456 raft_consensus.cc:3037] T e1770e6ac12d420eadb2be38ba010bba P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:13.703965 6456 raft_consensus.cc:2454] T e1770e6ac12d420eadb2be38ba010bba P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:13.704078 6720 raft_consensus.cc:2454] T e1770e6ac12d420eadb2be38ba010bba P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:13.704447 9112 leader_election.cc:302] T e1770e6ac12d420eadb2be38ba010bba P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:13.704653 13814 raft_consensus.cc:2781] T e1770e6ac12d420eadb2be38ba010bba P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:13.704723 13814 raft_consensus.cc:686] T e1770e6ac12d420eadb2be38ba010bba P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Becoming Leader. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Running, Role: LEADER
I20240503 02:00:13.704866 13814 consensus_queue.cc:237] T e1770e6ac12d420eadb2be38ba010bba P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:13.707132 6313 catalog_manager.cc:5484] T e1770e6ac12d420eadb2be38ba010bba P 2a01ace80baf4057b6bf1995cf88468b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135). New cstate: current_term: 1 leader_uuid: "2a01ace80baf4057b6bf1995cf88468b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } }
W20240503 02:00:13.743497 6654 consensus_peers.cc:485] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:00:13.747087 6313 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:13.752944 13863 tablet.cc:1722] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:13.753482 13862 tablet.cc:1722] T 7b37a8876d0349f4a6885868f63ee6ce P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:13.753602 13892 tablet.cc:1722] T 7b37a8876d0349f4a6885868f63ee6ce P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:13.759773 6313 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=2e477496b2bb4a5d8f1af5475a756447] alter complete (version 3)
W20240503 02:00:13.760648 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
W20240503 02:00:13.857681 9113 consensus_peers.cc:485] T f44b59277c3b47adbd4dd6aff9350af1 P 2a01ace80baf4057b6bf1995cf88468b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:13.858315 6785 consensus_peers.cc:485] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:00:13.861311 6313 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:13.861462 6313 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:13.863600 6313 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet 7b37a8876d0349f4a6885868f63ee6ce
I20240503 02:00:13.864300 6569 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b37a8876d0349f4a6885868f63ee6ce with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:13 UTC) from {username='slave'} at 127.0.0.1:59846
I20240503 02:00:13.864692 13913 tablet_replica.cc:330] T 7b37a8876d0349f4a6885868f63ee6ce P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 02:00:13.864913 13913 raft_consensus.cc:2229] T 7b37a8876d0349f4a6885868f63ee6ce P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:13.864953 8897 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b37a8876d0349f4a6885868f63ee6ce with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:13 UTC) from {username='slave'} at 127.0.0.1:40296
I20240503 02:00:13.865073 13913 raft_consensus.cc:2258] T 7b37a8876d0349f4a6885868f63ee6ce P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:13.865481 13914 tablet_replica.cc:330] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2: stopping tablet replica
I20240503 02:00:13.865622 13914 raft_consensus.cc:2229] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:13.865866 13914 raft_consensus.cc:2258] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:13.865896 9028 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b37a8876d0349f4a6885868f63ee6ce with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:13 UTC) from {username='slave'} at 127.0.0.1:52584
I20240503 02:00:13.866168 13913 ts_tablet_manager.cc:1875] T 7b37a8876d0349f4a6885868f63ee6ce P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:13.866288 13915 tablet_replica.cc:330] T 7b37a8876d0349f4a6885868f63ee6ce P 99ead02915954b4fab23b033ad6d546b: stopping tablet replica
I20240503 02:00:13.866426 13915 raft_consensus.cc:2229] T 7b37a8876d0349f4a6885868f63ee6ce P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:13.866429 13914 ts_tablet_manager.cc:1875] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:13.866555 13915 raft_consensus.cc:2258] T 7b37a8876d0349f4a6885868f63ee6ce P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:13.867090 13915 ts_tablet_manager.cc:1875] T 7b37a8876d0349f4a6885868f63ee6ce P 99ead02915954b4fab23b033ad6d546b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:13.868983 13915 ts_tablet_manager.cc:1888] T 7b37a8876d0349f4a6885868f63ee6ce P 99ead02915954b4fab23b033ad6d546b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:13.868963 13913 ts_tablet_manager.cc:1888] T 7b37a8876d0349f4a6885868f63ee6ce P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:13.868974 13914 ts_tablet_manager.cc:1888] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:13.869056 13915 log.cc:1196] T 7b37a8876d0349f4a6885868f63ee6ce P 99ead02915954b4fab23b033ad6d546b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/7b37a8876d0349f4a6885868f63ee6ce
I20240503 02:00:13.869057 13914 log.cc:1196] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/7b37a8876d0349f4a6885868f63ee6ce
I20240503 02:00:13.869057 13913 log.cc:1196] T 7b37a8876d0349f4a6885868f63ee6ce P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/7b37a8876d0349f4a6885868f63ee6ce
I20240503 02:00:13.869316 13915 ts_tablet_manager.cc:1909] T 7b37a8876d0349f4a6885868f63ee6ce P 99ead02915954b4fab23b033ad6d546b: Deleting consensus metadata
I20240503 02:00:13.869324 13914 ts_tablet_manager.cc:1909] T 7b37a8876d0349f4a6885868f63ee6ce P 9942306855404cfd891adf227dd65eb2: Deleting consensus metadata
I20240503 02:00:13.869357 13913 ts_tablet_manager.cc:1909] T 7b37a8876d0349f4a6885868f63ee6ce P 80fd18ef4d604c86b909d5c008a46b4f: Deleting consensus metadata
I20240503 02:00:13.870093 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet 7b37a8876d0349f4a6885868f63ee6ce (table rebalancer_dynamic_table_new_name [id=2e477496b2bb4a5d8f1af5475a756447]) successfully deleted
I20240503 02:00:13.870160 6303 catalog_manager.cc:4830] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211): tablet 7b37a8876d0349f4a6885868f63ee6ce (table rebalancer_dynamic_table_new_name [id=2e477496b2bb4a5d8f1af5475a756447]) successfully deleted
I20240503 02:00:13.870532 6301 catalog_manager.cc:4830] TS 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157): tablet 7b37a8876d0349f4a6885868f63ee6ce (table rebalancer_dynamic_table_new_name [id=2e477496b2bb4a5d8f1af5475a756447]) successfully deleted
W20240503 02:00:13.879943 6785 consensus_peers.cc:485] T d897c21e630d4770832bb10f43778e82 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:13.913137 6523 consensus_peers.cc:485] T 925612b660ec471da5b3c548802ede6e P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:13.928684 8851 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 02:00:13.938019 6785 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240503 02:00:13.940716 6654 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 02:00:13.945075 8982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240503 02:00:13.965508 6313 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
W20240503 02:00:13.970527 6785 consensus_peers.cc:485] T 8246488dec204bf6b09e868d7951ac5a P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:00:13.972178 6436 tablet_service.cc:1467] Processing CreateTablet for tablet 308236ce104a41baa84cf1d8e6f68c0e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d3615bb8f356419e9ff1adfb5631ddc8]), partition=
I20240503 02:00:13.972326 6569 tablet_service.cc:1467] Processing CreateTablet for tablet 308236ce104a41baa84cf1d8e6f68c0e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d3615bb8f356419e9ff1adfb5631ddc8]), partition=
I20240503 02:00:13.972496 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 308236ce104a41baa84cf1d8e6f68c0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:13.972616 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 308236ce104a41baa84cf1d8e6f68c0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:13.974699 13901 tablet_bootstrap.cc:492] T 308236ce104a41baa84cf1d8e6f68c0e P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:13.974804 13881 tablet_bootstrap.cc:492] T 308236ce104a41baa84cf1d8e6f68c0e P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:13.975472 9028 tablet_service.cc:1467] Processing CreateTablet for tablet 308236ce104a41baa84cf1d8e6f68c0e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d3615bb8f356419e9ff1adfb5631ddc8]), partition=
I20240503 02:00:13.975685 13881 tablet_bootstrap.cc:654] T 308236ce104a41baa84cf1d8e6f68c0e P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:13.975687 13901 tablet_bootstrap.cc:654] T 308236ce104a41baa84cf1d8e6f68c0e P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:13.975711 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 308236ce104a41baa84cf1d8e6f68c0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:13.977161 13881 tablet_bootstrap.cc:492] T 308236ce104a41baa84cf1d8e6f68c0e P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:13.977241 13881 ts_tablet_manager.cc:1367] T 308236ce104a41baa84cf1d8e6f68c0e P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:00:13.977665 13881 raft_consensus.cc:348] T 308236ce104a41baa84cf1d8e6f68c0e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:13.977762 13881 raft_consensus.cc:374] T 308236ce104a41baa84cf1d8e6f68c0e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:13.977787 13881 raft_consensus.cc:729] T 308236ce104a41baa84cf1d8e6f68c0e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:13.977907 13881 consensus_queue.cc:260] T 308236ce104a41baa84cf1d8e6f68c0e P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:13.978049 13884 tablet_bootstrap.cc:492] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:13.978152 13881 ts_tablet_manager.cc:1398] T 308236ce104a41baa84cf1d8e6f68c0e P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:13.979136 13884 tablet_bootstrap.cc:654] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:13.980881 13901 tablet_bootstrap.cc:492] T 308236ce104a41baa84cf1d8e6f68c0e P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:13.980990 13901 ts_tablet_manager.cc:1367] T 308236ce104a41baa84cf1d8e6f68c0e P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:00:13.981508 13901 raft_consensus.cc:348] T 308236ce104a41baa84cf1d8e6f68c0e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:13.981631 13901 raft_consensus.cc:374] T 308236ce104a41baa84cf1d8e6f68c0e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:13.981660 13901 raft_consensus.cc:729] T 308236ce104a41baa84cf1d8e6f68c0e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:13.981798 13901 consensus_queue.cc:260] T 308236ce104a41baa84cf1d8e6f68c0e P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:13.983023 13901 ts_tablet_manager.cc:1398] T 308236ce104a41baa84cf1d8e6f68c0e P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:13.985600 13884 tablet_bootstrap.cc:492] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:13.985723 13884 ts_tablet_manager.cc:1367] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20240503 02:00:13.986181 13884 raft_consensus.cc:348] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:13.986303 13884 raft_consensus.cc:374] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:13.986338 13884 raft_consensus.cc:729] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:13.986464 13884 consensus_queue.cc:260] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:13.986752 13884 ts_tablet_manager.cc:1398] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20240503 02:00:13.991488 6523 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 02:00:13.992280 6523 consensus_peers.cc:485] T 4385d51d11574192883d01376e2d5620 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:14.004704 6523 consensus_peers.cc:485] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:14.015327 8851 consensus_peers.cc:485] T 5f51fe5f799f4e479b62f5917a2c2606 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:00:14.037117 13887 consensus_queue.cc:1035] T 849308d0b7e744f7946c892dc444f1f0 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:00:14.037592 8851 consensus_peers.cc:485] T f28586e70bd9473fbdc6b9126b666fa9 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:00:14.041064 13887 consensus_queue.cc:1035] T 849308d0b7e744f7946c892dc444f1f0 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:00:14.089943 9113 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 02:00:14.090626 9113 consensus_peers.cc:485] T b1740b877c1e4372a0337d99f5bcab0e P 2a01ace80baf4057b6bf1995cf88468b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:14.108647 8851 consensus_peers.cc:485] T 7da8f614a70e4177b70e7c2f31d22a10 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:14.110893 8982 consensus_peers.cc:485] T 13756a6e44814fe4b09a6b0f087546c3 P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:00:14.113708 13924 raft_consensus.cc:482] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:14.113852 13924 raft_consensus.cc:504] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:14.114260 13924 leader_election.cc:288] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:14.114712 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308236ce104a41baa84cf1d8e6f68c0e" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 02:00:14.114867 6456 raft_consensus.cc:2454] T 308236ce104a41baa84cf1d8e6f68c0e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:14.115018 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308236ce104a41baa84cf1d8e6f68c0e" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 02:00:14.115144 6589 raft_consensus.cc:2454] T 308236ce104a41baa84cf1d8e6f68c0e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:14.115396 8983 leader_election.cc:302] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:14.115589 13924 raft_consensus.cc:2781] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:14.115646 13924 raft_consensus.cc:482] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:14.115690 13924 raft_consensus.cc:3037] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:14.116561 13924 raft_consensus.cc:504] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:14.116943 13924 leader_election.cc:288] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Requested vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:14.117357 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308236ce104a41baa84cf1d8e6f68c0e" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 02:00:14.117345 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308236ce104a41baa84cf1d8e6f68c0e" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 02:00:14.117477 6456 raft_consensus.cc:3037] T 308236ce104a41baa84cf1d8e6f68c0e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:14.117502 6589 raft_consensus.cc:3037] T 308236ce104a41baa84cf1d8e6f68c0e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:14.118299 6456 raft_consensus.cc:2454] T 308236ce104a41baa84cf1d8e6f68c0e P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:14.118352 6589 raft_consensus.cc:2454] T 308236ce104a41baa84cf1d8e6f68c0e P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:14.118680 8983 leader_election.cc:302] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, 99ead02915954b4fab23b033ad6d546b, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:14.118984 13924 raft_consensus.cc:2781] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:14.119081 13924 raft_consensus.cc:686] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Becoming Leader. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Running, Role: LEADER
I20240503 02:00:14.119261 13924 consensus_queue.cc:237] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
W20240503 02:00:14.120415 6523 consensus_peers.cc:485] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:00:14.122707 6309 catalog_manager.cc:5484] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b reported cstate change: term changed from 0 to 1, leader changed from <none> to 99ead02915954b4fab23b033ad6d546b (127.0.99.134). New cstate: current_term: 1 leader_uuid: "99ead02915954b4fab23b033ad6d546b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } }
W20240503 02:00:14.139240 8982 consensus_peers.cc:485] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:14.173719 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:14.200642 13814 consensus_queue.cc:1035] T e1770e6ac12d420eadb2be38ba010bba P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:14.205544 13814 consensus_queue.cc:1035] T e1770e6ac12d420eadb2be38ba010bba P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:14.221732 6309 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_extra_table_36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:14.227696 6569 tablet_service.cc:1467] Processing CreateTablet for tablet 0fc624ba9c6940a3a7eedcf9a7d79dde (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=5d8b1e5b1c3a44018823bf52234cd57b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:14.228041 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fc624ba9c6940a3a7eedcf9a7d79dde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:14.228369 6831 tablet_service.cc:1467] Processing CreateTablet for tablet 37190356d40d40c7aed6607b87c676fd (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=5d8b1e5b1c3a44018823bf52234cd57b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:14.228680 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37190356d40d40c7aed6607b87c676fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:14.228719 6699 tablet_service.cc:1467] Processing CreateTablet for tablet 0fc624ba9c6940a3a7eedcf9a7d79dde (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=5d8b1e5b1c3a44018823bf52234cd57b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:14.228967 6699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fc624ba9c6940a3a7eedcf9a7d79dde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:14.228950 9025 tablet_service.cc:1467] Processing CreateTablet for tablet d9c6a7e041014192b2a8e9bced641391 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=5d8b1e5b1c3a44018823bf52234cd57b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:14.229220 9025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9c6a7e041014192b2a8e9bced641391. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:14.230664 13881 tablet_bootstrap.cc:492] T 0fc624ba9c6940a3a7eedcf9a7d79dde P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:14.231206 6829 tablet_service.cc:1467] Processing CreateTablet for tablet d9c6a7e041014192b2a8e9bced641391 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=5d8b1e5b1c3a44018823bf52234cd57b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:14.231336 9028 tablet_service.cc:1467] Processing CreateTablet for tablet 37190356d40d40c7aed6607b87c676fd (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=5d8b1e5b1c3a44018823bf52234cd57b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:14.231482 6829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9c6a7e041014192b2a8e9bced641391. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:14.231578 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37190356d40d40c7aed6607b87c676fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:14.231712 13881 tablet_bootstrap.cc:654] T 0fc624ba9c6940a3a7eedcf9a7d79dde P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:14.232079 13933 tablet_bootstrap.cc:492] T 0fc624ba9c6940a3a7eedcf9a7d79dde P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:14.233112 13933 tablet_bootstrap.cc:654] T 0fc624ba9c6940a3a7eedcf9a7d79dde P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:14.233583 13884 tablet_bootstrap.cc:492] T d9c6a7e041014192b2a8e9bced641391 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:14.234525 13884 tablet_bootstrap.cc:654] T d9c6a7e041014192b2a8e9bced641391 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:14.234866 6699 tablet_service.cc:1467] Processing CreateTablet for tablet d9c6a7e041014192b2a8e9bced641391 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=5d8b1e5b1c3a44018823bf52234cd57b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:14.235077 13881 tablet_bootstrap.cc:492] T 0fc624ba9c6940a3a7eedcf9a7d79dde P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:14.235126 6699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9c6a7e041014192b2a8e9bced641391. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:14.235167 13881 ts_tablet_manager.cc:1367] T 0fc624ba9c6940a3a7eedcf9a7d79dde P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240503 02:00:14.235662 13881 raft_consensus.cc:348] T 0fc624ba9c6940a3a7eedcf9a7d79dde P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:14.235780 13881 raft_consensus.cc:374] T 0fc624ba9c6940a3a7eedcf9a7d79dde P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:14.235810 13881 raft_consensus.cc:729] T 0fc624ba9c6940a3a7eedcf9a7d79dde P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:14.235932 13881 consensus_queue.cc:260] T 0fc624ba9c6940a3a7eedcf9a7d79dde P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:14.235978 8897 tablet_service.cc:1467] Processing CreateTablet for tablet 0fc624ba9c6940a3a7eedcf9a7d79dde (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=5d8b1e5b1c3a44018823bf52234cd57b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:14.236235 8897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fc624ba9c6940a3a7eedcf9a7d79dde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:14.236227 13881 ts_tablet_manager.cc:1398] T 0fc624ba9c6940a3a7eedcf9a7d79dde P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:14.236320 13884 tablet_bootstrap.cc:492] T d9c6a7e041014192b2a8e9bced641391 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:14.236428 13884 ts_tablet_manager.cc:1367] T d9c6a7e041014192b2a8e9bced641391 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:00:14.236894 13884 raft_consensus.cc:348] T d9c6a7e041014192b2a8e9bced641391 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:14.236994 13884 raft_consensus.cc:374] T d9c6a7e041014192b2a8e9bced641391 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:14.237020 13884 raft_consensus.cc:729] T d9c6a7e041014192b2a8e9bced641391 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:14.237088 9158 tablet_service.cc:1467] Processing CreateTablet for tablet 37190356d40d40c7aed6607b87c676fd (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=5d8b1e5b1c3a44018823bf52234cd57b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:14.237135 13884 consensus_queue.cc:260] T d9c6a7e041014192b2a8e9bced641391 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:14.237334 9158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37190356d40d40c7aed6607b87c676fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:14.237408 13884 ts_tablet_manager.cc:1398] T d9c6a7e041014192b2a8e9bced641391 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:14.237833 13776 consensus_queue.cc:1035] T 603eb0789cfb44ea930f447cd2190e6a P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:14.239691 13884 tablet_bootstrap.cc:492] T 37190356d40d40c7aed6607b87c676fd P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:14.241334 13884 tablet_bootstrap.cc:654] T 37190356d40d40c7aed6607b87c676fd P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:14.241257 13814 consensus_queue.cc:1035] T 603eb0789cfb44ea930f447cd2190e6a P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:14.242273 6309 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:14.245647 13932 tablet_bootstrap.cc:492] T 37190356d40d40c7aed6607b87c676fd P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:14.246040 13939 tablet_bootstrap.cc:492] T 37190356d40d40c7aed6607b87c676fd P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:14.246220 13884 tablet_bootstrap.cc:492] T 37190356d40d40c7aed6607b87c676fd P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:14.246313 13884 ts_tablet_manager.cc:1367] T 37190356d40d40c7aed6607b87c676fd P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:00:14.248346 13932 tablet_bootstrap.cc:654] T 37190356d40d40c7aed6607b87c676fd P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:14.249073 13939 tablet_bootstrap.cc:654] T 37190356d40d40c7aed6607b87c676fd P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:14.249894 13884 raft_consensus.cc:348] T 37190356d40d40c7aed6607b87c676fd P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:14.250341 13884 raft_consensus.cc:374] T 37190356d40d40c7aed6607b87c676fd P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:14.250440 13884 raft_consensus.cc:729] T 37190356d40d40c7aed6607b87c676fd P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:14.250777 13884 consensus_queue.cc:260] T 37190356d40d40c7aed6607b87c676fd P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:14.251274 13884 ts_tablet_manager.cc:1398] T 37190356d40d40c7aed6607b87c676fd P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240503 02:00:14.251797 6458 raft_consensus.cc:1261] T 308236ce104a41baa84cf1d8e6f68c0e P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 99ead02915954b4fab23b033ad6d546b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:14.252208 6589 raft_consensus.cc:1261] T 308236ce104a41baa84cf1d8e6f68c0e P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer 99ead02915954b4fab23b033ad6d546b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:14.252936 13924 consensus_queue.cc:1035] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:14.253211 13937 tablet_bootstrap.cc:492] T 0fc624ba9c6940a3a7eedcf9a7d79dde P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:14.253995 13924 consensus_queue.cc:1035] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:14.255728 6308 catalog_manager.cc:5576] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863) does not have the latest schema for tablet 308236ce104a41baa84cf1d8e6f68c0e (table rebalancer_dynamic_table [id=d3615bb8f356419e9ff1adfb5631ddc8]). Expected version 1 got 0
I20240503 02:00:14.256233 13933 tablet_bootstrap.cc:492] T 0fc624ba9c6940a3a7eedcf9a7d79dde P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:14.256331 13933 ts_tablet_manager.cc:1367] T 0fc624ba9c6940a3a7eedcf9a7d79dde P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.000s
I20240503 02:00:14.256882 13933 raft_consensus.cc:348] T 0fc624ba9c6940a3a7eedcf9a7d79dde P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:14.257331 13933 raft_consensus.cc:374] T 0fc624ba9c6940a3a7eedcf9a7d79dde P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:14.257411 13932 tablet_bootstrap.cc:492] T 37190356d40d40c7aed6607b87c676fd P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:14.257570 13932 ts_tablet_manager.cc:1367] T 37190356d40d40c7aed6607b87c676fd P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 02:00:14.258093 13932 raft_consensus.cc:348] T 37190356d40d40c7aed6607b87c676fd P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:14.258229 6309 catalog_manager.cc:5576] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093) does not have the latest schema for tablet 308236ce104a41baa84cf1d8e6f68c0e (table rebalancer_dynamic_table [id=d3615bb8f356419e9ff1adfb5631ddc8]). Expected version 1 got 0
I20240503 02:00:14.258915 13937 tablet_bootstrap.cc:654] T 0fc624ba9c6940a3a7eedcf9a7d79dde P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:14.259405 13933 raft_consensus.cc:729] T 0fc624ba9c6940a3a7eedcf9a7d79dde P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:14.259485 13932 raft_consensus.cc:374] T 37190356d40d40c7aed6607b87c676fd P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:14.259555 13932 raft_consensus.cc:729] T 37190356d40d40c7aed6607b87c676fd P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:14.259706 13932 consensus_queue.cc:260] T 37190356d40d40c7aed6607b87c676fd P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:14.259765 13933 consensus_queue.cc:260] T 0fc624ba9c6940a3a7eedcf9a7d79dde P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:14.260032 13932 ts_tablet_manager.cc:1398] T 37190356d40d40c7aed6607b87c676fd P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:14.260174 13933 ts_tablet_manager.cc:1398] T 0fc624ba9c6940a3a7eedcf9a7d79dde P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 02:00:14.260219 13932 tablet_bootstrap.cc:492] T d9c6a7e041014192b2a8e9bced641391 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:14.261260 13932 tablet_bootstrap.cc:654] T d9c6a7e041014192b2a8e9bced641391 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:14.261518 13933 tablet_bootstrap.cc:492] T d9c6a7e041014192b2a8e9bced641391 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:14.262668 13933 tablet_bootstrap.cc:654] T d9c6a7e041014192b2a8e9bced641391 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:14.262797 13937 tablet_bootstrap.cc:492] T 0fc624ba9c6940a3a7eedcf9a7d79dde P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:14.262822 13932 tablet_bootstrap.cc:492] T d9c6a7e041014192b2a8e9bced641391 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:14.262905 13932 ts_tablet_manager.cc:1367] T d9c6a7e041014192b2a8e9bced641391 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:14.262934 13937 ts_tablet_manager.cc:1367] T 0fc624ba9c6940a3a7eedcf9a7d79dde P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 02:00:14.263460 13937 raft_consensus.cc:348] T 0fc624ba9c6940a3a7eedcf9a7d79dde P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:14.263459 13932 raft_consensus.cc:348] T d9c6a7e041014192b2a8e9bced641391 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:14.263588 13932 raft_consensus.cc:374] T d9c6a7e041014192b2a8e9bced641391 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:14.263724 13932 raft_consensus.cc:729] T d9c6a7e041014192b2a8e9bced641391 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:14.263901 13932 consensus_queue.cc:260] T d9c6a7e041014192b2a8e9bced641391 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:14.264250 13932 ts_tablet_manager.cc:1398] T d9c6a7e041014192b2a8e9bced641391 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:14.263624 13937 raft_consensus.cc:374] T 0fc624ba9c6940a3a7eedcf9a7d79dde P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:14.264726 13937 raft_consensus.cc:729] T 0fc624ba9c6940a3a7eedcf9a7d79dde P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:14.266183 13937 consensus_queue.cc:260] T 0fc624ba9c6940a3a7eedcf9a7d79dde P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:14.266649 13933 tablet_bootstrap.cc:492] T d9c6a7e041014192b2a8e9bced641391 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:14.266745 13933 ts_tablet_manager.cc:1367] T d9c6a7e041014192b2a8e9bced641391 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240503 02:00:14.267230 13933 raft_consensus.cc:348] T d9c6a7e041014192b2a8e9bced641391 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:14.267334 13933 raft_consensus.cc:374] T d9c6a7e041014192b2a8e9bced641391 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:14.267361 13933 raft_consensus.cc:729] T d9c6a7e041014192b2a8e9bced641391 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:14.267611 13950 tablet.cc:1722] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:14.267935 13950 tablet.cc:1620] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b: MemRowSet was empty: no flush needed.
I20240503 02:00:14.269068 13933 consensus_queue.cc:260] T d9c6a7e041014192b2a8e9bced641391 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:14.270792 13933 ts_tablet_manager.cc:1398] T d9c6a7e041014192b2a8e9bced641391 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.004s user 0.000s sys 0.000s
I20240503 02:00:14.271836 6309 catalog_manager.cc:5576] TS 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157) does not have the latest schema for tablet 308236ce104a41baa84cf1d8e6f68c0e (table rebalancer_dynamic_table [id=d3615bb8f356419e9ff1adfb5631ddc8]). Expected version 1 got 0
I20240503 02:00:14.272539 13939 tablet_bootstrap.cc:492] T 37190356d40d40c7aed6607b87c676fd P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:14.272639 13939 ts_tablet_manager.cc:1367] T 37190356d40d40c7aed6607b87c676fd P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.000s
I20240503 02:00:14.273192 13939 raft_consensus.cc:348] T 37190356d40d40c7aed6607b87c676fd P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:14.273308 13939 raft_consensus.cc:374] T 37190356d40d40c7aed6607b87c676fd P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:14.273339 13939 raft_consensus.cc:729] T 37190356d40d40c7aed6607b87c676fd P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:14.273455 13939 consensus_queue.cc:260] T 37190356d40d40c7aed6607b87c676fd P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:14.273749 13939 ts_tablet_manager.cc:1398] T 37190356d40d40c7aed6607b87c676fd P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:14.274029 13937 ts_tablet_manager.cc:1398] T 0fc624ba9c6940a3a7eedcf9a7d79dde P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20240503 02:00:14.275005 6313 catalog_manager.cc:5576] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863) does not have the latest schema for tablet 308236ce104a41baa84cf1d8e6f68c0e (table rebalancer_dynamic_table [id=d3615bb8f356419e9ff1adfb5631ddc8]). Expected version 1 got 0
I20240503 02:00:14.275112 13951 tablet.cc:1722] T 308236ce104a41baa84cf1d8e6f68c0e P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:14.276151 13951 tablet.cc:1620] T 308236ce104a41baa84cf1d8e6f68c0e P dec3e4a04d604f7eaaa3f077d081218a: MemRowSet was empty: no flush needed.
I20240503 02:00:14.277295 6312 catalog_manager.cc:5576] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093) does not have the latest schema for tablet 308236ce104a41baa84cf1d8e6f68c0e (table rebalancer_dynamic_table [id=d3615bb8f356419e9ff1adfb5631ddc8]). Expected version 1 got 0
I20240503 02:00:14.278475 6308 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=d3615bb8f356419e9ff1adfb5631ddc8] alter complete (version 1)
I20240503 02:00:14.280073 13952 tablet.cc:1722] T 308236ce104a41baa84cf1d8e6f68c0e P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:14.280490 13952 tablet.cc:1620] T 308236ce104a41baa84cf1d8e6f68c0e P 80fd18ef4d604c86b909d5c008a46b4f: MemRowSet was empty: no flush needed.
I20240503 02:00:14.308444 13911 raft_consensus.cc:482] T 0fc624ba9c6940a3a7eedcf9a7d79dde P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:14.308594 13911 raft_consensus.cc:504] T 0fc624ba9c6940a3a7eedcf9a7d79dde P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:14.309015 13911 leader_election.cc:288] T 0fc624ba9c6940a3a7eedcf9a7d79dde P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:14.309453 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc624ba9c6940a3a7eedcf9a7d79dde" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:14.309515 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc624ba9c6940a3a7eedcf9a7d79dde" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 02:00:14.309593 8916 raft_consensus.cc:2454] T 0fc624ba9c6940a3a7eedcf9a7d79dde P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:14.309665 6589 raft_consensus.cc:2454] T 0fc624ba9c6940a3a7eedcf9a7d79dde P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:14.309937 6656 leader_election.cc:302] T 0fc624ba9c6940a3a7eedcf9a7d79dde P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9942306855404cfd891adf227dd65eb2, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:14.310214 13911 raft_consensus.cc:2781] T 0fc624ba9c6940a3a7eedcf9a7d79dde P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:14.310276 13911 raft_consensus.cc:482] T 0fc624ba9c6940a3a7eedcf9a7d79dde P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:14.310302 13911 raft_consensus.cc:3037] T 0fc624ba9c6940a3a7eedcf9a7d79dde P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:14.311195 13911 raft_consensus.cc:504] T 0fc624ba9c6940a3a7eedcf9a7d79dde P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:14.311623 13911 leader_election.cc:288] T 0fc624ba9c6940a3a7eedcf9a7d79dde P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Requested vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:14.312065 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc624ba9c6940a3a7eedcf9a7d79dde" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:14.312193 8916 raft_consensus.cc:3037] T 0fc624ba9c6940a3a7eedcf9a7d79dde P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:14.312191 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc624ba9c6940a3a7eedcf9a7d79dde" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 02:00:14.312330 6589 raft_consensus.cc:3037] T 0fc624ba9c6940a3a7eedcf9a7d79dde P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:14.313182 8916 raft_consensus.cc:2454] T 0fc624ba9c6940a3a7eedcf9a7d79dde P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:14.313359 6589 raft_consensus.cc:2454] T 0fc624ba9c6940a3a7eedcf9a7d79dde P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:14.313490 6656 leader_election.cc:302] T 0fc624ba9c6940a3a7eedcf9a7d79dde P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9942306855404cfd891adf227dd65eb2, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:14.313690 13911 raft_consensus.cc:2781] T 0fc624ba9c6940a3a7eedcf9a7d79dde P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:14.313864 13911 raft_consensus.cc:686] T 0fc624ba9c6940a3a7eedcf9a7d79dde P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Becoming Leader. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Running, Role: LEADER
I20240503 02:00:14.314019 13911 consensus_queue.cc:237] T 0fc624ba9c6940a3a7eedcf9a7d79dde P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:14.316128 6309 catalog_manager.cc:5484] T 0fc624ba9c6940a3a7eedcf9a7d79dde P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } }
W20240503 02:00:14.324738 6301 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111) [suppressed 56 similar messages]
W20240503 02:00:14.325449 6301 catalog_manager.cc:4573] TS e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): CreateTablet RPC failed for tablet 56824b553788471c9a2fcd6ec82d507e: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
W20240503 02:00:14.329160 6654 consensus_peers.cc:485] T cea1ea50fa434d31b42451f84f26bc76 P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:00:14.383385 6309 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:14.394821 13952 tablet.cc:1722] T 308236ce104a41baa84cf1d8e6f68c0e P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:14.395202 13952 tablet.cc:1620] T 308236ce104a41baa84cf1d8e6f68c0e P 80fd18ef4d604c86b909d5c008a46b4f: MemRowSet was empty: no flush needed.
I20240503 02:00:14.395514 13950 tablet.cc:1722] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:14.395833 13950 tablet.cc:1620] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b: MemRowSet was empty: no flush needed.
I20240503 02:00:14.401247 13951 tablet.cc:1722] T 308236ce104a41baa84cf1d8e6f68c0e P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:14.401621 13951 tablet.cc:1620] T 308236ce104a41baa84cf1d8e6f68c0e P dec3e4a04d604f7eaaa3f077d081218a: MemRowSet was empty: no flush needed.
I20240503 02:00:14.408622 6312 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=d3615bb8f356419e9ff1adfb5631ddc8] alter complete (version 2)
I20240503 02:00:14.409536 13679 raft_consensus.cc:482] T d9c6a7e041014192b2a8e9bced641391 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:14.409643 13679 raft_consensus.cc:504] T d9c6a7e041014192b2a8e9bced641391 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:14.410024 13679 leader_election.cc:288] T d9c6a7e041014192b2a8e9bced641391 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:14.410410 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9c6a7e041014192b2a8e9bced641391" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 02:00:14.410547 6851 raft_consensus.cc:2454] T d9c6a7e041014192b2a8e9bced641391 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:14.410604 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9c6a7e041014192b2a8e9bced641391" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:14.410735 9048 raft_consensus.cc:2454] T d9c6a7e041014192b2a8e9bced641391 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:14.410986 6656 leader_election.cc:302] T d9c6a7e041014192b2a8e9bced641391 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:14.411259 13679 raft_consensus.cc:2781] T d9c6a7e041014192b2a8e9bced641391 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:14.411319 13679 raft_consensus.cc:482] T d9c6a7e041014192b2a8e9bced641391 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:14.411346 13679 raft_consensus.cc:3037] T d9c6a7e041014192b2a8e9bced641391 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:14.412189 13679 raft_consensus.cc:504] T d9c6a7e041014192b2a8e9bced641391 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:14.412603 13679 leader_election.cc:288] T d9c6a7e041014192b2a8e9bced641391 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Requested vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:14.412968 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9c6a7e041014192b2a8e9bced641391" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:14.412989 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9c6a7e041014192b2a8e9bced641391" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 02:00:14.413081 9048 raft_consensus.cc:3037] T d9c6a7e041014192b2a8e9bced641391 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:14.413100 6851 raft_consensus.cc:3037] T d9c6a7e041014192b2a8e9bced641391 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:14.413931 6851 raft_consensus.cc:2454] T d9c6a7e041014192b2a8e9bced641391 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:14.413928 9048 raft_consensus.cc:2454] T d9c6a7e041014192b2a8e9bced641391 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:14.414350 6654 leader_election.cc:302] T d9c6a7e041014192b2a8e9bced641391 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99ead02915954b4fab23b033ad6d546b, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:14.414914 13679 raft_consensus.cc:2781] T d9c6a7e041014192b2a8e9bced641391 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:14.415007 13679 raft_consensus.cc:686] T d9c6a7e041014192b2a8e9bced641391 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Becoming Leader. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Running, Role: LEADER
I20240503 02:00:14.415151 13679 consensus_queue.cc:237] T d9c6a7e041014192b2a8e9bced641391 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:14.417402 6308 catalog_manager.cc:5484] T d9c6a7e041014192b2a8e9bced641391 P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } }
W20240503 02:00:14.470124 8982 consensus_peers.cc:485] T beeea71f02ae4c89a1af13d3584ef70a P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:14.483481 8982 consensus_peers.cc:485] T 1a7ee79312ea4bad816264375ca5bb2d P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:00:14.512534 6308 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:14.516469 13924 raft_consensus.cc:482] T 37190356d40d40c7aed6607b87c676fd P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:14.516589 13924 raft_consensus.cc:504] T 37190356d40d40c7aed6607b87c676fd P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:14.516976 13924 leader_election.cc:288] T 37190356d40d40c7aed6607b87c676fd P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:14.517905 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37190356d40d40c7aed6607b87c676fd" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:14.518059 9178 raft_consensus.cc:2454] T 37190356d40d40c7aed6607b87c676fd P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:14.518431 8981 leader_election.cc:302] T 37190356d40d40c7aed6607b87c676fd P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:14.518571 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37190356d40d40c7aed6607b87c676fd" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 02:00:14.518733 6851 raft_consensus.cc:2454] T 37190356d40d40c7aed6607b87c676fd P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:14.518739 13952 tablet.cc:1722] T 308236ce104a41baa84cf1d8e6f68c0e P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:14.519153 13900 raft_consensus.cc:2781] T 37190356d40d40c7aed6607b87c676fd P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:14.519212 13900 raft_consensus.cc:482] T 37190356d40d40c7aed6607b87c676fd P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:14.519251 13900 raft_consensus.cc:3037] T 37190356d40d40c7aed6607b87c676fd P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:14.519395 13950 tablet.cc:1722] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:14.520215 13900 raft_consensus.cc:504] T 37190356d40d40c7aed6607b87c676fd P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:14.520635 13900 leader_election.cc:288] T 37190356d40d40c7aed6607b87c676fd P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Requested vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:14.522167 13951 tablet.cc:1722] T 308236ce104a41baa84cf1d8e6f68c0e P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:14.522986 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37190356d40d40c7aed6607b87c676fd" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:14.523136 9178 raft_consensus.cc:3037] T 37190356d40d40c7aed6607b87c676fd P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:14.524210 9178 raft_consensus.cc:2454] T 37190356d40d40c7aed6607b87c676fd P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:14.524590 8981 leader_election.cc:302] T 37190356d40d40c7aed6607b87c676fd P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:14.525272 13956 raft_consensus.cc:2781] T 37190356d40d40c7aed6607b87c676fd P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:14.525417 13956 raft_consensus.cc:686] T 37190356d40d40c7aed6607b87c676fd P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Becoming Leader. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Running, Role: LEADER
I20240503 02:00:14.525609 13956 consensus_queue.cc:237] T 37190356d40d40c7aed6607b87c676fd P 99ead02915954b4fab23b033ad6d546b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:14.526402 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37190356d40d40c7aed6607b87c676fd" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 02:00:14.526533 6851 raft_consensus.cc:3037] T 37190356d40d40c7aed6607b87c676fd P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:14.527149 6309 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=d3615bb8f356419e9ff1adfb5631ddc8] alter complete (version 3)
I20240503 02:00:14.527714 6851 raft_consensus.cc:2454] T 37190356d40d40c7aed6607b87c676fd P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:14.529774 6308 catalog_manager.cc:5484] T 37190356d40d40c7aed6607b87c676fd P 99ead02915954b4fab23b033ad6d546b reported cstate change: term changed from 0 to 1, leader changed from <none> to 99ead02915954b4fab23b033ad6d546b (127.0.99.134). New cstate: current_term: 1 leader_uuid: "99ead02915954b4fab23b033ad6d546b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:14.632862 6308 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:14.633026 6308 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:14.635115 6308 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet 308236ce104a41baa84cf1d8e6f68c0e
I20240503 02:00:14.635776 9028 tablet_service.cc:1514] Processing DeleteTablet for tablet 308236ce104a41baa84cf1d8e6f68c0e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:14 UTC) from {username='slave'} at 127.0.0.1:52584
I20240503 02:00:14.635859 6569 tablet_service.cc:1514] Processing DeleteTablet for tablet 308236ce104a41baa84cf1d8e6f68c0e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:14 UTC) from {username='slave'} at 127.0.0.1:59846
I20240503 02:00:14.635861 6436 tablet_service.cc:1514] Processing DeleteTablet for tablet 308236ce104a41baa84cf1d8e6f68c0e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:14 UTC) from {username='slave'} at 127.0.0.1:51338
I20240503 02:00:14.636179 13961 tablet_replica.cc:330] T 308236ce104a41baa84cf1d8e6f68c0e P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 02:00:14.636196 13962 tablet_replica.cc:330] T 308236ce104a41baa84cf1d8e6f68c0e P dec3e4a04d604f7eaaa3f077d081218a: stopping tablet replica
I20240503 02:00:14.636344 13960 tablet_replica.cc:330] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b: stopping tablet replica
I20240503 02:00:14.636341 13961 raft_consensus.cc:2229] T 308236ce104a41baa84cf1d8e6f68c0e P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:14.636355 13962 raft_consensus.cc:2229] T 308236ce104a41baa84cf1d8e6f68c0e P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:14.636497 13960 raft_consensus.cc:2229] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:14.636560 13962 raft_consensus.cc:2258] T 308236ce104a41baa84cf1d8e6f68c0e P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:14.636497 13961 raft_consensus.cc:2258] T 308236ce104a41baa84cf1d8e6f68c0e P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:14.636965 13960 raft_consensus.cc:2258] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:14.637156 13962 ts_tablet_manager.cc:1875] T 308236ce104a41baa84cf1d8e6f68c0e P dec3e4a04d604f7eaaa3f077d081218a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:14.637166 13961 ts_tablet_manager.cc:1875] T 308236ce104a41baa84cf1d8e6f68c0e P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:14.637501 13960 ts_tablet_manager.cc:1875] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:14.639355 13961 ts_tablet_manager.cc:1888] T 308236ce104a41baa84cf1d8e6f68c0e P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:14.639439 13961 log.cc:1196] T 308236ce104a41baa84cf1d8e6f68c0e P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/308236ce104a41baa84cf1d8e6f68c0e
I20240503 02:00:14.639617 13962 ts_tablet_manager.cc:1888] T 308236ce104a41baa84cf1d8e6f68c0e P dec3e4a04d604f7eaaa3f077d081218a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:14.639742 13961 ts_tablet_manager.cc:1909] T 308236ce104a41baa84cf1d8e6f68c0e P 80fd18ef4d604c86b909d5c008a46b4f: Deleting consensus metadata
I20240503 02:00:14.639907 13960 ts_tablet_manager.cc:1888] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:14.639945 13962 log.cc:1196] T 308236ce104a41baa84cf1d8e6f68c0e P dec3e4a04d604f7eaaa3f077d081218a: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/308236ce104a41baa84cf1d8e6f68c0e
I20240503 02:00:14.639989 13960 log.cc:1196] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/308236ce104a41baa84cf1d8e6f68c0e
I20240503 02:00:14.640235 13960 ts_tablet_manager.cc:1909] T 308236ce104a41baa84cf1d8e6f68c0e P 99ead02915954b4fab23b033ad6d546b: Deleting consensus metadata
I20240503 02:00:14.640275 13962 ts_tablet_manager.cc:1909] T 308236ce104a41baa84cf1d8e6f68c0e P dec3e4a04d604f7eaaa3f077d081218a: Deleting consensus metadata
I20240503 02:00:14.640877 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet 308236ce104a41baa84cf1d8e6f68c0e (table rebalancer_dynamic_table_new_name [id=d3615bb8f356419e9ff1adfb5631ddc8]) successfully deleted
I20240503 02:00:14.641057 6300 catalog_manager.cc:4830] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): tablet 308236ce104a41baa84cf1d8e6f68c0e (table rebalancer_dynamic_table_new_name [id=d3615bb8f356419e9ff1adfb5631ddc8]) successfully deleted
I20240503 02:00:14.641381 6301 catalog_manager.cc:4830] TS 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157): tablet 308236ce104a41baa84cf1d8e6f68c0e (table rebalancer_dynamic_table_new_name [id=d3615bb8f356419e9ff1adfb5631ddc8]) successfully deleted
W20240503 02:00:14.727628 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:14.737404 6308 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:00:14.743417 8897 tablet_service.cc:1467] Processing CreateTablet for tablet aaec090e44114ab38e49cc24814453fe (DEFAULT_TABLE table=rebalancer_dynamic_table [id=66c21d25deab4cceb8c1e9ff09440b2b]), partition=
I20240503 02:00:14.743556 6569 tablet_service.cc:1467] Processing CreateTablet for tablet aaec090e44114ab38e49cc24814453fe (DEFAULT_TABLE table=rebalancer_dynamic_table [id=66c21d25deab4cceb8c1e9ff09440b2b]), partition=
I20240503 02:00:14.743690 8897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaec090e44114ab38e49cc24814453fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:14.743817 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaec090e44114ab38e49cc24814453fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:14.745245 9028 tablet_service.cc:1467] Processing CreateTablet for tablet aaec090e44114ab38e49cc24814453fe (DEFAULT_TABLE table=rebalancer_dynamic_table [id=66c21d25deab4cceb8c1e9ff09440b2b]), partition=
I20240503 02:00:14.745554 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaec090e44114ab38e49cc24814453fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:14.746312 13963 tablet_bootstrap.cc:492] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:14.746320 13937 tablet_bootstrap.cc:492] T aaec090e44114ab38e49cc24814453fe P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:14.747273 13937 tablet_bootstrap.cc:654] T aaec090e44114ab38e49cc24814453fe P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:14.747339 13963 tablet_bootstrap.cc:654] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:14.747831 13884 tablet_bootstrap.cc:492] T aaec090e44114ab38e49cc24814453fe P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:14.748667 13963 tablet_bootstrap.cc:492] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:14.748772 13963 ts_tablet_manager.cc:1367] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:14.749096 13884 tablet_bootstrap.cc:654] T aaec090e44114ab38e49cc24814453fe P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:14.749375 13937 tablet_bootstrap.cc:492] T aaec090e44114ab38e49cc24814453fe P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:14.749460 13937 ts_tablet_manager.cc:1367] T aaec090e44114ab38e49cc24814453fe P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:14.749629 13963 raft_consensus.cc:348] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:14.749872 13963 raft_consensus.cc:374] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:14.749920 13963 raft_consensus.cc:729] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:14.749928 13937 raft_consensus.cc:348] T aaec090e44114ab38e49cc24814453fe P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:14.750047 13937 raft_consensus.cc:374] T aaec090e44114ab38e49cc24814453fe P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:14.750087 13937 raft_consensus.cc:729] T aaec090e44114ab38e49cc24814453fe P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:14.750169 13963 consensus_queue.cc:260] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:14.750207 13937 consensus_queue.cc:260] T aaec090e44114ab38e49cc24814453fe P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:14.750473 13937 ts_tablet_manager.cc:1398] T aaec090e44114ab38e49cc24814453fe P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:14.751610 13963 ts_tablet_manager.cc:1398] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:00:14.751928 13884 tablet_bootstrap.cc:492] T aaec090e44114ab38e49cc24814453fe P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:14.752043 13884 ts_tablet_manager.cc:1367] T aaec090e44114ab38e49cc24814453fe P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:14.752586 13884 raft_consensus.cc:348] T aaec090e44114ab38e49cc24814453fe P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:14.752714 13884 raft_consensus.cc:374] T aaec090e44114ab38e49cc24814453fe P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:14.752753 13884 raft_consensus.cc:729] T aaec090e44114ab38e49cc24814453fe P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:14.752878 13884 consensus_queue.cc:260] T aaec090e44114ab38e49cc24814453fe P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:14.753152 13884 ts_tablet_manager.cc:1398] T aaec090e44114ab38e49cc24814453fe P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:14.827914 13969 consensus_queue.cc:1035] T d9c6a7e041014192b2a8e9bced641391 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:14.832425 13969 consensus_queue.cc:1035] T d9c6a7e041014192b2a8e9bced641391 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:14.887717 13969 consensus_queue.cc:1035] T 0fc624ba9c6940a3a7eedcf9a7d79dde P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:14.891083 13911 consensus_queue.cc:1035] T 0fc624ba9c6940a3a7eedcf9a7d79dde P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:14.923871 13943 consensus_queue.cc:1035] T 37190356d40d40c7aed6607b87c676fd P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:14.927531 13956 consensus_queue.cc:1035] T 37190356d40d40c7aed6607b87c676fd P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:15.045089 13978 raft_consensus.cc:482] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:15.045279 13978 raft_consensus.cc:504] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:15.045509 6309 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_extra_table_38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:15.045835 13978 leader_election.cc:288] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:15.046497 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaec090e44114ab38e49cc24814453fe" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:15.046509 8915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaec090e44114ab38e49cc24814453fe" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:15.046635 9048 raft_consensus.cc:2454] T aaec090e44114ab38e49cc24814453fe P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:15.046651 8915 raft_consensus.cc:2454] T aaec090e44114ab38e49cc24814453fe P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:15.047029 6525 leader_election.cc:302] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:15.047300 13978 raft_consensus.cc:2781] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:15.047351 13978 raft_consensus.cc:482] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:15.047376 13978 raft_consensus.cc:3037] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:15.048588 13978 raft_consensus.cc:504] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:15.049018 13978 leader_election.cc:288] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Requested vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:15.049515 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaec090e44114ab38e49cc24814453fe" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:15.049661 9048 raft_consensus.cc:3037] T aaec090e44114ab38e49cc24814453fe P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:15.050861 9048 raft_consensus.cc:2454] T aaec090e44114ab38e49cc24814453fe P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:15.050969 8915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaec090e44114ab38e49cc24814453fe" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:15.051091 8915 raft_consensus.cc:3037] T aaec090e44114ab38e49cc24814453fe P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:15.051235 6523 leader_election.cc:302] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:15.052194 8915 raft_consensus.cc:2454] T aaec090e44114ab38e49cc24814453fe P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:15.053357 9028 tablet_service.cc:1467] Processing CreateTablet for tablet 5ca282a13b714b63954c2812308904b4 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=f84ba9cbcf8f4e91af818979cee96b3c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:15.053364 8880 tablet_service.cc:1467] Processing CreateTablet for tablet 5ca282a13b714b63954c2812308904b4 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=f84ba9cbcf8f4e91af818979cee96b3c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:15.053689 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ca282a13b714b63954c2812308904b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:15.053709 8880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ca282a13b714b63954c2812308904b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:15.054262 6438 tablet_service.cc:1467] Processing CreateTablet for tablet 6f2e5973824b4d7a8b0d31faa622e6aa (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=f84ba9cbcf8f4e91af818979cee96b3c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:15.054545 6438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f2e5973824b4d7a8b0d31faa622e6aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:15.055925 13884 tablet_bootstrap.cc:492] T 5ca282a13b714b63954c2812308904b4 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:15.055959 6436 tablet_service.cc:1467] Processing CreateTablet for tablet 01b63d6a79314a86bd64f7618c86716d (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=f84ba9cbcf8f4e91af818979cee96b3c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:15.056219 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01b63d6a79314a86bd64f7618c86716d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:15.056954 13884 tablet_bootstrap.cc:654] T 5ca282a13b714b63954c2812308904b4 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:15.057562 9028 tablet_service.cc:1467] Processing CreateTablet for tablet 6f2e5973824b4d7a8b0d31faa622e6aa (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=f84ba9cbcf8f4e91af818979cee96b3c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:15.057823 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f2e5973824b4d7a8b0d31faa622e6aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:15.058452 13937 tablet_bootstrap.cc:492] T 5ca282a13b714b63954c2812308904b4 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:15.058586 13884 tablet_bootstrap.cc:492] T 5ca282a13b714b63954c2812308904b4 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:15.058660 13884 ts_tablet_manager.cc:1367] T 5ca282a13b714b63954c2812308904b4 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:15.059034 13978 raft_consensus.cc:2781] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:15.059134 13884 raft_consensus.cc:348] T 5ca282a13b714b63954c2812308904b4 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:15.059207 13978 raft_consensus.cc:686] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Becoming Leader. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Running, Role: LEADER
I20240503 02:00:15.059239 13884 raft_consensus.cc:374] T 5ca282a13b714b63954c2812308904b4 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:15.059270 13884 raft_consensus.cc:729] T 5ca282a13b714b63954c2812308904b4 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:15.059347 13978 consensus_queue.cc:237] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:15.059379 13884 consensus_queue.cc:260] T 5ca282a13b714b63954c2812308904b4 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:15.059468 13937 tablet_bootstrap.cc:654] T 5ca282a13b714b63954c2812308904b4 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:15.059669 13884 ts_tablet_manager.cc:1398] T 5ca282a13b714b63954c2812308904b4 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:15.060227 6569 tablet_service.cc:1467] Processing CreateTablet for tablet 01b63d6a79314a86bd64f7618c86716d (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=f84ba9cbcf8f4e91af818979cee96b3c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:15.060449 6567 tablet_service.cc:1467] Processing CreateTablet for tablet 5ca282a13b714b63954c2812308904b4 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=f84ba9cbcf8f4e91af818979cee96b3c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:15.060614 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01b63d6a79314a86bd64f7618c86716d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:15.053357 8897 tablet_service.cc:1467] Processing CreateTablet for tablet 01b63d6a79314a86bd64f7618c86716d (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=f84ba9cbcf8f4e91af818979cee96b3c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:15.060767 6567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ca282a13b714b63954c2812308904b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:15.060899 8897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01b63d6a79314a86bd64f7618c86716d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:15.061363 13884 tablet_bootstrap.cc:492] T 6f2e5973824b4d7a8b0d31faa622e6aa P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:15.062307 13884 tablet_bootstrap.cc:654] T 6f2e5973824b4d7a8b0d31faa622e6aa P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:15.063079 13963 tablet_bootstrap.cc:492] T 5ca282a13b714b63954c2812308904b4 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:15.064007 13937 tablet_bootstrap.cc:492] T 5ca282a13b714b63954c2812308904b4 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:15.064100 13937 ts_tablet_manager.cc:1367] T 5ca282a13b714b63954c2812308904b4 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:00:15.064241 13963 tablet_bootstrap.cc:654] T 5ca282a13b714b63954c2812308904b4 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:15.064641 13937 raft_consensus.cc:348] T 5ca282a13b714b63954c2812308904b4 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:15.064769 13937 raft_consensus.cc:374] T 5ca282a13b714b63954c2812308904b4 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:15.064800 13937 raft_consensus.cc:729] T 5ca282a13b714b63954c2812308904b4 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:15.064916 13937 consensus_queue.cc:260] T 5ca282a13b714b63954c2812308904b4 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:15.065204 13937 ts_tablet_manager.cc:1398] T 5ca282a13b714b63954c2812308904b4 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:15.065371 13937 tablet_bootstrap.cc:492] T 01b63d6a79314a86bd64f7618c86716d P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:15.066399 13937 tablet_bootstrap.cc:654] T 01b63d6a79314a86bd64f7618c86716d P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:15.066676 9158 tablet_service.cc:1467] Processing CreateTablet for tablet 6f2e5973824b4d7a8b0d31faa622e6aa (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=f84ba9cbcf8f4e91af818979cee96b3c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:15.066970 9158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f2e5973824b4d7a8b0d31faa622e6aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:15.067739 13937 tablet_bootstrap.cc:492] T 01b63d6a79314a86bd64f7618c86716d P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:15.067826 13937 ts_tablet_manager.cc:1367] T 01b63d6a79314a86bd64f7618c86716d P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:15.068353 13937 raft_consensus.cc:348] T 01b63d6a79314a86bd64f7618c86716d P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:15.068488 13937 raft_consensus.cc:374] T 01b63d6a79314a86bd64f7618c86716d P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:15.068519 13937 raft_consensus.cc:729] T 01b63d6a79314a86bd64f7618c86716d P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:15.068639 13937 consensus_queue.cc:260] T 01b63d6a79314a86bd64f7618c86716d P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:15.068928 13937 ts_tablet_manager.cc:1398] T 01b63d6a79314a86bd64f7618c86716d P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:15.069126 13963 tablet_bootstrap.cc:492] T 5ca282a13b714b63954c2812308904b4 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:15.069242 13963 ts_tablet_manager.cc:1367] T 5ca282a13b714b63954c2812308904b4 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:00:15.069710 13963 raft_consensus.cc:348] T 5ca282a13b714b63954c2812308904b4 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:15.069820 13963 raft_consensus.cc:374] T 5ca282a13b714b63954c2812308904b4 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:15.069852 13963 raft_consensus.cc:729] T 5ca282a13b714b63954c2812308904b4 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:15.069988 13963 consensus_queue.cc:260] T 5ca282a13b714b63954c2812308904b4 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:15.070295 13963 ts_tablet_manager.cc:1398] T 5ca282a13b714b63954c2812308904b4 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:15.070493 13963 tablet_bootstrap.cc:492] T 01b63d6a79314a86bd64f7618c86716d P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:15.071184 13992 tablet_bootstrap.cc:492] T 6f2e5973824b4d7a8b0d31faa622e6aa P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:15.071533 13963 tablet_bootstrap.cc:654] T 01b63d6a79314a86bd64f7618c86716d P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:15.072237 13992 tablet_bootstrap.cc:654] T 6f2e5973824b4d7a8b0d31faa622e6aa P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:15.074033 13985 tablet_bootstrap.cc:492] T 6f2e5973824b4d7a8b0d31faa622e6aa P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:15.075141 13985 tablet_bootstrap.cc:654] T 6f2e5973824b4d7a8b0d31faa622e6aa P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:15.075188 6309 catalog_manager.cc:5484] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:15.076663 13985 tablet_bootstrap.cc:492] T 6f2e5973824b4d7a8b0d31faa622e6aa P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:15.076864 13985 ts_tablet_manager.cc:1367] T 6f2e5973824b4d7a8b0d31faa622e6aa P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:15.077266 13963 tablet_bootstrap.cc:492] T 01b63d6a79314a86bd64f7618c86716d P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:15.077374 13963 ts_tablet_manager.cc:1367] T 01b63d6a79314a86bd64f7618c86716d P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:00:15.077425 13985 raft_consensus.cc:348] T 6f2e5973824b4d7a8b0d31faa622e6aa P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:15.077534 13985 raft_consensus.cc:374] T 6f2e5973824b4d7a8b0d31faa622e6aa P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:15.077562 13985 raft_consensus.cc:729] T 6f2e5973824b4d7a8b0d31faa622e6aa P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:15.077703 13985 consensus_queue.cc:260] T 6f2e5973824b4d7a8b0d31faa622e6aa P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:15.077855 13963 raft_consensus.cc:348] T 01b63d6a79314a86bd64f7618c86716d P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:15.077956 13963 raft_consensus.cc:374] T 01b63d6a79314a86bd64f7618c86716d P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:15.077987 13963 raft_consensus.cc:729] T 01b63d6a79314a86bd64f7618c86716d P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:15.078002 13985 ts_tablet_manager.cc:1398] T 6f2e5973824b4d7a8b0d31faa622e6aa P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:15.078104 13963 consensus_queue.cc:260] T 01b63d6a79314a86bd64f7618c86716d P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:15.078387 13963 ts_tablet_manager.cc:1398] T 01b63d6a79314a86bd64f7618c86716d P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:15.078572 13985 tablet_bootstrap.cc:492] T 01b63d6a79314a86bd64f7618c86716d P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:15.081099 13884 tablet_bootstrap.cc:492] T 6f2e5973824b4d7a8b0d31faa622e6aa P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:15.081236 13884 ts_tablet_manager.cc:1367] T 6f2e5973824b4d7a8b0d31faa622e6aa P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20240503 02:00:15.081511 13985 tablet_bootstrap.cc:654] T 01b63d6a79314a86bd64f7618c86716d P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:15.081772 13884 raft_consensus.cc:348] T 6f2e5973824b4d7a8b0d31faa622e6aa P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:15.081889 13884 raft_consensus.cc:374] T 6f2e5973824b4d7a8b0d31faa622e6aa P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:15.081918 13884 raft_consensus.cc:729] T 6f2e5973824b4d7a8b0d31faa622e6aa P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:15.082049 13884 consensus_queue.cc:260] T 6f2e5973824b4d7a8b0d31faa622e6aa P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:15.082334 13884 ts_tablet_manager.cc:1398] T 6f2e5973824b4d7a8b0d31faa622e6aa P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:15.084327 13992 tablet_bootstrap.cc:492] T 6f2e5973824b4d7a8b0d31faa622e6aa P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:15.084555 13992 ts_tablet_manager.cc:1367] T 6f2e5973824b4d7a8b0d31faa622e6aa P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.002s
I20240503 02:00:15.084699 13985 tablet_bootstrap.cc:492] T 01b63d6a79314a86bd64f7618c86716d P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:15.084812 13985 ts_tablet_manager.cc:1367] T 01b63d6a79314a86bd64f7618c86716d P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:00:15.085175 13992 raft_consensus.cc:348] T 6f2e5973824b4d7a8b0d31faa622e6aa P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:15.085379 13992 raft_consensus.cc:374] T 6f2e5973824b4d7a8b0d31faa622e6aa P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:15.085472 13985 raft_consensus.cc:348] T 01b63d6a79314a86bd64f7618c86716d P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:15.085579 13985 raft_consensus.cc:374] T 01b63d6a79314a86bd64f7618c86716d P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:15.085608 13985 raft_consensus.cc:729] T 01b63d6a79314a86bd64f7618c86716d P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:15.085738 13985 consensus_queue.cc:260] T 01b63d6a79314a86bd64f7618c86716d P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:15.086014 13985 ts_tablet_manager.cc:1398] T 01b63d6a79314a86bd64f7618c86716d P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:15.086318 13992 raft_consensus.cc:729] T 6f2e5973824b4d7a8b0d31faa622e6aa P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:15.086510 13992 consensus_queue.cc:260] T 6f2e5973824b4d7a8b0d31faa622e6aa P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:15.086925 13992 ts_tablet_manager.cc:1398] T 6f2e5973824b4d7a8b0d31faa622e6aa P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
W20240503 02:00:15.203860 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:15.213878 6312 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:15.219329 9048 raft_consensus.cc:1261] T aaec090e44114ab38e49cc24814453fe P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:15.219503 8916 raft_consensus.cc:1261] T aaec090e44114ab38e49cc24814453fe P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:15.219957 13954 consensus_queue.cc:1035] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:15.220353 13997 consensus_queue.cc:1035] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:15.223644 14001 tablet.cc:1722] T aaec090e44114ab38e49cc24814453fe P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:15.224112 14001 tablet.cc:1620] T aaec090e44114ab38e49cc24814453fe P 99ead02915954b4fab23b033ad6d546b: MemRowSet was empty: no flush needed.
I20240503 02:00:15.224539 6309 catalog_manager.cc:5576] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211) does not have the latest schema for tablet aaec090e44114ab38e49cc24814453fe (table rebalancer_dynamic_table [id=66c21d25deab4cceb8c1e9ff09440b2b]). Expected version 1 got 0
I20240503 02:00:15.225278 6308 catalog_manager.cc:5576] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863) does not have the latest schema for tablet aaec090e44114ab38e49cc24814453fe (table rebalancer_dynamic_table [id=66c21d25deab4cceb8c1e9ff09440b2b]). Expected version 1 got 0
I20240503 02:00:15.225908 6312 catalog_manager.cc:5576] TS 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157) does not have the latest schema for tablet aaec090e44114ab38e49cc24814453fe (table rebalancer_dynamic_table [id=66c21d25deab4cceb8c1e9ff09440b2b]). Expected version 1 got 0
I20240503 02:00:15.226426 14000 tablet.cc:1722] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:15.227753 14000 tablet.cc:1620] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f: MemRowSet was empty: no flush needed.
I20240503 02:00:15.228147 14002 tablet.cc:1722] T aaec090e44114ab38e49cc24814453fe P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:15.228492 14002 tablet.cc:1620] T aaec090e44114ab38e49cc24814453fe P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
I20240503 02:00:15.230638 6308 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=66c21d25deab4cceb8c1e9ff09440b2b] alter complete (version 1)
I20240503 02:00:15.232579 6308 catalog_manager.cc:5576] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211) does not have the latest schema for tablet aaec090e44114ab38e49cc24814453fe (table rebalancer_dynamic_table [id=66c21d25deab4cceb8c1e9ff09440b2b]). Expected version 1 got 0
I20240503 02:00:15.238692 13991 raft_consensus.cc:482] T 01b63d6a79314a86bd64f7618c86716d P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:15.238822 13991 raft_consensus.cc:504] T 01b63d6a79314a86bd64f7618c86716d P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:15.239284 13991 leader_election.cc:288] T 01b63d6a79314a86bd64f7618c86716d P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:15.239675 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b63d6a79314a86bd64f7618c86716d" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:15.239677 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b63d6a79314a86bd64f7618c86716d" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 02:00:15.239799 8916 raft_consensus.cc:2454] T 01b63d6a79314a86bd64f7618c86716d P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:15.239812 6589 raft_consensus.cc:2454] T 01b63d6a79314a86bd64f7618c86716d P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:15.240075 6394 leader_election.cc:302] T 01b63d6a79314a86bd64f7618c86716d P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9942306855404cfd891adf227dd65eb2, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:15.240242 13991 raft_consensus.cc:2781] T 01b63d6a79314a86bd64f7618c86716d P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:15.240293 13991 raft_consensus.cc:482] T 01b63d6a79314a86bd64f7618c86716d P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:15.240322 13991 raft_consensus.cc:3037] T 01b63d6a79314a86bd64f7618c86716d P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:15.241400 13991 raft_consensus.cc:504] T 01b63d6a79314a86bd64f7618c86716d P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:15.241782 13991 leader_election.cc:288] T 01b63d6a79314a86bd64f7618c86716d P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Requested vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:15.242087 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b63d6a79314a86bd64f7618c86716d" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:15.242173 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b63d6a79314a86bd64f7618c86716d" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 02:00:15.242234 8916 raft_consensus.cc:3037] T 01b63d6a79314a86bd64f7618c86716d P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:15.242282 6589 raft_consensus.cc:3037] T 01b63d6a79314a86bd64f7618c86716d P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:15.243299 6589 raft_consensus.cc:2454] T 01b63d6a79314a86bd64f7618c86716d P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:15.243485 8916 raft_consensus.cc:2454] T 01b63d6a79314a86bd64f7618c86716d P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:15.243775 6393 leader_election.cc:302] T 01b63d6a79314a86bd64f7618c86716d P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:15.244030 13991 raft_consensus.cc:2781] T 01b63d6a79314a86bd64f7618c86716d P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:15.244125 13991 raft_consensus.cc:686] T 01b63d6a79314a86bd64f7618c86716d P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Becoming Leader. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Running, Role: LEADER
I20240503 02:00:15.244264 13991 consensus_queue.cc:237] T 01b63d6a79314a86bd64f7618c86716d P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:15.246294 6312 catalog_manager.cc:5484] T 01b63d6a79314a86bd64f7618c86716d P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: term changed from 0 to 1, leader changed from <none> to dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129). New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:15.248795 13956 raft_consensus.cc:482] T 6f2e5973824b4d7a8b0d31faa622e6aa P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:15.248911 13956 raft_consensus.cc:504] T 6f2e5973824b4d7a8b0d31faa622e6aa P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:15.249281 13956 leader_election.cc:288] T 6f2e5973824b4d7a8b0d31faa622e6aa P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 02:00:15.249752 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f2e5973824b4d7a8b0d31faa622e6aa" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 02:00:15.249895 6458 raft_consensus.cc:2454] T 6f2e5973824b4d7a8b0d31faa622e6aa P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:15.250128 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f2e5973824b4d7a8b0d31faa622e6aa" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:15.250241 9179 raft_consensus.cc:2454] T 6f2e5973824b4d7a8b0d31faa622e6aa P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:15.250299 8981 leader_election.cc:302] T 6f2e5973824b4d7a8b0d31faa622e6aa P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99ead02915954b4fab23b033ad6d546b, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:15.250653 13956 raft_consensus.cc:2781] T 6f2e5973824b4d7a8b0d31faa622e6aa P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:15.250720 13956 raft_consensus.cc:482] T 6f2e5973824b4d7a8b0d31faa622e6aa P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:15.250764 13956 raft_consensus.cc:3037] T 6f2e5973824b4d7a8b0d31faa622e6aa P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:15.251708 13956 raft_consensus.cc:504] T 6f2e5973824b4d7a8b0d31faa622e6aa P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:15.252075 13956 leader_election.cc:288] T 6f2e5973824b4d7a8b0d31faa622e6aa P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Requested vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 02:00:15.252501 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f2e5973824b4d7a8b0d31faa622e6aa" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:15.252501 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f2e5973824b4d7a8b0d31faa622e6aa" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 02:00:15.252624 9179 raft_consensus.cc:3037] T 6f2e5973824b4d7a8b0d31faa622e6aa P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:15.252625 6458 raft_consensus.cc:3037] T 6f2e5973824b4d7a8b0d31faa622e6aa P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:15.253624 6458 raft_consensus.cc:2454] T 6f2e5973824b4d7a8b0d31faa622e6aa P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:15.253623 9179 raft_consensus.cc:2454] T 6f2e5973824b4d7a8b0d31faa622e6aa P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:15.253957 8981 leader_election.cc:302] T 6f2e5973824b4d7a8b0d31faa622e6aa P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:15.254156 13956 raft_consensus.cc:2781] T 6f2e5973824b4d7a8b0d31faa622e6aa P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:15.254242 13956 raft_consensus.cc:686] T 6f2e5973824b4d7a8b0d31faa622e6aa P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Becoming Leader. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Running, Role: LEADER
I20240503 02:00:15.254390 13956 consensus_queue.cc:237] T 6f2e5973824b4d7a8b0d31faa622e6aa P 99ead02915954b4fab23b033ad6d546b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:15.256824 6308 catalog_manager.cc:5484] T 6f2e5973824b4d7a8b0d31faa622e6aa P 99ead02915954b4fab23b033ad6d546b reported cstate change: term changed from 0 to 1, leader changed from <none> to 99ead02915954b4fab23b033ad6d546b (127.0.99.134). New cstate: current_term: 1 leader_uuid: "99ead02915954b4fab23b033ad6d546b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:15.292024 13102 raft_consensus.cc:482] T 5ca282a13b714b63954c2812308904b4 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:15.292158 13102 raft_consensus.cc:504] T 5ca282a13b714b63954c2812308904b4 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:15.292697 13102 leader_election.cc:288] T 5ca282a13b714b63954c2812308904b4 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:15.293143 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ca282a13b714b63954c2812308904b4" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 02:00:15.293143 9039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ca282a13b714b63954c2812308904b4" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:15.293298 9039 raft_consensus.cc:2454] T 5ca282a13b714b63954c2812308904b4 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 0.
I20240503 02:00:15.293298 6589 raft_consensus.cc:2454] T 5ca282a13b714b63954c2812308904b4 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 0.
I20240503 02:00:15.293583 8852 leader_election.cc:302] T 5ca282a13b714b63954c2812308904b4 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:15.293757 13102 raft_consensus.cc:2781] T 5ca282a13b714b63954c2812308904b4 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:15.293817 13102 raft_consensus.cc:482] T 5ca282a13b714b63954c2812308904b4 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:15.293844 13102 raft_consensus.cc:3037] T 5ca282a13b714b63954c2812308904b4 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:15.294896 13102 raft_consensus.cc:504] T 5ca282a13b714b63954c2812308904b4 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:15.295289 13102 leader_election.cc:288] T 5ca282a13b714b63954c2812308904b4 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 election: Requested vote from peers 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:15.295629 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ca282a13b714b63954c2812308904b4" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 02:00:15.295751 6589 raft_consensus.cc:3037] T 5ca282a13b714b63954c2812308904b4 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:15.295725 9039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ca282a13b714b63954c2812308904b4" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:15.295876 9039 raft_consensus.cc:3037] T 5ca282a13b714b63954c2812308904b4 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:15.296705 6589 raft_consensus.cc:2454] T 5ca282a13b714b63954c2812308904b4 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 1.
I20240503 02:00:15.296875 9039 raft_consensus.cc:2454] T 5ca282a13b714b63954c2812308904b4 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 1.
I20240503 02:00:15.296995 8852 leader_election.cc:302] T 5ca282a13b714b63954c2812308904b4 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:15.297168 13102 raft_consensus.cc:2781] T 5ca282a13b714b63954c2812308904b4 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:15.297255 13102 raft_consensus.cc:686] T 5ca282a13b714b63954c2812308904b4 P 9942306855404cfd891adf227dd65eb2 [term 1 LEADER]: Becoming Leader. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Running, Role: LEADER
I20240503 02:00:15.297395 13102 consensus_queue.cc:237] T 5ca282a13b714b63954c2812308904b4 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:15.299533 6308 catalog_manager.cc:5484] T 5ca282a13b714b63954c2812308904b4 P 9942306855404cfd891adf227dd65eb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9942306855404cfd891adf227dd65eb2 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "9942306855404cfd891adf227dd65eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:15.336645 6312 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:15.342357 14000 tablet.cc:1722] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:15.342721 14000 tablet.cc:1620] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f: MemRowSet was empty: no flush needed.
I20240503 02:00:15.342720 14001 tablet.cc:1722] T aaec090e44114ab38e49cc24814453fe P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:15.343060 14001 tablet.cc:1620] T aaec090e44114ab38e49cc24814453fe P 99ead02915954b4fab23b033ad6d546b: MemRowSet was empty: no flush needed.
I20240503 02:00:15.343438 14002 tablet.cc:1722] T aaec090e44114ab38e49cc24814453fe P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:15.343731 14002 tablet.cc:1620] T aaec090e44114ab38e49cc24814453fe P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
I20240503 02:00:15.349637 6309 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=66c21d25deab4cceb8c1e9ff09440b2b] alter complete (version 2)
I20240503 02:00:15.452988 6308 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:15.458621 14000 tablet.cc:1722] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:15.459162 14001 tablet.cc:1722] T aaec090e44114ab38e49cc24814453fe P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:15.459447 14002 tablet.cc:1722] T aaec090e44114ab38e49cc24814453fe P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:15.464571 6309 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=66c21d25deab4cceb8c1e9ff09440b2b] alter complete (version 3)
I20240503 02:00:15.568696 6309 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:15.568892 6309 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:15.570972 6309 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet aaec090e44114ab38e49cc24814453fe
I20240503 02:00:15.571699 9028 tablet_service.cc:1514] Processing DeleteTablet for tablet aaec090e44114ab38e49cc24814453fe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:15 UTC) from {username='slave'} at 127.0.0.1:52584
I20240503 02:00:15.571699 8897 tablet_service.cc:1514] Processing DeleteTablet for tablet aaec090e44114ab38e49cc24814453fe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:15 UTC) from {username='slave'} at 127.0.0.1:40296
I20240503 02:00:15.571985 6569 tablet_service.cc:1514] Processing DeleteTablet for tablet aaec090e44114ab38e49cc24814453fe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:15 UTC) from {username='slave'} at 127.0.0.1:59846
I20240503 02:00:15.572063 14007 tablet_replica.cc:330] T aaec090e44114ab38e49cc24814453fe P 9942306855404cfd891adf227dd65eb2: stopping tablet replica
I20240503 02:00:15.572229 14007 raft_consensus.cc:2229] T aaec090e44114ab38e49cc24814453fe P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:15.572309 14009 tablet_replica.cc:330] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 02:00:15.572410 14007 raft_consensus.cc:2258] T aaec090e44114ab38e49cc24814453fe P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:15.572472 14009 raft_consensus.cc:2229] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:15.572774 14009 raft_consensus.cc:2258] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:15.573262 14007 ts_tablet_manager.cc:1875] T aaec090e44114ab38e49cc24814453fe P 9942306855404cfd891adf227dd65eb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:15.573349 14009 ts_tablet_manager.cc:1875] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:15.573419 14008 tablet_replica.cc:330] T aaec090e44114ab38e49cc24814453fe P 99ead02915954b4fab23b033ad6d546b: stopping tablet replica
I20240503 02:00:15.573575 14008 raft_consensus.cc:2229] T aaec090e44114ab38e49cc24814453fe P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:15.573700 14008 raft_consensus.cc:2258] T aaec090e44114ab38e49cc24814453fe P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:15.574219 14008 ts_tablet_manager.cc:1875] T aaec090e44114ab38e49cc24814453fe P 99ead02915954b4fab23b033ad6d546b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:15.576042 14007 ts_tablet_manager.cc:1888] T aaec090e44114ab38e49cc24814453fe P 9942306855404cfd891adf227dd65eb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:15.576051 14009 ts_tablet_manager.cc:1888] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:15.576138 14007 log.cc:1196] T aaec090e44114ab38e49cc24814453fe P 9942306855404cfd891adf227dd65eb2: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/aaec090e44114ab38e49cc24814453fe
I20240503 02:00:15.576138 14009 log.cc:1196] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/aaec090e44114ab38e49cc24814453fe
I20240503 02:00:15.576177 14008 ts_tablet_manager.cc:1888] T aaec090e44114ab38e49cc24814453fe P 99ead02915954b4fab23b033ad6d546b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:15.576448 14007 ts_tablet_manager.cc:1909] T aaec090e44114ab38e49cc24814453fe P 9942306855404cfd891adf227dd65eb2: Deleting consensus metadata
I20240503 02:00:15.576469 14008 log.cc:1196] T aaec090e44114ab38e49cc24814453fe P 99ead02915954b4fab23b033ad6d546b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/aaec090e44114ab38e49cc24814453fe
I20240503 02:00:15.576534 14009 ts_tablet_manager.cc:1909] T aaec090e44114ab38e49cc24814453fe P 80fd18ef4d604c86b909d5c008a46b4f: Deleting consensus metadata
I20240503 02:00:15.576958 14008 ts_tablet_manager.cc:1909] T aaec090e44114ab38e49cc24814453fe P 99ead02915954b4fab23b033ad6d546b: Deleting consensus metadata
I20240503 02:00:15.577279 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet aaec090e44114ab38e49cc24814453fe (table rebalancer_dynamic_table_new_name [id=66c21d25deab4cceb8c1e9ff09440b2b]) successfully deleted
I20240503 02:00:15.577392 6303 catalog_manager.cc:4830] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211): tablet aaec090e44114ab38e49cc24814453fe (table rebalancer_dynamic_table_new_name [id=66c21d25deab4cceb8c1e9ff09440b2b]) successfully deleted
I20240503 02:00:15.577641 6301 catalog_manager.cc:4830] TS 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157): tablet aaec090e44114ab38e49cc24814453fe (table rebalancer_dynamic_table_new_name [id=66c21d25deab4cceb8c1e9ff09440b2b]) successfully deleted
I20240503 02:00:15.629885 13829 consensus_queue.cc:1035] T 01b63d6a79314a86bd64f7618c86716d P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:15.633460 13991 consensus_queue.cc:1035] T 01b63d6a79314a86bd64f7618c86716d P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:15.672569 6312 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
W20240503 02:00:15.676455 8982 consensus_peers.cc:485] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:00:15.680270 6829 tablet_service.cc:1467] Processing CreateTablet for tablet 61ea3cc712c54205955c7e585d02025f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=73b1d9c7686d4af099d8797f3009a419]), partition=
I20240503 02:00:15.680258 9028 tablet_service.cc:1467] Processing CreateTablet for tablet 61ea3cc712c54205955c7e585d02025f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=73b1d9c7686d4af099d8797f3009a419]), partition=
I20240503 02:00:15.680258 6699 tablet_service.cc:1467] Processing CreateTablet for tablet 61ea3cc712c54205955c7e585d02025f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=73b1d9c7686d4af099d8797f3009a419]), partition=
I20240503 02:00:15.680579 6699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61ea3cc712c54205955c7e585d02025f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:15.680578 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61ea3cc712c54205955c7e585d02025f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:15.680578 6829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61ea3cc712c54205955c7e585d02025f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:15.683624 14014 tablet_bootstrap.cc:492] T 61ea3cc712c54205955c7e585d02025f P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:15.683671 14015 tablet_bootstrap.cc:492] T 61ea3cc712c54205955c7e585d02025f P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:15.684055 14016 tablet_bootstrap.cc:492] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:15.684828 14014 tablet_bootstrap.cc:654] T 61ea3cc712c54205955c7e585d02025f P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:15.685510 14015 tablet_bootstrap.cc:654] T 61ea3cc712c54205955c7e585d02025f P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:15.685660 14016 tablet_bootstrap.cc:654] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:15.687355 14016 tablet_bootstrap.cc:492] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:15.687465 14016 ts_tablet_manager.cc:1367] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:15.687940 14014 tablet_bootstrap.cc:492] T 61ea3cc712c54205955c7e585d02025f P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:15.687889 14016 raft_consensus.cc:348] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:15.687989 14016 raft_consensus.cc:374] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:15.688009 14016 raft_consensus.cc:729] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:15.688035 14014 ts_tablet_manager.cc:1367] T 61ea3cc712c54205955c7e585d02025f P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:15.688119 14016 consensus_queue.cc:260] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:15.688417 14016 ts_tablet_manager.cc:1398] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:15.688511 14014 raft_consensus.cc:348] T 61ea3cc712c54205955c7e585d02025f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:15.688617 14014 raft_consensus.cc:374] T 61ea3cc712c54205955c7e585d02025f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:15.688647 14014 raft_consensus.cc:729] T 61ea3cc712c54205955c7e585d02025f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:15.688764 14014 consensus_queue.cc:260] T 61ea3cc712c54205955c7e585d02025f P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:15.689038 14015 tablet_bootstrap.cc:492] T 61ea3cc712c54205955c7e585d02025f P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:15.689054 14014 ts_tablet_manager.cc:1398] T 61ea3cc712c54205955c7e585d02025f P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:15.689139 14015 ts_tablet_manager.cc:1367] T 61ea3cc712c54205955c7e585d02025f P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:00:15.689702 14015 raft_consensus.cc:348] T 61ea3cc712c54205955c7e585d02025f P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:15.689833 14015 raft_consensus.cc:374] T 61ea3cc712c54205955c7e585d02025f P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:15.689868 14015 raft_consensus.cc:729] T 61ea3cc712c54205955c7e585d02025f P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:15.689998 14015 consensus_queue.cc:260] T 61ea3cc712c54205955c7e585d02025f P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:15.690327 14015 ts_tablet_manager.cc:1398] T 61ea3cc712c54205955c7e585d02025f P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240503 02:00:15.698967 6785 consensus_peers.cc:485] T c3485701bf3b48f5ae2e1568afeed133 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:00:15.711455 13956 raft_consensus.cc:482] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:15.711589 13956 raft_consensus.cc:504] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:15.712083 13956 leader_election.cc:288] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:15.712553 6719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61ea3cc712c54205955c7e585d02025f" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 02:00:15.712612 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61ea3cc712c54205955c7e585d02025f" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 02:00:15.712714 6719 raft_consensus.cc:2454] T 61ea3cc712c54205955c7e585d02025f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:15.712726 6849 raft_consensus.cc:2454] T 61ea3cc712c54205955c7e585d02025f P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:15.712996 8984 leader_election.cc:302] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:15.713169 13956 raft_consensus.cc:2781] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:15.713229 13956 raft_consensus.cc:482] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:15.713255 13956 raft_consensus.cc:3037] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:15.714259 13956 raft_consensus.cc:504] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:15.714599 13956 leader_election.cc:288] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Requested vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:15.715015 6719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61ea3cc712c54205955c7e585d02025f" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 02:00:15.715016 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61ea3cc712c54205955c7e585d02025f" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 02:00:15.715147 6849 raft_consensus.cc:3037] T 61ea3cc712c54205955c7e585d02025f P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:15.715145 6719 raft_consensus.cc:3037] T 61ea3cc712c54205955c7e585d02025f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:15.715929 6849 raft_consensus.cc:2454] T 61ea3cc712c54205955c7e585d02025f P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:15.715929 6719 raft_consensus.cc:2454] T 61ea3cc712c54205955c7e585d02025f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:15.716328 8984 leader_election.cc:302] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99ead02915954b4fab23b033ad6d546b, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:15.716647 13956 raft_consensus.cc:2781] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:15.716738 13956 raft_consensus.cc:686] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Becoming Leader. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Running, Role: LEADER
I20240503 02:00:15.716884 13956 consensus_queue.cc:237] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:15.719502 6308 catalog_manager.cc:5484] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b reported cstate change: term changed from 0 to 1, leader changed from <none> to 99ead02915954b4fab23b033ad6d546b (127.0.99.134). New cstate: current_term: 1 leader_uuid: "99ead02915954b4fab23b033ad6d546b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:15.725975 13102 consensus_queue.cc:1035] T 5ca282a13b714b63954c2812308904b4 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:15.726390 13900 consensus_queue.cc:1035] T 6f2e5973824b4d7a8b0d31faa622e6aa P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:15.729998 13943 consensus_queue.cc:1035] T 6f2e5973824b4d7a8b0d31faa622e6aa P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:15.730335 13910 consensus_queue.cc:1035] T 5ca282a13b714b63954c2812308904b4 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:00:15.732780 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:15.803247 6312 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_extra_table_40"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:15.809813 6699 tablet_service.cc:1467] Processing CreateTablet for tablet 0f53daddf5fd44a3a9d8febcf0792ce8 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=4b0e62f32b2e4d32b6c0c4ef08932d08]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:15.810143 6699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f53daddf5fd44a3a9d8febcf0792ce8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:15.810724 9158 tablet_service.cc:1467] Processing CreateTablet for tablet 350fc92a3e034564800e8fbd8f6914a5 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=4b0e62f32b2e4d32b6c0c4ef08932d08]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:15.810900 6438 tablet_service.cc:1467] Processing CreateTablet for tablet 4966bb684477438e96d337e2320810f9 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=4b0e62f32b2e4d32b6c0c4ef08932d08]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:15.810982 9158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 350fc92a3e034564800e8fbd8f6914a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:15.811175 6438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4966bb684477438e96d337e2320810f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:15.812521 14014 tablet_bootstrap.cc:492] T 0f53daddf5fd44a3a9d8febcf0792ce8 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:15.812788 6436 tablet_service.cc:1467] Processing CreateTablet for tablet 0f53daddf5fd44a3a9d8febcf0792ce8 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=4b0e62f32b2e4d32b6c0c4ef08932d08]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:15.813058 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f53daddf5fd44a3a9d8febcf0792ce8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:15.813738 14014 tablet_bootstrap.cc:654] T 0f53daddf5fd44a3a9d8febcf0792ce8 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:15.814190 6569 tablet_service.cc:1467] Processing CreateTablet for tablet 4966bb684477438e96d337e2320810f9 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=4b0e62f32b2e4d32b6c0c4ef08932d08]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:15.814453 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4966bb684477438e96d337e2320810f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:15.814550 9028 tablet_service.cc:1467] Processing CreateTablet for tablet 0f53daddf5fd44a3a9d8febcf0792ce8 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=4b0e62f32b2e4d32b6c0c4ef08932d08]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:15.814600 14027 tablet_bootstrap.cc:492] T 350fc92a3e034564800e8fbd8f6914a5 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:15.814786 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f53daddf5fd44a3a9d8febcf0792ce8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:15.815768 14027 tablet_bootstrap.cc:654] T 350fc92a3e034564800e8fbd8f6914a5 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:15.816882 14016 tablet_bootstrap.cc:492] T 0f53daddf5fd44a3a9d8febcf0792ce8 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:15.816962 14030 tablet_bootstrap.cc:492] T 4966bb684477438e96d337e2320810f9 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:15.816978 9025 tablet_service.cc:1467] Processing CreateTablet for tablet 4966bb684477438e96d337e2320810f9 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=4b0e62f32b2e4d32b6c0c4ef08932d08]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:15.817195 9025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4966bb684477438e96d337e2320810f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:15.817801 14016 tablet_bootstrap.cc:654] T 0f53daddf5fd44a3a9d8febcf0792ce8 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:15.817991 6699 tablet_service.cc:1467] Processing CreateTablet for tablet 350fc92a3e034564800e8fbd8f6914a5 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=4b0e62f32b2e4d32b6c0c4ef08932d08]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:15.818289 6699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 350fc92a3e034564800e8fbd8f6914a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:15.818679 14030 tablet_bootstrap.cc:654] T 4966bb684477438e96d337e2320810f9 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:15.819365 6829 tablet_service.cc:1467] Processing CreateTablet for tablet 350fc92a3e034564800e8fbd8f6914a5 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=4b0e62f32b2e4d32b6c0c4ef08932d08]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:15.819675 6829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 350fc92a3e034564800e8fbd8f6914a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:15.820235 14014 tablet_bootstrap.cc:492] T 0f53daddf5fd44a3a9d8febcf0792ce8 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:15.820326 14014 ts_tablet_manager.cc:1367] T 0f53daddf5fd44a3a9d8febcf0792ce8 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 02:00:15.820936 14014 raft_consensus.cc:348] T 0f53daddf5fd44a3a9d8febcf0792ce8 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:15.821056 14014 raft_consensus.cc:374] T 0f53daddf5fd44a3a9d8febcf0792ce8 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:15.821101 14014 raft_consensus.cc:729] T 0f53daddf5fd44a3a9d8febcf0792ce8 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:15.821316 14014 consensus_queue.cc:260] T 0f53daddf5fd44a3a9d8febcf0792ce8 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:15.821372 14026 tablet_bootstrap.cc:492] T 4966bb684477438e96d337e2320810f9 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:15.821636 14014 ts_tablet_manager.cc:1398] T 0f53daddf5fd44a3a9d8febcf0792ce8 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:15.821808 14014 tablet_bootstrap.cc:492] T 350fc92a3e034564800e8fbd8f6914a5 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:15.822012 14015 tablet_bootstrap.cc:492] T 350fc92a3e034564800e8fbd8f6914a5 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:15.822599 14026 tablet_bootstrap.cc:654] T 4966bb684477438e96d337e2320810f9 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:15.822726 14014 tablet_bootstrap.cc:654] T 350fc92a3e034564800e8fbd8f6914a5 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:15.822947 14015 tablet_bootstrap.cc:654] T 350fc92a3e034564800e8fbd8f6914a5 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:15.824085 14027 tablet_bootstrap.cc:492] T 350fc92a3e034564800e8fbd8f6914a5 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:15.824175 14027 ts_tablet_manager.cc:1367] T 350fc92a3e034564800e8fbd8f6914a5 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 02:00:15.824363 14016 tablet_bootstrap.cc:492] T 0f53daddf5fd44a3a9d8febcf0792ce8 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:15.824477 14016 ts_tablet_manager.cc:1367] T 0f53daddf5fd44a3a9d8febcf0792ce8 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240503 02:00:15.824723 14027 raft_consensus.cc:348] T 350fc92a3e034564800e8fbd8f6914a5 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:15.824829 14027 raft_consensus.cc:374] T 350fc92a3e034564800e8fbd8f6914a5 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:15.824863 14027 raft_consensus.cc:729] T 350fc92a3e034564800e8fbd8f6914a5 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:15.824928 14016 raft_consensus.cc:348] T 0f53daddf5fd44a3a9d8febcf0792ce8 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:15.825021 14016 raft_consensus.cc:374] T 0f53daddf5fd44a3a9d8febcf0792ce8 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:15.824982 14027 consensus_queue.cc:260] T 350fc92a3e034564800e8fbd8f6914a5 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:15.825049 14016 raft_consensus.cc:729] T 0f53daddf5fd44a3a9d8febcf0792ce8 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:15.825155 14016 consensus_queue.cc:260] T 0f53daddf5fd44a3a9d8febcf0792ce8 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:15.825274 14027 ts_tablet_manager.cc:1398] T 350fc92a3e034564800e8fbd8f6914a5 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:15.825423 14016 ts_tablet_manager.cc:1398] T 0f53daddf5fd44a3a9d8febcf0792ce8 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:15.825588 14016 tablet_bootstrap.cc:492] T 4966bb684477438e96d337e2320810f9 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:15.826531 14016 tablet_bootstrap.cc:654] T 4966bb684477438e96d337e2320810f9 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:15.827327 14014 tablet_bootstrap.cc:492] T 350fc92a3e034564800e8fbd8f6914a5 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:15.827430 14014 ts_tablet_manager.cc:1367] T 350fc92a3e034564800e8fbd8f6914a5 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:00:15.825867 14030 tablet_bootstrap.cc:492] T 4966bb684477438e96d337e2320810f9 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:15.827651 14030 ts_tablet_manager.cc:1367] T 4966bb684477438e96d337e2320810f9 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 02:00:15.828298 14014 raft_consensus.cc:348] T 350fc92a3e034564800e8fbd8f6914a5 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:15.828303 14030 raft_consensus.cc:348] T 4966bb684477438e96d337e2320810f9 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:15.828449 14030 raft_consensus.cc:374] T 4966bb684477438e96d337e2320810f9 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:15.828498 14030 raft_consensus.cc:729] T 4966bb684477438e96d337e2320810f9 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:15.828481 14014 raft_consensus.cc:374] T 350fc92a3e034564800e8fbd8f6914a5 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:15.828573 14014 raft_consensus.cc:729] T 350fc92a3e034564800e8fbd8f6914a5 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:15.828629 14030 consensus_queue.cc:260] T 4966bb684477438e96d337e2320810f9 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:15.828747 14014 consensus_queue.cc:260] T 350fc92a3e034564800e8fbd8f6914a5 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:15.829591 14030 ts_tablet_manager.cc:1398] T 4966bb684477438e96d337e2320810f9 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:15.830127 14014 ts_tablet_manager.cc:1398] T 350fc92a3e034564800e8fbd8f6914a5 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:00:15.831409 14026 tablet_bootstrap.cc:492] T 4966bb684477438e96d337e2320810f9 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:15.831581 14026 ts_tablet_manager.cc:1367] T 4966bb684477438e96d337e2320810f9 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 02:00:15.832198 14015 tablet_bootstrap.cc:492] T 350fc92a3e034564800e8fbd8f6914a5 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:15.832298 14015 ts_tablet_manager.cc:1367] T 350fc92a3e034564800e8fbd8f6914a5 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 02:00:15.832473 14026 raft_consensus.cc:348] T 4966bb684477438e96d337e2320810f9 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:15.832579 14026 raft_consensus.cc:374] T 4966bb684477438e96d337e2320810f9 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:15.832609 14026 raft_consensus.cc:729] T 4966bb684477438e96d337e2320810f9 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:15.832769 14026 consensus_queue.cc:260] T 4966bb684477438e96d337e2320810f9 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:15.832808 14015 raft_consensus.cc:348] T 350fc92a3e034564800e8fbd8f6914a5 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:15.832895 14015 raft_consensus.cc:374] T 350fc92a3e034564800e8fbd8f6914a5 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:15.832944 14015 raft_consensus.cc:729] T 350fc92a3e034564800e8fbd8f6914a5 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:15.833063 14026 ts_tablet_manager.cc:1398] T 4966bb684477438e96d337e2320810f9 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:15.833047 14015 consensus_queue.cc:260] T 350fc92a3e034564800e8fbd8f6914a5 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:15.833329 14015 ts_tablet_manager.cc:1398] T 350fc92a3e034564800e8fbd8f6914a5 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:15.833959 14026 tablet_bootstrap.cc:492] T 0f53daddf5fd44a3a9d8febcf0792ce8 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:15.836145 14026 tablet_bootstrap.cc:654] T 0f53daddf5fd44a3a9d8febcf0792ce8 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:15.838500 6308 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:15.838979 14026 tablet_bootstrap.cc:492] T 0f53daddf5fd44a3a9d8febcf0792ce8 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:15.839103 14026 ts_tablet_manager.cc:1367] T 0f53daddf5fd44a3a9d8febcf0792ce8 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:00:15.839623 14026 raft_consensus.cc:348] T 0f53daddf5fd44a3a9d8febcf0792ce8 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:15.839960 14026 raft_consensus.cc:374] T 0f53daddf5fd44a3a9d8febcf0792ce8 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:15.840035 14026 raft_consensus.cc:729] T 0f53daddf5fd44a3a9d8febcf0792ce8 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:15.840193 14026 consensus_queue.cc:260] T 0f53daddf5fd44a3a9d8febcf0792ce8 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:15.840528 14026 ts_tablet_manager.cc:1398] T 0f53daddf5fd44a3a9d8febcf0792ce8 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:15.843935 6719 raft_consensus.cc:1261] T 61ea3cc712c54205955c7e585d02025f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer 99ead02915954b4fab23b033ad6d546b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:15.844754 6849 raft_consensus.cc:1261] T 61ea3cc712c54205955c7e585d02025f P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer 99ead02915954b4fab23b033ad6d546b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:15.845125 13943 consensus_queue.cc:1035] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:15.845404 13956 consensus_queue.cc:1035] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:15.850194 6309 catalog_manager.cc:5576] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719) does not have the latest schema for tablet 61ea3cc712c54205955c7e585d02025f (table rebalancer_dynamic_table [id=73b1d9c7686d4af099d8797f3009a419]). Expected version 1 got 0
I20240503 02:00:15.850281 14001 tablet.cc:1722] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:15.850651 14001 tablet.cc:1620] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b: MemRowSet was empty: no flush needed.
I20240503 02:00:15.850764 14042 tablet.cc:1722] T 61ea3cc712c54205955c7e585d02025f P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:15.851094 14042 tablet.cc:1620] T 61ea3cc712c54205955c7e585d02025f P 492448570cc94ed1879feb2ec3b9051d: MemRowSet was empty: no flush needed.
I20240503 02:00:15.854935 6313 catalog_manager.cc:5576] TS c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655) does not have the latest schema for tablet 61ea3cc712c54205955c7e585d02025f (table rebalancer_dynamic_table [id=73b1d9c7686d4af099d8797f3009a419]). Expected version 1 got 0
I20240503 02:00:15.855268 6308 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=73b1d9c7686d4af099d8797f3009a419] alter complete (version 1)
I20240503 02:00:15.855544 14043 tablet.cc:1722] T 61ea3cc712c54205955c7e585d02025f P c2ac2eb7cc264047b9ab32fe3a4fc025: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:15.855876 14043 tablet.cc:1620] T 61ea3cc712c54205955c7e585d02025f P c2ac2eb7cc264047b9ab32fe3a4fc025: MemRowSet was empty: no flush needed.
I20240503 02:00:15.856918 14016 tablet_bootstrap.cc:492] T 4966bb684477438e96d337e2320810f9 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:15.857033 14016 ts_tablet_manager.cc:1367] T 4966bb684477438e96d337e2320810f9 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.031s user 0.000s sys 0.002s
I20240503 02:00:15.857538 14016 raft_consensus.cc:348] T 4966bb684477438e96d337e2320810f9 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:15.857630 14016 raft_consensus.cc:374] T 4966bb684477438e96d337e2320810f9 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:15.857650 14016 raft_consensus.cc:729] T 4966bb684477438e96d337e2320810f9 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:15.857757 14016 consensus_queue.cc:260] T 4966bb684477438e96d337e2320810f9 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:15.858070 14016 ts_tablet_manager.cc:1398] T 4966bb684477438e96d337e2320810f9 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:15.879215 13900 raft_consensus.cc:482] T 0f53daddf5fd44a3a9d8febcf0792ce8 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:15.879393 13900 raft_consensus.cc:504] T 0f53daddf5fd44a3a9d8febcf0792ce8 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:15.880091 13900 leader_election.cc:288] T 0f53daddf5fd44a3a9d8febcf0792ce8 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 02:00:15.880550 6719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f53daddf5fd44a3a9d8febcf0792ce8" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 02:00:15.880532 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f53daddf5fd44a3a9d8febcf0792ce8" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 02:00:15.880699 6458 raft_consensus.cc:2454] T 0f53daddf5fd44a3a9d8febcf0792ce8 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:15.880702 6719 raft_consensus.cc:2454] T 0f53daddf5fd44a3a9d8febcf0792ce8 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:15.881039 8981 leader_election.cc:302] T 0f53daddf5fd44a3a9d8febcf0792ce8 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99ead02915954b4fab23b033ad6d546b, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:15.881242 13900 raft_consensus.cc:2781] T 0f53daddf5fd44a3a9d8febcf0792ce8 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:15.881314 13900 raft_consensus.cc:482] T 0f53daddf5fd44a3a9d8febcf0792ce8 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:15.881369 13900 raft_consensus.cc:3037] T 0f53daddf5fd44a3a9d8febcf0792ce8 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:15.882229 13900 raft_consensus.cc:504] T 0f53daddf5fd44a3a9d8febcf0792ce8 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:15.882715 13900 leader_election.cc:288] T 0f53daddf5fd44a3a9d8febcf0792ce8 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Requested vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 02:00:15.883172 6719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f53daddf5fd44a3a9d8febcf0792ce8" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 02:00:15.883172 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f53daddf5fd44a3a9d8febcf0792ce8" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 02:00:15.883313 6719 raft_consensus.cc:3037] T 0f53daddf5fd44a3a9d8febcf0792ce8 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:15.883313 6458 raft_consensus.cc:3037] T 0f53daddf5fd44a3a9d8febcf0792ce8 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:15.883318 13829 raft_consensus.cc:482] T 4966bb684477438e96d337e2320810f9 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:15.883493 13829 raft_consensus.cc:504] T 4966bb684477438e96d337e2320810f9 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:15.883957 13829 leader_election.cc:288] T 4966bb684477438e96d337e2320810f9 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:15.884346 6582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4966bb684477438e96d337e2320810f9" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 02:00:15.884346 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4966bb684477438e96d337e2320810f9" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:15.884457 6458 raft_consensus.cc:2454] T 0f53daddf5fd44a3a9d8febcf0792ce8 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:15.884500 9048 raft_consensus.cc:2454] T 4966bb684477438e96d337e2320810f9 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:15.884500 6582 raft_consensus.cc:2454] T 4966bb684477438e96d337e2320810f9 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:15.884649 6719 raft_consensus.cc:2454] T 0f53daddf5fd44a3a9d8febcf0792ce8 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:15.884785 8981 leader_election.cc:302] T 0f53daddf5fd44a3a9d8febcf0792ce8 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99ead02915954b4fab23b033ad6d546b, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:15.884887 6393 leader_election.cc:302] T 4966bb684477438e96d337e2320810f9 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:15.885075 13829 raft_consensus.cc:2781] T 4966bb684477438e96d337e2320810f9 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:15.885139 13829 raft_consensus.cc:482] T 4966bb684477438e96d337e2320810f9 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:15.885182 13829 raft_consensus.cc:3037] T 4966bb684477438e96d337e2320810f9 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:15.885399 13900 raft_consensus.cc:2781] T 0f53daddf5fd44a3a9d8febcf0792ce8 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:15.885481 13900 raft_consensus.cc:686] T 0f53daddf5fd44a3a9d8febcf0792ce8 P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Becoming Leader. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Running, Role: LEADER
I20240503 02:00:15.885604 13900 consensus_queue.cc:237] T 0f53daddf5fd44a3a9d8febcf0792ce8 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:15.886224 13829 raft_consensus.cc:504] T 4966bb684477438e96d337e2320810f9 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:15.886531 13829 leader_election.cc:288] T 4966bb684477438e96d337e2320810f9 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Requested vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:15.886878 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4966bb684477438e96d337e2320810f9" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:15.886973 9048 raft_consensus.cc:3037] T 4966bb684477438e96d337e2320810f9 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:15.887266 6582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4966bb684477438e96d337e2320810f9" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 02:00:15.887385 6582 raft_consensus.cc:3037] T 4966bb684477438e96d337e2320810f9 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:15.887883 9048 raft_consensus.cc:2454] T 4966bb684477438e96d337e2320810f9 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:15.887988 6308 catalog_manager.cc:5484] T 0f53daddf5fd44a3a9d8febcf0792ce8 P 99ead02915954b4fab23b033ad6d546b reported cstate change: term changed from 0 to 1, leader changed from <none> to 99ead02915954b4fab23b033ad6d546b (127.0.99.134). New cstate: current_term: 1 leader_uuid: "99ead02915954b4fab23b033ad6d546b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:15.888336 6582 raft_consensus.cc:2454] T 4966bb684477438e96d337e2320810f9 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:15.888675 6393 leader_election.cc:302] T 4966bb684477438e96d337e2320810f9 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:15.888953 13829 raft_consensus.cc:2781] T 4966bb684477438e96d337e2320810f9 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:15.889047 13829 raft_consensus.cc:686] T 4966bb684477438e96d337e2320810f9 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Becoming Leader. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Running, Role: LEADER
I20240503 02:00:15.889191 13829 consensus_queue.cc:237] T 4966bb684477438e96d337e2320810f9 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:15.891600 6308 catalog_manager.cc:5484] T 4966bb684477438e96d337e2320810f9 P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: term changed from 0 to 1, leader changed from <none> to dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129). New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:15.907866 13975 raft_consensus.cc:482] T 350fc92a3e034564800e8fbd8f6914a5 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:15.908020 13975 raft_consensus.cc:504] T 350fc92a3e034564800e8fbd8f6914a5 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:15.908634 13975 leader_election.cc:288] T 350fc92a3e034564800e8fbd8f6914a5 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:15.909093 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "350fc92a3e034564800e8fbd8f6914a5" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:15.909132 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "350fc92a3e034564800e8fbd8f6914a5" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 02:00:15.909245 9178 raft_consensus.cc:2454] T 350fc92a3e034564800e8fbd8f6914a5 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:15.909263 6849 raft_consensus.cc:2454] T 350fc92a3e034564800e8fbd8f6914a5 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:15.909526 6656 leader_election.cc:302] T 350fc92a3e034564800e8fbd8f6914a5 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:15.909718 13975 raft_consensus.cc:2781] T 350fc92a3e034564800e8fbd8f6914a5 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:15.909776 13975 raft_consensus.cc:482] T 350fc92a3e034564800e8fbd8f6914a5 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:15.909812 13975 raft_consensus.cc:3037] T 350fc92a3e034564800e8fbd8f6914a5 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:15.910676 13975 raft_consensus.cc:504] T 350fc92a3e034564800e8fbd8f6914a5 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:15.911024 13975 leader_election.cc:288] T 350fc92a3e034564800e8fbd8f6914a5 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Requested vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:15.911419 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "350fc92a3e034564800e8fbd8f6914a5" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:15.911419 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "350fc92a3e034564800e8fbd8f6914a5" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 02:00:15.911551 6849 raft_consensus.cc:3037] T 350fc92a3e034564800e8fbd8f6914a5 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:15.911551 9178 raft_consensus.cc:3037] T 350fc92a3e034564800e8fbd8f6914a5 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:15.912601 9178 raft_consensus.cc:2454] T 350fc92a3e034564800e8fbd8f6914a5 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:15.912601 6849 raft_consensus.cc:2454] T 350fc92a3e034564800e8fbd8f6914a5 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:15.912891 6653 leader_election.cc:302] T 350fc92a3e034564800e8fbd8f6914a5 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:15.913081 13975 raft_consensus.cc:2781] T 350fc92a3e034564800e8fbd8f6914a5 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:15.913195 13975 raft_consensus.cc:686] T 350fc92a3e034564800e8fbd8f6914a5 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Becoming Leader. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Running, Role: LEADER
I20240503 02:00:15.913359 13975 consensus_queue.cc:237] T 350fc92a3e034564800e8fbd8f6914a5 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:15.915706 6308 catalog_manager.cc:5484] T 350fc92a3e034564800e8fbd8f6914a5 P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:15.959856 6308 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:15.965605 14001 tablet.cc:1722] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:15.965946 14001 tablet.cc:1620] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b: MemRowSet was empty: no flush needed.
I20240503 02:00:15.966488 14042 tablet.cc:1722] T 61ea3cc712c54205955c7e585d02025f P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:15.966832 14042 tablet.cc:1620] T 61ea3cc712c54205955c7e585d02025f P 492448570cc94ed1879feb2ec3b9051d: MemRowSet was empty: no flush needed.
W20240503 02:00:15.969913 6654 consensus_peers.cc:485] T ba17d4d3e0754b2798eb95484d37229a P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:00:15.970688 14043 tablet.cc:1722] T 61ea3cc712c54205955c7e585d02025f P c2ac2eb7cc264047b9ab32fe3a4fc025: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:15.970999 14043 tablet.cc:1620] T 61ea3cc712c54205955c7e585d02025f P c2ac2eb7cc264047b9ab32fe3a4fc025: MemRowSet was empty: no flush needed.
I20240503 02:00:15.973464 6309 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=73b1d9c7686d4af099d8797f3009a419] alter complete (version 2)
W20240503 02:00:16.014598 6523 consensus_peers.cc:485] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:16.028046 6654 consensus_peers.cc:485] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:00:16.077383 6309 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
W20240503 02:00:16.081918 6523 consensus_peers.cc:485] T d0d5dd42db3e4883af7fd16d8813b550 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:16.081991 8851 consensus_peers.cc:485] T 067f54011a37432d99157884e70fc601 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:00:16.083861 14001 tablet.cc:1722] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
W20240503 02:00:16.084163 6785 consensus_peers.cc:485] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:00:16.084722 14043 tablet.cc:1722] T 61ea3cc712c54205955c7e585d02025f P c2ac2eb7cc264047b9ab32fe3a4fc025: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:16.085592 14042 tablet.cc:1722] T 61ea3cc712c54205955c7e585d02025f P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:16.089983 6309 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=73b1d9c7686d4af099d8797f3009a419] alter complete (version 3)
W20240503 02:00:16.115957 9113 consensus_peers.cc:485] T f44b59277c3b47adbd4dd6aff9350af1 P 2a01ace80baf4057b6bf1995cf88468b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:00:16.192467 6313 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:16.192616 6313 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:16.194583 6313 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet 61ea3cc712c54205955c7e585d02025f
I20240503 02:00:16.195214 6699 tablet_service.cc:1514] Processing DeleteTablet for tablet 61ea3cc712c54205955c7e585d02025f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:16 UTC) from {username='slave'} at 127.0.0.1:40336
I20240503 02:00:16.195272 6829 tablet_service.cc:1514] Processing DeleteTablet for tablet 61ea3cc712c54205955c7e585d02025f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:16 UTC) from {username='slave'} at 127.0.0.1:52812
I20240503 02:00:16.195437 9025 tablet_service.cc:1514] Processing DeleteTablet for tablet 61ea3cc712c54205955c7e585d02025f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:16 UTC) from {username='slave'} at 127.0.0.1:52584
I20240503 02:00:16.195562 14049 tablet_replica.cc:330] T 61ea3cc712c54205955c7e585d02025f P c2ac2eb7cc264047b9ab32fe3a4fc025: stopping tablet replica
I20240503 02:00:16.195583 14050 tablet_replica.cc:330] T 61ea3cc712c54205955c7e585d02025f P 492448570cc94ed1879feb2ec3b9051d: stopping tablet replica
I20240503 02:00:16.195716 14050 raft_consensus.cc:2229] T 61ea3cc712c54205955c7e585d02025f P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:16.195721 14049 raft_consensus.cc:2229] T 61ea3cc712c54205955c7e585d02025f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:16.195740 14051 tablet_replica.cc:330] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b: stopping tablet replica
I20240503 02:00:16.195845 14049 raft_consensus.cc:2258] T 61ea3cc712c54205955c7e585d02025f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:16.195868 14050 raft_consensus.cc:2258] T 61ea3cc712c54205955c7e585d02025f P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:16.195863 14051 raft_consensus.cc:2229] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:16.196115 14051 raft_consensus.cc:2258] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:16.196612 14049 ts_tablet_manager.cc:1875] T 61ea3cc712c54205955c7e585d02025f P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:16.196691 14051 ts_tablet_manager.cc:1875] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:16.196691 14050 ts_tablet_manager.cc:1875] T 61ea3cc712c54205955c7e585d02025f P 492448570cc94ed1879feb2ec3b9051d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:16.198971 14051 ts_tablet_manager.cc:1888] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:16.198977 14050 ts_tablet_manager.cc:1888] T 61ea3cc712c54205955c7e585d02025f P 492448570cc94ed1879feb2ec3b9051d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:16.199065 14050 log.cc:1196] T 61ea3cc712c54205955c7e585d02025f P 492448570cc94ed1879feb2ec3b9051d: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/61ea3cc712c54205955c7e585d02025f
I20240503 02:00:16.199090 14051 log.cc:1196] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/61ea3cc712c54205955c7e585d02025f
I20240503 02:00:16.199234 14049 ts_tablet_manager.cc:1888] T 61ea3cc712c54205955c7e585d02025f P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:16.199339 14049 log.cc:1196] T 61ea3cc712c54205955c7e585d02025f P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/61ea3cc712c54205955c7e585d02025f
I20240503 02:00:16.199355 14051 ts_tablet_manager.cc:1909] T 61ea3cc712c54205955c7e585d02025f P 99ead02915954b4fab23b033ad6d546b: Deleting consensus metadata
I20240503 02:00:16.199373 14050 ts_tablet_manager.cc:1909] T 61ea3cc712c54205955c7e585d02025f P 492448570cc94ed1879feb2ec3b9051d: Deleting consensus metadata
I20240503 02:00:16.199663 14049 ts_tablet_manager.cc:1909] T 61ea3cc712c54205955c7e585d02025f P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting consensus metadata
I20240503 02:00:16.200430 6303 catalog_manager.cc:4830] TS c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655): tablet 61ea3cc712c54205955c7e585d02025f (table rebalancer_dynamic_table_new_name [id=73b1d9c7686d4af099d8797f3009a419]) successfully deleted
I20240503 02:00:16.200553 6301 catalog_manager.cc:4830] TS 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157): tablet 61ea3cc712c54205955c7e585d02025f (table rebalancer_dynamic_table_new_name [id=73b1d9c7686d4af099d8797f3009a419]) successfully deleted
I20240503 02:00:16.200835 6303 catalog_manager.cc:4830] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719): tablet 61ea3cc712c54205955c7e585d02025f (table rebalancer_dynamic_table_new_name [id=73b1d9c7686d4af099d8797f3009a419]) successfully deleted
W20240503 02:00:16.222460 6785 consensus_peers.cc:485] T 46ad540deb624a25b97d3b967fe1f6f4 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:16.224550 8851 consensus_peers.cc:485] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:16.259510 6523 consensus_peers.cc:485] T 925612b660ec471da5b3c548802ede6e P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:00:16.279611 14052 consensus_queue.cc:1035] T 4966bb684477438e96d337e2320810f9 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:16.283103 13991 consensus_queue.cc:1035] T 4966bb684477438e96d337e2320810f9 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:00:16.289844 6785 consensus_peers.cc:485] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:00:16.296200 6309 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:00:16.301681 9025 tablet_service.cc:1467] Processing CreateTablet for tablet 90a0a91005714af39c8f0e28b4446e01 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7d9cea0b1a7a4cddabd18ed44a365181]), partition=
I20240503 02:00:16.301697 8897 tablet_service.cc:1467] Processing CreateTablet for tablet 90a0a91005714af39c8f0e28b4446e01 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7d9cea0b1a7a4cddabd18ed44a365181]), partition=
I20240503 02:00:16.301967 8897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90a0a91005714af39c8f0e28b4446e01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:16.301972 9025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90a0a91005714af39c8f0e28b4446e01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:16.304126 14016 tablet_bootstrap.cc:492] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:16.304729 9158 tablet_service.cc:1467] Processing CreateTablet for tablet 90a0a91005714af39c8f0e28b4446e01 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7d9cea0b1a7a4cddabd18ed44a365181]), partition=
I20240503 02:00:16.304884 14059 tablet_bootstrap.cc:492] T 90a0a91005714af39c8f0e28b4446e01 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:16.304934 9158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90a0a91005714af39c8f0e28b4446e01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:16.305155 14016 tablet_bootstrap.cc:654] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:16.305774 14059 tablet_bootstrap.cc:654] T 90a0a91005714af39c8f0e28b4446e01 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:16.306864 14059 tablet_bootstrap.cc:492] T 90a0a91005714af39c8f0e28b4446e01 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:16.306960 14059 ts_tablet_manager.cc:1367] T 90a0a91005714af39c8f0e28b4446e01 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:16.307030 14027 tablet_bootstrap.cc:492] T 90a0a91005714af39c8f0e28b4446e01 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:16.307379 14016 tablet_bootstrap.cc:492] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:16.307344 14059 raft_consensus.cc:348] T 90a0a91005714af39c8f0e28b4446e01 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:16.307431 14059 raft_consensus.cc:374] T 90a0a91005714af39c8f0e28b4446e01 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:16.307451 14059 raft_consensus.cc:729] T 90a0a91005714af39c8f0e28b4446e01 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:16.307466 14016 ts_tablet_manager.cc:1367] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:16.307535 14059 consensus_queue.cc:260] T 90a0a91005714af39c8f0e28b4446e01 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:16.307787 14059 ts_tablet_manager.cc:1398] T 90a0a91005714af39c8f0e28b4446e01 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:16.307950 14027 tablet_bootstrap.cc:654] T 90a0a91005714af39c8f0e28b4446e01 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:16.307910 14016 raft_consensus.cc:348] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:16.308013 14016 raft_consensus.cc:374] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:16.308041 14016 raft_consensus.cc:729] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:16.308177 14016 consensus_queue.cc:260] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:16.308487 14016 ts_tablet_manager.cc:1398] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:16.321663 13900 consensus_queue.cc:1035] T 0f53daddf5fd44a3a9d8febcf0792ce8 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:16.325084 13943 consensus_queue.cc:1035] T 0f53daddf5fd44a3a9d8febcf0792ce8 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:16.340945 14027 tablet_bootstrap.cc:492] T 90a0a91005714af39c8f0e28b4446e01 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:16.341091 14027 ts_tablet_manager.cc:1367] T 90a0a91005714af39c8f0e28b4446e01 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.034s user 0.002s sys 0.000s
I20240503 02:00:16.341665 14027 raft_consensus.cc:348] T 90a0a91005714af39c8f0e28b4446e01 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:16.341787 14027 raft_consensus.cc:374] T 90a0a91005714af39c8f0e28b4446e01 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:16.341817 14027 raft_consensus.cc:729] T 90a0a91005714af39c8f0e28b4446e01 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:16.341949 14027 consensus_queue.cc:260] T 90a0a91005714af39c8f0e28b4446e01 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:16.342260 14027 ts_tablet_manager.cc:1398] T 90a0a91005714af39c8f0e28b4446e01 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20240503 02:00:16.353497 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
W20240503 02:00:16.361248 6785 consensus_peers.cc:485] T 8246488dec204bf6b09e868d7951ac5a P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:00:16.375476 13975 consensus_queue.cc:1035] T 350fc92a3e034564800e8fbd8f6914a5 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:16.379936 14066 consensus_queue.cc:1035] T 350fc92a3e034564800e8fbd8f6914a5 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:16.404758 13943 raft_consensus.cc:482] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:16.404911 13943 raft_consensus.cc:504] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:16.405419 13943 leader_election.cc:288] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:16.405875 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90a0a91005714af39c8f0e28b4446e01" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:16.405934 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90a0a91005714af39c8f0e28b4446e01" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:16.406049 8916 raft_consensus.cc:2454] T 90a0a91005714af39c8f0e28b4446e01 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:16.406085 9178 raft_consensus.cc:2454] T 90a0a91005714af39c8f0e28b4446e01 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:16.406494 8981 leader_election.cc:302] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:16.406734 13943 raft_consensus.cc:2781] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:16.406781 13943 raft_consensus.cc:482] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:16.406813 13943 raft_consensus.cc:3037] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:16.407789 13943 raft_consensus.cc:504] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:16.408157 13943 leader_election.cc:288] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Requested vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:16.408597 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90a0a91005714af39c8f0e28b4446e01" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:16.408764 8916 raft_consensus.cc:3037] T 90a0a91005714af39c8f0e28b4446e01 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:16.409608 8916 raft_consensus.cc:2454] T 90a0a91005714af39c8f0e28b4446e01 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:16.409873 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90a0a91005714af39c8f0e28b4446e01" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:16.409914 8984 leader_election.cc:302] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9942306855404cfd891adf227dd65eb2, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:16.410008 9178 raft_consensus.cc:3037] T 90a0a91005714af39c8f0e28b4446e01 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:16.410080 13943 raft_consensus.cc:2781] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:16.410163 13943 raft_consensus.cc:686] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Becoming Leader. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Running, Role: LEADER
I20240503 02:00:16.410312 13943 consensus_queue.cc:237] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:16.410985 9178 raft_consensus.cc:2454] T 90a0a91005714af39c8f0e28b4446e01 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:16.413386 6312 catalog_manager.cc:5484] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b reported cstate change: term changed from 0 to 1, leader changed from <none> to 99ead02915954b4fab23b033ad6d546b (127.0.99.134). New cstate: current_term: 1 leader_uuid: "99ead02915954b4fab23b033ad6d546b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } }
W20240503 02:00:16.432929 6785 consensus_peers.cc:485] T d897c21e630d4770832bb10f43778e82 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:00:16.438104 6312 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_extra_table_42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:16.444370 6829 tablet_service.cc:1467] Processing CreateTablet for tablet 2c14900a27ce4c7eb9c0ad0f0c34488d (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=eb0be093febd40a99ecffa1fbe86b63e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:16.444759 6829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c14900a27ce4c7eb9c0ad0f0c34488d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:16.444957 6436 tablet_service.cc:1467] Processing CreateTablet for tablet 2c14900a27ce4c7eb9c0ad0f0c34488d (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=eb0be093febd40a99ecffa1fbe86b63e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:16.445223 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c14900a27ce4c7eb9c0ad0f0c34488d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:16.445915 9028 tablet_service.cc:1467] Processing CreateTablet for tablet f93281ad992049c38b8d446b09f821c0 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=eb0be093febd40a99ecffa1fbe86b63e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:16.446221 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f93281ad992049c38b8d446b09f821c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:16.446944 6569 tablet_service.cc:1467] Processing CreateTablet for tablet c280f4c27f17437fa7012deca745307b (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=eb0be093febd40a99ecffa1fbe86b63e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:16.447201 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c280f4c27f17437fa7012deca745307b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:16.447306 6831 tablet_service.cc:1467] Processing CreateTablet for tablet f93281ad992049c38b8d446b09f821c0 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=eb0be093febd40a99ecffa1fbe86b63e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:16.447542 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f93281ad992049c38b8d446b09f821c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:16.448184 6699 tablet_service.cc:1467] Processing CreateTablet for tablet c280f4c27f17437fa7012deca745307b (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=eb0be093febd40a99ecffa1fbe86b63e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:16.448483 6699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c280f4c27f17437fa7012deca745307b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:16.448503 14016 tablet_bootstrap.cc:492] T f93281ad992049c38b8d446b09f821c0 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:16.449504 14016 tablet_bootstrap.cc:654] T f93281ad992049c38b8d446b09f821c0 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:16.449986 9158 tablet_service.cc:1467] Processing CreateTablet for tablet f93281ad992049c38b8d446b09f821c0 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=eb0be093febd40a99ecffa1fbe86b63e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:16.450213 14070 tablet_bootstrap.cc:492] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:16.450274 9158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f93281ad992049c38b8d446b09f821c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:16.450580 14072 tablet_bootstrap.cc:492] T c280f4c27f17437fa7012deca745307b P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:16.451589 14072 tablet_bootstrap.cc:654] T c280f4c27f17437fa7012deca745307b P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:16.451790 14070 tablet_bootstrap.cc:654] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:16.452173 14071 tablet_bootstrap.cc:492] T 2c14900a27ce4c7eb9c0ad0f0c34488d P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:16.452256 8897 tablet_service.cc:1467] Processing CreateTablet for tablet c280f4c27f17437fa7012deca745307b (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=eb0be093febd40a99ecffa1fbe86b63e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:16.452520 8897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c280f4c27f17437fa7012deca745307b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:16.452688 9025 tablet_service.cc:1467] Processing CreateTablet for tablet 2c14900a27ce4c7eb9c0ad0f0c34488d (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=eb0be093febd40a99ecffa1fbe86b63e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:16.452966 9025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c14900a27ce4c7eb9c0ad0f0c34488d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:16.453562 14074 tablet_bootstrap.cc:492] T c280f4c27f17437fa7012deca745307b P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:16.454530 14074 tablet_bootstrap.cc:654] T c280f4c27f17437fa7012deca745307b P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:16.454995 14059 tablet_bootstrap.cc:492] T c280f4c27f17437fa7012deca745307b P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:16.455931 14059 tablet_bootstrap.cc:654] T c280f4c27f17437fa7012deca745307b P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:16.456012 14072 tablet_bootstrap.cc:492] T c280f4c27f17437fa7012deca745307b P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:16.456105 14072 ts_tablet_manager.cc:1367] T c280f4c27f17437fa7012deca745307b P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:00:16.456326 14027 tablet_bootstrap.cc:492] T f93281ad992049c38b8d446b09f821c0 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:16.456631 14072 raft_consensus.cc:348] T c280f4c27f17437fa7012deca745307b P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:16.456749 14072 raft_consensus.cc:374] T c280f4c27f17437fa7012deca745307b P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:16.456784 14072 raft_consensus.cc:729] T c280f4c27f17437fa7012deca745307b P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:16.456904 14072 consensus_queue.cc:260] T c280f4c27f17437fa7012deca745307b P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:16.457180 14072 ts_tablet_manager.cc:1398] T c280f4c27f17437fa7012deca745307b P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:16.457345 14027 tablet_bootstrap.cc:654] T f93281ad992049c38b8d446b09f821c0 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:16.459208 14071 tablet_bootstrap.cc:654] T 2c14900a27ce4c7eb9c0ad0f0c34488d P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:16.460144 14016 tablet_bootstrap.cc:492] T f93281ad992049c38b8d446b09f821c0 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:16.460242 14016 ts_tablet_manager.cc:1367] T f93281ad992049c38b8d446b09f821c0 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 02:00:16.460215 14070 tablet_bootstrap.cc:492] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:16.460333 14070 ts_tablet_manager.cc:1367] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 02:00:16.460743 14016 raft_consensus.cc:348] T f93281ad992049c38b8d446b09f821c0 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:16.460863 14016 raft_consensus.cc:374] T f93281ad992049c38b8d446b09f821c0 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:16.460894 14016 raft_consensus.cc:729] T f93281ad992049c38b8d446b09f821c0 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:16.460892 14070 raft_consensus.cc:348] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:16.461007 14070 raft_consensus.cc:374] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:16.461000 14016 consensus_queue.cc:260] T f93281ad992049c38b8d446b09f821c0 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:16.461061 14070 raft_consensus.cc:729] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:16.461180 14070 consensus_queue.cc:260] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:16.461290 14016 ts_tablet_manager.cc:1398] T f93281ad992049c38b8d446b09f821c0 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:16.461464 14070 ts_tablet_manager.cc:1398] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:16.461697 14016 tablet_bootstrap.cc:492] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:16.461891 14070 tablet_bootstrap.cc:492] T f93281ad992049c38b8d446b09f821c0 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:16.462497 14027 tablet_bootstrap.cc:492] T f93281ad992049c38b8d446b09f821c0 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:16.462604 14027 ts_tablet_manager.cc:1367] T f93281ad992049c38b8d446b09f821c0 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:00:16.462672 14016 tablet_bootstrap.cc:654] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:16.462699 14071 tablet_bootstrap.cc:492] T 2c14900a27ce4c7eb9c0ad0f0c34488d P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:16.462790 14071 ts_tablet_manager.cc:1367] T 2c14900a27ce4c7eb9c0ad0f0c34488d P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 02:00:16.462857 14070 tablet_bootstrap.cc:654] T f93281ad992049c38b8d446b09f821c0 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:16.463094 14027 raft_consensus.cc:348] T f93281ad992049c38b8d446b09f821c0 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:16.463215 14027 raft_consensus.cc:374] T f93281ad992049c38b8d446b09f821c0 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:16.463248 14027 raft_consensus.cc:729] T f93281ad992049c38b8d446b09f821c0 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:16.463297 14071 raft_consensus.cc:348] T 2c14900a27ce4c7eb9c0ad0f0c34488d P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:16.463366 14027 consensus_queue.cc:260] T f93281ad992049c38b8d446b09f821c0 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:16.463407 14071 raft_consensus.cc:374] T 2c14900a27ce4c7eb9c0ad0f0c34488d P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:16.463439 14071 raft_consensus.cc:729] T 2c14900a27ce4c7eb9c0ad0f0c34488d P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:16.463546 14071 consensus_queue.cc:260] T 2c14900a27ce4c7eb9c0ad0f0c34488d P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:16.463636 14027 ts_tablet_manager.cc:1398] T f93281ad992049c38b8d446b09f821c0 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:16.463881 14071 ts_tablet_manager.cc:1398] T 2c14900a27ce4c7eb9c0ad0f0c34488d P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:16.464818 14074 tablet_bootstrap.cc:492] T c280f4c27f17437fa7012deca745307b P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:16.465175 14074 ts_tablet_manager.cc:1367] T c280f4c27f17437fa7012deca745307b P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.001s
I20240503 02:00:16.465725 14074 raft_consensus.cc:348] T c280f4c27f17437fa7012deca745307b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:16.465921 14074 raft_consensus.cc:374] T c280f4c27f17437fa7012deca745307b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:16.466001 14074 raft_consensus.cc:729] T c280f4c27f17437fa7012deca745307b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:16.466483 14074 consensus_queue.cc:260] T c280f4c27f17437fa7012deca745307b P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:16.467864 14074 ts_tablet_manager.cc:1398] T c280f4c27f17437fa7012deca745307b P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 02:00:16.470026 14016 tablet_bootstrap.cc:492] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:16.470214 14016 ts_tablet_manager.cc:1367] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.001s
I20240503 02:00:16.473268 14016 raft_consensus.cc:348] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:16.473433 14070 tablet_bootstrap.cc:492] T f93281ad992049c38b8d446b09f821c0 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:16.473500 14016 raft_consensus.cc:374] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:16.473544 14016 raft_consensus.cc:729] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:16.473567 14070 ts_tablet_manager.cc:1367] T f93281ad992049c38b8d446b09f821c0 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 02:00:16.473668 14016 consensus_queue.cc:260] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:16.473953 14016 ts_tablet_manager.cc:1398] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.004s user 0.000s sys 0.000s
I20240503 02:00:16.474074 14070 raft_consensus.cc:348] T f93281ad992049c38b8d446b09f821c0 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:16.474195 14070 raft_consensus.cc:374] T f93281ad992049c38b8d446b09f821c0 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:16.474225 14070 raft_consensus.cc:729] T f93281ad992049c38b8d446b09f821c0 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:16.475705 14070 consensus_queue.cc:260] T f93281ad992049c38b8d446b09f821c0 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:16.476464 14070 ts_tablet_manager.cc:1398] T f93281ad992049c38b8d446b09f821c0 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:00:16.476909 14059 tablet_bootstrap.cc:492] T c280f4c27f17437fa7012deca745307b P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:16.477017 14059 ts_tablet_manager.cc:1367] T c280f4c27f17437fa7012deca745307b P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.001s
I20240503 02:00:16.477658 14059 raft_consensus.cc:348] T c280f4c27f17437fa7012deca745307b P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:16.477800 14059 raft_consensus.cc:374] T c280f4c27f17437fa7012deca745307b P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:16.477833 14059 raft_consensus.cc:729] T c280f4c27f17437fa7012deca745307b P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:16.477963 14059 consensus_queue.cc:260] T c280f4c27f17437fa7012deca745307b P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:16.478272 14059 ts_tablet_manager.cc:1398] T c280f4c27f17437fa7012deca745307b P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20240503 02:00:16.494074 8982 consensus_peers.cc:485] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:00:16.515985 14053 raft_consensus.cc:482] T f93281ad992049c38b8d446b09f821c0 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:16.516132 14053 raft_consensus.cc:504] T f93281ad992049c38b8d446b09f821c0 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:16.516682 14053 leader_election.cc:288] T f93281ad992049c38b8d446b09f821c0 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:16.517277 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f93281ad992049c38b8d446b09f821c0" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:16.517266 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f93281ad992049c38b8d446b09f821c0" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 02:00:16.517406 6851 raft_consensus.cc:2454] T f93281ad992049c38b8d446b09f821c0 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:16.517411 9048 raft_consensus.cc:2454] T f93281ad992049c38b8d446b09f821c0 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:16.517674 9115 leader_election.cc:302] T f93281ad992049c38b8d446b09f821c0 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 492448570cc94ed1879feb2ec3b9051d; no voters:
I20240503 02:00:16.517838 14053 raft_consensus.cc:2781] T f93281ad992049c38b8d446b09f821c0 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:16.517899 14053 raft_consensus.cc:482] T f93281ad992049c38b8d446b09f821c0 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:16.517938 14053 raft_consensus.cc:3037] T f93281ad992049c38b8d446b09f821c0 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:16.518962 14053 raft_consensus.cc:504] T f93281ad992049c38b8d446b09f821c0 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:16.519342 14053 leader_election.cc:288] T f93281ad992049c38b8d446b09f821c0 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Requested vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:16.519713 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f93281ad992049c38b8d446b09f821c0" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 02:00:16.519732 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f93281ad992049c38b8d446b09f821c0" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:16.519826 6849 raft_consensus.cc:3037] T f93281ad992049c38b8d446b09f821c0 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:16.519834 9048 raft_consensus.cc:3037] T f93281ad992049c38b8d446b09f821c0 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:16.520694 6849 raft_consensus.cc:2454] T f93281ad992049c38b8d446b09f821c0 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:16.520730 9048 raft_consensus.cc:2454] T f93281ad992049c38b8d446b09f821c0 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:16.521020 9115 leader_election.cc:302] T f93281ad992049c38b8d446b09f821c0 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 492448570cc94ed1879feb2ec3b9051d; no voters:
I20240503 02:00:16.521214 14053 raft_consensus.cc:2781] T f93281ad992049c38b8d446b09f821c0 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:16.521319 14053 raft_consensus.cc:686] T f93281ad992049c38b8d446b09f821c0 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Becoming Leader. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Running, Role: LEADER
I20240503 02:00:16.521481 14053 consensus_queue.cc:237] T f93281ad992049c38b8d446b09f821c0 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
W20240503 02:00:16.522233 6767 tablet.cc:2369] T c280f4c27f17437fa7012deca745307b P c2ac2eb7cc264047b9ab32fe3a4fc025: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:00:16.523916 6309 catalog_manager.cc:5484] T f93281ad992049c38b8d446b09f821c0 P 2a01ace80baf4057b6bf1995cf88468b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135). New cstate: current_term: 1 leader_uuid: "2a01ace80baf4057b6bf1995cf88468b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: HEALTHY } } }
W20240503 02:00:16.526990 6523 consensus_peers.cc:485] T 4385d51d11574192883d01376e2d5620 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:00:16.533735 6309 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:16.537921 13969 raft_consensus.cc:482] T c280f4c27f17437fa7012deca745307b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:16.538056 13969 raft_consensus.cc:504] T c280f4c27f17437fa7012deca745307b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:16.538650 9178 raft_consensus.cc:1261] T 90a0a91005714af39c8f0e28b4446e01 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer 99ead02915954b4fab23b033ad6d546b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:16.538749 13969 leader_election.cc:288] T c280f4c27f17437fa7012deca745307b P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:16.538841 8916 raft_consensus.cc:1261] T 90a0a91005714af39c8f0e28b4446e01 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Refusing update from remote peer 99ead02915954b4fab23b033ad6d546b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:16.539157 13900 consensus_queue.cc:1035] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:16.539255 6582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c280f4c27f17437fa7012deca745307b" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 02:00:16.539325 13956 consensus_queue.cc:1035] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:16.539388 6582 raft_consensus.cc:2454] T c280f4c27f17437fa7012deca745307b P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:16.539763 6655 leader_election.cc:302] T c280f4c27f17437fa7012deca745307b P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:16.540009 13969 raft_consensus.cc:2781] T c280f4c27f17437fa7012deca745307b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:16.540217 8917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c280f4c27f17437fa7012deca745307b" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:16.540428 8917 raft_consensus.cc:2454] T c280f4c27f17437fa7012deca745307b P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:16.540695 13969 raft_consensus.cc:482] T c280f4c27f17437fa7012deca745307b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:16.540805 13969 raft_consensus.cc:3037] T c280f4c27f17437fa7012deca745307b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:16.542454 13969 raft_consensus.cc:504] T c280f4c27f17437fa7012deca745307b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:16.543447 8917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c280f4c27f17437fa7012deca745307b" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:16.543589 8917 raft_consensus.cc:3037] T c280f4c27f17437fa7012deca745307b P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:16.544198 14001 tablet.cc:1722] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:16.544556 14001 tablet.cc:1620] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b: MemRowSet was empty: no flush needed.
I20240503 02:00:16.544673 8917 raft_consensus.cc:2454] T c280f4c27f17437fa7012deca745307b P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:16.545131 6656 leader_election.cc:302] T c280f4c27f17437fa7012deca745307b P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9942306855404cfd891adf227dd65eb2, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:16.546456 6582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c280f4c27f17437fa7012deca745307b" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 02:00:16.546612 6582 raft_consensus.cc:3037] T c280f4c27f17437fa7012deca745307b P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:16.547000 14066 raft_consensus.cc:2781] T c280f4c27f17437fa7012deca745307b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:16.547086 14066 raft_consensus.cc:686] T c280f4c27f17437fa7012deca745307b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Becoming Leader. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Running, Role: LEADER
I20240503 02:00:16.547238 14066 consensus_queue.cc:237] T c280f4c27f17437fa7012deca745307b P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:16.547729 6582 raft_consensus.cc:2454] T c280f4c27f17437fa7012deca745307b P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:16.549646 13900 raft_consensus.cc:482] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:16.549742 13900 raft_consensus.cc:504] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:16.550093 13900 leader_election.cc:288] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 02:00:16.550515 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c14900a27ce4c7eb9c0ad0f0c34488d" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 02:00:16.550473 6312 catalog_manager.cc:5484] T c280f4c27f17437fa7012deca745307b P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:16.550637 6458 raft_consensus.cc:2454] T 2c14900a27ce4c7eb9c0ad0f0c34488d P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:16.550916 8981 leader_election.cc:302] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99ead02915954b4fab23b033ad6d546b, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:16.551065 13900 raft_consensus.cc:2781] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:16.551101 13900 raft_consensus.cc:482] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:16.551131 13900 raft_consensus.cc:3037] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:16.552156 13900 raft_consensus.cc:504] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:16.552592 13900 leader_election.cc:288] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Requested vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 02:00:16.554039 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c14900a27ce4c7eb9c0ad0f0c34488d" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 02:00:16.554188 6458 raft_consensus.cc:3037] T 2c14900a27ce4c7eb9c0ad0f0c34488d P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:16.554275 13969 leader_election.cc:288] T c280f4c27f17437fa7012deca745307b P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Requested vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:16.555240 6458 raft_consensus.cc:2454] T 2c14900a27ce4c7eb9c0ad0f0c34488d P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:16.555580 8981 leader_election.cc:302] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99ead02915954b4fab23b033ad6d546b, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:16.556318 13956 raft_consensus.cc:2781] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:16.556542 13956 raft_consensus.cc:686] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Becoming Leader. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Running, Role: LEADER
I20240503 02:00:16.556684 13956 consensus_queue.cc:237] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 99ead02915954b4fab23b033ad6d546b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:16.557754 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c14900a27ce4c7eb9c0ad0f0c34488d" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 02:00:16.557878 6851 raft_consensus.cc:3037] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:16.558828 6851 raft_consensus.cc:2454] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:16.557770 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c14900a27ce4c7eb9c0ad0f0c34488d" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 02:00:16.559094 6849 raft_consensus.cc:2362] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1. Re-sending same reply.
W20240503 02:00:16.560447 8851 consensus_peers.cc:485] T 5f51fe5f799f4e479b62f5917a2c2606 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:00:16.561012 14090 tablet.cc:1722] T 90a0a91005714af39c8f0e28b4446e01 P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:16.561389 14090 tablet.cc:1620] T 90a0a91005714af39c8f0e28b4446e01 P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
I20240503 02:00:16.561591 6308 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=7d9cea0b1a7a4cddabd18ed44a365181] alter complete (version 1)
I20240503 02:00:16.564208 6309 catalog_manager.cc:5576] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211) does not have the latest schema for tablet 90a0a91005714af39c8f0e28b4446e01 (table rebalancer_dynamic_table [id=7d9cea0b1a7a4cddabd18ed44a365181]). Expected version 1 got 0
I20240503 02:00:16.564904 6308 catalog_manager.cc:5484] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 99ead02915954b4fab23b033ad6d546b reported cstate change: term changed from 0 to 1, leader changed from <none> to 99ead02915954b4fab23b033ad6d546b (127.0.99.134). New cstate: current_term: 1 leader_uuid: "99ead02915954b4fab23b033ad6d546b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:16.569403 6312 catalog_manager.cc:5576] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211) does not have the latest schema for tablet 90a0a91005714af39c8f0e28b4446e01 (table rebalancer_dynamic_table [id=7d9cea0b1a7a4cddabd18ed44a365181]). Expected version 1 got 0
I20240503 02:00:16.573462 6312 catalog_manager.cc:5576] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079) does not have the latest schema for tablet 90a0a91005714af39c8f0e28b4446e01 (table rebalancer_dynamic_table [id=7d9cea0b1a7a4cddabd18ed44a365181]). Expected version 1 got 0
I20240503 02:00:16.576496 14091 tablet.cc:1722] T 90a0a91005714af39c8f0e28b4446e01 P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:16.576871 14091 tablet.cc:1620] T 90a0a91005714af39c8f0e28b4446e01 P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
W20240503 02:00:16.586419 6636 tablet.cc:2369] T c280f4c27f17437fa7012deca745307b P 80fd18ef4d604c86b909d5c008a46b4f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 02:00:16.638167 9113 consensus_peers.cc:485] T b1740b877c1e4372a0337d99f5bcab0e P 2a01ace80baf4057b6bf1995cf88468b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:16.656208 6505 tablet.cc:2369] T 2c14900a27ce4c7eb9c0ad0f0c34488d P dec3e4a04d604f7eaaa3f077d081218a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:00:16.676721 6309 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
W20240503 02:00:16.677179 8982 consensus_peers.cc:485] T 13756a6e44814fe4b09a6b0f087546c3 P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:00:16.682971 14001 tablet.cc:1722] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:16.683363 14001 tablet.cc:1620] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b: MemRowSet was empty: no flush needed.
I20240503 02:00:16.683799 14091 tablet.cc:1722] T 90a0a91005714af39c8f0e28b4446e01 P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:16.684198 14091 tablet.cc:1620] T 90a0a91005714af39c8f0e28b4446e01 P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
W20240503 02:00:16.687206 6898 tablet.cc:2369] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 492448570cc94ed1879feb2ec3b9051d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:00:16.687014 14090 tablet.cc:1722] T 90a0a91005714af39c8f0e28b4446e01 P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:16.689318 14090 tablet.cc:1620] T 90a0a91005714af39c8f0e28b4446e01 P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
I20240503 02:00:16.690321 6309 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=7d9cea0b1a7a4cddabd18ed44a365181] alter complete (version 2)
W20240503 02:00:16.718788 6523 consensus_peers.cc:485] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:16.772617 6523 consensus_peers.cc:485] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:16.789618 8851 consensus_peers.cc:485] T f28586e70bd9473fbdc6b9126b666fa9 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:00:16.793658 6309 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:16.799192 14001 tablet.cc:1722] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:16.799604 14091 tablet.cc:1722] T 90a0a91005714af39c8f0e28b4446e01 P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:16.799928 14090 tablet.cc:1722] T 90a0a91005714af39c8f0e28b4446e01 P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:16.805409 6309 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=7d9cea0b1a7a4cddabd18ed44a365181] alter complete (version 3)
W20240503 02:00:16.821132 8851 consensus_peers.cc:485] T 7da8f614a70e4177b70e7c2f31d22a10 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:00:16.908059 6308 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:16.908246 6308 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:16.910475 6308 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet 90a0a91005714af39c8f0e28b4446e01
I20240503 02:00:16.911142 9025 tablet_service.cc:1514] Processing DeleteTablet for tablet 90a0a91005714af39c8f0e28b4446e01 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:16 UTC) from {username='slave'} at 127.0.0.1:52584
I20240503 02:00:16.911199 8897 tablet_service.cc:1514] Processing DeleteTablet for tablet 90a0a91005714af39c8f0e28b4446e01 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:16 UTC) from {username='slave'} at 127.0.0.1:40296
I20240503 02:00:16.911561 14092 tablet_replica.cc:330] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b: stopping tablet replica
I20240503 02:00:16.911561 14093 tablet_replica.cc:330] T 90a0a91005714af39c8f0e28b4446e01 P 9942306855404cfd891adf227dd65eb2: stopping tablet replica
I20240503 02:00:16.911710 14093 raft_consensus.cc:2229] T 90a0a91005714af39c8f0e28b4446e01 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:16.911710 14092 raft_consensus.cc:2229] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:16.911885 9158 tablet_service.cc:1514] Processing DeleteTablet for tablet 90a0a91005714af39c8f0e28b4446e01 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:16 UTC) from {username='slave'} at 127.0.0.1:51940
I20240503 02:00:16.911938 14092 raft_consensus.cc:2258] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:16.912226 14093 raft_consensus.cc:2258] T 90a0a91005714af39c8f0e28b4446e01 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:16.912362 14094 tablet_replica.cc:330] T 90a0a91005714af39c8f0e28b4446e01 P 2a01ace80baf4057b6bf1995cf88468b: stopping tablet replica
I20240503 02:00:16.912564 14094 raft_consensus.cc:2229] T 90a0a91005714af39c8f0e28b4446e01 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:16.912664 14092 ts_tablet_manager.cc:1875] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:16.912698 14094 raft_consensus.cc:2258] T 90a0a91005714af39c8f0e28b4446e01 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:16.912881 14093 ts_tablet_manager.cc:1875] T 90a0a91005714af39c8f0e28b4446e01 P 9942306855404cfd891adf227dd65eb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:16.913209 14094 ts_tablet_manager.cc:1875] T 90a0a91005714af39c8f0e28b4446e01 P 2a01ace80baf4057b6bf1995cf88468b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:16.915357 14094 ts_tablet_manager.cc:1888] T 90a0a91005714af39c8f0e28b4446e01 P 2a01ace80baf4057b6bf1995cf88468b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:16.915391 14092 ts_tablet_manager.cc:1888] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:16.915400 14093 ts_tablet_manager.cc:1888] T 90a0a91005714af39c8f0e28b4446e01 P 9942306855404cfd891adf227dd65eb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:16.915441 14094 log.cc:1196] T 90a0a91005714af39c8f0e28b4446e01 P 2a01ace80baf4057b6bf1995cf88468b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/90a0a91005714af39c8f0e28b4446e01
I20240503 02:00:16.915488 14092 log.cc:1196] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/90a0a91005714af39c8f0e28b4446e01
I20240503 02:00:16.915488 14093 log.cc:1196] T 90a0a91005714af39c8f0e28b4446e01 P 9942306855404cfd891adf227dd65eb2: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/90a0a91005714af39c8f0e28b4446e01
I20240503 02:00:16.915745 14094 ts_tablet_manager.cc:1909] T 90a0a91005714af39c8f0e28b4446e01 P 2a01ace80baf4057b6bf1995cf88468b: Deleting consensus metadata
I20240503 02:00:16.915803 14092 ts_tablet_manager.cc:1909] T 90a0a91005714af39c8f0e28b4446e01 P 99ead02915954b4fab23b033ad6d546b: Deleting consensus metadata
I20240503 02:00:16.915803 14093 ts_tablet_manager.cc:1909] T 90a0a91005714af39c8f0e28b4446e01 P 9942306855404cfd891adf227dd65eb2: Deleting consensus metadata
I20240503 02:00:16.916632 6301 catalog_manager.cc:4830] TS 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157): tablet 90a0a91005714af39c8f0e28b4446e01 (table rebalancer_dynamic_table_new_name [id=7d9cea0b1a7a4cddabd18ed44a365181]) successfully deleted
I20240503 02:00:16.916639 6300 catalog_manager.cc:4830] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079): tablet 90a0a91005714af39c8f0e28b4446e01 (table rebalancer_dynamic_table_new_name [id=7d9cea0b1a7a4cddabd18ed44a365181]) successfully deleted
I20240503 02:00:16.916898 6303 catalog_manager.cc:4830] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211): tablet 90a0a91005714af39c8f0e28b4446e01 (table rebalancer_dynamic_table_new_name [id=7d9cea0b1a7a4cddabd18ed44a365181]) successfully deleted
I20240503 02:00:16.929770 13975 consensus_queue.cc:1035] T c280f4c27f17437fa7012deca745307b P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:16.933979 13975 consensus_queue.cc:1035] T c280f4c27f17437fa7012deca745307b P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:00:16.935741 6654 consensus_peers.cc:485] T cea1ea50fa434d31b42451f84f26bc76 P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:16.968093 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:16.974040 13956 consensus_queue.cc:1035] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:16.978891 13943 consensus_queue.cc:1035] T 2c14900a27ce4c7eb9c0ad0f0c34488d P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:16.991117 14053 consensus_queue.cc:1035] T f93281ad992049c38b8d446b09f821c0 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:16.997408 13968 consensus_queue.cc:1035] T f93281ad992049c38b8d446b09f821c0 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:00:17.009279 8982 consensus_peers.cc:485] T 1a7ee79312ea4bad816264375ca5bb2d P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:00:17.013185 6312 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:00:17.018968 9158 tablet_service.cc:1467] Processing CreateTablet for tablet e90a9c9fbf844f5b8ff12e0973ab4ebc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=db4fa84dbe7b41dfbee6df24fc2fcabc]), partition=
I20240503 02:00:17.019287 9158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e90a9c9fbf844f5b8ff12e0973ab4ebc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:17.019475 8897 tablet_service.cc:1467] Processing CreateTablet for tablet e90a9c9fbf844f5b8ff12e0973ab4ebc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=db4fa84dbe7b41dfbee6df24fc2fcabc]), partition=
I20240503 02:00:17.019721 8897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e90a9c9fbf844f5b8ff12e0973ab4ebc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:17.021378 14107 tablet_bootstrap.cc:492] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:17.022663 14107 tablet_bootstrap.cc:654] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:17.024735 14107 tablet_bootstrap.cc:492] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:17.024976 14107 ts_tablet_manager.cc:1367] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:17.025429 6569 tablet_service.cc:1467] Processing CreateTablet for tablet e90a9c9fbf844f5b8ff12e0973ab4ebc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=db4fa84dbe7b41dfbee6df24fc2fcabc]), partition=
I20240503 02:00:17.025544 14107 raft_consensus.cc:348] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.025660 14107 raft_consensus.cc:374] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:17.025688 14107 raft_consensus.cc:729] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:17.025700 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e90a9c9fbf844f5b8ff12e0973ab4ebc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:17.025826 14107 consensus_queue.cc:260] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.027223 14107 ts_tablet_manager.cc:1398] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:17.028754 14111 tablet_bootstrap.cc:492] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:17.029875 14111 tablet_bootstrap.cc:654] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:17.032233 14111 tablet_bootstrap.cc:492] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:17.032450 14111 ts_tablet_manager.cc:1367] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:17.033006 14111 raft_consensus.cc:348] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.033217 14111 raft_consensus.cc:374] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:17.033296 14111 raft_consensus.cc:729] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:17.033478 14111 consensus_queue.cc:260] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.034010 14111 ts_tablet_manager.cc:1398] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:17.038473 14108 tablet_bootstrap.cc:492] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:17.039407 14108 tablet_bootstrap.cc:654] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:17.040511 14108 tablet_bootstrap.cc:492] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:17.040611 14108 ts_tablet_manager.cc:1367] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:17.041110 14108 raft_consensus.cc:348] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.041273 14108 raft_consensus.cc:374] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:17.041340 14108 raft_consensus.cc:729] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:17.041515 14108 consensus_queue.cc:260] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.043776 14108 ts_tablet_manager.cc:1398] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
W20240503 02:00:17.056495 8982 consensus_peers.cc:485] T beeea71f02ae4c89a1af13d3584ef70a P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:00:17.091604 13968 raft_consensus.cc:482] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:17.091769 13968 raft_consensus.cc:504] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.092278 13968 leader_election.cc:288] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:17.092630 6582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e90a9c9fbf844f5b8ff12e0973ab4ebc" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 02:00:17.092682 8917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e90a9c9fbf844f5b8ff12e0973ab4ebc" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:17.092767 6582 raft_consensus.cc:2454] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:17.092845 8917 raft_consensus.cc:2454] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:17.093055 9114 leader_election.cc:302] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 80fd18ef4d604c86b909d5c008a46b4f; no voters:
I20240503 02:00:17.093277 13968 raft_consensus.cc:2781] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:17.093322 13968 raft_consensus.cc:482] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:17.093358 13968 raft_consensus.cc:3037] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:17.094273 13968 raft_consensus.cc:504] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.094606 13968 leader_election.cc:288] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Requested vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:17.094946 8917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e90a9c9fbf844f5b8ff12e0973ab4ebc" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:17.094969 6582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e90a9c9fbf844f5b8ff12e0973ab4ebc" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 02:00:17.095080 8917 raft_consensus.cc:3037] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:17.095091 6582 raft_consensus.cc:3037] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:17.095917 6312 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_extra_table_44"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:17.096017 8917 raft_consensus.cc:2454] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:17.096021 6582 raft_consensus.cc:2454] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:17.096297 9115 leader_election.cc:302] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:17.096798 13968 raft_consensus.cc:2781] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:17.096977 13968 raft_consensus.cc:686] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Becoming Leader. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Running, Role: LEADER
I20240503 02:00:17.097126 13968 consensus_queue.cc:237] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.099771 6308 catalog_manager.cc:5484] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135). New cstate: current_term: 1 leader_uuid: "2a01ace80baf4057b6bf1995cf88468b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:17.104457 9158 tablet_service.cc:1467] Processing CreateTablet for tablet 83f5452445214999951ede92cea0d569 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=21134e2d123046cd9d1503cb044e8384]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:17.104692 6569 tablet_service.cc:1467] Processing CreateTablet for tablet b40e5d2709bf488c98f0f65c13cfce03 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=21134e2d123046cd9d1503cb044e8384]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:17.104801 9158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83f5452445214999951ede92cea0d569. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:17.104964 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b40e5d2709bf488c98f0f65c13cfce03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:17.106768 14107 tablet_bootstrap.cc:492] T 83f5452445214999951ede92cea0d569 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:17.107028 9025 tablet_service.cc:1467] Processing CreateTablet for tablet 2cf429a418a6493588685b18d864d6b1 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=21134e2d123046cd9d1503cb044e8384]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:17.107306 9025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cf429a418a6493588685b18d864d6b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:17.107252 6436 tablet_service.cc:1467] Processing CreateTablet for tablet 83f5452445214999951ede92cea0d569 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=21134e2d123046cd9d1503cb044e8384]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:17.107354 6438 tablet_service.cc:1467] Processing CreateTablet for tablet 2cf429a418a6493588685b18d864d6b1 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=21134e2d123046cd9d1503cb044e8384]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:17.107486 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83f5452445214999951ede92cea0d569. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:17.107595 6438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cf429a418a6493588685b18d864d6b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:17.108247 8897 tablet_service.cc:1467] Processing CreateTablet for tablet 83f5452445214999951ede92cea0d569 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=21134e2d123046cd9d1503cb044e8384]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:17.108533 8897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83f5452445214999951ede92cea0d569. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:17.108656 8880 tablet_service.cc:1467] Processing CreateTablet for tablet b40e5d2709bf488c98f0f65c13cfce03 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=21134e2d123046cd9d1503cb044e8384]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:17.108886 8880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b40e5d2709bf488c98f0f65c13cfce03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:17.109874 6831 tablet_service.cc:1467] Processing CreateTablet for tablet b40e5d2709bf488c98f0f65c13cfce03 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=21134e2d123046cd9d1503cb044e8384]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:17.109897 14116 tablet_bootstrap.cc:492] T 2cf429a418a6493588685b18d864d6b1 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:17.110147 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b40e5d2709bf488c98f0f65c13cfce03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:17.110958 8887 tablet_service.cc:1467] Processing CreateTablet for tablet 2cf429a418a6493588685b18d864d6b1 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=21134e2d123046cd9d1503cb044e8384]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:17.111187 8887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cf429a418a6493588685b18d864d6b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:17.112504 14108 tablet_bootstrap.cc:492] T 83f5452445214999951ede92cea0d569 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:17.112686 14116 tablet_bootstrap.cc:654] T 2cf429a418a6493588685b18d864d6b1 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:17.113490 14108 tablet_bootstrap.cc:654] T 83f5452445214999951ede92cea0d569 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:17.113720 14111 tablet_bootstrap.cc:492] T b40e5d2709bf488c98f0f65c13cfce03 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:17.114739 14111 tablet_bootstrap.cc:654] T b40e5d2709bf488c98f0f65c13cfce03 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:17.115569 14108 tablet_bootstrap.cc:492] T 83f5452445214999951ede92cea0d569 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:17.115887 14108 ts_tablet_manager.cc:1367] T 83f5452445214999951ede92cea0d569 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:17.117913 14107 tablet_bootstrap.cc:654] T 83f5452445214999951ede92cea0d569 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:17.117404 14108 raft_consensus.cc:348] T 83f5452445214999951ede92cea0d569 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.119376 14108 raft_consensus.cc:374] T 83f5452445214999951ede92cea0d569 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:17.119647 14108 raft_consensus.cc:729] T 83f5452445214999951ede92cea0d569 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:17.120332 14115 tablet_bootstrap.cc:492] T 2cf429a418a6493588685b18d864d6b1 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:17.120332 14108 consensus_queue.cc:260] T 83f5452445214999951ede92cea0d569 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.124405 14116 tablet_bootstrap.cc:492] T 2cf429a418a6493588685b18d864d6b1 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:17.124490 14115 tablet_bootstrap.cc:654] T 2cf429a418a6493588685b18d864d6b1 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:17.124531 14116 ts_tablet_manager.cc:1367] T 2cf429a418a6493588685b18d864d6b1 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240503 02:00:17.125101 14116 raft_consensus.cc:348] T 2cf429a418a6493588685b18d864d6b1 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.125229 14116 raft_consensus.cc:374] T 2cf429a418a6493588685b18d864d6b1 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:17.125259 14116 raft_consensus.cc:729] T 2cf429a418a6493588685b18d864d6b1 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:17.125383 14116 consensus_queue.cc:260] T 2cf429a418a6493588685b18d864d6b1 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.125684 14115 tablet_bootstrap.cc:492] T 2cf429a418a6493588685b18d864d6b1 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:17.125690 14116 ts_tablet_manager.cc:1398] T 2cf429a418a6493588685b18d864d6b1 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:17.125773 14115 ts_tablet_manager.cc:1367] T 2cf429a418a6493588685b18d864d6b1 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20240503 02:00:17.125772 14107 tablet_bootstrap.cc:492] T 83f5452445214999951ede92cea0d569 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:17.125854 14107 ts_tablet_manager.cc:1367] T 83f5452445214999951ede92cea0d569 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.004s
I20240503 02:00:17.126267 14115 raft_consensus.cc:348] T 2cf429a418a6493588685b18d864d6b1 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.126309 14107 raft_consensus.cc:348] T 83f5452445214999951ede92cea0d569 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.126374 14115 raft_consensus.cc:374] T 2cf429a418a6493588685b18d864d6b1 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:17.126406 14115 raft_consensus.cc:729] T 2cf429a418a6493588685b18d864d6b1 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:17.126410 14107 raft_consensus.cc:374] T 83f5452445214999951ede92cea0d569 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:17.126438 14107 raft_consensus.cc:729] T 83f5452445214999951ede92cea0d569 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:17.126523 14115 consensus_queue.cc:260] T 2cf429a418a6493588685b18d864d6b1 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.126559 14107 consensus_queue.cc:260] T 83f5452445214999951ede92cea0d569 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.126763 14116 tablet_bootstrap.cc:492] T 83f5452445214999951ede92cea0d569 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:17.126821 14115 ts_tablet_manager.cc:1398] T 2cf429a418a6493588685b18d864d6b1 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:17.126845 14107 ts_tablet_manager.cc:1398] T 83f5452445214999951ede92cea0d569 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 02:00:17.128144 14116 tablet_bootstrap.cc:654] T 83f5452445214999951ede92cea0d569 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:17.128686 14108 ts_tablet_manager.cc:1398] T 83f5452445214999951ede92cea0d569 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.013s user 0.000s sys 0.003s
I20240503 02:00:17.128926 14108 tablet_bootstrap.cc:492] T b40e5d2709bf488c98f0f65c13cfce03 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:17.131194 14108 tablet_bootstrap.cc:654] T b40e5d2709bf488c98f0f65c13cfce03 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:17.132444 14117 tablet_bootstrap.cc:492] T b40e5d2709bf488c98f0f65c13cfce03 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:17.133796 14117 tablet_bootstrap.cc:654] T b40e5d2709bf488c98f0f65c13cfce03 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:17.135171 14116 tablet_bootstrap.cc:492] T 83f5452445214999951ede92cea0d569 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:17.135283 14116 ts_tablet_manager.cc:1367] T 83f5452445214999951ede92cea0d569 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.001s
I20240503 02:00:17.135941 14116 raft_consensus.cc:348] T 83f5452445214999951ede92cea0d569 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.136056 14116 raft_consensus.cc:374] T 83f5452445214999951ede92cea0d569 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:17.136086 14116 raft_consensus.cc:729] T 83f5452445214999951ede92cea0d569 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:17.136260 14116 consensus_queue.cc:260] T 83f5452445214999951ede92cea0d569 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.136554 14116 ts_tablet_manager.cc:1398] T 83f5452445214999951ede92cea0d569 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:17.136768 14108 tablet_bootstrap.cc:492] T b40e5d2709bf488c98f0f65c13cfce03 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:17.136893 14108 ts_tablet_manager.cc:1367] T b40e5d2709bf488c98f0f65c13cfce03 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.001s
I20240503 02:00:17.137308 14108 raft_consensus.cc:348] T b40e5d2709bf488c98f0f65c13cfce03 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.137405 14108 raft_consensus.cc:374] T b40e5d2709bf488c98f0f65c13cfce03 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:17.137428 14108 raft_consensus.cc:729] T b40e5d2709bf488c98f0f65c13cfce03 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:17.137548 14108 consensus_queue.cc:260] T b40e5d2709bf488c98f0f65c13cfce03 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.137814 14108 ts_tablet_manager.cc:1398] T b40e5d2709bf488c98f0f65c13cfce03 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:17.138073 14108 tablet_bootstrap.cc:492] T 2cf429a418a6493588685b18d864d6b1 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:17.139264 14108 tablet_bootstrap.cc:654] T 2cf429a418a6493588685b18d864d6b1 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:17.140630 14111 tablet_bootstrap.cc:492] T b40e5d2709bf488c98f0f65c13cfce03 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:17.140802 14111 ts_tablet_manager.cc:1367] T b40e5d2709bf488c98f0f65c13cfce03 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.000s
I20240503 02:00:17.140913 14108 tablet_bootstrap.cc:492] T 2cf429a418a6493588685b18d864d6b1 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:17.141047 14108 ts_tablet_manager.cc:1367] T 2cf429a418a6493588685b18d864d6b1 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:00:17.141489 14111 raft_consensus.cc:348] T b40e5d2709bf488c98f0f65c13cfce03 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.141613 14111 raft_consensus.cc:374] T b40e5d2709bf488c98f0f65c13cfce03 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:17.141640 14111 raft_consensus.cc:729] T b40e5d2709bf488c98f0f65c13cfce03 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:17.141628 14108 raft_consensus.cc:348] T 2cf429a418a6493588685b18d864d6b1 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.141748 14108 raft_consensus.cc:374] T 2cf429a418a6493588685b18d864d6b1 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:17.141779 14108 raft_consensus.cc:729] T 2cf429a418a6493588685b18d864d6b1 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:17.141796 14111 consensus_queue.cc:260] T b40e5d2709bf488c98f0f65c13cfce03 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.141902 14108 consensus_queue.cc:260] T 2cf429a418a6493588685b18d864d6b1 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.142149 14111 ts_tablet_manager.cc:1398] T b40e5d2709bf488c98f0f65c13cfce03 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:17.142308 14108 ts_tablet_manager.cc:1398] T 2cf429a418a6493588685b18d864d6b1 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:17.143401 14117 tablet_bootstrap.cc:492] T b40e5d2709bf488c98f0f65c13cfce03 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:17.143507 14117 ts_tablet_manager.cc:1367] T b40e5d2709bf488c98f0f65c13cfce03 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 02:00:17.144001 14117 raft_consensus.cc:348] T b40e5d2709bf488c98f0f65c13cfce03 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.144698 14117 raft_consensus.cc:374] T b40e5d2709bf488c98f0f65c13cfce03 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:17.144795 14117 raft_consensus.cc:729] T b40e5d2709bf488c98f0f65c13cfce03 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:17.145737 14117 consensus_queue.cc:260] T b40e5d2709bf488c98f0f65c13cfce03 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.146136 14117 ts_tablet_manager.cc:1398] T b40e5d2709bf488c98f0f65c13cfce03 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:00:17.216567 6309 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:17.221279 6582 raft_consensus.cc:1261] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer 2a01ace80baf4057b6bf1995cf88468b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:17.221279 8917 raft_consensus.cc:1261] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Refusing update from remote peer 2a01ace80baf4057b6bf1995cf88468b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:17.221663 13968 consensus_queue.cc:1035] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:17.221794 14053 consensus_queue.cc:1035] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:17.225106 6309 catalog_manager.cc:5576] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211) does not have the latest schema for tablet e90a9c9fbf844f5b8ff12e0973ab4ebc (table rebalancer_dynamic_table [id=db4fa84dbe7b41dfbee6df24fc2fcabc]). Expected version 1 got 0
I20240503 02:00:17.225052 14090 tablet.cc:1722] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:17.225456 14090 tablet.cc:1620] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
I20240503 02:00:17.225499 6312 catalog_manager.cc:5576] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863) does not have the latest schema for tablet e90a9c9fbf844f5b8ff12e0973ab4ebc (table rebalancer_dynamic_table [id=db4fa84dbe7b41dfbee6df24fc2fcabc]). Expected version 1 got 0
I20240503 02:00:17.225979 14047 raft_consensus.cc:482] T b40e5d2709bf488c98f0f65c13cfce03 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:17.226136 14047 raft_consensus.cc:504] T b40e5d2709bf488c98f0f65c13cfce03 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.226640 14047 leader_election.cc:288] T b40e5d2709bf488c98f0f65c13cfce03 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:17.227066 6582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b40e5d2709bf488c98f0f65c13cfce03" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 02:00:17.227210 6582 raft_consensus.cc:2454] T b40e5d2709bf488c98f0f65c13cfce03 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 02:00:17.227389 14091 tablet.cc:1722] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:17.227743 14091 tablet.cc:1620] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
I20240503 02:00:17.228282 6786 leader_election.cc:302] T b40e5d2709bf488c98f0f65c13cfce03 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 80fd18ef4d604c86b909d5c008a46b4f; no voters:
I20240503 02:00:17.228521 14047 raft_consensus.cc:2781] T b40e5d2709bf488c98f0f65c13cfce03 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:17.228577 14047 raft_consensus.cc:482] T b40e5d2709bf488c98f0f65c13cfce03 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:17.228610 14047 raft_consensus.cc:3037] T b40e5d2709bf488c98f0f65c13cfce03 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:17.229415 8917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b40e5d2709bf488c98f0f65c13cfce03" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:17.229566 8917 raft_consensus.cc:2454] T b40e5d2709bf488c98f0f65c13cfce03 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 02:00:17.229717 14047 raft_consensus.cc:504] T b40e5d2709bf488c98f0f65c13cfce03 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.230094 14047 leader_election.cc:288] T b40e5d2709bf488c98f0f65c13cfce03 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Requested vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:17.230615 6582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b40e5d2709bf488c98f0f65c13cfce03" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 02:00:17.230753 6582 raft_consensus.cc:3037] T b40e5d2709bf488c98f0f65c13cfce03 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:17.231398 8917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b40e5d2709bf488c98f0f65c13cfce03" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:17.231519 8917 raft_consensus.cc:3037] T b40e5d2709bf488c98f0f65c13cfce03 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:17.231886 6582 raft_consensus.cc:2454] T b40e5d2709bf488c98f0f65c13cfce03 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 02:00:17.232297 6786 leader_election.cc:302] T b40e5d2709bf488c98f0f65c13cfce03 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 80fd18ef4d604c86b909d5c008a46b4f; no voters:
I20240503 02:00:17.232511 8917 raft_consensus.cc:2454] T b40e5d2709bf488c98f0f65c13cfce03 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 02:00:17.233364 14047 raft_consensus.cc:2781] T b40e5d2709bf488c98f0f65c13cfce03 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:17.233454 14047 raft_consensus.cc:686] T b40e5d2709bf488c98f0f65c13cfce03 P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Becoming Leader. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Running, Role: LEADER
I20240503 02:00:17.233597 14047 consensus_queue.cc:237] T b40e5d2709bf488c98f0f65c13cfce03 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.236629 14053 raft_consensus.cc:482] T 83f5452445214999951ede92cea0d569 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:17.236754 14053 raft_consensus.cc:504] T 83f5452445214999951ede92cea0d569 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.237089 14098 mvcc.cc:205] Tried to move back new op lower bound from 7023417824134352896 to 7023417823631486976. Current Snapshot: MvccSnapshot[applied={T|T < 7023417824134352896}]
I20240503 02:00:17.237207 14053 leader_election.cc:288] T 83f5452445214999951ede92cea0d569 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:17.237214 14132 tablet.cc:1722] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:17.237545 14132 tablet.cc:1620] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 80fd18ef4d604c86b909d5c008a46b4f: MemRowSet was empty: no flush needed.
I20240503 02:00:17.238103 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f5452445214999951ede92cea0d569" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 02:00:17.238252 6458 raft_consensus.cc:2454] T 83f5452445214999951ede92cea0d569 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:17.238636 9112 leader_election.cc:302] T 83f5452445214999951ede92cea0d569 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:17.238663 8917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f5452445214999951ede92cea0d569" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:17.238797 8917 raft_consensus.cc:2454] T 83f5452445214999951ede92cea0d569 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:17.238852 14053 raft_consensus.cc:2781] T 83f5452445214999951ede92cea0d569 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:17.238903 14053 raft_consensus.cc:482] T 83f5452445214999951ede92cea0d569 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:17.238934 14053 raft_consensus.cc:3037] T 83f5452445214999951ede92cea0d569 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:17.240021 14053 raft_consensus.cc:504] T 83f5452445214999951ede92cea0d569 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.241626 6308 catalog_manager.cc:5484] T b40e5d2709bf488c98f0f65c13cfce03 P 492448570cc94ed1879feb2ec3b9051d reported cstate change: term changed from 0 to 1, leader changed from <none> to 492448570cc94ed1879feb2ec3b9051d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:17.242256 6309 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=db4fa84dbe7b41dfbee6df24fc2fcabc] alter complete (version 1)
I20240503 02:00:17.242563 14053 leader_election.cc:288] T 83f5452445214999951ede92cea0d569 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Requested vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:17.242321 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f5452445214999951ede92cea0d569" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 02:00:17.242674 6458 raft_consensus.cc:3037] T 83f5452445214999951ede92cea0d569 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:17.243767 6458 raft_consensus.cc:2454] T 83f5452445214999951ede92cea0d569 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:17.244021 8917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f5452445214999951ede92cea0d569" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:17.244102 9112 leader_election.cc:302] T 83f5452445214999951ede92cea0d569 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:17.244156 8917 raft_consensus.cc:3037] T 83f5452445214999951ede92cea0d569 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:17.244364 14053 raft_consensus.cc:2781] T 83f5452445214999951ede92cea0d569 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:17.244482 14053 raft_consensus.cc:686] T 83f5452445214999951ede92cea0d569 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Becoming Leader. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Running, Role: LEADER
I20240503 02:00:17.244616 14053 consensus_queue.cc:237] T 83f5452445214999951ede92cea0d569 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.245194 8917 raft_consensus.cc:2454] T 83f5452445214999951ede92cea0d569 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:17.249415 6308 catalog_manager.cc:5484] T 83f5452445214999951ede92cea0d569 P 2a01ace80baf4057b6bf1995cf88468b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135). New cstate: current_term: 1 leader_uuid: "2a01ace80baf4057b6bf1995cf88468b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:17.274847 14056 raft_consensus.cc:482] T 2cf429a418a6493588685b18d864d6b1 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:17.274973 14056 raft_consensus.cc:504] T 2cf429a418a6493588685b18d864d6b1 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.275424 14056 leader_election.cc:288] T 2cf429a418a6493588685b18d864d6b1 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:17.275876 9039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cf429a418a6493588685b18d864d6b1" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:17.276052 9039 raft_consensus.cc:2454] T 2cf429a418a6493588685b18d864d6b1 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:17.276430 6392 leader_election.cc:302] T 2cf429a418a6493588685b18d864d6b1 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99ead02915954b4fab23b033ad6d546b, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:17.276446 8917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cf429a418a6493588685b18d864d6b1" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:17.276600 8917 raft_consensus.cc:2454] T 2cf429a418a6493588685b18d864d6b1 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:17.276727 14056 raft_consensus.cc:2781] T 2cf429a418a6493588685b18d864d6b1 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:17.276778 14056 raft_consensus.cc:482] T 2cf429a418a6493588685b18d864d6b1 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:17.276813 14056 raft_consensus.cc:3037] T 2cf429a418a6493588685b18d864d6b1 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:17.277935 14056 raft_consensus.cc:504] T 2cf429a418a6493588685b18d864d6b1 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.278306 14056 leader_election.cc:288] T 2cf429a418a6493588685b18d864d6b1 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Requested vote from peers 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:17.278945 9047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cf429a418a6493588685b18d864d6b1" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:17.279058 9047 raft_consensus.cc:3037] T 2cf429a418a6493588685b18d864d6b1 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:17.280094 9047 raft_consensus.cc:2454] T 2cf429a418a6493588685b18d864d6b1 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:17.280464 6392 leader_election.cc:302] T 2cf429a418a6493588685b18d864d6b1 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99ead02915954b4fab23b033ad6d546b, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:17.280894 8917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cf429a418a6493588685b18d864d6b1" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:17.281029 8917 raft_consensus.cc:3037] T 2cf429a418a6493588685b18d864d6b1 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:17.281788 8917 raft_consensus.cc:2454] T 2cf429a418a6493588685b18d864d6b1 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:17.281994 14056 raft_consensus.cc:2781] T 2cf429a418a6493588685b18d864d6b1 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:17.282172 14056 raft_consensus.cc:686] T 2cf429a418a6493588685b18d864d6b1 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Becoming Leader. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Running, Role: LEADER
I20240503 02:00:17.282341 14056 consensus_queue.cc:237] T 2cf429a418a6493588685b18d864d6b1 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.285866 6308 catalog_manager.cc:5484] T 2cf429a418a6493588685b18d864d6b1 P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: term changed from 0 to 1, leader changed from <none> to dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129). New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:17.347667 6308 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:17.353247 14090 tablet.cc:1722] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:17.353619 14090 tablet.cc:1620] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
I20240503 02:00:17.353634 14132 tablet.cc:1722] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:17.353969 14132 tablet.cc:1620] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 80fd18ef4d604c86b909d5c008a46b4f: MemRowSet was empty: no flush needed.
I20240503 02:00:17.356916 14091 tablet.cc:1722] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:17.357283 14091 tablet.cc:1620] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
I20240503 02:00:17.359094 6313 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=db4fa84dbe7b41dfbee6df24fc2fcabc] alter complete (version 2)
W20240503 02:00:17.394699 6301 catalog_manager.cc:4573] TS e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3d3463f1ff2b41a396bb1b039ddf1946: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:17.462828 6312 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:17.468237 14090 tablet.cc:1722] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:17.468791 14091 tablet.cc:1722] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:17.471163 14132 tablet.cc:1722] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:17.474341 6312 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=db4fa84dbe7b41dfbee6df24fc2fcabc] alter complete (version 3)
W20240503 02:00:17.539021 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:17.578639 6309 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:17.578817 6309 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:17.580955 6309 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet e90a9c9fbf844f5b8ff12e0973ab4ebc
I20240503 02:00:17.581604 9158 tablet_service.cc:1514] Processing DeleteTablet for tablet e90a9c9fbf844f5b8ff12e0973ab4ebc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:17 UTC) from {username='slave'} at 127.0.0.1:51940
I20240503 02:00:17.581696 8887 tablet_service.cc:1514] Processing DeleteTablet for tablet e90a9c9fbf844f5b8ff12e0973ab4ebc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:17 UTC) from {username='slave'} at 127.0.0.1:40296
I20240503 02:00:17.581966 14139 tablet_replica.cc:330] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b: stopping tablet replica
I20240503 02:00:17.582106 14139 raft_consensus.cc:2229] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:17.582103 6569 tablet_service.cc:1514] Processing DeleteTablet for tablet e90a9c9fbf844f5b8ff12e0973ab4ebc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:17 UTC) from {username='slave'} at 127.0.0.1:59846
I20240503 02:00:17.582336 14139 raft_consensus.cc:2258] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:17.582381 14141 tablet_replica.cc:330] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 02:00:17.582556 14141 raft_consensus.cc:2229] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:17.582690 14141 raft_consensus.cc:2258] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:17.582407 14140 tablet_replica.cc:330] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 9942306855404cfd891adf227dd65eb2: stopping tablet replica
I20240503 02:00:17.582861 14140 raft_consensus.cc:2229] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:17.583081 14139 ts_tablet_manager.cc:1875] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:17.583424 14141 ts_tablet_manager.cc:1875] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:17.584542 14140 raft_consensus.cc:2258] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:17.585150 14140 ts_tablet_manager.cc:1875] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 9942306855404cfd891adf227dd65eb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:17.585644 14141 ts_tablet_manager.cc:1888] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:17.585651 14139 ts_tablet_manager.cc:1888] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:17.585724 14139 log.cc:1196] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/e90a9c9fbf844f5b8ff12e0973ab4ebc
I20240503 02:00:17.585726 14141 log.cc:1196] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/e90a9c9fbf844f5b8ff12e0973ab4ebc
I20240503 02:00:17.586004 14139 ts_tablet_manager.cc:1909] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 2a01ace80baf4057b6bf1995cf88468b: Deleting consensus metadata
I20240503 02:00:17.586042 14141 ts_tablet_manager.cc:1909] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 80fd18ef4d604c86b909d5c008a46b4f: Deleting consensus metadata
I20240503 02:00:17.586822 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet e90a9c9fbf844f5b8ff12e0973ab4ebc (table rebalancer_dynamic_table_new_name [id=db4fa84dbe7b41dfbee6df24fc2fcabc]) successfully deleted
I20240503 02:00:17.586822 6300 catalog_manager.cc:4830] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079): tablet e90a9c9fbf844f5b8ff12e0973ab4ebc (table rebalancer_dynamic_table_new_name [id=db4fa84dbe7b41dfbee6df24fc2fcabc]) successfully deleted
I20240503 02:00:17.587491 14140 ts_tablet_manager.cc:1888] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 9942306855404cfd891adf227dd65eb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:17.587566 14140 log.cc:1196] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 9942306855404cfd891adf227dd65eb2: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/e90a9c9fbf844f5b8ff12e0973ab4ebc
I20240503 02:00:17.587775 14140 ts_tablet_manager.cc:1909] T e90a9c9fbf844f5b8ff12e0973ab4ebc P 9942306855404cfd891adf227dd65eb2: Deleting consensus metadata
I20240503 02:00:17.588786 6303 catalog_manager.cc:4830] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211): tablet e90a9c9fbf844f5b8ff12e0973ab4ebc (table rebalancer_dynamic_table_new_name [id=db4fa84dbe7b41dfbee6df24fc2fcabc]) successfully deleted
I20240503 02:00:17.682777 6309 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:00:17.688422 6699 tablet_service.cc:1467] Processing CreateTablet for tablet 89aa8aa378d540a5aec5f61b21b4695c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0d3383ad1add444f973b21aa7d708ad4]), partition=
I20240503 02:00:17.688422 6436 tablet_service.cc:1467] Processing CreateTablet for tablet 89aa8aa378d540a5aec5f61b21b4695c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0d3383ad1add444f973b21aa7d708ad4]), partition=
I20240503 02:00:17.688422 9025 tablet_service.cc:1467] Processing CreateTablet for tablet 89aa8aa378d540a5aec5f61b21b4695c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0d3383ad1add444f973b21aa7d708ad4]), partition=
I20240503 02:00:17.688719 6699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89aa8aa378d540a5aec5f61b21b4695c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:17.688727 9025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89aa8aa378d540a5aec5f61b21b4695c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:17.688727 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89aa8aa378d540a5aec5f61b21b4695c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:17.691222 14144 tablet_bootstrap.cc:492] T 89aa8aa378d540a5aec5f61b21b4695c P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:17.691373 14142 tablet_bootstrap.cc:492] T 89aa8aa378d540a5aec5f61b21b4695c P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:17.692447 14142 tablet_bootstrap.cc:654] T 89aa8aa378d540a5aec5f61b21b4695c P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:17.692476 14144 tablet_bootstrap.cc:654] T 89aa8aa378d540a5aec5f61b21b4695c P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:17.693153 14143 tablet_bootstrap.cc:492] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:17.694137 14143 tablet_bootstrap.cc:654] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:17.694638 14144 tablet_bootstrap.cc:492] T 89aa8aa378d540a5aec5f61b21b4695c P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:17.694636 14142 tablet_bootstrap.cc:492] T 89aa8aa378d540a5aec5f61b21b4695c P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:17.694742 14142 ts_tablet_manager.cc:1367] T 89aa8aa378d540a5aec5f61b21b4695c P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:00:17.694772 14144 ts_tablet_manager.cc:1367] T 89aa8aa378d540a5aec5f61b21b4695c P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 02:00:17.695176 14144 raft_consensus.cc:348] T 89aa8aa378d540a5aec5f61b21b4695c P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:17.695284 14144 raft_consensus.cc:374] T 89aa8aa378d540a5aec5f61b21b4695c P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:17.695255 14142 raft_consensus.cc:348] T 89aa8aa378d540a5aec5f61b21b4695c P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:17.695315 14144 raft_consensus.cc:729] T 89aa8aa378d540a5aec5f61b21b4695c P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:17.695366 14142 raft_consensus.cc:374] T 89aa8aa378d540a5aec5f61b21b4695c P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:17.695405 14142 raft_consensus.cc:729] T 89aa8aa378d540a5aec5f61b21b4695c P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:17.695466 14144 consensus_queue.cc:260] T 89aa8aa378d540a5aec5f61b21b4695c P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:17.695581 14142 consensus_queue.cc:260] T 89aa8aa378d540a5aec5f61b21b4695c P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:17.695760 14144 ts_tablet_manager.cc:1398] T 89aa8aa378d540a5aec5f61b21b4695c P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:17.695892 14142 ts_tablet_manager.cc:1398] T 89aa8aa378d540a5aec5f61b21b4695c P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:17.697423 14143 tablet_bootstrap.cc:492] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:17.697535 14143 ts_tablet_manager.cc:1367] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 02:00:17.698035 14143 raft_consensus.cc:348] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:17.698158 14143 raft_consensus.cc:374] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:17.698187 14143 raft_consensus.cc:729] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:17.699101 14143 consensus_queue.cc:260] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:17.699553 14143 ts_tablet_manager.cc:1398] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:00:17.733987 13968 consensus_queue.cc:1035] T 83f5452445214999951ede92cea0d569 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:17.737313 14135 consensus_queue.cc:1035] T 83f5452445214999951ede92cea0d569 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:17.776531 14056 consensus_queue.cc:1035] T 2cf429a418a6493588685b18d864d6b1 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:17.779151 13991 consensus_queue.cc:1035] T 2cf429a418a6493588685b18d864d6b1 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:17.815986 14106 consensus_queue.cc:1035] T b40e5d2709bf488c98f0f65c13cfce03 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:17.817005 6309 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_extra_table_46"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:17.820365 14106 consensus_queue.cc:1035] T b40e5d2709bf488c98f0f65c13cfce03 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:17.825361 13991 raft_consensus.cc:482] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:17.825495 13991 raft_consensus.cc:504] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:17.825980 13991 leader_election.cc:288] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:17.826565 6720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89aa8aa378d540a5aec5f61b21b4695c" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 02:00:17.826536 9047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89aa8aa378d540a5aec5f61b21b4695c" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:17.826694 9047 raft_consensus.cc:2454] T 89aa8aa378d540a5aec5f61b21b4695c P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:17.826694 6720 raft_consensus.cc:2454] T 89aa8aa378d540a5aec5f61b21b4695c P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:17.826989 6394 leader_election.cc:302] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2ac2eb7cc264047b9ab32fe3a4fc025, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:17.827232 13991 raft_consensus.cc:2781] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:17.827289 13991 raft_consensus.cc:482] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:17.827332 13991 raft_consensus.cc:3037] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:17.828332 6569 tablet_service.cc:1467] Processing CreateTablet for tablet 53b328f875b94dc8906adfdf763821a8 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=b8371b74307b4001945492dae50b8bf4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:17.828666 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53b328f875b94dc8906adfdf763821a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:17.828837 6699 tablet_service.cc:1467] Processing CreateTablet for tablet 0d03b39b002d44e38df27382b107cbbf (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=b8371b74307b4001945492dae50b8bf4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:17.829102 6699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d03b39b002d44e38df27382b107cbbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:17.829067 6567 tablet_service.cc:1467] Processing CreateTablet for tablet c6690a8e5aaa4d83b47899cc8eb96d3d (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=b8371b74307b4001945492dae50b8bf4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:17.829161 9158 tablet_service.cc:1467] Processing CreateTablet for tablet 0d03b39b002d44e38df27382b107cbbf (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=b8371b74307b4001945492dae50b8bf4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:17.829279 6567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6690a8e5aaa4d83b47899cc8eb96d3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:17.829406 9158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d03b39b002d44e38df27382b107cbbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:17.831387 14142 tablet_bootstrap.cc:492] T 0d03b39b002d44e38df27382b107cbbf P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:17.831962 8887 tablet_service.cc:1467] Processing CreateTablet for tablet 0d03b39b002d44e38df27382b107cbbf (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=b8371b74307b4001945492dae50b8bf4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:17.832234 8887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d03b39b002d44e38df27382b107cbbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:17.832408 14142 tablet_bootstrap.cc:654] T 0d03b39b002d44e38df27382b107cbbf P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:17.833235 9025 tablet_service.cc:1467] Processing CreateTablet for tablet 53b328f875b94dc8906adfdf763821a8 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=b8371b74307b4001945492dae50b8bf4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:17.833499 9025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53b328f875b94dc8906adfdf763821a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:17.833659 13991 raft_consensus.cc:504] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:17.834122 13991 leader_election.cc:288] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Requested vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:17.834501 6720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89aa8aa378d540a5aec5f61b21b4695c" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 02:00:17.834616 6720 raft_consensus.cc:3037] T 89aa8aa378d540a5aec5f61b21b4695c P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:17.834723 8880 tablet_service.cc:1467] Processing CreateTablet for tablet c6690a8e5aaa4d83b47899cc8eb96d3d (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=b8371b74307b4001945492dae50b8bf4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:17.834950 8880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6690a8e5aaa4d83b47899cc8eb96d3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:17.835487 6720 raft_consensus.cc:2454] T 89aa8aa378d540a5aec5f61b21b4695c P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:17.835620 14144 tablet_bootstrap.cc:492] T 53b328f875b94dc8906adfdf763821a8 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:17.835839 6831 tablet_service.cc:1467] Processing CreateTablet for tablet 53b328f875b94dc8906adfdf763821a8 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=b8371b74307b4001945492dae50b8bf4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:17.836212 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53b328f875b94dc8906adfdf763821a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:17.836225 6394 leader_election.cc:302] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2ac2eb7cc264047b9ab32fe3a4fc025, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:17.836444 13991 raft_consensus.cc:2781] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:17.836520 13991 raft_consensus.cc:686] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Becoming Leader. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Running, Role: LEADER
I20240503 02:00:17.836663 13991 consensus_queue.cc:237] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:17.836748 14144 tablet_bootstrap.cc:654] T 53b328f875b94dc8906adfdf763821a8 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:17.838084 14159 tablet_bootstrap.cc:492] T 53b328f875b94dc8906adfdf763821a8 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:17.838773 6829 tablet_service.cc:1467] Processing CreateTablet for tablet c6690a8e5aaa4d83b47899cc8eb96d3d (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=b8371b74307b4001945492dae50b8bf4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:17.839015 6829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6690a8e5aaa4d83b47899cc8eb96d3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:17.839118 14159 tablet_bootstrap.cc:654] T 53b328f875b94dc8906adfdf763821a8 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:17.839692 6312 catalog_manager.cc:5484] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: term changed from 0 to 1, leader changed from <none> to dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129). New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:17.841305 9047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89aa8aa378d540a5aec5f61b21b4695c" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:17.841452 9047 raft_consensus.cc:3037] T 89aa8aa378d540a5aec5f61b21b4695c P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:17.842187 14165 tablet_bootstrap.cc:492] T 53b328f875b94dc8906adfdf763821a8 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:17.842468 9047 raft_consensus.cc:2454] T 89aa8aa378d540a5aec5f61b21b4695c P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:17.843277 14162 tablet_bootstrap.cc:492] T 0d03b39b002d44e38df27382b107cbbf P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:17.843418 14165 tablet_bootstrap.cc:654] T 53b328f875b94dc8906adfdf763821a8 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:17.844957 14162 tablet_bootstrap.cc:654] T 0d03b39b002d44e38df27382b107cbbf P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:17.845995 14144 tablet_bootstrap.cc:492] T 53b328f875b94dc8906adfdf763821a8 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:17.846101 14144 ts_tablet_manager.cc:1367] T 53b328f875b94dc8906adfdf763821a8 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.001s
I20240503 02:00:17.846438 14162 tablet_bootstrap.cc:492] T 0d03b39b002d44e38df27382b107cbbf P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:17.846521 14144 raft_consensus.cc:348] T 53b328f875b94dc8906adfdf763821a8 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:17.846607 14162 ts_tablet_manager.cc:1367] T 0d03b39b002d44e38df27382b107cbbf P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:17.846645 14144 raft_consensus.cc:374] T 53b328f875b94dc8906adfdf763821a8 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:17.846679 14144 raft_consensus.cc:729] T 53b328f875b94dc8906adfdf763821a8 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:17.846745 14160 tablet_bootstrap.cc:492] T 0d03b39b002d44e38df27382b107cbbf P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:17.846799 14144 consensus_queue.cc:260] T 53b328f875b94dc8906adfdf763821a8 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:17.847080 14144 ts_tablet_manager.cc:1398] T 53b328f875b94dc8906adfdf763821a8 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:17.847977 14165 tablet_bootstrap.cc:492] T 53b328f875b94dc8906adfdf763821a8 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:17.848109 14165 ts_tablet_manager.cc:1367] T 53b328f875b94dc8906adfdf763821a8 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:00:17.848629 14165 raft_consensus.cc:348] T 53b328f875b94dc8906adfdf763821a8 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:17.848771 14165 raft_consensus.cc:374] T 53b328f875b94dc8906adfdf763821a8 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:17.848814 14165 raft_consensus.cc:729] T 53b328f875b94dc8906adfdf763821a8 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:17.848940 14165 consensus_queue.cc:260] T 53b328f875b94dc8906adfdf763821a8 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:17.849246 14165 ts_tablet_manager.cc:1398] T 53b328f875b94dc8906adfdf763821a8 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:17.849426 14165 tablet_bootstrap.cc:492] T c6690a8e5aaa4d83b47899cc8eb96d3d P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:17.850785 14165 tablet_bootstrap.cc:654] T c6690a8e5aaa4d83b47899cc8eb96d3d P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:17.850845 14160 tablet_bootstrap.cc:654] T 0d03b39b002d44e38df27382b107cbbf P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:17.851148 14142 tablet_bootstrap.cc:492] T 0d03b39b002d44e38df27382b107cbbf P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:17.851341 14142 ts_tablet_manager.cc:1367] T 0d03b39b002d44e38df27382b107cbbf P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.001s
I20240503 02:00:17.851941 14142 raft_consensus.cc:348] T 0d03b39b002d44e38df27382b107cbbf P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.852058 14142 raft_consensus.cc:374] T 0d03b39b002d44e38df27382b107cbbf P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:17.852059 14160 tablet_bootstrap.cc:492] T 0d03b39b002d44e38df27382b107cbbf P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:17.852090 14142 raft_consensus.cc:729] T 0d03b39b002d44e38df27382b107cbbf P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:17.852097 14159 tablet_bootstrap.cc:492] T 53b328f875b94dc8906adfdf763821a8 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:17.852149 14160 ts_tablet_manager.cc:1367] T 0d03b39b002d44e38df27382b107cbbf P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:00:17.852185 14159 ts_tablet_manager.cc:1367] T 53b328f875b94dc8906adfdf763821a8 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 02:00:17.852208 14142 consensus_queue.cc:260] T 0d03b39b002d44e38df27382b107cbbf P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.852550 14142 ts_tablet_manager.cc:1398] T 0d03b39b002d44e38df27382b107cbbf P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:17.852634 14160 raft_consensus.cc:348] T 0d03b39b002d44e38df27382b107cbbf P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.852671 14159 raft_consensus.cc:348] T 53b328f875b94dc8906adfdf763821a8 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:17.852746 14160 raft_consensus.cc:374] T 0d03b39b002d44e38df27382b107cbbf P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:17.852770 14159 raft_consensus.cc:374] T 53b328f875b94dc8906adfdf763821a8 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:17.852782 14160 raft_consensus.cc:729] T 0d03b39b002d44e38df27382b107cbbf P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:17.852808 14159 raft_consensus.cc:729] T 53b328f875b94dc8906adfdf763821a8 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:17.852905 14160 consensus_queue.cc:260] T 0d03b39b002d44e38df27382b107cbbf P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.852918 14159 consensus_queue.cc:260] T 53b328f875b94dc8906adfdf763821a8 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:17.853191 14159 ts_tablet_manager.cc:1398] T 53b328f875b94dc8906adfdf763821a8 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:17.853195 14160 ts_tablet_manager.cc:1398] T 0d03b39b002d44e38df27382b107cbbf P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:17.853364 14159 tablet_bootstrap.cc:492] T c6690a8e5aaa4d83b47899cc8eb96d3d P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:17.854362 14159 tablet_bootstrap.cc:654] T c6690a8e5aaa4d83b47899cc8eb96d3d P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:17.858774 14162 raft_consensus.cc:348] T 0d03b39b002d44e38df27382b107cbbf P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.859136 14162 raft_consensus.cc:374] T 0d03b39b002d44e38df27382b107cbbf P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:17.859472 14162 raft_consensus.cc:729] T 0d03b39b002d44e38df27382b107cbbf P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:17.859660 14162 consensus_queue.cc:260] T 0d03b39b002d44e38df27382b107cbbf P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.859987 14162 ts_tablet_manager.cc:1398] T 0d03b39b002d44e38df27382b107cbbf P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.013s user 0.001s sys 0.000s
I20240503 02:00:17.860118 14159 tablet_bootstrap.cc:492] T c6690a8e5aaa4d83b47899cc8eb96d3d P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:17.860153 14162 tablet_bootstrap.cc:492] T c6690a8e5aaa4d83b47899cc8eb96d3d P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:17.860226 14159 ts_tablet_manager.cc:1367] T c6690a8e5aaa4d83b47899cc8eb96d3d P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:00:17.860960 14159 raft_consensus.cc:348] T c6690a8e5aaa4d83b47899cc8eb96d3d P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:17.861194 14159 raft_consensus.cc:374] T c6690a8e5aaa4d83b47899cc8eb96d3d P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:17.861219 14162 tablet_bootstrap.cc:654] T c6690a8e5aaa4d83b47899cc8eb96d3d P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:17.861250 14165 tablet_bootstrap.cc:492] T c6690a8e5aaa4d83b47899cc8eb96d3d P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:17.861378 14159 raft_consensus.cc:729] T c6690a8e5aaa4d83b47899cc8eb96d3d P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:17.861467 14165 ts_tablet_manager.cc:1367] T c6690a8e5aaa4d83b47899cc8eb96d3d P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 02:00:17.861617 14159 consensus_queue.cc:260] T c6690a8e5aaa4d83b47899cc8eb96d3d P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:17.862381 14162 tablet_bootstrap.cc:492] T c6690a8e5aaa4d83b47899cc8eb96d3d P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:17.862470 14162 ts_tablet_manager.cc:1367] T c6690a8e5aaa4d83b47899cc8eb96d3d P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:17.862740 14159 ts_tablet_manager.cc:1398] T c6690a8e5aaa4d83b47899cc8eb96d3d P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:17.862917 14162 raft_consensus.cc:348] T c6690a8e5aaa4d83b47899cc8eb96d3d P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:17.863018 14162 raft_consensus.cc:374] T c6690a8e5aaa4d83b47899cc8eb96d3d P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:17.863049 14162 raft_consensus.cc:729] T c6690a8e5aaa4d83b47899cc8eb96d3d P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:17.863109 14165 raft_consensus.cc:348] T c6690a8e5aaa4d83b47899cc8eb96d3d P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:17.863174 14162 consensus_queue.cc:260] T c6690a8e5aaa4d83b47899cc8eb96d3d P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:17.863221 14165 raft_consensus.cc:374] T c6690a8e5aaa4d83b47899cc8eb96d3d P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:17.863449 14162 ts_tablet_manager.cc:1398] T c6690a8e5aaa4d83b47899cc8eb96d3d P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:17.863255 14165 raft_consensus.cc:729] T c6690a8e5aaa4d83b47899cc8eb96d3d P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:17.864279 14165 consensus_queue.cc:260] T c6690a8e5aaa4d83b47899cc8eb96d3d P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:17.864665 14165 ts_tablet_manager.cc:1398] T c6690a8e5aaa4d83b47899cc8eb96d3d P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:00:17.896737 13954 raft_consensus.cc:482] T c6690a8e5aaa4d83b47899cc8eb96d3d P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:17.896862 13954 raft_consensus.cc:504] T c6690a8e5aaa4d83b47899cc8eb96d3d P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:17.897339 13954 leader_election.cc:288] T c6690a8e5aaa4d83b47899cc8eb96d3d P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:17.897706 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6690a8e5aaa4d83b47899cc8eb96d3d" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 02:00:17.897759 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6690a8e5aaa4d83b47899cc8eb96d3d" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:17.897836 6849 raft_consensus.cc:2454] T c6690a8e5aaa4d83b47899cc8eb96d3d P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:17.897907 8916 raft_consensus.cc:2454] T c6690a8e5aaa4d83b47899cc8eb96d3d P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:17.898133 6525 leader_election.cc:302] T c6690a8e5aaa4d83b47899cc8eb96d3d P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 80fd18ef4d604c86b909d5c008a46b4f; no voters:
I20240503 02:00:17.898293 13954 raft_consensus.cc:2781] T c6690a8e5aaa4d83b47899cc8eb96d3d P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:17.898347 13954 raft_consensus.cc:482] T c6690a8e5aaa4d83b47899cc8eb96d3d P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:17.898391 13954 raft_consensus.cc:3037] T c6690a8e5aaa4d83b47899cc8eb96d3d P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:17.899238 13954 raft_consensus.cc:504] T c6690a8e5aaa4d83b47899cc8eb96d3d P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:17.899562 13954 leader_election.cc:288] T c6690a8e5aaa4d83b47899cc8eb96d3d P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Requested vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:17.899972 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6690a8e5aaa4d83b47899cc8eb96d3d" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 02:00:17.899972 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6690a8e5aaa4d83b47899cc8eb96d3d" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:17.900104 6849 raft_consensus.cc:3037] T c6690a8e5aaa4d83b47899cc8eb96d3d P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:17.900104 8916 raft_consensus.cc:3037] T c6690a8e5aaa4d83b47899cc8eb96d3d P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:17.901118 8916 raft_consensus.cc:2454] T c6690a8e5aaa4d83b47899cc8eb96d3d P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:17.901144 6849 raft_consensus.cc:2454] T c6690a8e5aaa4d83b47899cc8eb96d3d P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:17.901413 6525 leader_election.cc:302] T c6690a8e5aaa4d83b47899cc8eb96d3d P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:17.901672 13954 raft_consensus.cc:2781] T c6690a8e5aaa4d83b47899cc8eb96d3d P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:17.901758 13954 raft_consensus.cc:686] T c6690a8e5aaa4d83b47899cc8eb96d3d P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Becoming Leader. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Running, Role: LEADER
I20240503 02:00:17.901894 13954 consensus_queue.cc:237] T c6690a8e5aaa4d83b47899cc8eb96d3d P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:17.904089 6309 catalog_manager.cc:5484] T c6690a8e5aaa4d83b47899cc8eb96d3d P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:17.907795 14138 raft_consensus.cc:482] T 0d03b39b002d44e38df27382b107cbbf P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:17.907930 14138 raft_consensus.cc:504] T 0d03b39b002d44e38df27382b107cbbf P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.908365 14138 leader_election.cc:288] T 0d03b39b002d44e38df27382b107cbbf P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:17.908773 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d03b39b002d44e38df27382b107cbbf" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:17.908916 9178 raft_consensus.cc:2454] T 0d03b39b002d44e38df27382b107cbbf P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:17.908900 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d03b39b002d44e38df27382b107cbbf" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:17.909020 8916 raft_consensus.cc:2454] T 0d03b39b002d44e38df27382b107cbbf P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:17.909343 6656 leader_election.cc:302] T 0d03b39b002d44e38df27382b107cbbf P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9942306855404cfd891adf227dd65eb2, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:17.909509 14138 raft_consensus.cc:2781] T 0d03b39b002d44e38df27382b107cbbf P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:17.909560 14138 raft_consensus.cc:482] T 0d03b39b002d44e38df27382b107cbbf P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:17.909595 14138 raft_consensus.cc:3037] T 0d03b39b002d44e38df27382b107cbbf P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:17.910559 14138 raft_consensus.cc:504] T 0d03b39b002d44e38df27382b107cbbf P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.910881 14138 leader_election.cc:288] T 0d03b39b002d44e38df27382b107cbbf P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Requested vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:17.911237 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d03b39b002d44e38df27382b107cbbf" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:17.911237 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d03b39b002d44e38df27382b107cbbf" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:17.911365 9178 raft_consensus.cc:3037] T 0d03b39b002d44e38df27382b107cbbf P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:17.911365 8916 raft_consensus.cc:3037] T 0d03b39b002d44e38df27382b107cbbf P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:17.912351 8916 raft_consensus.cc:2454] T 0d03b39b002d44e38df27382b107cbbf P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:17.912351 9178 raft_consensus.cc:2454] T 0d03b39b002d44e38df27382b107cbbf P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:17.912694 6653 leader_election.cc:302] T 0d03b39b002d44e38df27382b107cbbf P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:17.912851 14138 raft_consensus.cc:2781] T 0d03b39b002d44e38df27382b107cbbf P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:17.912928 14138 raft_consensus.cc:686] T 0d03b39b002d44e38df27382b107cbbf P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Becoming Leader. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Running, Role: LEADER
I20240503 02:00:17.913089 14138 consensus_queue.cc:237] T 0d03b39b002d44e38df27382b107cbbf P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:17.915343 6309 catalog_manager.cc:5484] T 0d03b39b002d44e38df27382b107cbbf P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } }
W20240503 02:00:17.949244 6785 consensus_peers.cc:485] T c3485701bf3b48f5ae2e1568afeed133 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:00:17.952591 6309 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:17.956645 9048 raft_consensus.cc:1261] T 89aa8aa378d540a5aec5f61b21b4695c P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:17.956959 6720 raft_consensus.cc:1261] T 89aa8aa378d540a5aec5f61b21b4695c P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:17.957034 13991 consensus_queue.cc:1035] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:17.957520 13991 consensus_queue.cc:1035] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:17.961328 14179 tablet.cc:1722] T 89aa8aa378d540a5aec5f61b21b4695c P c2ac2eb7cc264047b9ab32fe3a4fc025: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:17.961566 6308 catalog_manager.cc:5576] TS c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655) does not have the latest schema for tablet 89aa8aa378d540a5aec5f61b21b4695c (table rebalancer_dynamic_table [id=0d3383ad1add444f973b21aa7d708ad4]). Expected version 1 got 0
I20240503 02:00:17.962558 6312 catalog_manager.cc:5576] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093) does not have the latest schema for tablet 89aa8aa378d540a5aec5f61b21b4695c (table rebalancer_dynamic_table [id=0d3383ad1add444f973b21aa7d708ad4]). Expected version 1 got 0
I20240503 02:00:17.963254 14179 tablet.cc:1620] T 89aa8aa378d540a5aec5f61b21b4695c P c2ac2eb7cc264047b9ab32fe3a4fc025: MemRowSet was empty: no flush needed.
I20240503 02:00:17.963521 14180 tablet.cc:1722] T 89aa8aa378d540a5aec5f61b21b4695c P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:17.963893 14180 tablet.cc:1620] T 89aa8aa378d540a5aec5f61b21b4695c P 99ead02915954b4fab23b033ad6d546b: MemRowSet was empty: no flush needed.
I20240503 02:00:17.966733 6312 catalog_manager.cc:5576] TS 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157) does not have the latest schema for tablet 89aa8aa378d540a5aec5f61b21b4695c (table rebalancer_dynamic_table [id=0d3383ad1add444f973b21aa7d708ad4]). Expected version 1 got 0
I20240503 02:00:17.969127 6309 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=0d3383ad1add444f973b21aa7d708ad4] alter complete (version 1)
I20240503 02:00:17.969586 14178 tablet.cc:1722] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:17.969928 14178 tablet.cc:1620] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a: MemRowSet was empty: no flush needed.
I20240503 02:00:17.970834 13900 raft_consensus.cc:482] T 53b328f875b94dc8906adfdf763821a8 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:17.970934 13900 raft_consensus.cc:504] T 53b328f875b94dc8906adfdf763821a8 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:17.971393 13900 leader_election.cc:288] T 53b328f875b94dc8906adfdf763821a8 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:17.971997 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b328f875b94dc8906adfdf763821a8" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 02:00:17.972134 6849 raft_consensus.cc:2454] T 53b328f875b94dc8906adfdf763821a8 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:17.972457 8984 leader_election.cc:302] T 53b328f875b94dc8906adfdf763821a8 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:17.972548 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b328f875b94dc8906adfdf763821a8" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 02:00:17.972615 13900 raft_consensus.cc:2781] T 53b328f875b94dc8906adfdf763821a8 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:17.972659 13900 raft_consensus.cc:482] T 53b328f875b94dc8906adfdf763821a8 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:17.972674 6589 raft_consensus.cc:2454] T 53b328f875b94dc8906adfdf763821a8 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:17.972693 13900 raft_consensus.cc:3037] T 53b328f875b94dc8906adfdf763821a8 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:17.973798 13900 raft_consensus.cc:504] T 53b328f875b94dc8906adfdf763821a8 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:17.974205 13900 leader_election.cc:288] T 53b328f875b94dc8906adfdf763821a8 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Requested vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:17.974480 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b328f875b94dc8906adfdf763821a8" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 02:00:17.974596 6589 raft_consensus.cc:3037] T 53b328f875b94dc8906adfdf763821a8 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:17.974807 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b328f875b94dc8906adfdf763821a8" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 02:00:17.974925 6849 raft_consensus.cc:3037] T 53b328f875b94dc8906adfdf763821a8 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:17.975697 6589 raft_consensus.cc:2454] T 53b328f875b94dc8906adfdf763821a8 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:17.976028 6849 raft_consensus.cc:2454] T 53b328f875b94dc8906adfdf763821a8 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:17.976143 8983 leader_election.cc:302] T 53b328f875b94dc8906adfdf763821a8 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:17.976470 13900 raft_consensus.cc:2781] T 53b328f875b94dc8906adfdf763821a8 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:17.976572 13900 raft_consensus.cc:686] T 53b328f875b94dc8906adfdf763821a8 P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Becoming Leader. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Running, Role: LEADER
I20240503 02:00:17.976706 13900 consensus_queue.cc:237] T 53b328f875b94dc8906adfdf763821a8 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:17.979898 6312 catalog_manager.cc:5484] T 53b328f875b94dc8906adfdf763821a8 P 99ead02915954b4fab23b033ad6d546b reported cstate change: term changed from 0 to 1, leader changed from <none> to 99ead02915954b4fab23b033ad6d546b (127.0.99.134). New cstate: current_term: 1 leader_uuid: "99ead02915954b4fab23b033ad6d546b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } }
W20240503 02:00:18.040329 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:18.075299 6312 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:18.080706 14178 tablet.cc:1722] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:18.081166 14180 tablet.cc:1722] T 89aa8aa378d540a5aec5f61b21b4695c P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:18.081252 14178 tablet.cc:1620] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a: MemRowSet was empty: no flush needed.
I20240503 02:00:18.081250 14179 tablet.cc:1722] T 89aa8aa378d540a5aec5f61b21b4695c P c2ac2eb7cc264047b9ab32fe3a4fc025: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:18.081518 14179 tablet.cc:1620] T 89aa8aa378d540a5aec5f61b21b4695c P c2ac2eb7cc264047b9ab32fe3a4fc025: MemRowSet was empty: no flush needed.
I20240503 02:00:18.081512 14180 tablet.cc:1620] T 89aa8aa378d540a5aec5f61b21b4695c P 99ead02915954b4fab23b033ad6d546b: MemRowSet was empty: no flush needed.
I20240503 02:00:18.087538 6308 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=0d3383ad1add444f973b21aa7d708ad4] alter complete (version 2)
W20240503 02:00:18.144089 8982 consensus_peers.cc:485] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:00:18.190490 6312 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:18.196534 14178 tablet.cc:1722] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:18.197516 14179 tablet.cc:1722] T 89aa8aa378d540a5aec5f61b21b4695c P c2ac2eb7cc264047b9ab32fe3a4fc025: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:18.200986 14180 tablet.cc:1722] T 89aa8aa378d540a5aec5f61b21b4695c P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:18.202361 6312 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=0d3383ad1add444f973b21aa7d708ad4] alter complete (version 3)
I20240503 02:00:18.305567 6312 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:18.305724 6312 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:18.307997 6312 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet 89aa8aa378d540a5aec5f61b21b4695c
I20240503 02:00:18.308635 9025 tablet_service.cc:1514] Processing DeleteTablet for tablet 89aa8aa378d540a5aec5f61b21b4695c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:18 UTC) from {username='slave'} at 127.0.0.1:52584
I20240503 02:00:18.308701 6436 tablet_service.cc:1514] Processing DeleteTablet for tablet 89aa8aa378d540a5aec5f61b21b4695c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:18 UTC) from {username='slave'} at 127.0.0.1:51338
I20240503 02:00:18.309037 14185 tablet_replica.cc:330] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a: stopping tablet replica
I20240503 02:00:18.309053 14186 tablet_replica.cc:330] T 89aa8aa378d540a5aec5f61b21b4695c P 99ead02915954b4fab23b033ad6d546b: stopping tablet replica
I20240503 02:00:18.309196 14186 raft_consensus.cc:2229] T 89aa8aa378d540a5aec5f61b21b4695c P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:18.309211 14185 raft_consensus.cc:2229] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:18.309177 6699 tablet_service.cc:1514] Processing DeleteTablet for tablet 89aa8aa378d540a5aec5f61b21b4695c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:18 UTC) from {username='slave'} at 127.0.0.1:40336
I20240503 02:00:18.309340 14186 raft_consensus.cc:2258] T 89aa8aa378d540a5aec5f61b21b4695c P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:18.309427 14185 raft_consensus.cc:2258] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:18.309515 14187 tablet_replica.cc:330] T 89aa8aa378d540a5aec5f61b21b4695c P c2ac2eb7cc264047b9ab32fe3a4fc025: stopping tablet replica
I20240503 02:00:18.309641 14187 raft_consensus.cc:2229] T 89aa8aa378d540a5aec5f61b21b4695c P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:18.309760 14187 raft_consensus.cc:2258] T 89aa8aa378d540a5aec5f61b21b4695c P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:18.310186 14186 ts_tablet_manager.cc:1875] T 89aa8aa378d540a5aec5f61b21b4695c P 99ead02915954b4fab23b033ad6d546b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:18.310307 14187 ts_tablet_manager.cc:1875] T 89aa8aa378d540a5aec5f61b21b4695c P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:18.310308 14185 ts_tablet_manager.cc:1875] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:18.312835 14185 ts_tablet_manager.cc:1888] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:18.312846 14186 ts_tablet_manager.cc:1888] T 89aa8aa378d540a5aec5f61b21b4695c P 99ead02915954b4fab23b033ad6d546b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:18.312862 14187 ts_tablet_manager.cc:1888] T 89aa8aa378d540a5aec5f61b21b4695c P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:18.312933 14187 log.cc:1196] T 89aa8aa378d540a5aec5f61b21b4695c P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/89aa8aa378d540a5aec5f61b21b4695c
I20240503 02:00:18.312935 14186 log.cc:1196] T 89aa8aa378d540a5aec5f61b21b4695c P 99ead02915954b4fab23b033ad6d546b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/89aa8aa378d540a5aec5f61b21b4695c
I20240503 02:00:18.312999 14185 log.cc:1196] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/89aa8aa378d540a5aec5f61b21b4695c
I20240503 02:00:18.313336 14185 ts_tablet_manager.cc:1909] T 89aa8aa378d540a5aec5f61b21b4695c P dec3e4a04d604f7eaaa3f077d081218a: Deleting consensus metadata
I20240503 02:00:18.313336 14187 ts_tablet_manager.cc:1909] T 89aa8aa378d540a5aec5f61b21b4695c P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting consensus metadata
I20240503 02:00:18.313576 14186 ts_tablet_manager.cc:1909] T 89aa8aa378d540a5aec5f61b21b4695c P 99ead02915954b4fab23b033ad6d546b: Deleting consensus metadata
I20240503 02:00:18.314338 6300 catalog_manager.cc:4830] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): tablet 89aa8aa378d540a5aec5f61b21b4695c (table rebalancer_dynamic_table_new_name [id=0d3383ad1add444f973b21aa7d708ad4]) successfully deleted
I20240503 02:00:18.314426 6303 catalog_manager.cc:4830] TS c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655): tablet 89aa8aa378d540a5aec5f61b21b4695c (table rebalancer_dynamic_table_new_name [id=0d3383ad1add444f973b21aa7d708ad4]) successfully deleted
I20240503 02:00:18.314361 6301 catalog_manager.cc:4830] TS 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157): tablet 89aa8aa378d540a5aec5f61b21b4695c (table rebalancer_dynamic_table_new_name [id=0d3383ad1add444f973b21aa7d708ad4]) successfully deleted
I20240503 02:00:18.379634 14138 consensus_queue.cc:1035] T 0d03b39b002d44e38df27382b107cbbf P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:00:18.382259 6654 consensus_peers.cc:485] T ba17d4d3e0754b2798eb95484d37229a P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:00:18.382709 13969 consensus_queue.cc:1035] T 0d03b39b002d44e38df27382b107cbbf P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:18.400043 13900 consensus_queue.cc:1035] T 53b328f875b94dc8906adfdf763821a8 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:18.407176 14195 consensus_queue.cc:1035] T 53b328f875b94dc8906adfdf763821a8 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:18.410064 6308 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
W20240503 02:00:18.411206 6523 consensus_peers.cc:485] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:00:18.416102 6567 tablet_service.cc:1467] Processing CreateTablet for tablet 4f191e65b61c499fb0a821ec9d5488a1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f2514b38a37b435aaaba4bc751e7ee6e]), partition=
I20240503 02:00:18.416102 6436 tablet_service.cc:1467] Processing CreateTablet for tablet 4f191e65b61c499fb0a821ec9d5488a1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f2514b38a37b435aaaba4bc751e7ee6e]), partition=
I20240503 02:00:18.416431 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f191e65b61c499fb0a821ec9d5488a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:18.416431 6567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f191e65b61c499fb0a821ec9d5488a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:18.416895 9158 tablet_service.cc:1467] Processing CreateTablet for tablet 4f191e65b61c499fb0a821ec9d5488a1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f2514b38a37b435aaaba4bc751e7ee6e]), partition=
I20240503 02:00:18.417155 9158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f191e65b61c499fb0a821ec9d5488a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:18.419922 14201 tablet_bootstrap.cc:492] T 4f191e65b61c499fb0a821ec9d5488a1 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:18.419880 14199 tablet_bootstrap.cc:492] T 4f191e65b61c499fb0a821ec9d5488a1 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:18.421083 14201 tablet_bootstrap.cc:654] T 4f191e65b61c499fb0a821ec9d5488a1 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:18.422739 14199 tablet_bootstrap.cc:654] T 4f191e65b61c499fb0a821ec9d5488a1 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:18.422868 14201 tablet_bootstrap.cc:492] T 4f191e65b61c499fb0a821ec9d5488a1 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:18.422968 14201 ts_tablet_manager.cc:1367] T 4f191e65b61c499fb0a821ec9d5488a1 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:00:18.423463 14200 tablet_bootstrap.cc:492] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:18.423457 14201 raft_consensus.cc:348] T 4f191e65b61c499fb0a821ec9d5488a1 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:18.423606 14201 raft_consensus.cc:374] T 4f191e65b61c499fb0a821ec9d5488a1 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:18.423655 14201 raft_consensus.cc:729] T 4f191e65b61c499fb0a821ec9d5488a1 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:18.423883 14199 tablet_bootstrap.cc:492] T 4f191e65b61c499fb0a821ec9d5488a1 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:18.423862 14201 consensus_queue.cc:260] T 4f191e65b61c499fb0a821ec9d5488a1 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:18.423979 14199 ts_tablet_manager.cc:1367] T 4f191e65b61c499fb0a821ec9d5488a1 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:18.424427 14200 tablet_bootstrap.cc:654] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:18.424484 14199 raft_consensus.cc:348] T 4f191e65b61c499fb0a821ec9d5488a1 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:18.424602 14199 raft_consensus.cc:374] T 4f191e65b61c499fb0a821ec9d5488a1 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:18.424666 14199 raft_consensus.cc:729] T 4f191e65b61c499fb0a821ec9d5488a1 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:18.424800 14199 consensus_queue.cc:260] T 4f191e65b61c499fb0a821ec9d5488a1 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:18.425172 14199 ts_tablet_manager.cc:1398] T 4f191e65b61c499fb0a821ec9d5488a1 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:18.425910 14201 ts_tablet_manager.cc:1398] T 4f191e65b61c499fb0a821ec9d5488a1 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:00:18.430229 14200 tablet_bootstrap.cc:492] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:18.430457 14200 ts_tablet_manager.cc:1367] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240503 02:00:18.431257 14200 raft_consensus.cc:348] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:18.431416 14200 raft_consensus.cc:374] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:18.431460 14200 raft_consensus.cc:729] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:18.431584 14200 consensus_queue.cc:260] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:18.431892 14200 ts_tablet_manager.cc:1398] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:18.435761 14202 consensus_queue.cc:1035] T c6690a8e5aaa4d83b47899cc8eb96d3d P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:00:18.437105 6785 consensus_peers.cc:485] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:00:18.439603 14202 consensus_queue.cc:1035] T c6690a8e5aaa4d83b47899cc8eb96d3d P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:00:18.442914 6301 catalog_manager.cc:4573] TS e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): CreateTablet RPC failed for tablet 56824b553788471c9a2fcd6ec82d507e: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:18.496364 6314 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_extra_table_48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:18.503767 6700 tablet_service.cc:1467] Processing CreateTablet for tablet fa74c52d8f6f43d7ab61ae500fec62d4 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=52f31af1c95b4104b394062d67c75c08]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:18.503809 8880 tablet_service.cc:1467] Processing CreateTablet for tablet 1029229ea6924a31b637639a95a04f9e (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=52f31af1c95b4104b394062d67c75c08]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:18.503767 6699 tablet_service.cc:1467] Processing CreateTablet for tablet 6338547c9784461d80cee2c69aaf905b (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=52f31af1c95b4104b394062d67c75c08]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:18.504154 8880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1029229ea6924a31b637639a95a04f9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:18.504160 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa74c52d8f6f43d7ab61ae500fec62d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:18.504760 9025 tablet_service.cc:1467] Processing CreateTablet for tablet 6338547c9784461d80cee2c69aaf905b (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=52f31af1c95b4104b394062d67c75c08]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:18.504837 9028 tablet_service.cc:1467] Processing CreateTablet for tablet fa74c52d8f6f43d7ab61ae500fec62d4 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=52f31af1c95b4104b394062d67c75c08]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:18.505079 9025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6338547c9784461d80cee2c69aaf905b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:18.505472 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa74c52d8f6f43d7ab61ae500fec62d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:18.506799 9027 tablet_service.cc:1467] Processing CreateTablet for tablet 1029229ea6924a31b637639a95a04f9e (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=52f31af1c95b4104b394062d67c75c08]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:18.507349 9027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1029229ea6924a31b637639a95a04f9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:18.507690 9158 tablet_service.cc:1467] Processing CreateTablet for tablet 6338547c9784461d80cee2c69aaf905b (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=52f31af1c95b4104b394062d67c75c08]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:18.507740 9157 tablet_service.cc:1467] Processing CreateTablet for tablet fa74c52d8f6f43d7ab61ae500fec62d4 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=52f31af1c95b4104b394062d67c75c08]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:18.507953 9158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6338547c9784461d80cee2c69aaf905b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:18.507951 6699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6338547c9784461d80cee2c69aaf905b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:18.508229 9157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa74c52d8f6f43d7ab61ae500fec62d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:18.510056 14201 tablet_bootstrap.cc:492] T 6338547c9784461d80cee2c69aaf905b P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:18.511224 14210 tablet_bootstrap.cc:492] T fa74c52d8f6f43d7ab61ae500fec62d4 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:18.511279 14201 tablet_bootstrap.cc:654] T 6338547c9784461d80cee2c69aaf905b P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:18.511343 6436 tablet_service.cc:1467] Processing CreateTablet for tablet 1029229ea6924a31b637639a95a04f9e (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=52f31af1c95b4104b394062d67c75c08]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:18.511643 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1029229ea6924a31b637639a95a04f9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:18.512513 14210 tablet_bootstrap.cc:654] T fa74c52d8f6f43d7ab61ae500fec62d4 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:18.513289 14209 tablet_bootstrap.cc:492] T 6338547c9784461d80cee2c69aaf905b P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:18.513456 14201 tablet_bootstrap.cc:492] T 6338547c9784461d80cee2c69aaf905b P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:18.513561 14201 ts_tablet_manager.cc:1367] T 6338547c9784461d80cee2c69aaf905b P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 02:00:18.514039 14201 raft_consensus.cc:348] T 6338547c9784461d80cee2c69aaf905b P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:18.514147 14201 raft_consensus.cc:374] T 6338547c9784461d80cee2c69aaf905b P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:18.514178 14201 raft_consensus.cc:729] T 6338547c9784461d80cee2c69aaf905b P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:18.514294 14201 consensus_queue.cc:260] T 6338547c9784461d80cee2c69aaf905b P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:18.514433 14199 tablet_bootstrap.cc:492] T 1029229ea6924a31b637639a95a04f9e P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:18.514587 14201 ts_tablet_manager.cc:1398] T 6338547c9784461d80cee2c69aaf905b P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:18.514767 14201 tablet_bootstrap.cc:492] T fa74c52d8f6f43d7ab61ae500fec62d4 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:18.514936 14209 tablet_bootstrap.cc:654] T 6338547c9784461d80cee2c69aaf905b P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:18.515663 14201 tablet_bootstrap.cc:654] T fa74c52d8f6f43d7ab61ae500fec62d4 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:18.515899 14199 tablet_bootstrap.cc:654] T 1029229ea6924a31b637639a95a04f9e P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:18.516520 14208 tablet_bootstrap.cc:492] T 1029229ea6924a31b637639a95a04f9e P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:18.517072 14209 tablet_bootstrap.cc:492] T 6338547c9784461d80cee2c69aaf905b P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:18.517176 14209 ts_tablet_manager.cc:1367] T 6338547c9784461d80cee2c69aaf905b P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:18.517642 14209 raft_consensus.cc:348] T 6338547c9784461d80cee2c69aaf905b P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:18.517762 14208 tablet_bootstrap.cc:654] T 1029229ea6924a31b637639a95a04f9e P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:18.517771 14209 raft_consensus.cc:374] T 6338547c9784461d80cee2c69aaf905b P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:18.517800 14209 raft_consensus.cc:729] T 6338547c9784461d80cee2c69aaf905b P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:18.517911 14209 consensus_queue.cc:260] T 6338547c9784461d80cee2c69aaf905b P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:18.518236 14209 ts_tablet_manager.cc:1398] T 6338547c9784461d80cee2c69aaf905b P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:18.518417 14209 tablet_bootstrap.cc:492] T fa74c52d8f6f43d7ab61ae500fec62d4 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:18.518967 14201 tablet_bootstrap.cc:492] T fa74c52d8f6f43d7ab61ae500fec62d4 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:18.519044 14201 ts_tablet_manager.cc:1367] T fa74c52d8f6f43d7ab61ae500fec62d4 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240503 02:00:18.519098 14210 tablet_bootstrap.cc:492] T fa74c52d8f6f43d7ab61ae500fec62d4 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:18.519253 14210 ts_tablet_manager.cc:1367] T fa74c52d8f6f43d7ab61ae500fec62d4 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20240503 02:00:18.519368 14209 tablet_bootstrap.cc:654] T fa74c52d8f6f43d7ab61ae500fec62d4 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:18.519480 14201 raft_consensus.cc:348] T fa74c52d8f6f43d7ab61ae500fec62d4 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:18.519596 14201 raft_consensus.cc:374] T fa74c52d8f6f43d7ab61ae500fec62d4 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:18.519629 14201 raft_consensus.cc:729] T fa74c52d8f6f43d7ab61ae500fec62d4 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:18.519739 14201 consensus_queue.cc:260] T fa74c52d8f6f43d7ab61ae500fec62d4 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:18.520006 14201 ts_tablet_manager.cc:1398] T fa74c52d8f6f43d7ab61ae500fec62d4 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:18.519950 14210 raft_consensus.cc:348] T fa74c52d8f6f43d7ab61ae500fec62d4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:18.520058 14210 raft_consensus.cc:374] T fa74c52d8f6f43d7ab61ae500fec62d4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:18.520093 14210 raft_consensus.cc:729] T fa74c52d8f6f43d7ab61ae500fec62d4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:18.520226 14210 consensus_queue.cc:260] T fa74c52d8f6f43d7ab61ae500fec62d4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:18.520857 14210 ts_tablet_manager.cc:1398] T fa74c52d8f6f43d7ab61ae500fec62d4 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:18.520954 14209 tablet_bootstrap.cc:492] T fa74c52d8f6f43d7ab61ae500fec62d4 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:18.521029 14209 ts_tablet_manager.cc:1367] T fa74c52d8f6f43d7ab61ae500fec62d4 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:00:18.521044 14210 tablet_bootstrap.cc:492] T 6338547c9784461d80cee2c69aaf905b P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:18.521454 14209 raft_consensus.cc:348] T fa74c52d8f6f43d7ab61ae500fec62d4 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:18.521603 14209 raft_consensus.cc:374] T fa74c52d8f6f43d7ab61ae500fec62d4 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:18.521632 14209 raft_consensus.cc:729] T fa74c52d8f6f43d7ab61ae500fec62d4 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:18.521734 14209 consensus_queue.cc:260] T fa74c52d8f6f43d7ab61ae500fec62d4 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:18.521992 14209 ts_tablet_manager.cc:1398] T fa74c52d8f6f43d7ab61ae500fec62d4 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:18.522166 14209 tablet_bootstrap.cc:492] T 1029229ea6924a31b637639a95a04f9e P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:18.522198 14210 tablet_bootstrap.cc:654] T 6338547c9784461d80cee2c69aaf905b P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:18.523059 14209 tablet_bootstrap.cc:654] T 1029229ea6924a31b637639a95a04f9e P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:18.524114 14209 tablet_bootstrap.cc:492] T 1029229ea6924a31b637639a95a04f9e P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:18.524235 14209 ts_tablet_manager.cc:1367] T 1029229ea6924a31b637639a95a04f9e P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:00:18.524869 14209 raft_consensus.cc:348] T 1029229ea6924a31b637639a95a04f9e P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:18.525010 14209 raft_consensus.cc:374] T 1029229ea6924a31b637639a95a04f9e P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:18.525045 14209 raft_consensus.cc:729] T 1029229ea6924a31b637639a95a04f9e P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:18.525185 14209 consensus_queue.cc:260] T 1029229ea6924a31b637639a95a04f9e P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:18.525565 14209 ts_tablet_manager.cc:1398] T 1029229ea6924a31b637639a95a04f9e P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:18.528671 14199 tablet_bootstrap.cc:492] T 1029229ea6924a31b637639a95a04f9e P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:18.528800 14199 ts_tablet_manager.cc:1367] T 1029229ea6924a31b637639a95a04f9e P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 02:00:18.529284 14199 raft_consensus.cc:348] T 1029229ea6924a31b637639a95a04f9e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:18.529420 14199 raft_consensus.cc:374] T 1029229ea6924a31b637639a95a04f9e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:18.529454 14199 raft_consensus.cc:729] T 1029229ea6924a31b637639a95a04f9e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:18.529587 14199 consensus_queue.cc:260] T 1029229ea6924a31b637639a95a04f9e P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:18.530620 14199 ts_tablet_manager.cc:1398] T 1029229ea6924a31b637639a95a04f9e P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:18.531240 14208 tablet_bootstrap.cc:492] T 1029229ea6924a31b637639a95a04f9e P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:18.531360 14208 ts_tablet_manager.cc:1367] T 1029229ea6924a31b637639a95a04f9e P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240503 02:00:18.532029 14208 raft_consensus.cc:348] T 1029229ea6924a31b637639a95a04f9e P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:18.532153 14208 raft_consensus.cc:374] T 1029229ea6924a31b637639a95a04f9e P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:18.532186 14208 raft_consensus.cc:729] T 1029229ea6924a31b637639a95a04f9e P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:18.532307 14208 consensus_queue.cc:260] T 1029229ea6924a31b637639a95a04f9e P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:18.532641 14208 ts_tablet_manager.cc:1398] T 1029229ea6924a31b637639a95a04f9e P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:18.537848 14210 tablet_bootstrap.cc:492] T 6338547c9784461d80cee2c69aaf905b P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:18.537983 14210 ts_tablet_manager.cc:1367] T 6338547c9784461d80cee2c69aaf905b P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.001s
I20240503 02:00:18.538460 14210 raft_consensus.cc:348] T 6338547c9784461d80cee2c69aaf905b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:18.538566 14210 raft_consensus.cc:374] T 6338547c9784461d80cee2c69aaf905b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:18.538587 14210 raft_consensus.cc:729] T 6338547c9784461d80cee2c69aaf905b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:18.538710 14210 consensus_queue.cc:260] T 6338547c9784461d80cee2c69aaf905b P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:18.539829 14210 ts_tablet_manager.cc:1398] T 6338547c9784461d80cee2c69aaf905b P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:00:18.574126 14190 raft_consensus.cc:482] T fa74c52d8f6f43d7ab61ae500fec62d4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:18.574271 14190 raft_consensus.cc:504] T fa74c52d8f6f43d7ab61ae500fec62d4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:18.574790 14190 leader_election.cc:288] T fa74c52d8f6f43d7ab61ae500fec62d4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:18.575147 9047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa74c52d8f6f43d7ab61ae500fec62d4" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:18.575219 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa74c52d8f6f43d7ab61ae500fec62d4" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:18.575292 9047 raft_consensus.cc:2454] T fa74c52d8f6f43d7ab61ae500fec62d4 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:18.575371 9178 raft_consensus.cc:2454] T fa74c52d8f6f43d7ab61ae500fec62d4 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:18.575618 6654 leader_election.cc:302] T fa74c52d8f6f43d7ab61ae500fec62d4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99ead02915954b4fab23b033ad6d546b, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:18.575794 14190 raft_consensus.cc:2781] T fa74c52d8f6f43d7ab61ae500fec62d4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:18.575860 14190 raft_consensus.cc:482] T fa74c52d8f6f43d7ab61ae500fec62d4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:18.575903 14190 raft_consensus.cc:3037] T fa74c52d8f6f43d7ab61ae500fec62d4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:18.576906 14190 raft_consensus.cc:504] T fa74c52d8f6f43d7ab61ae500fec62d4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:18.577268 14190 leader_election.cc:288] T fa74c52d8f6f43d7ab61ae500fec62d4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Requested vote from peers 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:18.577675 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa74c52d8f6f43d7ab61ae500fec62d4" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:18.577831 9178 raft_consensus.cc:3037] T fa74c52d8f6f43d7ab61ae500fec62d4 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:18.577836 9047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa74c52d8f6f43d7ab61ae500fec62d4" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:18.577975 9047 raft_consensus.cc:3037] T fa74c52d8f6f43d7ab61ae500fec62d4 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:18.578902 9047 raft_consensus.cc:2454] T fa74c52d8f6f43d7ab61ae500fec62d4 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:18.578902 9178 raft_consensus.cc:2454] T fa74c52d8f6f43d7ab61ae500fec62d4 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:18.579252 6653 leader_election.cc:302] T fa74c52d8f6f43d7ab61ae500fec62d4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:18.579574 14190 raft_consensus.cc:2781] T fa74c52d8f6f43d7ab61ae500fec62d4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:18.579712 14190 raft_consensus.cc:686] T fa74c52d8f6f43d7ab61ae500fec62d4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Becoming Leader. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Running, Role: LEADER
I20240503 02:00:18.579886 14190 consensus_queue.cc:237] T fa74c52d8f6f43d7ab61ae500fec62d4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:18.582201 6312 catalog_manager.cc:5484] T fa74c52d8f6f43d7ab61ae500fec62d4 P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } }
W20240503 02:00:18.608253 6654 consensus_peers.cc:485] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:18.614535 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
W20240503 02:00:18.617149 6785 consensus_peers.cc:485] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:00:18.638736 14190 raft_consensus.cc:482] T 6338547c9784461d80cee2c69aaf905b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:18.638857 14190 raft_consensus.cc:504] T 6338547c9784461d80cee2c69aaf905b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:18.639264 14190 leader_election.cc:288] T 6338547c9784461d80cee2c69aaf905b P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:18.639680 9047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6338547c9784461d80cee2c69aaf905b" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:18.639683 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6338547c9784461d80cee2c69aaf905b" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:18.639822 9179 raft_consensus.cc:2454] T 6338547c9784461d80cee2c69aaf905b P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:18.639822 9047 raft_consensus.cc:2454] T 6338547c9784461d80cee2c69aaf905b P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:18.640201 6654 leader_election.cc:302] T 6338547c9784461d80cee2c69aaf905b P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99ead02915954b4fab23b033ad6d546b, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:18.640439 14190 raft_consensus.cc:2781] T 6338547c9784461d80cee2c69aaf905b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:18.640512 14190 raft_consensus.cc:482] T 6338547c9784461d80cee2c69aaf905b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:18.640559 14190 raft_consensus.cc:3037] T 6338547c9784461d80cee2c69aaf905b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:18.642122 14190 raft_consensus.cc:504] T 6338547c9784461d80cee2c69aaf905b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:18.642649 14190 leader_election.cc:288] T 6338547c9784461d80cee2c69aaf905b P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Requested vote from peers 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
W20240503 02:00:18.642692 6523 consensus_peers.cc:485] T d0d5dd42db3e4883af7fd16d8813b550 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:00:18.643152 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6338547c9784461d80cee2c69aaf905b" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:18.643291 9047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6338547c9784461d80cee2c69aaf905b" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:18.643291 9179 raft_consensus.cc:3037] T 6338547c9784461d80cee2c69aaf905b P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:18.643397 9047 raft_consensus.cc:3037] T 6338547c9784461d80cee2c69aaf905b P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:18.644204 9047 raft_consensus.cc:2454] T 6338547c9784461d80cee2c69aaf905b P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:18.644203 9179 raft_consensus.cc:2454] T 6338547c9784461d80cee2c69aaf905b P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:18.644635 6654 leader_election.cc:302] T 6338547c9784461d80cee2c69aaf905b P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99ead02915954b4fab23b033ad6d546b, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:18.644841 14190 raft_consensus.cc:2781] T 6338547c9784461d80cee2c69aaf905b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:18.644915 14190 raft_consensus.cc:686] T 6338547c9784461d80cee2c69aaf905b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Becoming Leader. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Running, Role: LEADER
I20240503 02:00:18.645083 14190 consensus_queue.cc:237] T 6338547c9784461d80cee2c69aaf905b P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:18.647436 6312 catalog_manager.cc:5484] T 6338547c9784461d80cee2c69aaf905b P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:18.662559 14202 raft_consensus.cc:482] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:18.662711 14202 raft_consensus.cc:504] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:18.663384 14202 leader_election.cc:288] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
W20240503 02:00:18.663486 8851 consensus_peers.cc:485] T 067f54011a37432d99157884e70fc601 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:00:18.663758 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f191e65b61c499fb0a821ec9d5488a1" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 02:00:18.663879 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f191e65b61c499fb0a821ec9d5488a1" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:18.663913 6458 raft_consensus.cc:2454] T 4f191e65b61c499fb0a821ec9d5488a1 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:18.664016 9179 raft_consensus.cc:2454] T 4f191e65b61c499fb0a821ec9d5488a1 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:18.664312 6522 leader_election.cc:302] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:18.664511 14202 raft_consensus.cc:2781] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:18.664568 14202 raft_consensus.cc:482] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:18.664610 14202 raft_consensus.cc:3037] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:18.665505 14202 raft_consensus.cc:504] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:18.665865 14202 leader_election.cc:288] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Requested vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:18.666237 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f191e65b61c499fb0a821ec9d5488a1" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 02:00:18.666319 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f191e65b61c499fb0a821ec9d5488a1" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:18.666373 6458 raft_consensus.cc:3037] T 4f191e65b61c499fb0a821ec9d5488a1 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:18.666450 9179 raft_consensus.cc:3037] T 4f191e65b61c499fb0a821ec9d5488a1 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:18.667414 9179 raft_consensus.cc:2454] T 4f191e65b61c499fb0a821ec9d5488a1 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:18.667414 6458 raft_consensus.cc:2454] T 4f191e65b61c499fb0a821ec9d5488a1 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:18.667867 6522 leader_election.cc:302] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 80fd18ef4d604c86b909d5c008a46b4f; no voters:
I20240503 02:00:18.668176 14202 raft_consensus.cc:2781] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:18.668336 14202 raft_consensus.cc:686] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Becoming Leader. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Running, Role: LEADER
I20240503 02:00:18.668527 14202 consensus_queue.cc:237] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:18.671768 6312 catalog_manager.cc:5484] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } }
W20240503 02:00:18.686398 9113 consensus_peers.cc:485] T f44b59277c3b47adbd4dd6aff9350af1 P 2a01ace80baf4057b6bf1995cf88468b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:00:18.767609 14195 raft_consensus.cc:482] T 1029229ea6924a31b637639a95a04f9e P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:18.767753 14195 raft_consensus.cc:504] T 1029229ea6924a31b637639a95a04f9e P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:18.768173 14195 leader_election.cc:288] T 1029229ea6924a31b637639a95a04f9e P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 02:00:18.768561 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1029229ea6924a31b637639a95a04f9e" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 02:00:18.768709 6458 raft_consensus.cc:2454] T 1029229ea6924a31b637639a95a04f9e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:18.768733 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1029229ea6924a31b637639a95a04f9e" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:18.768847 8916 raft_consensus.cc:2454] T 1029229ea6924a31b637639a95a04f9e P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:18.769045 8981 leader_election.cc:302] T 1029229ea6924a31b637639a95a04f9e P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99ead02915954b4fab23b033ad6d546b, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:18.769284 14195 raft_consensus.cc:2781] T 1029229ea6924a31b637639a95a04f9e P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:18.769332 14195 raft_consensus.cc:482] T 1029229ea6924a31b637639a95a04f9e P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:18.769356 14195 raft_consensus.cc:3037] T 1029229ea6924a31b637639a95a04f9e P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:18.770248 14195 raft_consensus.cc:504] T 1029229ea6924a31b637639a95a04f9e P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:18.770633 14195 leader_election.cc:288] T 1029229ea6924a31b637639a95a04f9e P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Requested vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 02:00:18.770973 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1029229ea6924a31b637639a95a04f9e" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:18.771034 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1029229ea6924a31b637639a95a04f9e" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 02:00:18.771109 8916 raft_consensus.cc:3037] T 1029229ea6924a31b637639a95a04f9e P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:18.771157 6458 raft_consensus.cc:3037] T 1029229ea6924a31b637639a95a04f9e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:18.772141 8916 raft_consensus.cc:2454] T 1029229ea6924a31b637639a95a04f9e P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:18.772141 6458 raft_consensus.cc:2454] T 1029229ea6924a31b637639a95a04f9e P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:18.772636 8984 leader_election.cc:302] T 1029229ea6924a31b637639a95a04f9e P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9942306855404cfd891adf227dd65eb2, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:18.772833 14195 raft_consensus.cc:2781] T 1029229ea6924a31b637639a95a04f9e P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:18.772902 14195 raft_consensus.cc:686] T 1029229ea6924a31b637639a95a04f9e P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Becoming Leader. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Running, Role: LEADER
I20240503 02:00:18.773051 14195 consensus_queue.cc:237] T 1029229ea6924a31b637639a95a04f9e P 99ead02915954b4fab23b033ad6d546b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:18.778419 6312 catalog_manager.cc:5484] T 1029229ea6924a31b637639a95a04f9e P 99ead02915954b4fab23b033ad6d546b reported cstate change: term changed from 0 to 1, leader changed from <none> to 99ead02915954b4fab23b033ad6d546b (127.0.99.134). New cstate: current_term: 1 leader_uuid: "99ead02915954b4fab23b033ad6d546b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:18.820266 6312 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:18.824810 9179 raft_consensus.cc:1261] T 4f191e65b61c499fb0a821ec9d5488a1 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:18.825474 6456 raft_consensus.cc:1261] T 4f191e65b61c499fb0a821ec9d5488a1 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 80fd18ef4d604c86b909d5c008a46b4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:18.825428 13954 consensus_queue.cc:1035] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:18.826009 13954 consensus_queue.cc:1035] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:18.828787 14227 tablet.cc:1722] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:18.829227 6308 catalog_manager.cc:5576] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079) does not have the latest schema for tablet 4f191e65b61c499fb0a821ec9d5488a1 (table rebalancer_dynamic_table [id=f2514b38a37b435aaaba4bc751e7ee6e]). Expected version 1 got 0
I20240503 02:00:18.830174 14193 mvcc.cc:205] Tried to move back new op lower bound from 7023417830702645248 to 7023417830068908032. Current Snapshot: MvccSnapshot[applied={T|T < 7023417830702645248}]
I20240503 02:00:18.830931 14227 tablet.cc:1620] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f: MemRowSet was empty: no flush needed.
I20240503 02:00:18.831017 6308 catalog_manager.cc:5576] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093) does not have the latest schema for tablet 4f191e65b61c499fb0a821ec9d5488a1 (table rebalancer_dynamic_table [id=f2514b38a37b435aaaba4bc751e7ee6e]). Expected version 1 got 0
I20240503 02:00:18.831921 6312 catalog_manager.cc:5576] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863) does not have the latest schema for tablet 4f191e65b61c499fb0a821ec9d5488a1 (table rebalancer_dynamic_table [id=f2514b38a37b435aaaba4bc751e7ee6e]). Expected version 1 got 0
I20240503 02:00:18.832237 6314 catalog_manager.cc:5576] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079) does not have the latest schema for tablet 4f191e65b61c499fb0a821ec9d5488a1 (table rebalancer_dynamic_table [id=f2514b38a37b435aaaba4bc751e7ee6e]). Expected version 1 got 0
I20240503 02:00:18.830353 14229 tablet.cc:1722] T 4f191e65b61c499fb0a821ec9d5488a1 P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:18.832607 14229 tablet.cc:1620] T 4f191e65b61c499fb0a821ec9d5488a1 P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
I20240503 02:00:18.833527 14226 mvcc.cc:205] Tried to move back new op lower bound from 7023417830702645248 to 7023417830068908032. Current Snapshot: MvccSnapshot[applied={T|T < 7023417830702645248}]
W20240503 02:00:18.835520 8851 consensus_peers.cc:485] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:00:18.837497 6312 catalog_manager.cc:5576] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093) does not have the latest schema for tablet 4f191e65b61c499fb0a821ec9d5488a1 (table rebalancer_dynamic_table [id=f2514b38a37b435aaaba4bc751e7ee6e]). Expected version 1 got 0
I20240503 02:00:18.837746 6314 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=f2514b38a37b435aaaba4bc751e7ee6e] alter complete (version 1)
W20240503 02:00:18.838284 6523 consensus_peers.cc:485] T 4385d51d11574192883d01376e2d5620 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:00:18.839368 14231 tablet.cc:1722] T 4f191e65b61c499fb0a821ec9d5488a1 P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:18.839756 14231 tablet.cc:1620] T 4f191e65b61c499fb0a821ec9d5488a1 P dec3e4a04d604f7eaaa3f077d081218a: MemRowSet was empty: no flush needed.
W20240503 02:00:18.908777 8982 consensus_peers.cc:485] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:18.933183 6523 consensus_peers.cc:485] T 925612b660ec471da5b3c548802ede6e P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:00:18.941829 6314 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:18.948406 14227 tablet.cc:1722] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:18.948787 14227 tablet.cc:1620] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f: MemRowSet was empty: no flush needed.
I20240503 02:00:18.949031 14229 tablet.cc:1722] T 4f191e65b61c499fb0a821ec9d5488a1 P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:18.949334 14229 tablet.cc:1620] T 4f191e65b61c499fb0a821ec9d5488a1 P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
W20240503 02:00:18.951365 6785 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240503 02:00:18.951395 8851 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111) [suppressed 48 similar messages]
I20240503 02:00:18.951395 14231 tablet.cc:1722] T 4f191e65b61c499fb0a821ec9d5488a1 P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:18.951746 14231 tablet.cc:1620] T 4f191e65b61c499fb0a821ec9d5488a1 P dec3e4a04d604f7eaaa3f077d081218a: MemRowSet was empty: no flush needed.
I20240503 02:00:18.955374 6314 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=f2514b38a37b435aaaba4bc751e7ee6e] alter complete (version 2)
W20240503 02:00:18.958595 6785 consensus_peers.cc:485] T 8246488dec204bf6b09e868d7951ac5a P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:00:18.962817 13969 consensus_queue.cc:1035] T fa74c52d8f6f43d7ab61ae500fec62d4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:18.973165 13969 consensus_queue.cc:1035] T fa74c52d8f6f43d7ab61ae500fec62d4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:00:18.999888 6785 consensus_peers.cc:485] T 46ad540deb624a25b97d3b967fe1f6f4 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:19.015749 8982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 02:00:19.016360 8982 consensus_peers.cc:485] T 13756a6e44814fe4b09a6b0f087546c3 P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:19.024590 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
W20240503 02:00:19.025542 6654 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 02:00:19.026162 6654 consensus_peers.cc:485] T cea1ea50fa434d31b42451f84f26bc76 P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:19.048748 6785 consensus_peers.cc:485] T d897c21e630d4770832bb10f43778e82 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:00:19.057284 6308 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:19.062883 14227 tablet.cc:1722] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:19.063702 14229 tablet.cc:1722] T 4f191e65b61c499fb0a821ec9d5488a1 P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:19.064074 14231 tablet.cc:1722] T 4f191e65b61c499fb0a821ec9d5488a1 P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:19.069939 6314 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=f2514b38a37b435aaaba4bc751e7ee6e] alter complete (version 3)
I20240503 02:00:19.073679 14190 consensus_queue.cc:1035] T 6338547c9784461d80cee2c69aaf905b P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:19.077131 14190 consensus_queue.cc:1035] T 6338547c9784461d80cee2c69aaf905b P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:00:19.081373 6523 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240503 02:00:19.166998 13943 consensus_queue.cc:1035] T 1029229ea6924a31b637639a95a04f9e P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:19.170635 13900 consensus_queue.cc:1035] T 1029229ea6924a31b637639a95a04f9e P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:19.173913 6308 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:19.174085 6308 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
W20240503 02:00:19.174924 8851 consensus_peers.cc:485] T 5f51fe5f799f4e479b62f5917a2c2606 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:00:19.177659 6308 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet 4f191e65b61c499fb0a821ec9d5488a1
I20240503 02:00:19.178231 6436 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f191e65b61c499fb0a821ec9d5488a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:19 UTC) from {username='slave'} at 127.0.0.1:51338
I20240503 02:00:19.178422 6567 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f191e65b61c499fb0a821ec9d5488a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:19 UTC) from {username='slave'} at 127.0.0.1:59846
I20240503 02:00:19.178611 9157 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f191e65b61c499fb0a821ec9d5488a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:19 UTC) from {username='slave'} at 127.0.0.1:51940
I20240503 02:00:19.179023 14244 tablet_replica.cc:330] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 02:00:19.179211 14244 raft_consensus.cc:2229] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:19.179479 14244 raft_consensus.cc:2258] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:19.180162 14245 tablet_replica.cc:330] T 4f191e65b61c499fb0a821ec9d5488a1 P 2a01ace80baf4057b6bf1995cf88468b: stopping tablet replica
I20240503 02:00:19.180562 14245 raft_consensus.cc:2229] T 4f191e65b61c499fb0a821ec9d5488a1 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:19.180693 14244 ts_tablet_manager.cc:1875] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:19.180717 14245 raft_consensus.cc:2258] T 4f191e65b61c499fb0a821ec9d5488a1 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:19.181284 14245 ts_tablet_manager.cc:1875] T 4f191e65b61c499fb0a821ec9d5488a1 P 2a01ace80baf4057b6bf1995cf88468b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:19.183243 14244 ts_tablet_manager.cc:1888] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:19.183391 14244 log.cc:1196] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/4f191e65b61c499fb0a821ec9d5488a1
I20240503 02:00:19.183859 14244 ts_tablet_manager.cc:1909] T 4f191e65b61c499fb0a821ec9d5488a1 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting consensus metadata
I20240503 02:00:19.184068 14245 ts_tablet_manager.cc:1888] T 4f191e65b61c499fb0a821ec9d5488a1 P 2a01ace80baf4057b6bf1995cf88468b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:19.184238 14245 log.cc:1196] T 4f191e65b61c499fb0a821ec9d5488a1 P 2a01ace80baf4057b6bf1995cf88468b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/4f191e65b61c499fb0a821ec9d5488a1
I20240503 02:00:19.184818 14245 ts_tablet_manager.cc:1909] T 4f191e65b61c499fb0a821ec9d5488a1 P 2a01ace80baf4057b6bf1995cf88468b: Deleting consensus metadata
I20240503 02:00:19.184932 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet 4f191e65b61c499fb0a821ec9d5488a1 (table rebalancer_dynamic_table_new_name [id=f2514b38a37b435aaaba4bc751e7ee6e]) successfully deleted
I20240503 02:00:19.185681 6300 catalog_manager.cc:4830] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079): tablet 4f191e65b61c499fb0a821ec9d5488a1 (table rebalancer_dynamic_table_new_name [id=f2514b38a37b435aaaba4bc751e7ee6e]) successfully deleted
I20240503 02:00:19.188797 14243 tablet_replica.cc:330] T 4f191e65b61c499fb0a821ec9d5488a1 P dec3e4a04d604f7eaaa3f077d081218a: stopping tablet replica
I20240503 02:00:19.188964 14243 raft_consensus.cc:2229] T 4f191e65b61c499fb0a821ec9d5488a1 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:19.189095 14243 raft_consensus.cc:2258] T 4f191e65b61c499fb0a821ec9d5488a1 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:19.189783 14243 ts_tablet_manager.cc:1875] T 4f191e65b61c499fb0a821ec9d5488a1 P dec3e4a04d604f7eaaa3f077d081218a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:19.192313 14243 ts_tablet_manager.cc:1888] T 4f191e65b61c499fb0a821ec9d5488a1 P dec3e4a04d604f7eaaa3f077d081218a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:19.192427 14243 log.cc:1196] T 4f191e65b61c499fb0a821ec9d5488a1 P dec3e4a04d604f7eaaa3f077d081218a: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/4f191e65b61c499fb0a821ec9d5488a1
I20240503 02:00:19.192723 14243 ts_tablet_manager.cc:1909] T 4f191e65b61c499fb0a821ec9d5488a1 P dec3e4a04d604f7eaaa3f077d081218a: Deleting consensus metadata
I20240503 02:00:19.193389 6300 catalog_manager.cc:4830] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): tablet 4f191e65b61c499fb0a821ec9d5488a1 (table rebalancer_dynamic_table_new_name [id=f2514b38a37b435aaaba4bc751e7ee6e]) successfully deleted
W20240503 02:00:19.269558 6523 consensus_peers.cc:485] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:00:19.280041 6312 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
W20240503 02:00:19.281942 9113 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240503 02:00:19.285883 6829 tablet_service.cc:1467] Processing CreateTablet for tablet 32dd15923a55417c97d54822f930bbdd (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d6007f227ea147b58a002001972f41cf]), partition=
I20240503 02:00:19.286185 6829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32dd15923a55417c97d54822f930bbdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:19.286437 8880 tablet_service.cc:1467] Processing CreateTablet for tablet 32dd15923a55417c97d54822f930bbdd (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d6007f227ea147b58a002001972f41cf]), partition=
I20240503 02:00:19.286655 8880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32dd15923a55417c97d54822f930bbdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:19.288781 14247 tablet_bootstrap.cc:492] T 32dd15923a55417c97d54822f930bbdd P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:19.289300 6436 tablet_service.cc:1467] Processing CreateTablet for tablet 32dd15923a55417c97d54822f930bbdd (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d6007f227ea147b58a002001972f41cf]), partition=
I20240503 02:00:19.289487 14246 tablet_bootstrap.cc:492] T 32dd15923a55417c97d54822f930bbdd P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:19.289620 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32dd15923a55417c97d54822f930bbdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:19.290423 14246 tablet_bootstrap.cc:654] T 32dd15923a55417c97d54822f930bbdd P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:19.291646 14246 tablet_bootstrap.cc:492] T 32dd15923a55417c97d54822f930bbdd P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:19.291738 14246 ts_tablet_manager.cc:1367] T 32dd15923a55417c97d54822f930bbdd P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:19.292147 14249 tablet_bootstrap.cc:492] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:19.293124 14249 tablet_bootstrap.cc:654] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
W20240503 02:00:19.293432 8851 consensus_peers.cc:485] T 7da8f614a70e4177b70e7c2f31d22a10 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:00:19.294420 14246 raft_consensus.cc:348] T 32dd15923a55417c97d54822f930bbdd P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:19.294574 14246 raft_consensus.cc:374] T 32dd15923a55417c97d54822f930bbdd P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:19.294586 14247 tablet_bootstrap.cc:654] T 32dd15923a55417c97d54822f930bbdd P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:19.294623 14246 raft_consensus.cc:729] T 32dd15923a55417c97d54822f930bbdd P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:19.295286 14246 consensus_queue.cc:260] T 32dd15923a55417c97d54822f930bbdd P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:19.295661 14246 ts_tablet_manager.cc:1398] T 32dd15923a55417c97d54822f930bbdd P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:19.295980 14247 tablet_bootstrap.cc:492] T 32dd15923a55417c97d54822f930bbdd P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:19.296092 14247 ts_tablet_manager.cc:1367] T 32dd15923a55417c97d54822f930bbdd P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:00:19.296911 14247 raft_consensus.cc:348] T 32dd15923a55417c97d54822f930bbdd P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:19.297039 14247 raft_consensus.cc:374] T 32dd15923a55417c97d54822f930bbdd P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:19.297080 14247 raft_consensus.cc:729] T 32dd15923a55417c97d54822f930bbdd P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:19.298473 14249 tablet_bootstrap.cc:492] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:19.298583 14249 ts_tablet_manager.cc:1367] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:00:19.298561 14247 consensus_queue.cc:260] T 32dd15923a55417c97d54822f930bbdd P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:19.298878 14247 ts_tablet_manager.cc:1398] T 32dd15923a55417c97d54822f930bbdd P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:00:19.299067 14249 raft_consensus.cc:348] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:19.299163 14249 raft_consensus.cc:374] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:19.299192 14249 raft_consensus.cc:729] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:19.299309 14249 consensus_queue.cc:260] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:19.299537 14249 ts_tablet_manager.cc:1398] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:19.317831 6308 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_extra_table_50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:19.325495 6699 tablet_service.cc:1467] Processing CreateTablet for tablet 2b7a65bae2c5459c9a378b80f2255ed3 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=2490acf99a114d9296316c67e2418726]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:19.325495 6436 tablet_service.cc:1467] Processing CreateTablet for tablet 2b7a65bae2c5459c9a378b80f2255ed3 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=2490acf99a114d9296316c67e2418726]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:19.325908 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b7a65bae2c5459c9a378b80f2255ed3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:19.325908 6699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b7a65bae2c5459c9a378b80f2255ed3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:19.326144 6829 tablet_service.cc:1467] Processing CreateTablet for tablet 5d8e8972c10241108d716d92a38f4c17 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=2490acf99a114d9296316c67e2418726]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:19.326144 6831 tablet_service.cc:1467] Processing CreateTablet for tablet 72aea5be835849d9a392fbab35357b7b (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=2490acf99a114d9296316c67e2418726]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:19.326442 6829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d8e8972c10241108d716d92a38f4c17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:19.326911 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72aea5be835849d9a392fbab35357b7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:19.328522 6700 tablet_service.cc:1467] Processing CreateTablet for tablet 5d8e8972c10241108d716d92a38f4c17 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=2490acf99a114d9296316c67e2418726]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:19.328783 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d8e8972c10241108d716d92a38f4c17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:19.328806 14249 tablet_bootstrap.cc:492] T 2b7a65bae2c5459c9a378b80f2255ed3 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:19.329066 14246 tablet_bootstrap.cc:492] T 5d8e8972c10241108d716d92a38f4c17 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:19.329788 14249 tablet_bootstrap.cc:654] T 2b7a65bae2c5459c9a378b80f2255ed3 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:19.330045 14246 tablet_bootstrap.cc:654] T 5d8e8972c10241108d716d92a38f4c17 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:19.331267 9027 tablet_service.cc:1467] Processing CreateTablet for tablet 2b7a65bae2c5459c9a378b80f2255ed3 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=2490acf99a114d9296316c67e2418726]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:19.331564 9027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b7a65bae2c5459c9a378b80f2255ed3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:19.331585 14246 tablet_bootstrap.cc:492] T 5d8e8972c10241108d716d92a38f4c17 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:19.331606 14252 tablet_bootstrap.cc:492] T 2b7a65bae2c5459c9a378b80f2255ed3 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:19.331662 14246 ts_tablet_manager.cc:1367] T 5d8e8972c10241108d716d92a38f4c17 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:19.332152 14246 raft_consensus.cc:348] T 5d8e8972c10241108d716d92a38f4c17 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:19.332259 14246 raft_consensus.cc:374] T 5d8e8972c10241108d716d92a38f4c17 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:19.332285 14246 raft_consensus.cc:729] T 5d8e8972c10241108d716d92a38f4c17 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:19.332422 14246 consensus_queue.cc:260] T 5d8e8972c10241108d716d92a38f4c17 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:19.332460 14252 tablet_bootstrap.cc:654] T 2b7a65bae2c5459c9a378b80f2255ed3 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:19.333187 14249 tablet_bootstrap.cc:492] T 2b7a65bae2c5459c9a378b80f2255ed3 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:19.333283 14249 ts_tablet_manager.cc:1367] T 2b7a65bae2c5459c9a378b80f2255ed3 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:00:19.333577 14252 tablet_bootstrap.cc:492] T 2b7a65bae2c5459c9a378b80f2255ed3 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:19.333657 14252 ts_tablet_manager.cc:1367] T 2b7a65bae2c5459c9a378b80f2255ed3 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:00:19.333679 14246 ts_tablet_manager.cc:1398] T 5d8e8972c10241108d716d92a38f4c17 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:19.333768 14249 raft_consensus.cc:348] T 2b7a65bae2c5459c9a378b80f2255ed3 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:19.333873 14249 raft_consensus.cc:374] T 2b7a65bae2c5459c9a378b80f2255ed3 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:19.333884 14246 tablet_bootstrap.cc:492] T 72aea5be835849d9a392fbab35357b7b P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:19.333904 14249 raft_consensus.cc:729] T 2b7a65bae2c5459c9a378b80f2255ed3 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:19.334038 14249 consensus_queue.cc:260] T 2b7a65bae2c5459c9a378b80f2255ed3 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:19.334082 14252 raft_consensus.cc:348] T 2b7a65bae2c5459c9a378b80f2255ed3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:19.334169 14252 raft_consensus.cc:374] T 2b7a65bae2c5459c9a378b80f2255ed3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:19.334195 14252 raft_consensus.cc:729] T 2b7a65bae2c5459c9a378b80f2255ed3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:19.334323 14249 ts_tablet_manager.cc:1398] T 2b7a65bae2c5459c9a378b80f2255ed3 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:19.334300 14252 consensus_queue.cc:260] T 2b7a65bae2c5459c9a378b80f2255ed3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:19.334544 14252 ts_tablet_manager.cc:1398] T 2b7a65bae2c5459c9a378b80f2255ed3 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:19.334698 14252 tablet_bootstrap.cc:492] T 5d8e8972c10241108d716d92a38f4c17 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:19.334900 9157 tablet_service.cc:1467] Processing CreateTablet for tablet 72aea5be835849d9a392fbab35357b7b (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=2490acf99a114d9296316c67e2418726]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:19.335175 9157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72aea5be835849d9a392fbab35357b7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:19.335111 6567 tablet_service.cc:1467] Processing CreateTablet for tablet 5d8e8972c10241108d716d92a38f4c17 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=2490acf99a114d9296316c67e2418726]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:19.335407 6567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d8e8972c10241108d716d92a38f4c17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:19.335568 14246 tablet_bootstrap.cc:654] T 72aea5be835849d9a392fbab35357b7b P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:19.335574 14252 tablet_bootstrap.cc:654] T 5d8e8972c10241108d716d92a38f4c17 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:19.336524 14256 tablet_bootstrap.cc:492] T 2b7a65bae2c5459c9a378b80f2255ed3 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:19.337108 14246 tablet_bootstrap.cc:492] T 72aea5be835849d9a392fbab35357b7b P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:19.337266 14246 ts_tablet_manager.cc:1367] T 72aea5be835849d9a392fbab35357b7b P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:19.337795 14246 raft_consensus.cc:348] T 72aea5be835849d9a392fbab35357b7b P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:19.337936 14246 raft_consensus.cc:374] T 72aea5be835849d9a392fbab35357b7b P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:19.337944 14260 tablet_bootstrap.cc:492] T 5d8e8972c10241108d716d92a38f4c17 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:19.337973 14246 raft_consensus.cc:729] T 72aea5be835849d9a392fbab35357b7b P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:19.338162 14246 consensus_queue.cc:260] T 72aea5be835849d9a392fbab35357b7b P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:19.338452 14246 ts_tablet_manager.cc:1398] T 72aea5be835849d9a392fbab35357b7b P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:19.338666 14259 tablet_bootstrap.cc:492] T 72aea5be835849d9a392fbab35357b7b P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:19.339803 14256 tablet_bootstrap.cc:654] T 2b7a65bae2c5459c9a378b80f2255ed3 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:19.339812 6700 tablet_service.cc:1467] Processing CreateTablet for tablet 72aea5be835849d9a392fbab35357b7b (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=2490acf99a114d9296316c67e2418726]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:19.340036 14259 tablet_bootstrap.cc:654] T 72aea5be835849d9a392fbab35357b7b P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:19.340085 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72aea5be835849d9a392fbab35357b7b. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:00:19.342213 9113 consensus_peers.cc:485] T b1740b877c1e4372a0337d99f5bcab0e P 2a01ace80baf4057b6bf1995cf88468b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:00:19.342859 14252 tablet_bootstrap.cc:492] T 5d8e8972c10241108d716d92a38f4c17 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:19.342952 14252 ts_tablet_manager.cc:1367] T 5d8e8972c10241108d716d92a38f4c17 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.001s
I20240503 02:00:19.343437 14252 raft_consensus.cc:348] T 5d8e8972c10241108d716d92a38f4c17 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:19.343562 14252 raft_consensus.cc:374] T 5d8e8972c10241108d716d92a38f4c17 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:19.343606 14252 raft_consensus.cc:729] T 5d8e8972c10241108d716d92a38f4c17 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:19.343732 14252 consensus_queue.cc:260] T 5d8e8972c10241108d716d92a38f4c17 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:19.343995 14252 ts_tablet_manager.cc:1398] T 5d8e8972c10241108d716d92a38f4c17 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:19.344139 14252 tablet_bootstrap.cc:492] T 72aea5be835849d9a392fbab35357b7b P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:19.344528 14256 tablet_bootstrap.cc:492] T 2b7a65bae2c5459c9a378b80f2255ed3 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:19.344544 14056 raft_consensus.cc:482] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:19.344642 14056 raft_consensus.cc:504] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:19.344774 14256 ts_tablet_manager.cc:1367] T 2b7a65bae2c5459c9a378b80f2255ed3 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 02:00:19.345108 14252 tablet_bootstrap.cc:654] T 72aea5be835849d9a392fbab35357b7b P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:19.345160 14056 leader_election.cc:288] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:19.345634 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32dd15923a55417c97d54822f930bbdd" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 02:00:19.345759 6849 raft_consensus.cc:2454] T 32dd15923a55417c97d54822f930bbdd P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:19.346074 6394 leader_election.cc:302] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:19.346295 8917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32dd15923a55417c97d54822f930bbdd" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:19.346355 14056 raft_consensus.cc:2781] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:19.346410 14056 raft_consensus.cc:482] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:19.346444 14056 raft_consensus.cc:3037] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:19.346427 8917 raft_consensus.cc:2454] T 32dd15923a55417c97d54822f930bbdd P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:19.346839 14260 tablet_bootstrap.cc:654] T 5d8e8972c10241108d716d92a38f4c17 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:19.347543 14056 raft_consensus.cc:504] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:19.347904 14260 tablet_bootstrap.cc:492] T 5d8e8972c10241108d716d92a38f4c17 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:19.347906 14056 leader_election.cc:288] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Requested vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:19.347996 14260 ts_tablet_manager.cc:1367] T 5d8e8972c10241108d716d92a38f4c17 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20240503 02:00:19.348526 14252 tablet_bootstrap.cc:492] T 72aea5be835849d9a392fbab35357b7b P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:19.348489 8917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32dd15923a55417c97d54822f930bbdd" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:19.348481 14260 raft_consensus.cc:348] T 5d8e8972c10241108d716d92a38f4c17 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:19.348583 14260 raft_consensus.cc:374] T 5d8e8972c10241108d716d92a38f4c17 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:19.348613 14260 raft_consensus.cc:729] T 5d8e8972c10241108d716d92a38f4c17 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:19.348614 14252 ts_tablet_manager.cc:1367] T 72aea5be835849d9a392fbab35357b7b P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:00:19.348608 8917 raft_consensus.cc:3037] T 32dd15923a55417c97d54822f930bbdd P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:19.348733 14260 consensus_queue.cc:260] T 5d8e8972c10241108d716d92a38f4c17 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:19.348737 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32dd15923a55417c97d54822f930bbdd" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 02:00:19.348861 6849 raft_consensus.cc:3037] T 32dd15923a55417c97d54822f930bbdd P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:19.349002 14260 ts_tablet_manager.cc:1398] T 5d8e8972c10241108d716d92a38f4c17 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:19.349157 14252 raft_consensus.cc:348] T 72aea5be835849d9a392fbab35357b7b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:19.349264 14252 raft_consensus.cc:374] T 72aea5be835849d9a392fbab35357b7b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:19.349290 14252 raft_consensus.cc:729] T 72aea5be835849d9a392fbab35357b7b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:19.349401 14252 consensus_queue.cc:260] T 72aea5be835849d9a392fbab35357b7b P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:19.349618 8917 raft_consensus.cc:2454] T 32dd15923a55417c97d54822f930bbdd P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:19.349664 14252 ts_tablet_manager.cc:1398] T 72aea5be835849d9a392fbab35357b7b P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:19.349859 6849 raft_consensus.cc:2454] T 32dd15923a55417c97d54822f930bbdd P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:19.350246 14259 tablet_bootstrap.cc:492] T 72aea5be835849d9a392fbab35357b7b P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:19.350270 6394 leader_election.cc:302] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:19.350335 14259 ts_tablet_manager.cc:1367] T 72aea5be835849d9a392fbab35357b7b P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 02:00:19.350944 14259 raft_consensus.cc:348] T 72aea5be835849d9a392fbab35357b7b P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:19.351058 14259 raft_consensus.cc:374] T 72aea5be835849d9a392fbab35357b7b P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:19.351123 14259 raft_consensus.cc:729] T 72aea5be835849d9a392fbab35357b7b P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:19.351171 14056 raft_consensus.cc:2781] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:19.351272 14056 raft_consensus.cc:686] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Becoming Leader. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Running, Role: LEADER
I20240503 02:00:19.351311 14259 consensus_queue.cc:260] T 72aea5be835849d9a392fbab35357b7b P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:19.351409 14056 consensus_queue.cc:237] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:19.351742 14259 ts_tablet_manager.cc:1398] T 72aea5be835849d9a392fbab35357b7b P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 02:00:19.345325 14256 raft_consensus.cc:348] T 2b7a65bae2c5459c9a378b80f2255ed3 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:19.353992 14256 raft_consensus.cc:374] T 2b7a65bae2c5459c9a378b80f2255ed3 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:19.354110 14256 raft_consensus.cc:729] T 2b7a65bae2c5459c9a378b80f2255ed3 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:19.354321 14256 consensus_queue.cc:260] T 2b7a65bae2c5459c9a378b80f2255ed3 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:19.354540 6309 catalog_manager.cc:5484] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: term changed from 0 to 1, leader changed from <none> to dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129). New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:19.357532 14256 ts_tablet_manager.cc:1398] T 2b7a65bae2c5459c9a378b80f2255ed3 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.013s user 0.001s sys 0.000s
I20240503 02:00:19.360335 13954 raft_consensus.cc:482] T 5d8e8972c10241108d716d92a38f4c17 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:19.360497 13954 raft_consensus.cc:504] T 5d8e8972c10241108d716d92a38f4c17 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:19.360942 13954 leader_election.cc:288] T 5d8e8972c10241108d716d92a38f4c17 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:19.361353 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d8e8972c10241108d716d92a38f4c17" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 02:00:19.361403 6720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d8e8972c10241108d716d92a38f4c17" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 02:00:19.361503 6849 raft_consensus.cc:2454] T 5d8e8972c10241108d716d92a38f4c17 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:19.361567 6720 raft_consensus.cc:2454] T 5d8e8972c10241108d716d92a38f4c17 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:19.361788 6525 leader_election.cc:302] T 5d8e8972c10241108d716d92a38f4c17 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 80fd18ef4d604c86b909d5c008a46b4f; no voters:
I20240503 02:00:19.361963 13954 raft_consensus.cc:2781] T 5d8e8972c10241108d716d92a38f4c17 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:19.362023 13954 raft_consensus.cc:482] T 5d8e8972c10241108d716d92a38f4c17 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:19.362057 13954 raft_consensus.cc:3037] T 5d8e8972c10241108d716d92a38f4c17 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:19.362979 13954 raft_consensus.cc:504] T 5d8e8972c10241108d716d92a38f4c17 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:19.363327 13954 leader_election.cc:288] T 5d8e8972c10241108d716d92a38f4c17 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Requested vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:19.363795 6720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d8e8972c10241108d716d92a38f4c17" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 02:00:19.363770 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d8e8972c10241108d716d92a38f4c17" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 02:00:19.363915 6720 raft_consensus.cc:3037] T 5d8e8972c10241108d716d92a38f4c17 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:19.363924 6849 raft_consensus.cc:3037] T 5d8e8972c10241108d716d92a38f4c17 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:19.364768 6720 raft_consensus.cc:2454] T 5d8e8972c10241108d716d92a38f4c17 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:19.364768 6849 raft_consensus.cc:2454] T 5d8e8972c10241108d716d92a38f4c17 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:19.365231 6525 leader_election.cc:302] T 5d8e8972c10241108d716d92a38f4c17 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 80fd18ef4d604c86b909d5c008a46b4f; no voters:
I20240503 02:00:19.365620 13954 raft_consensus.cc:2781] T 5d8e8972c10241108d716d92a38f4c17 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:19.365731 13954 raft_consensus.cc:686] T 5d8e8972c10241108d716d92a38f4c17 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Becoming Leader. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Running, Role: LEADER
I20240503 02:00:19.365906 13954 consensus_queue.cc:237] T 5d8e8972c10241108d716d92a38f4c17 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:19.368669 6312 catalog_manager.cc:5484] T 5d8e8972c10241108d716d92a38f4c17 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } }
W20240503 02:00:19.400913 6301 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20240503 02:00:19.421371 8982 consensus_peers.cc:485] T beeea71f02ae4c89a1af13d3584ef70a P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:00:19.464830 13969 raft_consensus.cc:482] T 2b7a65bae2c5459c9a378b80f2255ed3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:19.464984 13969 raft_consensus.cc:504] T 2b7a65bae2c5459c9a378b80f2255ed3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:19.465502 13969 leader_election.cc:288] T 2b7a65bae2c5459c9a378b80f2255ed3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:19.465941 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b7a65bae2c5459c9a378b80f2255ed3" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 02:00:19.465958 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b7a65bae2c5459c9a378b80f2255ed3" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:19.466092 9048 raft_consensus.cc:2454] T 2b7a65bae2c5459c9a378b80f2255ed3 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:19.466104 6456 raft_consensus.cc:2454] T 2b7a65bae2c5459c9a378b80f2255ed3 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:19.466413 6653 leader_election.cc:302] T 2b7a65bae2c5459c9a378b80f2255ed3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2ac2eb7cc264047b9ab32fe3a4fc025, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:19.466588 13969 raft_consensus.cc:2781] T 2b7a65bae2c5459c9a378b80f2255ed3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:19.466668 13969 raft_consensus.cc:482] T 2b7a65bae2c5459c9a378b80f2255ed3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:19.466701 13969 raft_consensus.cc:3037] T 2b7a65bae2c5459c9a378b80f2255ed3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:19.467762 13969 raft_consensus.cc:504] T 2b7a65bae2c5459c9a378b80f2255ed3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:19.469170 6309 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:19.469205 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b7a65bae2c5459c9a378b80f2255ed3" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 02:00:19.469398 6456 raft_consensus.cc:3037] T 2b7a65bae2c5459c9a378b80f2255ed3 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:19.470297 6456 raft_consensus.cc:2454] T 2b7a65bae2c5459c9a378b80f2255ed3 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:19.468771 13969 leader_election.cc:288] T 2b7a65bae2c5459c9a378b80f2255ed3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Requested vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:19.470662 6653 leader_election.cc:302] T 2b7a65bae2c5459c9a378b80f2255ed3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2ac2eb7cc264047b9ab32fe3a4fc025, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:19.470815 13969 raft_consensus.cc:2781] T 2b7a65bae2c5459c9a378b80f2255ed3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:19.470940 13969 raft_consensus.cc:686] T 2b7a65bae2c5459c9a378b80f2255ed3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Becoming Leader. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Running, Role: LEADER
I20240503 02:00:19.471052 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b7a65bae2c5459c9a378b80f2255ed3" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:19.471172 13969 consensus_queue.cc:237] T 2b7a65bae2c5459c9a378b80f2255ed3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:19.471266 9048 raft_consensus.cc:3037] T 2b7a65bae2c5459c9a378b80f2255ed3 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:19.472852 9048 raft_consensus.cc:2454] T 2b7a65bae2c5459c9a378b80f2255ed3 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:19.473603 6849 raft_consensus.cc:1261] T 32dd15923a55417c97d54822f930bbdd P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:19.474016 8917 raft_consensus.cc:1261] T 32dd15923a55417c97d54822f930bbdd P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:19.474133 14056 consensus_queue.cc:1035] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:19.474762 14153 consensus_queue.cc:1035] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:19.475209 6309 catalog_manager.cc:5484] T 2b7a65bae2c5459c9a378b80f2255ed3 P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:19.478020 14231 tablet.cc:1722] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:19.478397 14231 tablet.cc:1620] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a: MemRowSet was empty: no flush needed.
I20240503 02:00:19.479257 6312 catalog_manager.cc:5576] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211) does not have the latest schema for tablet 32dd15923a55417c97d54822f930bbdd (table rebalancer_dynamic_table [id=d6007f227ea147b58a002001972f41cf]). Expected version 1 got 0
I20240503 02:00:19.480370 6314 catalog_manager.cc:5576] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719) does not have the latest schema for tablet 32dd15923a55417c97d54822f930bbdd (table rebalancer_dynamic_table [id=d6007f227ea147b58a002001972f41cf]). Expected version 1 got 0
I20240503 02:00:19.480705 14269 mvcc.cc:205] Tried to move back new op lower bound from 7023417833358508032 to 7023417832868155392. Current Snapshot: MvccSnapshot[applied={T|T < 7023417833358508032}]
I20240503 02:00:19.480943 14272 tablet.cc:1722] T 32dd15923a55417c97d54822f930bbdd P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:19.481245 14272 tablet.cc:1620] T 32dd15923a55417c97d54822f930bbdd P 492448570cc94ed1879feb2ec3b9051d: MemRowSet was empty: no flush needed.
I20240503 02:00:19.481793 14274 tablet.cc:1722] T 32dd15923a55417c97d54822f930bbdd P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:19.483021 14274 tablet.cc:1620] T 32dd15923a55417c97d54822f930bbdd P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
I20240503 02:00:19.486246 6314 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=d6007f227ea147b58a002001972f41cf] alter complete (version 1)
W20240503 02:00:19.526684 6523 consensus_peers.cc:485] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:19.538025 8851 consensus_peers.cc:485] T f28586e70bd9473fbdc6b9126b666fa9 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:19.542378 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:19.591318 6314 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:19.596755 14231 tablet.cc:1722] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:19.597190 14231 tablet.cc:1620] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a: MemRowSet was empty: no flush needed.
I20240503 02:00:19.600607 14272 tablet.cc:1722] T 32dd15923a55417c97d54822f930bbdd P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:19.600965 14272 tablet.cc:1620] T 32dd15923a55417c97d54822f930bbdd P 492448570cc94ed1879feb2ec3b9051d: MemRowSet was empty: no flush needed.
I20240503 02:00:19.603163 14274 tablet.cc:1722] T 32dd15923a55417c97d54822f930bbdd P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:19.603480 14274 tablet.cc:1620] T 32dd15923a55417c97d54822f930bbdd P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
I20240503 02:00:19.604059 6314 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=d6007f227ea147b58a002001972f41cf] alter complete (version 2)
W20240503 02:00:19.609004 8982 consensus_peers.cc:485] T 1a7ee79312ea4bad816264375ca5bb2d P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:00:19.613512 14106 raft_consensus.cc:482] T 72aea5be835849d9a392fbab35357b7b P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:19.613667 14106 raft_consensus.cc:504] T 72aea5be835849d9a392fbab35357b7b P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:19.614194 14106 leader_election.cc:288] T 72aea5be835849d9a392fbab35357b7b P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:19.614650 6720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72aea5be835849d9a392fbab35357b7b" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 02:00:19.614794 6720 raft_consensus.cc:2454] T 72aea5be835849d9a392fbab35357b7b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 02:00:19.614765 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72aea5be835849d9a392fbab35357b7b" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:19.614894 9178 raft_consensus.cc:2454] T 72aea5be835849d9a392fbab35357b7b P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 02:00:19.615077 6787 leader_election.cc:302] T 72aea5be835849d9a392fbab35357b7b P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:19.615397 14106 raft_consensus.cc:2781] T 72aea5be835849d9a392fbab35357b7b P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:19.615461 14106 raft_consensus.cc:482] T 72aea5be835849d9a392fbab35357b7b P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:19.615489 14106 raft_consensus.cc:3037] T 72aea5be835849d9a392fbab35357b7b P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:19.616408 14106 raft_consensus.cc:504] T 72aea5be835849d9a392fbab35357b7b P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:19.616772 14106 leader_election.cc:288] T 72aea5be835849d9a392fbab35357b7b P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Requested vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:19.617137 6720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72aea5be835849d9a392fbab35357b7b" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 02:00:19.617146 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72aea5be835849d9a392fbab35357b7b" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:19.617353 6720 raft_consensus.cc:3037] T 72aea5be835849d9a392fbab35357b7b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:19.617362 9178 raft_consensus.cc:3037] T 72aea5be835849d9a392fbab35357b7b P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:19.618392 9178 raft_consensus.cc:2454] T 72aea5be835849d9a392fbab35357b7b P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 02:00:19.618425 6720 raft_consensus.cc:2454] T 72aea5be835849d9a392fbab35357b7b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 02:00:19.618734 6787 leader_election.cc:302] T 72aea5be835849d9a392fbab35357b7b P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:19.618947 14106 raft_consensus.cc:2781] T 72aea5be835849d9a392fbab35357b7b P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:19.619140 14106 raft_consensus.cc:686] T 72aea5be835849d9a392fbab35357b7b P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Becoming Leader. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Running, Role: LEADER
I20240503 02:00:19.619290 14106 consensus_queue.cc:237] T 72aea5be835849d9a392fbab35357b7b P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:19.621780 6308 catalog_manager.cc:5484] T 72aea5be835849d9a392fbab35357b7b P 492448570cc94ed1879feb2ec3b9051d reported cstate change: term changed from 0 to 1, leader changed from <none> to 492448570cc94ed1879feb2ec3b9051d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:19.708534 6308 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:19.714170 14231 tablet.cc:1722] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:19.714623 14274 tablet.cc:1722] T 32dd15923a55417c97d54822f930bbdd P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:19.714658 14272 tablet.cc:1722] T 32dd15923a55417c97d54822f930bbdd P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:19.720556 6308 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=d6007f227ea147b58a002001972f41cf] alter complete (version 3)
I20240503 02:00:19.824115 6314 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:19.824278 6314 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:19.826346 6314 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet 32dd15923a55417c97d54822f930bbdd
I20240503 02:00:19.826916 8880 tablet_service.cc:1514] Processing DeleteTablet for tablet 32dd15923a55417c97d54822f930bbdd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:19 UTC) from {username='slave'} at 127.0.0.1:40296
I20240503 02:00:19.827025 6436 tablet_service.cc:1514] Processing DeleteTablet for tablet 32dd15923a55417c97d54822f930bbdd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:19 UTC) from {username='slave'} at 127.0.0.1:51338
I20240503 02:00:19.827304 14278 tablet_replica.cc:330] T 32dd15923a55417c97d54822f930bbdd P 9942306855404cfd891adf227dd65eb2: stopping tablet replica
I20240503 02:00:19.827278 6831 tablet_service.cc:1514] Processing DeleteTablet for tablet 32dd15923a55417c97d54822f930bbdd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:19 UTC) from {username='slave'} at 127.0.0.1:52812
I20240503 02:00:19.827358 14279 tablet_replica.cc:330] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a: stopping tablet replica
I20240503 02:00:19.827466 14278 raft_consensus.cc:2229] T 32dd15923a55417c97d54822f930bbdd P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:19.827491 14279 raft_consensus.cc:2229] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:19.827653 14280 tablet_replica.cc:330] T 32dd15923a55417c97d54822f930bbdd P 492448570cc94ed1879feb2ec3b9051d: stopping tablet replica
I20240503 02:00:19.827677 14278 raft_consensus.cc:2258] T 32dd15923a55417c97d54822f930bbdd P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:19.827787 14279 raft_consensus.cc:2258] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:19.827832 14280 raft_consensus.cc:2229] T 32dd15923a55417c97d54822f930bbdd P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:19.827963 14280 raft_consensus.cc:2258] T 32dd15923a55417c97d54822f930bbdd P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:19.828290 14278 ts_tablet_manager.cc:1875] T 32dd15923a55417c97d54822f930bbdd P 9942306855404cfd891adf227dd65eb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:19.828474 14279 ts_tablet_manager.cc:1875] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:19.828598 14280 ts_tablet_manager.cc:1875] T 32dd15923a55417c97d54822f930bbdd P 492448570cc94ed1879feb2ec3b9051d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:19.830646 14279 ts_tablet_manager.cc:1888] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:19.830734 14279 log.cc:1196] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/32dd15923a55417c97d54822f930bbdd
I20240503 02:00:19.830783 14280 ts_tablet_manager.cc:1888] T 32dd15923a55417c97d54822f930bbdd P 492448570cc94ed1879feb2ec3b9051d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:19.830828 14278 ts_tablet_manager.cc:1888] T 32dd15923a55417c97d54822f930bbdd P 9942306855404cfd891adf227dd65eb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:19.830896 14280 log.cc:1196] T 32dd15923a55417c97d54822f930bbdd P 492448570cc94ed1879feb2ec3b9051d: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/32dd15923a55417c97d54822f930bbdd
I20240503 02:00:19.830889 14278 log.cc:1196] T 32dd15923a55417c97d54822f930bbdd P 9942306855404cfd891adf227dd65eb2: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/32dd15923a55417c97d54822f930bbdd
I20240503 02:00:19.831140 14279 ts_tablet_manager.cc:1909] T 32dd15923a55417c97d54822f930bbdd P dec3e4a04d604f7eaaa3f077d081218a: Deleting consensus metadata
I20240503 02:00:19.831259 14280 ts_tablet_manager.cc:1909] T 32dd15923a55417c97d54822f930bbdd P 492448570cc94ed1879feb2ec3b9051d: Deleting consensus metadata
I20240503 02:00:19.831288 14278 ts_tablet_manager.cc:1909] T 32dd15923a55417c97d54822f930bbdd P 9942306855404cfd891adf227dd65eb2: Deleting consensus metadata
I20240503 02:00:19.832036 6303 catalog_manager.cc:4830] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719): tablet 32dd15923a55417c97d54822f930bbdd (table rebalancer_dynamic_table_new_name [id=d6007f227ea147b58a002001972f41cf]) successfully deleted
I20240503 02:00:19.832036 6300 catalog_manager.cc:4830] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): tablet 32dd15923a55417c97d54822f930bbdd (table rebalancer_dynamic_table_new_name [id=d6007f227ea147b58a002001972f41cf]) successfully deleted
I20240503 02:00:19.832249 6303 catalog_manager.cc:4830] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211): tablet 32dd15923a55417c97d54822f930bbdd (table rebalancer_dynamic_table_new_name [id=d6007f227ea147b58a002001972f41cf]) successfully deleted
I20240503 02:00:19.892845 14190 consensus_queue.cc:1035] T 2b7a65bae2c5459c9a378b80f2255ed3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:19.896586 13969 consensus_queue.cc:1035] T 2b7a65bae2c5459c9a378b80f2255ed3 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:19.898124 14202 consensus_queue.cc:1035] T 5d8e8972c10241108d716d92a38f4c17 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:19.905500 14287 consensus_queue.cc:1035] T 5d8e8972c10241108d716d92a38f4c17 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:19.928005 6312 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:00:19.934177 9157 tablet_service.cc:1467] Processing CreateTablet for tablet 2ed12d2ca4c74194a99bd6e4f9c28834 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=78ad5e8d3227453fb447fb66f8d78a05]), partition=
I20240503 02:00:19.934177 6567 tablet_service.cc:1467] Processing CreateTablet for tablet 2ed12d2ca4c74194a99bd6e4f9c28834 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=78ad5e8d3227453fb447fb66f8d78a05]), partition=
I20240503 02:00:19.934473 6567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ed12d2ca4c74194a99bd6e4f9c28834. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:19.934473 9157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ed12d2ca4c74194a99bd6e4f9c28834. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:19.937086 6831 tablet_service.cc:1467] Processing CreateTablet for tablet 2ed12d2ca4c74194a99bd6e4f9c28834 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=78ad5e8d3227453fb447fb66f8d78a05]), partition=
I20240503 02:00:19.937325 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ed12d2ca4c74194a99bd6e4f9c28834. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:19.939164 14293 tablet_bootstrap.cc:492] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:19.939265 14291 tablet_bootstrap.cc:492] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:19.940114 14291 tablet_bootstrap.cc:654] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:19.940153 14293 tablet_bootstrap.cc:654] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:19.940867 14292 tablet_bootstrap.cc:492] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:19.941670 14293 tablet_bootstrap.cc:492] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:19.941834 14293 ts_tablet_manager.cc:1367] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:19.941895 14292 tablet_bootstrap.cc:654] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:19.942322 14293 raft_consensus.cc:348] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:19.942459 14293 raft_consensus.cc:374] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:19.942504 14293 raft_consensus.cc:729] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:19.942662 14293 consensus_queue.cc:260] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:19.943045 14291 tablet_bootstrap.cc:492] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:19.943204 14291 ts_tablet_manager.cc:1367] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:19.943325 14293 ts_tablet_manager.cc:1398] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:19.943688 14291 raft_consensus.cc:348] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:19.944362 14292 tablet_bootstrap.cc:492] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:19.944487 14292 ts_tablet_manager.cc:1367] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240503 02:00:19.944749 14291 raft_consensus.cc:374] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:19.945221 14292 raft_consensus.cc:348] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:19.945405 14291 raft_consensus.cc:729] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:19.945566 14291 consensus_queue.cc:260] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:19.945971 14291 ts_tablet_manager.cc:1398] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:00:19.946063 14292 raft_consensus.cc:374] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:19.946153 14292 raft_consensus.cc:729] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:19.946451 14292 consensus_queue.cc:260] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
W20240503 02:00:19.946756 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:19.946907 14292 ts_tablet_manager.cc:1398] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:00:19.998762 14276 consensus_queue.cc:1035] T 72aea5be835849d9a392fbab35357b7b P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:20.002166 14276 consensus_queue.cc:1035] T 72aea5be835849d9a392fbab35357b7b P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:20.134070 14106 raft_consensus.cc:482] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:20.134438 14106 raft_consensus.cc:504] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:20.135380 14106 leader_election.cc:288] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:20.135352 6582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ed12d2ca4c74194a99bd6e4f9c28834" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 02:00:20.135478 6582 raft_consensus.cc:2454] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 02:00:20.135746 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ed12d2ca4c74194a99bd6e4f9c28834" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:20.135809 6786 leader_election.cc:302] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 80fd18ef4d604c86b909d5c008a46b4f; no voters:
I20240503 02:00:20.135870 9178 raft_consensus.cc:2454] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 02:00:20.136047 14106 raft_consensus.cc:2781] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:20.136103 14106 raft_consensus.cc:482] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:20.136140 14106 raft_consensus.cc:3037] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:20.137259 14106 raft_consensus.cc:504] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:20.137585 14106 leader_election.cc:288] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Requested vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:20.137878 6582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ed12d2ca4c74194a99bd6e4f9c28834" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 02:00:20.137881 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ed12d2ca4c74194a99bd6e4f9c28834" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:20.137990 9178 raft_consensus.cc:3037] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:20.137989 6582 raft_consensus.cc:3037] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:20.138788 9178 raft_consensus.cc:2454] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 02:00:20.138862 6582 raft_consensus.cc:2454] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 02:00:20.139081 6784 leader_election.cc:302] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 492448570cc94ed1879feb2ec3b9051d; no voters:
I20240503 02:00:20.139250 14106 raft_consensus.cc:2781] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:20.139408 14106 raft_consensus.cc:686] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Becoming Leader. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Running, Role: LEADER
I20240503 02:00:20.139554 14106 consensus_queue.cc:237] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:20.140059 6314 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_extra_table_52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:20.141767 6308 catalog_manager.cc:5484] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d reported cstate change: term changed from 0 to 1, leader changed from <none> to 492448570cc94ed1879feb2ec3b9051d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:20.146596 9157 tablet_service.cc:1467] Processing CreateTablet for tablet 0433ba47dea84e2eb869df726191f722 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=63d2db6cf1d6486481601028d594114d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:20.146732 6700 tablet_service.cc:1467] Processing CreateTablet for tablet 0433ba47dea84e2eb869df726191f722 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=63d2db6cf1d6486481601028d594114d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:20.146934 9157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0433ba47dea84e2eb869df726191f722. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:20.146978 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0433ba47dea84e2eb869df726191f722. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:20.147053 8880 tablet_service.cc:1467] Processing CreateTablet for tablet 0433ba47dea84e2eb869df726191f722 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=63d2db6cf1d6486481601028d594114d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:20.147397 8880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0433ba47dea84e2eb869df726191f722. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:20.149415 14301 tablet_bootstrap.cc:492] T 0433ba47dea84e2eb869df726191f722 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:20.149538 9027 tablet_service.cc:1467] Processing CreateTablet for tablet db02d12f83a841f09588e7d6ce0a8398 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=63d2db6cf1d6486481601028d594114d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:20.149827 9027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db02d12f83a841f09588e7d6ce0a8398. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:20.150174 6831 tablet_service.cc:1467] Processing CreateTablet for tablet ae19b035d4594a29a770205dad7ac6ba (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=63d2db6cf1d6486481601028d594114d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:20.150413 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae19b035d4594a29a770205dad7ac6ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:20.150583 14301 tablet_bootstrap.cc:654] T 0433ba47dea84e2eb869df726191f722 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:20.152114 14291 tablet_bootstrap.cc:492] T 0433ba47dea84e2eb869df726191f722 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:20.152120 14304 tablet_bootstrap.cc:492] T db02d12f83a841f09588e7d6ce0a8398 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:20.152665 14293 tablet_bootstrap.cc:492] T ae19b035d4594a29a770205dad7ac6ba P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:20.153134 14291 tablet_bootstrap.cc:654] T 0433ba47dea84e2eb869df726191f722 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:20.153142 14304 tablet_bootstrap.cc:654] T db02d12f83a841f09588e7d6ce0a8398 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:20.154191 14293 tablet_bootstrap.cc:654] T ae19b035d4594a29a770205dad7ac6ba P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:20.154527 6438 tablet_service.cc:1467] Processing CreateTablet for tablet db02d12f83a841f09588e7d6ce0a8398 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=63d2db6cf1d6486481601028d594114d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:20.154798 6438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db02d12f83a841f09588e7d6ce0a8398. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:20.154789 6569 tablet_service.cc:1467] Processing CreateTablet for tablet db02d12f83a841f09588e7d6ce0a8398 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=63d2db6cf1d6486481601028d594114d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:20.154881 14302 tablet_bootstrap.cc:492] T 0433ba47dea84e2eb869df726191f722 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:20.154958 6436 tablet_service.cc:1467] Processing CreateTablet for tablet ae19b035d4594a29a770205dad7ac6ba (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=63d2db6cf1d6486481601028d594114d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:20.155050 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db02d12f83a841f09588e7d6ce0a8398. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:20.155179 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae19b035d4594a29a770205dad7ac6ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:20.155854 14302 tablet_bootstrap.cc:654] T 0433ba47dea84e2eb869df726191f722 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:20.157168 14293 tablet_bootstrap.cc:492] T ae19b035d4594a29a770205dad7ac6ba P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:20.157269 14293 ts_tablet_manager.cc:1367] T ae19b035d4594a29a770205dad7ac6ba P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:00:20.157433 14309 tablet_bootstrap.cc:492] T db02d12f83a841f09588e7d6ce0a8398 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:20.157750 14293 raft_consensus.cc:348] T ae19b035d4594a29a770205dad7ac6ba P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:20.157884 14293 raft_consensus.cc:374] T ae19b035d4594a29a770205dad7ac6ba P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:20.157914 14293 raft_consensus.cc:729] T ae19b035d4594a29a770205dad7ac6ba P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:20.157928 6567 tablet_service.cc:1467] Processing CreateTablet for tablet ae19b035d4594a29a770205dad7ac6ba (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=63d2db6cf1d6486481601028d594114d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:20.158025 14293 consensus_queue.cc:260] T ae19b035d4594a29a770205dad7ac6ba P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:20.158180 6567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae19b035d4594a29a770205dad7ac6ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:20.158288 14293 ts_tablet_manager.cc:1398] T ae19b035d4594a29a770205dad7ac6ba P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:20.158485 14309 tablet_bootstrap.cc:654] T db02d12f83a841f09588e7d6ce0a8398 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:20.160319 14292 tablet_bootstrap.cc:492] T db02d12f83a841f09588e7d6ce0a8398 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:20.161289 14292 tablet_bootstrap.cc:654] T db02d12f83a841f09588e7d6ce0a8398 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:20.165889 14292 tablet_bootstrap.cc:492] T db02d12f83a841f09588e7d6ce0a8398 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:20.166013 14292 ts_tablet_manager.cc:1367] T db02d12f83a841f09588e7d6ce0a8398 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20240503 02:00:20.166476 14292 raft_consensus.cc:348] T db02d12f83a841f09588e7d6ce0a8398 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:20.166597 14292 raft_consensus.cc:374] T db02d12f83a841f09588e7d6ce0a8398 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:20.166631 14292 raft_consensus.cc:729] T db02d12f83a841f09588e7d6ce0a8398 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:20.166747 14292 consensus_queue.cc:260] T db02d12f83a841f09588e7d6ce0a8398 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:20.167055 14292 ts_tablet_manager.cc:1398] T db02d12f83a841f09588e7d6ce0a8398 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:20.167227 14292 tablet_bootstrap.cc:492] T ae19b035d4594a29a770205dad7ac6ba P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:20.167310 14301 tablet_bootstrap.cc:492] T 0433ba47dea84e2eb869df726191f722 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:20.167410 14301 ts_tablet_manager.cc:1367] T 0433ba47dea84e2eb869df726191f722 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20240503 02:00:20.167857 14301 raft_consensus.cc:348] T 0433ba47dea84e2eb869df726191f722 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:20.167963 14301 raft_consensus.cc:374] T 0433ba47dea84e2eb869df726191f722 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:20.167991 14301 raft_consensus.cc:729] T 0433ba47dea84e2eb869df726191f722 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:20.168336 14301 consensus_queue.cc:260] T 0433ba47dea84e2eb869df726191f722 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:20.168558 14304 tablet_bootstrap.cc:492] T db02d12f83a841f09588e7d6ce0a8398 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:20.168653 14304 ts_tablet_manager.cc:1367] T db02d12f83a841f09588e7d6ce0a8398 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240503 02:00:20.168769 14301 ts_tablet_manager.cc:1398] T 0433ba47dea84e2eb869df726191f722 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:20.169116 14304 raft_consensus.cc:348] T db02d12f83a841f09588e7d6ce0a8398 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:20.169217 14304 raft_consensus.cc:374] T db02d12f83a841f09588e7d6ce0a8398 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:20.169248 14304 raft_consensus.cc:729] T db02d12f83a841f09588e7d6ce0a8398 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:20.169384 14304 consensus_queue.cc:260] T db02d12f83a841f09588e7d6ce0a8398 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:20.169596 14292 tablet_bootstrap.cc:654] T ae19b035d4594a29a770205dad7ac6ba P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:20.169679 14304 ts_tablet_manager.cc:1398] T db02d12f83a841f09588e7d6ce0a8398 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:20.176472 14302 tablet_bootstrap.cc:492] T 0433ba47dea84e2eb869df726191f722 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:20.176601 14302 ts_tablet_manager.cc:1367] T 0433ba47dea84e2eb869df726191f722 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20240503 02:00:20.176709 14291 tablet_bootstrap.cc:492] T 0433ba47dea84e2eb869df726191f722 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:20.176820 14291 ts_tablet_manager.cc:1367] T 0433ba47dea84e2eb869df726191f722 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.000s
I20240503 02:00:20.177042 14302 raft_consensus.cc:348] T 0433ba47dea84e2eb869df726191f722 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:20.177135 14302 raft_consensus.cc:374] T 0433ba47dea84e2eb869df726191f722 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:20.177155 14302 raft_consensus.cc:729] T 0433ba47dea84e2eb869df726191f722 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:20.177263 14302 consensus_queue.cc:260] T 0433ba47dea84e2eb869df726191f722 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:20.177259 14291 raft_consensus.cc:348] T 0433ba47dea84e2eb869df726191f722 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:20.177363 14291 raft_consensus.cc:374] T 0433ba47dea84e2eb869df726191f722 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:20.177388 14291 raft_consensus.cc:729] T 0433ba47dea84e2eb869df726191f722 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:20.177492 14291 consensus_queue.cc:260] T 0433ba47dea84e2eb869df726191f722 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:20.177556 14302 ts_tablet_manager.cc:1398] T 0433ba47dea84e2eb869df726191f722 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:20.177742 14291 ts_tablet_manager.cc:1398] T 0433ba47dea84e2eb869df726191f722 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:20.178704 14292 tablet_bootstrap.cc:492] T ae19b035d4594a29a770205dad7ac6ba P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:20.178807 14292 ts_tablet_manager.cc:1367] T ae19b035d4594a29a770205dad7ac6ba P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.001s
I20240503 02:00:20.179302 14292 raft_consensus.cc:348] T ae19b035d4594a29a770205dad7ac6ba P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:20.179416 14292 raft_consensus.cc:374] T ae19b035d4594a29a770205dad7ac6ba P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:20.179446 14292 raft_consensus.cc:729] T ae19b035d4594a29a770205dad7ac6ba P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:20.179564 14292 consensus_queue.cc:260] T ae19b035d4594a29a770205dad7ac6ba P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:20.179776 14309 tablet_bootstrap.cc:492] T db02d12f83a841f09588e7d6ce0a8398 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:20.179865 14309 ts_tablet_manager.cc:1367] T db02d12f83a841f09588e7d6ce0a8398 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20240503 02:00:20.179876 14292 ts_tablet_manager.cc:1398] T ae19b035d4594a29a770205dad7ac6ba P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:20.182152 14309 raft_consensus.cc:348] T db02d12f83a841f09588e7d6ce0a8398 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:20.182416 14309 raft_consensus.cc:374] T db02d12f83a841f09588e7d6ce0a8398 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:20.182510 14309 raft_consensus.cc:729] T db02d12f83a841f09588e7d6ce0a8398 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:20.182829 14309 consensus_queue.cc:260] T db02d12f83a841f09588e7d6ce0a8398 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:20.183231 14309 ts_tablet_manager.cc:1398] T db02d12f83a841f09588e7d6ce0a8398 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:20.184587 14309 tablet_bootstrap.cc:492] T ae19b035d4594a29a770205dad7ac6ba P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:20.185834 14309 tablet_bootstrap.cc:654] T ae19b035d4594a29a770205dad7ac6ba P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:20.187248 14309 tablet_bootstrap.cc:492] T ae19b035d4594a29a770205dad7ac6ba P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:20.187355 14309 ts_tablet_manager.cc:1367] T ae19b035d4594a29a770205dad7ac6ba P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:20.187734 14309 raft_consensus.cc:348] T ae19b035d4594a29a770205dad7ac6ba P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:20.187819 14309 raft_consensus.cc:374] T ae19b035d4594a29a770205dad7ac6ba P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:20.187846 14309 raft_consensus.cc:729] T ae19b035d4594a29a770205dad7ac6ba P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:20.187951 14309 consensus_queue.cc:260] T ae19b035d4594a29a770205dad7ac6ba P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:20.188200 14309 ts_tablet_manager.cc:1398] T ae19b035d4594a29a770205dad7ac6ba P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:20.193431 14106 raft_consensus.cc:482] T ae19b035d4594a29a770205dad7ac6ba P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:20.193557 14106 raft_consensus.cc:504] T ae19b035d4594a29a770205dad7ac6ba P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:20.194015 14106 leader_election.cc:288] T ae19b035d4594a29a770205dad7ac6ba P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 02:00:20.194375 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae19b035d4594a29a770205dad7ac6ba" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 02:00:20.194453 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae19b035d4594a29a770205dad7ac6ba" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 02:00:20.194535 6589 raft_consensus.cc:2454] T ae19b035d4594a29a770205dad7ac6ba P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 02:00:20.194592 6458 raft_consensus.cc:2454] T ae19b035d4594a29a770205dad7ac6ba P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 02:00:20.194856 6786 leader_election.cc:302] T ae19b035d4594a29a770205dad7ac6ba P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 80fd18ef4d604c86b909d5c008a46b4f; no voters:
I20240503 02:00:20.195081 14106 raft_consensus.cc:2781] T ae19b035d4594a29a770205dad7ac6ba P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:20.195128 14106 raft_consensus.cc:482] T ae19b035d4594a29a770205dad7ac6ba P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:20.195166 14106 raft_consensus.cc:3037] T ae19b035d4594a29a770205dad7ac6ba P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:20.196022 14106 raft_consensus.cc:504] T ae19b035d4594a29a770205dad7ac6ba P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:20.196411 14106 leader_election.cc:288] T ae19b035d4594a29a770205dad7ac6ba P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Requested vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 02:00:20.196740 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae19b035d4594a29a770205dad7ac6ba" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 02:00:20.196858 6458 raft_consensus.cc:3037] T ae19b035d4594a29a770205dad7ac6ba P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:20.197630 6458 raft_consensus.cc:2454] T ae19b035d4594a29a770205dad7ac6ba P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 02:00:20.197909 6784 leader_election.cc:302] T ae19b035d4594a29a770205dad7ac6ba P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:20.198089 14106 raft_consensus.cc:2781] T ae19b035d4594a29a770205dad7ac6ba P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:20.198170 14106 raft_consensus.cc:686] T ae19b035d4594a29a770205dad7ac6ba P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Becoming Leader. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Running, Role: LEADER
I20240503 02:00:20.198309 14106 consensus_queue.cc:237] T ae19b035d4594a29a770205dad7ac6ba P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:20.204865 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae19b035d4594a29a770205dad7ac6ba" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 02:00:20.205024 6589 raft_consensus.cc:3037] T ae19b035d4594a29a770205dad7ac6ba P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:20.206087 6589 raft_consensus.cc:2454] T ae19b035d4594a29a770205dad7ac6ba P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 02:00:20.206938 6309 catalog_manager.cc:5484] T ae19b035d4594a29a770205dad7ac6ba P 492448570cc94ed1879feb2ec3b9051d reported cstate change: term changed from 0 to 1, leader changed from <none> to 492448570cc94ed1879feb2ec3b9051d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:20.244220 14056 raft_consensus.cc:482] T db02d12f83a841f09588e7d6ce0a8398 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:20.244417 14056 raft_consensus.cc:504] T db02d12f83a841f09588e7d6ce0a8398 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:20.244928 14056 leader_election.cc:288] T db02d12f83a841f09588e7d6ce0a8398 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:20.245410 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db02d12f83a841f09588e7d6ce0a8398" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 02:00:20.245429 9047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db02d12f83a841f09588e7d6ce0a8398" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:20.245559 9047 raft_consensus.cc:2454] T db02d12f83a841f09588e7d6ce0a8398 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:20.245559 6589 raft_consensus.cc:2454] T db02d12f83a841f09588e7d6ce0a8398 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:20.245877 6393 leader_election.cc:302] T db02d12f83a841f09588e7d6ce0a8398 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:20.246059 14056 raft_consensus.cc:2781] T db02d12f83a841f09588e7d6ce0a8398 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:20.246114 14056 raft_consensus.cc:482] T db02d12f83a841f09588e7d6ce0a8398 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:20.246157 14056 raft_consensus.cc:3037] T db02d12f83a841f09588e7d6ce0a8398 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:20.247100 14056 raft_consensus.cc:504] T db02d12f83a841f09588e7d6ce0a8398 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:20.247437 14056 leader_election.cc:288] T db02d12f83a841f09588e7d6ce0a8398 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Requested vote from peers 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:20.247756 9047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db02d12f83a841f09588e7d6ce0a8398" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:20.247782 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db02d12f83a841f09588e7d6ce0a8398" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 02:00:20.247908 6589 raft_consensus.cc:3037] T db02d12f83a841f09588e7d6ce0a8398 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:20.247926 9047 raft_consensus.cc:3037] T db02d12f83a841f09588e7d6ce0a8398 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:20.249003 9047 raft_consensus.cc:2454] T db02d12f83a841f09588e7d6ce0a8398 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:20.249158 6589 raft_consensus.cc:2454] T db02d12f83a841f09588e7d6ce0a8398 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:20.249328 6392 leader_election.cc:302] T db02d12f83a841f09588e7d6ce0a8398 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99ead02915954b4fab23b033ad6d546b, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:20.249506 14056 raft_consensus.cc:2781] T db02d12f83a841f09588e7d6ce0a8398 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:20.249608 14056 raft_consensus.cc:686] T db02d12f83a841f09588e7d6ce0a8398 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Becoming Leader. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Running, Role: LEADER
I20240503 02:00:20.249748 14056 consensus_queue.cc:237] T db02d12f83a841f09588e7d6ce0a8398 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:20.252213 6314 catalog_manager.cc:5484] T db02d12f83a841f09588e7d6ce0a8398 P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: term changed from 0 to 1, leader changed from <none> to dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129). New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:20.275350 6314 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:20.279667 6589 raft_consensus.cc:1261] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:20.279721 9178 raft_consensus.cc:1261] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:20.280102 14276 consensus_queue.cc:1035] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:20.280232 14106 consensus_queue.cc:1035] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:20.283452 6314 catalog_manager.cc:5576] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863) does not have the latest schema for tablet 2ed12d2ca4c74194a99bd6e4f9c28834 (table rebalancer_dynamic_table [id=78ad5e8d3227453fb447fb66f8d78a05]). Expected version 1 got 0
I20240503 02:00:20.284984 6309 catalog_manager.cc:5576] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719) does not have the latest schema for tablet 2ed12d2ca4c74194a99bd6e4f9c28834 (table rebalancer_dynamic_table [id=78ad5e8d3227453fb447fb66f8d78a05]). Expected version 1 got 0
I20240503 02:00:20.285471 14318 tablet.cc:1722] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:20.285526 6314 catalog_manager.cc:5576] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079) does not have the latest schema for tablet 2ed12d2ca4c74194a99bd6e4f9c28834 (table rebalancer_dynamic_table [id=78ad5e8d3227453fb447fb66f8d78a05]). Expected version 1 got 0
I20240503 02:00:20.285840 14318 tablet.cc:1620] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d: MemRowSet was empty: no flush needed.
I20240503 02:00:20.284355 14319 tablet.cc:1722] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:20.286468 14319 tablet.cc:1620] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 80fd18ef4d604c86b909d5c008a46b4f: MemRowSet was empty: no flush needed.
I20240503 02:00:20.286868 6314 catalog_manager.cc:5576] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863) does not have the latest schema for tablet 2ed12d2ca4c74194a99bd6e4f9c28834 (table rebalancer_dynamic_table [id=78ad5e8d3227453fb447fb66f8d78a05]). Expected version 1 got 0
I20240503 02:00:20.287791 14320 tablet.cc:1722] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:20.288008 6309 catalog_manager.cc:5576] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079) does not have the latest schema for tablet 2ed12d2ca4c74194a99bd6e4f9c28834 (table rebalancer_dynamic_table [id=78ad5e8d3227453fb447fb66f8d78a05]). Expected version 1 got 0
I20240503 02:00:20.288134 14320 tablet.cc:1620] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
I20240503 02:00:20.291407 6308 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=78ad5e8d3227453fb447fb66f8d78a05] alter complete (version 1)
W20240503 02:00:20.354885 8982 consensus_peers.cc:485] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:00:20.362969 14284 raft_consensus.cc:482] T 0433ba47dea84e2eb869df726191f722 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:20.363109 14284 raft_consensus.cc:504] T 0433ba47dea84e2eb869df726191f722 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:20.363505 14284 leader_election.cc:288] T 0433ba47dea84e2eb869df726191f722 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:20.363809 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0433ba47dea84e2eb869df726191f722" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:20.363869 8917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0433ba47dea84e2eb869df726191f722" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:20.363952 9178 raft_consensus.cc:2454] T 0433ba47dea84e2eb869df726191f722 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:20.364015 8917 raft_consensus.cc:2454] T 0433ba47dea84e2eb869df726191f722 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:20.364257 6653 leader_election.cc:302] T 0433ba47dea84e2eb869df726191f722 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:20.364463 14284 raft_consensus.cc:2781] T 0433ba47dea84e2eb869df726191f722 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:20.364523 14284 raft_consensus.cc:482] T 0433ba47dea84e2eb869df726191f722 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:20.364552 14284 raft_consensus.cc:3037] T 0433ba47dea84e2eb869df726191f722 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:20.365538 14284 raft_consensus.cc:504] T 0433ba47dea84e2eb869df726191f722 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:20.365907 14284 leader_election.cc:288] T 0433ba47dea84e2eb869df726191f722 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Requested vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:20.366267 8917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0433ba47dea84e2eb869df726191f722" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:20.366405 8917 raft_consensus.cc:3037] T 0433ba47dea84e2eb869df726191f722 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:20.366380 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0433ba47dea84e2eb869df726191f722" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:20.366508 9178 raft_consensus.cc:3037] T 0433ba47dea84e2eb869df726191f722 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:20.367367 8917 raft_consensus.cc:2454] T 0433ba47dea84e2eb869df726191f722 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:20.367458 9178 raft_consensus.cc:2454] T 0433ba47dea84e2eb869df726191f722 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:20.367666 6656 leader_election.cc:302] T 0433ba47dea84e2eb869df726191f722 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9942306855404cfd891adf227dd65eb2, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:20.367832 14284 raft_consensus.cc:2781] T 0433ba47dea84e2eb869df726191f722 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:20.367925 14284 raft_consensus.cc:686] T 0433ba47dea84e2eb869df726191f722 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Becoming Leader. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Running, Role: LEADER
I20240503 02:00:20.368064 14284 consensus_queue.cc:237] T 0433ba47dea84e2eb869df726191f722 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:20.370608 6309 catalog_manager.cc:5484] T 0433ba47dea84e2eb869df726191f722 P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:20.403846 6309 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:20.413297 14318 tablet.cc:1722] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:20.413650 14318 tablet.cc:1620] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d: MemRowSet was empty: no flush needed.
I20240503 02:00:20.413921 14320 tablet.cc:1722] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:20.413657 14319 tablet.cc:1722] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:20.416023 14319 tablet.cc:1620] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 80fd18ef4d604c86b909d5c008a46b4f: MemRowSet was empty: no flush needed.
W20240503 02:00:20.416414 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:20.416642 14320 tablet.cc:1620] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
I20240503 02:00:20.419878 6309 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=78ad5e8d3227453fb447fb66f8d78a05] alter complete (version 2)
W20240503 02:00:20.454217 6785 consensus_peers.cc:485] T c3485701bf3b48f5ae2e1568afeed133 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:00:20.522709 6308 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:20.528911 14318 tablet.cc:1722] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:20.529829 14320 tablet.cc:1722] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:20.531289 14319 tablet.cc:1722] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:20.536793 6308 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=78ad5e8d3227453fb447fb66f8d78a05] alter complete (version 3)
I20240503 02:00:20.589056 14322 consensus_queue.cc:1035] T ae19b035d4594a29a770205dad7ac6ba P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:20.592350 14324 consensus_queue.cc:1035] T ae19b035d4594a29a770205dad7ac6ba P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:20.641884 6308 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:20.642060 6308 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:20.644143 6308 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet 2ed12d2ca4c74194a99bd6e4f9c28834
I20240503 02:00:20.644927 6831 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ed12d2ca4c74194a99bd6e4f9c28834 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:20 UTC) from {username='slave'} at 127.0.0.1:52812
I20240503 02:00:20.645022 6567 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ed12d2ca4c74194a99bd6e4f9c28834 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:20 UTC) from {username='slave'} at 127.0.0.1:59846
I20240503 02:00:20.645356 14328 tablet_replica.cc:330] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d: stopping tablet replica
I20240503 02:00:20.645378 14329 tablet_replica.cc:330] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 02:00:20.645524 14329 raft_consensus.cc:2229] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:20.645531 14328 raft_consensus.cc:2229] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:20.645640 9157 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ed12d2ca4c74194a99bd6e4f9c28834 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:20 UTC) from {username='slave'} at 127.0.0.1:51940
I20240503 02:00:20.645825 14329 raft_consensus.cc:2258] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:20.646144 14330 tablet_replica.cc:330] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 2a01ace80baf4057b6bf1995cf88468b: stopping tablet replica
I20240503 02:00:20.646320 14330 raft_consensus.cc:2229] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:20.646334 14328 raft_consensus.cc:2258] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:20.646472 14330 raft_consensus.cc:2258] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:20.646612 14329 ts_tablet_manager.cc:1875] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:20.646916 14328 ts_tablet_manager.cc:1875] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:20.647152 14330 ts_tablet_manager.cc:1875] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 2a01ace80baf4057b6bf1995cf88468b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:20.649324 14329 ts_tablet_manager.cc:1888] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:20.649423 14328 ts_tablet_manager.cc:1888] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:20.649451 14329 log.cc:1196] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/2ed12d2ca4c74194a99bd6e4f9c28834
I20240503 02:00:20.649463 14330 ts_tablet_manager.cc:1888] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 2a01ace80baf4057b6bf1995cf88468b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:20.649519 14330 log.cc:1196] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 2a01ace80baf4057b6bf1995cf88468b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/2ed12d2ca4c74194a99bd6e4f9c28834
I20240503 02:00:20.649539 14328 log.cc:1196] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/2ed12d2ca4c74194a99bd6e4f9c28834
I20240503 02:00:20.649780 14329 ts_tablet_manager.cc:1909] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting consensus metadata
I20240503 02:00:20.649780 14330 ts_tablet_manager.cc:1909] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 2a01ace80baf4057b6bf1995cf88468b: Deleting consensus metadata
I20240503 02:00:20.649894 14328 ts_tablet_manager.cc:1909] T 2ed12d2ca4c74194a99bd6e4f9c28834 P 492448570cc94ed1879feb2ec3b9051d: Deleting consensus metadata
I20240503 02:00:20.650579 6303 catalog_manager.cc:4830] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719): tablet 2ed12d2ca4c74194a99bd6e4f9c28834 (table rebalancer_dynamic_table_new_name [id=78ad5e8d3227453fb447fb66f8d78a05]) successfully deleted
I20240503 02:00:20.650792 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet 2ed12d2ca4c74194a99bd6e4f9c28834 (table rebalancer_dynamic_table_new_name [id=78ad5e8d3227453fb447fb66f8d78a05]) successfully deleted
I20240503 02:00:20.651173 6300 catalog_manager.cc:4830] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079): tablet 2ed12d2ca4c74194a99bd6e4f9c28834 (table rebalancer_dynamic_table_new_name [id=78ad5e8d3227453fb447fb66f8d78a05]) successfully deleted
I20240503 02:00:20.677201 14153 consensus_queue.cc:1035] T db02d12f83a841f09588e7d6ce0a8398 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:20.680784 14153 consensus_queue.cc:1035] T db02d12f83a841f09588e7d6ce0a8398 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:20.746182 6308 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:00:20.752024 6436 tablet_service.cc:1467] Processing CreateTablet for tablet 451cd82331904b84884f7fdaf2b7dec9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=364d2beeb13a4e77a5977e4c7d9786dc]), partition=
I20240503 02:00:20.752131 6567 tablet_service.cc:1467] Processing CreateTablet for tablet 451cd82331904b84884f7fdaf2b7dec9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=364d2beeb13a4e77a5977e4c7d9786dc]), partition=
I20240503 02:00:20.752301 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 451cd82331904b84884f7fdaf2b7dec9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:20.752422 6567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 451cd82331904b84884f7fdaf2b7dec9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:20.752357 8880 tablet_service.cc:1467] Processing CreateTablet for tablet 451cd82331904b84884f7fdaf2b7dec9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=364d2beeb13a4e77a5977e4c7d9786dc]), partition=
I20240503 02:00:20.752605 8880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 451cd82331904b84884f7fdaf2b7dec9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:20.754515 14334 tablet_bootstrap.cc:492] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:20.755326 14336 tablet_bootstrap.cc:492] T 451cd82331904b84884f7fdaf2b7dec9 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:20.755573 14335 tablet_bootstrap.cc:492] T 451cd82331904b84884f7fdaf2b7dec9 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:20.756290 14336 tablet_bootstrap.cc:654] T 451cd82331904b84884f7fdaf2b7dec9 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:20.756292 14334 tablet_bootstrap.cc:654] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:20.756479 14335 tablet_bootstrap.cc:654] T 451cd82331904b84884f7fdaf2b7dec9 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:20.757726 14336 tablet_bootstrap.cc:492] T 451cd82331904b84884f7fdaf2b7dec9 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:20.757827 14336 ts_tablet_manager.cc:1367] T 451cd82331904b84884f7fdaf2b7dec9 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:20.758085 14334 tablet_bootstrap.cc:492] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:20.758185 14334 ts_tablet_manager.cc:1367] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:20.758260 14336 raft_consensus.cc:348] T 451cd82331904b84884f7fdaf2b7dec9 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:20.758373 14336 raft_consensus.cc:374] T 451cd82331904b84884f7fdaf2b7dec9 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:20.758412 14336 raft_consensus.cc:729] T 451cd82331904b84884f7fdaf2b7dec9 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:20.758546 14336 consensus_queue.cc:260] T 451cd82331904b84884f7fdaf2b7dec9 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:20.758836 14334 raft_consensus.cc:348] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:20.758864 14336 ts_tablet_manager.cc:1398] T 451cd82331904b84884f7fdaf2b7dec9 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:20.758965 14334 raft_consensus.cc:374] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:20.758998 14334 raft_consensus.cc:729] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:20.759188 14334 consensus_queue.cc:260] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:20.759436 14335 tablet_bootstrap.cc:492] T 451cd82331904b84884f7fdaf2b7dec9 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:20.759538 14335 ts_tablet_manager.cc:1367] T 451cd82331904b84884f7fdaf2b7dec9 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240503 02:00:20.760027 14335 raft_consensus.cc:348] T 451cd82331904b84884f7fdaf2b7dec9 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:20.760141 14335 raft_consensus.cc:374] T 451cd82331904b84884f7fdaf2b7dec9 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:20.760176 14335 raft_consensus.cc:729] T 451cd82331904b84884f7fdaf2b7dec9 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:20.760300 14335 consensus_queue.cc:260] T 451cd82331904b84884f7fdaf2b7dec9 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:20.760603 14335 ts_tablet_manager.cc:1398] T 451cd82331904b84884f7fdaf2b7dec9 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:20.763206 14056 raft_consensus.cc:482] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:20.763317 14056 raft_consensus.cc:504] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:20.763695 14056 leader_election.cc:288] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:20.764240 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "451cd82331904b84884f7fdaf2b7dec9" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 02:00:20.764310 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "451cd82331904b84884f7fdaf2b7dec9" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:20.764370 6589 raft_consensus.cc:2454] T 451cd82331904b84884f7fdaf2b7dec9 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:20.764467 8916 raft_consensus.cc:2454] T 451cd82331904b84884f7fdaf2b7dec9 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
W20240503 02:00:20.764719 6785 consensus_peers.cc:485] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:00:20.765015 6393 leader_election.cc:302] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:20.765995 14153 raft_consensus.cc:2781] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:20.766057 14153 raft_consensus.cc:482] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:20.766094 14153 raft_consensus.cc:3037] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:20.767045 14153 raft_consensus.cc:504] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:20.767371 14153 leader_election.cc:288] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Requested vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:20.767765 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "451cd82331904b84884f7fdaf2b7dec9" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:20.767915 8916 raft_consensus.cc:3037] T 451cd82331904b84884f7fdaf2b7dec9 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:20.767875 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "451cd82331904b84884f7fdaf2b7dec9" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 02:00:20.767987 6589 raft_consensus.cc:3037] T 451cd82331904b84884f7fdaf2b7dec9 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:20.768838 6589 raft_consensus.cc:2454] T 451cd82331904b84884f7fdaf2b7dec9 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:20.768931 8916 raft_consensus.cc:2454] T 451cd82331904b84884f7fdaf2b7dec9 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:20.769176 6393 leader_election.cc:302] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:20.769361 14153 raft_consensus.cc:2781] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:20.769440 14153 raft_consensus.cc:686] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Becoming Leader. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Running, Role: LEADER
I20240503 02:00:20.769552 14153 consensus_queue.cc:237] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:20.777469 14334 ts_tablet_manager.cc:1398] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.019s user 0.001s sys 0.000s
I20240503 02:00:20.778122 6314 catalog_manager.cc:5484] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: term changed from 0 to 1, leader changed from <none> to dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129). New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:20.788797 14190 consensus_queue.cc:1035] T 0433ba47dea84e2eb869df726191f722 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:20.794026 14190 consensus_queue.cc:1035] T 0433ba47dea84e2eb869df726191f722 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:00:20.846863 6654 consensus_peers.cc:485] T ba17d4d3e0754b2798eb95484d37229a P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:00:20.889060 6314 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:20.893214 8916 raft_consensus.cc:1261] T 451cd82331904b84884f7fdaf2b7dec9 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:20.893754 6589 raft_consensus.cc:1261] T 451cd82331904b84884f7fdaf2b7dec9 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:20.893963 14153 consensus_queue.cc:1035] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:20.894388 14056 consensus_queue.cc:1035] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:20.894362 6314 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_extra_table_54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:20.897919 6308 catalog_manager.cc:5576] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863) does not have the latest schema for tablet 451cd82331904b84884f7fdaf2b7dec9 (table rebalancer_dynamic_table [id=364d2beeb13a4e77a5977e4c7d9786dc]). Expected version 1 got 0
I20240503 02:00:20.898679 6309 catalog_manager.cc:5576] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211) does not have the latest schema for tablet 451cd82331904b84884f7fdaf2b7dec9 (table rebalancer_dynamic_table [id=364d2beeb13a4e77a5977e4c7d9786dc]). Expected version 1 got 0
I20240503 02:00:20.899163 14319 tablet.cc:1722] T 451cd82331904b84884f7fdaf2b7dec9 P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:20.899524 14319 tablet.cc:1620] T 451cd82331904b84884f7fdaf2b7dec9 P 80fd18ef4d604c86b909d5c008a46b4f: MemRowSet was empty: no flush needed.
I20240503 02:00:20.900132 14348 tablet.cc:1722] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:20.900606 14348 tablet.cc:1620] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a: MemRowSet was empty: no flush needed.
I20240503 02:00:20.903079 6308 catalog_manager.cc:5576] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093) does not have the latest schema for tablet 451cd82331904b84884f7fdaf2b7dec9 (table rebalancer_dynamic_table [id=364d2beeb13a4e77a5977e4c7d9786dc]). Expected version 1 got 0
I20240503 02:00:20.903825 9027 tablet_service.cc:1467] Processing CreateTablet for tablet 3af89caa97634222ab834577a9217ce9 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=0293809e35df48c6bbb4a371a24b2499]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:20.903846 9157 tablet_service.cc:1467] Processing CreateTablet for tablet ca907fb5b04f4d2492e7f4cea7114c95 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=0293809e35df48c6bbb4a371a24b2499]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:20.903836 9028 tablet_service.cc:1467] Processing CreateTablet for tablet 87c4b630a58749e6be9f26cceda27573 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=0293809e35df48c6bbb4a371a24b2499]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:20.904131 9027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3af89caa97634222ab834577a9217ce9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:20.904146 9157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca907fb5b04f4d2492e7f4cea7114c95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:20.904714 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87c4b630a58749e6be9f26cceda27573. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:20.906142 14350 tablet_bootstrap.cc:492] T 3af89caa97634222ab834577a9217ce9 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:20.906368 9158 tablet_service.cc:1467] Processing CreateTablet for tablet 87c4b630a58749e6be9f26cceda27573 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=0293809e35df48c6bbb4a371a24b2499]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:20.906608 9158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87c4b630a58749e6be9f26cceda27573. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:20.907003 14351 tablet_bootstrap.cc:492] T ca907fb5b04f4d2492e7f4cea7114c95 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:20.907191 14350 tablet_bootstrap.cc:654] T 3af89caa97634222ab834577a9217ce9 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:20.907447 6309 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=364d2beeb13a4e77a5977e4c7d9786dc] alter complete (version 1)
I20240503 02:00:20.908399 14351 tablet_bootstrap.cc:654] T ca907fb5b04f4d2492e7f4cea7114c95 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:20.910374 6700 tablet_service.cc:1467] Processing CreateTablet for tablet ca907fb5b04f4d2492e7f4cea7114c95 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=0293809e35df48c6bbb4a371a24b2499]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:20.910674 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca907fb5b04f4d2492e7f4cea7114c95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:20.911375 6309 catalog_manager.cc:5576] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211) does not have the latest schema for tablet 451cd82331904b84884f7fdaf2b7dec9 (table rebalancer_dynamic_table [id=364d2beeb13a4e77a5977e4c7d9786dc]). Expected version 1 got 0
I20240503 02:00:20.912678 8880 tablet_service.cc:1467] Processing CreateTablet for tablet ca907fb5b04f4d2492e7f4cea7114c95 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=0293809e35df48c6bbb4a371a24b2499]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:20.912967 8880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca907fb5b04f4d2492e7f4cea7114c95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:20.914105 14343 mvcc.cc:205] Tried to move back new op lower bound from 7023417839174885376 to 7023417838673567744. Current Snapshot: MvccSnapshot[applied={T|T < 7023417839174885376}]
I20240503 02:00:20.914330 14351 tablet_bootstrap.cc:492] T ca907fb5b04f4d2492e7f4cea7114c95 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:20.914438 14351 ts_tablet_manager.cc:1367] T ca907fb5b04f4d2492e7f4cea7114c95 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 02:00:20.914495 14354 tablet.cc:1722] T 451cd82331904b84884f7fdaf2b7dec9 P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:20.914783 14354 tablet.cc:1620] T 451cd82331904b84884f7fdaf2b7dec9 P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
I20240503 02:00:20.914919 14351 raft_consensus.cc:348] T ca907fb5b04f4d2492e7f4cea7114c95 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:20.915014 14336 tablet_bootstrap.cc:492] T ca907fb5b04f4d2492e7f4cea7114c95 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:20.915035 14351 raft_consensus.cc:374] T ca907fb5b04f4d2492e7f4cea7114c95 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:20.915066 14351 raft_consensus.cc:729] T ca907fb5b04f4d2492e7f4cea7114c95 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:20.915185 14351 consensus_queue.cc:260] T ca907fb5b04f4d2492e7f4cea7114c95 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:20.915232 8887 tablet_service.cc:1467] Processing CreateTablet for tablet 3af89caa97634222ab834577a9217ce9 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=0293809e35df48c6bbb4a371a24b2499]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:20.915465 8887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3af89caa97634222ab834577a9217ce9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:20.915489 14351 ts_tablet_manager.cc:1398] T ca907fb5b04f4d2492e7f4cea7114c95 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:20.915761 14351 tablet_bootstrap.cc:492] T 87c4b630a58749e6be9f26cceda27573 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:20.915987 14336 tablet_bootstrap.cc:654] T ca907fb5b04f4d2492e7f4cea7114c95 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:20.916671 14351 tablet_bootstrap.cc:654] T 87c4b630a58749e6be9f26cceda27573 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:20.917021 14350 tablet_bootstrap.cc:492] T 3af89caa97634222ab834577a9217ce9 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:20.917117 14350 ts_tablet_manager.cc:1367] T 3af89caa97634222ab834577a9217ce9 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 02:00:20.917589 14350 raft_consensus.cc:348] T 3af89caa97634222ab834577a9217ce9 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:20.917672 14351 tablet_bootstrap.cc:492] T 87c4b630a58749e6be9f26cceda27573 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:20.917699 14350 raft_consensus.cc:374] T 3af89caa97634222ab834577a9217ce9 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:20.917733 14350 raft_consensus.cc:729] T 3af89caa97634222ab834577a9217ce9 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:20.917745 14351 ts_tablet_manager.cc:1367] T 87c4b630a58749e6be9f26cceda27573 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:20.917865 14350 consensus_queue.cc:260] T 3af89caa97634222ab834577a9217ce9 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:20.917981 6831 tablet_service.cc:1467] Processing CreateTablet for tablet 3af89caa97634222ab834577a9217ce9 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=0293809e35df48c6bbb4a371a24b2499]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:20.918180 14350 ts_tablet_manager.cc:1398] T 3af89caa97634222ab834577a9217ce9 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:20.918283 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3af89caa97634222ab834577a9217ce9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:20.918345 14350 tablet_bootstrap.cc:492] T 87c4b630a58749e6be9f26cceda27573 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:20.918309 14351 raft_consensus.cc:348] T 87c4b630a58749e6be9f26cceda27573 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:20.918406 14351 raft_consensus.cc:374] T 87c4b630a58749e6be9f26cceda27573 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:20.918433 14351 raft_consensus.cc:729] T 87c4b630a58749e6be9f26cceda27573 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:20.918545 14351 consensus_queue.cc:260] T 87c4b630a58749e6be9f26cceda27573 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:20.918918 14351 ts_tablet_manager.cc:1398] T 87c4b630a58749e6be9f26cceda27573 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:20.917984 6829 tablet_service.cc:1467] Processing CreateTablet for tablet 87c4b630a58749e6be9f26cceda27573 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=0293809e35df48c6bbb4a371a24b2499]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:20.919268 6829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87c4b630a58749e6be9f26cceda27573. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:20.919279 14350 tablet_bootstrap.cc:654] T 87c4b630a58749e6be9f26cceda27573 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:20.921962 14336 tablet_bootstrap.cc:492] T ca907fb5b04f4d2492e7f4cea7114c95 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:20.922073 14336 ts_tablet_manager.cc:1367] T ca907fb5b04f4d2492e7f4cea7114c95 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:00:20.922456 14357 tablet_bootstrap.cc:492] T 3af89caa97634222ab834577a9217ce9 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:20.922593 14336 raft_consensus.cc:348] T ca907fb5b04f4d2492e7f4cea7114c95 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:20.922708 14336 raft_consensus.cc:374] T ca907fb5b04f4d2492e7f4cea7114c95 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:20.922735 14336 raft_consensus.cc:729] T ca907fb5b04f4d2492e7f4cea7114c95 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:20.922868 14336 consensus_queue.cc:260] T ca907fb5b04f4d2492e7f4cea7114c95 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:20.923171 14336 ts_tablet_manager.cc:1398] T ca907fb5b04f4d2492e7f4cea7114c95 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:20.923609 14357 tablet_bootstrap.cc:654] T 3af89caa97634222ab834577a9217ce9 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:20.925228 14357 tablet_bootstrap.cc:492] T 3af89caa97634222ab834577a9217ce9 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:20.925323 14357 ts_tablet_manager.cc:1367] T 3af89caa97634222ab834577a9217ce9 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:20.925772 14357 raft_consensus.cc:348] T 3af89caa97634222ab834577a9217ce9 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:20.925884 14357 raft_consensus.cc:374] T 3af89caa97634222ab834577a9217ce9 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:20.925912 14357 raft_consensus.cc:729] T 3af89caa97634222ab834577a9217ce9 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:20.926028 14357 consensus_queue.cc:260] T 3af89caa97634222ab834577a9217ce9 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:20.923350 14336 tablet_bootstrap.cc:492] T 3af89caa97634222ab834577a9217ce9 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:20.927053 14357 ts_tablet_manager.cc:1398] T 3af89caa97634222ab834577a9217ce9 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:00:20.927229 14357 tablet_bootstrap.cc:492] T 87c4b630a58749e6be9f26cceda27573 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:20.928198 14357 tablet_bootstrap.cc:654] T 87c4b630a58749e6be9f26cceda27573 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:20.928512 14336 tablet_bootstrap.cc:654] T 3af89caa97634222ab834577a9217ce9 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:20.929657 14359 tablet_bootstrap.cc:492] T ca907fb5b04f4d2492e7f4cea7114c95 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:20.930253 14350 tablet_bootstrap.cc:492] T 87c4b630a58749e6be9f26cceda27573 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:20.930354 14350 ts_tablet_manager.cc:1367] T 87c4b630a58749e6be9f26cceda27573 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20240503 02:00:20.930482 14336 tablet_bootstrap.cc:492] T 3af89caa97634222ab834577a9217ce9 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:20.930550 14336 ts_tablet_manager.cc:1367] T 3af89caa97634222ab834577a9217ce9 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20240503 02:00:20.931159 14350 raft_consensus.cc:348] T 87c4b630a58749e6be9f26cceda27573 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:20.931277 14350 raft_consensus.cc:374] T 87c4b630a58749e6be9f26cceda27573 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:20.931308 14350 raft_consensus.cc:729] T 87c4b630a58749e6be9f26cceda27573 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:20.931425 14350 consensus_queue.cc:260] T 87c4b630a58749e6be9f26cceda27573 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:20.931727 14350 ts_tablet_manager.cc:1398] T 87c4b630a58749e6be9f26cceda27573 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:20.932001 14357 tablet_bootstrap.cc:492] T 87c4b630a58749e6be9f26cceda27573 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:20.932338 14357 ts_tablet_manager.cc:1367] T 87c4b630a58749e6be9f26cceda27573 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240503 02:00:20.932875 14357 raft_consensus.cc:348] T 87c4b630a58749e6be9f26cceda27573 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:20.933063 14357 raft_consensus.cc:374] T 87c4b630a58749e6be9f26cceda27573 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:20.933115 14357 raft_consensus.cc:729] T 87c4b630a58749e6be9f26cceda27573 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:20.933211 14336 raft_consensus.cc:348] T 3af89caa97634222ab834577a9217ce9 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:20.933266 14357 consensus_queue.cc:260] T 87c4b630a58749e6be9f26cceda27573 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:20.933342 14336 raft_consensus.cc:374] T 3af89caa97634222ab834577a9217ce9 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:20.933408 14336 raft_consensus.cc:729] T 3af89caa97634222ab834577a9217ce9 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:20.933547 14336 consensus_queue.cc:260] T 3af89caa97634222ab834577a9217ce9 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:20.933615 14357 ts_tablet_manager.cc:1398] T 87c4b630a58749e6be9f26cceda27573 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:20.933846 14336 ts_tablet_manager.cc:1398] T 3af89caa97634222ab834577a9217ce9 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 02:00:20.934881 14359 tablet_bootstrap.cc:654] T ca907fb5b04f4d2492e7f4cea7114c95 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
W20240503 02:00:20.936086 6654 consensus_peers.cc:485] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:00:20.936097 14359 tablet_bootstrap.cc:492] T ca907fb5b04f4d2492e7f4cea7114c95 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:20.936389 14359 ts_tablet_manager.cc:1367] T ca907fb5b04f4d2492e7f4cea7114c95 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:00:20.936885 14359 raft_consensus.cc:348] T ca907fb5b04f4d2492e7f4cea7114c95 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:20.936976 14359 raft_consensus.cc:374] T ca907fb5b04f4d2492e7f4cea7114c95 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:20.936995 14359 raft_consensus.cc:729] T ca907fb5b04f4d2492e7f4cea7114c95 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:20.937095 14359 consensus_queue.cc:260] T ca907fb5b04f4d2492e7f4cea7114c95 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:20.937373 14359 ts_tablet_manager.cc:1398] T ca907fb5b04f4d2492e7f4cea7114c95 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:20.972299 14220 raft_consensus.cc:482] T ca907fb5b04f4d2492e7f4cea7114c95 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:20.972476 14220 raft_consensus.cc:504] T ca907fb5b04f4d2492e7f4cea7114c95 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:20.973002 14220 leader_election.cc:288] T ca907fb5b04f4d2492e7f4cea7114c95 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:20.973428 6719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca907fb5b04f4d2492e7f4cea7114c95" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 02:00:20.973589 6719 raft_consensus.cc:2454] T ca907fb5b04f4d2492e7f4cea7114c95 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 0.
I20240503 02:00:20.973587 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca907fb5b04f4d2492e7f4cea7114c95" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:20.973703 9179 raft_consensus.cc:2454] T ca907fb5b04f4d2492e7f4cea7114c95 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 0.
I20240503 02:00:20.973958 8853 leader_election.cc:302] T ca907fb5b04f4d2492e7f4cea7114c95 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9942306855404cfd891adf227dd65eb2, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:20.974124 14220 raft_consensus.cc:2781] T ca907fb5b04f4d2492e7f4cea7114c95 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:20.974182 14220 raft_consensus.cc:482] T ca907fb5b04f4d2492e7f4cea7114c95 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:20.974215 14220 raft_consensus.cc:3037] T ca907fb5b04f4d2492e7f4cea7114c95 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:20.975301 14220 raft_consensus.cc:504] T ca907fb5b04f4d2492e7f4cea7114c95 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:20.975672 14220 leader_election.cc:288] T ca907fb5b04f4d2492e7f4cea7114c95 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 election: Requested vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:20.976017 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca907fb5b04f4d2492e7f4cea7114c95" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:20.976140 9179 raft_consensus.cc:3037] T ca907fb5b04f4d2492e7f4cea7114c95 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:20.976045 6719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca907fb5b04f4d2492e7f4cea7114c95" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 02:00:20.976254 6719 raft_consensus.cc:3037] T ca907fb5b04f4d2492e7f4cea7114c95 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:20.977144 9179 raft_consensus.cc:2454] T ca907fb5b04f4d2492e7f4cea7114c95 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 1.
I20240503 02:00:20.977315 6719 raft_consensus.cc:2454] T ca907fb5b04f4d2492e7f4cea7114c95 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 1.
I20240503 02:00:20.977479 8850 leader_election.cc:302] T ca907fb5b04f4d2492e7f4cea7114c95 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:20.977644 14220 raft_consensus.cc:2781] T ca907fb5b04f4d2492e7f4cea7114c95 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:20.977861 14220 raft_consensus.cc:686] T ca907fb5b04f4d2492e7f4cea7114c95 P 9942306855404cfd891adf227dd65eb2 [term 1 LEADER]: Becoming Leader. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Running, Role: LEADER
I20240503 02:00:20.978013 14220 consensus_queue.cc:237] T ca907fb5b04f4d2492e7f4cea7114c95 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:20.980813 6314 catalog_manager.cc:5484] T ca907fb5b04f4d2492e7f4cea7114c95 P 9942306855404cfd891adf227dd65eb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9942306855404cfd891adf227dd65eb2 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "9942306855404cfd891adf227dd65eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: HEALTHY } } }
W20240503 02:00:20.987742 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:21.011898 13968 raft_consensus.cc:482] T 87c4b630a58749e6be9f26cceda27573 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:21.012059 13968 raft_consensus.cc:504] T 87c4b630a58749e6be9f26cceda27573 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:21.012598 13968 leader_election.cc:288] T 87c4b630a58749e6be9f26cceda27573 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:21.012908 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87c4b630a58749e6be9f26cceda27573" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 02:00:21.013052 6851 raft_consensus.cc:2454] T 87c4b630a58749e6be9f26cceda27573 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:21.013129 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87c4b630a58749e6be9f26cceda27573" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:21.013280 9048 raft_consensus.cc:2454] T 87c4b630a58749e6be9f26cceda27573 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:21.013388 9115 leader_election.cc:302] T 87c4b630a58749e6be9f26cceda27573 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 492448570cc94ed1879feb2ec3b9051d; no voters:
I20240503 02:00:21.013586 13968 raft_consensus.cc:2781] T 87c4b630a58749e6be9f26cceda27573 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:21.013643 13968 raft_consensus.cc:482] T 87c4b630a58749e6be9f26cceda27573 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:21.013674 13968 raft_consensus.cc:3037] T 87c4b630a58749e6be9f26cceda27573 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:21.014564 13968 raft_consensus.cc:504] T 87c4b630a58749e6be9f26cceda27573 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:21.014973 13968 leader_election.cc:288] T 87c4b630a58749e6be9f26cceda27573 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Requested vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:21.015368 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87c4b630a58749e6be9f26cceda27573" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 02:00:21.015372 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87c4b630a58749e6be9f26cceda27573" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:21.015511 6851 raft_consensus.cc:3037] T 87c4b630a58749e6be9f26cceda27573 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:21.015511 9048 raft_consensus.cc:3037] T 87c4b630a58749e6be9f26cceda27573 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:21.016580 6851 raft_consensus.cc:2454] T 87c4b630a58749e6be9f26cceda27573 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:21.016580 9048 raft_consensus.cc:2454] T 87c4b630a58749e6be9f26cceda27573 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:21.016903 9115 leader_election.cc:302] T 87c4b630a58749e6be9f26cceda27573 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 492448570cc94ed1879feb2ec3b9051d; no voters:
I20240503 02:00:21.017648 13968 raft_consensus.cc:2781] T 87c4b630a58749e6be9f26cceda27573 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:21.017796 13968 raft_consensus.cc:686] T 87c4b630a58749e6be9f26cceda27573 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Becoming Leader. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Running, Role: LEADER
I20240503 02:00:21.018183 13968 consensus_queue.cc:237] T 87c4b630a58749e6be9f26cceda27573 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:21.020735 6314 catalog_manager.cc:5484] T 87c4b630a58749e6be9f26cceda27573 P 2a01ace80baf4057b6bf1995cf88468b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135). New cstate: current_term: 1 leader_uuid: "2a01ace80baf4057b6bf1995cf88468b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:21.022698 6309 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:21.029265 14348 tablet.cc:1722] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:21.029645 14348 tablet.cc:1620] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a: MemRowSet was empty: no flush needed.
I20240503 02:00:21.029887 14319 tablet.cc:1722] T 451cd82331904b84884f7fdaf2b7dec9 P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:21.030205 14319 tablet.cc:1620] T 451cd82331904b84884f7fdaf2b7dec9 P 80fd18ef4d604c86b909d5c008a46b4f: MemRowSet was empty: no flush needed.
I20240503 02:00:21.036324 6314 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=364d2beeb13a4e77a5977e4c7d9786dc] alter complete (version 2)
I20240503 02:00:21.036685 14354 tablet.cc:1722] T 451cd82331904b84884f7fdaf2b7dec9 P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:21.037016 14354 tablet.cc:1620] T 451cd82331904b84884f7fdaf2b7dec9 P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
I20240503 02:00:21.104014 13900 raft_consensus.cc:482] T 3af89caa97634222ab834577a9217ce9 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:21.104142 13900 raft_consensus.cc:504] T 3af89caa97634222ab834577a9217ce9 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:21.104566 13900 leader_election.cc:288] T 3af89caa97634222ab834577a9217ce9 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:21.104874 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3af89caa97634222ab834577a9217ce9" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:21.104948 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3af89caa97634222ab834577a9217ce9" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 02:00:21.105038 8916 raft_consensus.cc:2454] T 3af89caa97634222ab834577a9217ce9 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:21.105074 6851 raft_consensus.cc:2454] T 3af89caa97634222ab834577a9217ce9 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:21.105324 8984 leader_election.cc:302] T 3af89caa97634222ab834577a9217ce9 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:21.105480 13900 raft_consensus.cc:2781] T 3af89caa97634222ab834577a9217ce9 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:21.105540 13900 raft_consensus.cc:482] T 3af89caa97634222ab834577a9217ce9 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:21.105587 13900 raft_consensus.cc:3037] T 3af89caa97634222ab834577a9217ce9 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:21.106415 13900 raft_consensus.cc:504] T 3af89caa97634222ab834577a9217ce9 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:21.106762 13900 leader_election.cc:288] T 3af89caa97634222ab834577a9217ce9 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Requested vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:21.107164 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3af89caa97634222ab834577a9217ce9" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:21.107164 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3af89caa97634222ab834577a9217ce9" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 02:00:21.107303 6851 raft_consensus.cc:3037] T 3af89caa97634222ab834577a9217ce9 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:21.107299 8916 raft_consensus.cc:3037] T 3af89caa97634222ab834577a9217ce9 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:21.108299 6851 raft_consensus.cc:2454] T 3af89caa97634222ab834577a9217ce9 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:21.108299 8916 raft_consensus.cc:2454] T 3af89caa97634222ab834577a9217ce9 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:21.108700 8984 leader_election.cc:302] T 3af89caa97634222ab834577a9217ce9 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9942306855404cfd891adf227dd65eb2, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:21.108886 13900 raft_consensus.cc:2781] T 3af89caa97634222ab834577a9217ce9 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:21.108969 13900 raft_consensus.cc:686] T 3af89caa97634222ab834577a9217ce9 P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Becoming Leader. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Running, Role: LEADER
I20240503 02:00:21.109098 13900 consensus_queue.cc:237] T 3af89caa97634222ab834577a9217ce9 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
W20240503 02:00:21.112915 6523 consensus_peers.cc:485] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:00:21.113802 6309 catalog_manager.cc:5484] T 3af89caa97634222ab834577a9217ce9 P 99ead02915954b4fab23b033ad6d546b reported cstate change: term changed from 0 to 1, leader changed from <none> to 99ead02915954b4fab23b033ad6d546b (127.0.99.134). New cstate: current_term: 1 leader_uuid: "99ead02915954b4fab23b033ad6d546b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } }
W20240503 02:00:21.138303 6785 consensus_peers.cc:485] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:00:21.141315 6309 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:21.146302 14348 tablet.cc:1722] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:21.146957 14354 tablet.cc:1722] T 451cd82331904b84884f7fdaf2b7dec9 P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:21.148751 14319 tablet.cc:1722] T 451cd82331904b84884f7fdaf2b7dec9 P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
W20240503 02:00:21.150354 6523 consensus_peers.cc:485] T d0d5dd42db3e4883af7fd16d8813b550 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:00:21.152753 6309 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=364d2beeb13a4e77a5977e4c7d9786dc] alter complete (version 3)
W20240503 02:00:21.182763 6523 consensus_peers.cc:485] T 925612b660ec471da5b3c548802ede6e P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:21.255148 8982 consensus_peers.cc:485] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:00:21.256206 6308 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:21.256341 6308 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:21.258425 6308 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet 451cd82331904b84884f7fdaf2b7dec9
I20240503 02:00:21.259097 6436 tablet_service.cc:1514] Processing DeleteTablet for tablet 451cd82331904b84884f7fdaf2b7dec9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:21 UTC) from {username='slave'} at 127.0.0.1:51338
I20240503 02:00:21.259395 6567 tablet_service.cc:1514] Processing DeleteTablet for tablet 451cd82331904b84884f7fdaf2b7dec9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:21 UTC) from {username='slave'} at 127.0.0.1:59846
I20240503 02:00:21.259438 14368 tablet_replica.cc:330] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a: stopping tablet replica
I20240503 02:00:21.259580 14368 raft_consensus.cc:2229] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:21.259769 14369 tablet_replica.cc:330] T 451cd82331904b84884f7fdaf2b7dec9 P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 02:00:21.259904 14368 raft_consensus.cc:2258] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:21.260051 14369 raft_consensus.cc:2229] T 451cd82331904b84884f7fdaf2b7dec9 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:21.260421 8887 tablet_service.cc:1514] Processing DeleteTablet for tablet 451cd82331904b84884f7fdaf2b7dec9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:21 UTC) from {username='slave'} at 127.0.0.1:40296
I20240503 02:00:21.260658 14368 ts_tablet_manager.cc:1875] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:21.260666 14369 raft_consensus.cc:2258] T 451cd82331904b84884f7fdaf2b7dec9 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:21.261742 14370 tablet_replica.cc:330] T 451cd82331904b84884f7fdaf2b7dec9 P 9942306855404cfd891adf227dd65eb2: stopping tablet replica
I20240503 02:00:21.261953 14370 raft_consensus.cc:2229] T 451cd82331904b84884f7fdaf2b7dec9 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:21.262107 14370 raft_consensus.cc:2258] T 451cd82331904b84884f7fdaf2b7dec9 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:21.262440 14369 ts_tablet_manager.cc:1875] T 451cd82331904b84884f7fdaf2b7dec9 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:21.262735 14370 ts_tablet_manager.cc:1875] T 451cd82331904b84884f7fdaf2b7dec9 P 9942306855404cfd891adf227dd65eb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:21.264307 14368 ts_tablet_manager.cc:1888] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:21.264478 14368 log.cc:1196] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/451cd82331904b84884f7fdaf2b7dec9
I20240503 02:00:21.264814 14368 ts_tablet_manager.cc:1909] T 451cd82331904b84884f7fdaf2b7dec9 P dec3e4a04d604f7eaaa3f077d081218a: Deleting consensus metadata
I20240503 02:00:21.265072 14369 ts_tablet_manager.cc:1888] T 451cd82331904b84884f7fdaf2b7dec9 P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:21.265151 14369 log.cc:1196] T 451cd82331904b84884f7fdaf2b7dec9 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/451cd82331904b84884f7fdaf2b7dec9
I20240503 02:00:21.265509 14369 ts_tablet_manager.cc:1909] T 451cd82331904b84884f7fdaf2b7dec9 P 80fd18ef4d604c86b909d5c008a46b4f: Deleting consensus metadata
I20240503 02:00:21.265604 6300 catalog_manager.cc:4830] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): tablet 451cd82331904b84884f7fdaf2b7dec9 (table rebalancer_dynamic_table_new_name [id=364d2beeb13a4e77a5977e4c7d9786dc]) successfully deleted
I20240503 02:00:21.265666 14370 ts_tablet_manager.cc:1888] T 451cd82331904b84884f7fdaf2b7dec9 P 9942306855404cfd891adf227dd65eb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:21.265727 14370 log.cc:1196] T 451cd82331904b84884f7fdaf2b7dec9 P 9942306855404cfd891adf227dd65eb2: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/451cd82331904b84884f7fdaf2b7dec9
I20240503 02:00:21.266012 14370 ts_tablet_manager.cc:1909] T 451cd82331904b84884f7fdaf2b7dec9 P 9942306855404cfd891adf227dd65eb2: Deleting consensus metadata
I20240503 02:00:21.266448 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet 451cd82331904b84884f7fdaf2b7dec9 (table rebalancer_dynamic_table_new_name [id=364d2beeb13a4e77a5977e4c7d9786dc]) successfully deleted
I20240503 02:00:21.266707 6303 catalog_manager.cc:4830] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211): tablet 451cd82331904b84884f7fdaf2b7dec9 (table rebalancer_dynamic_table_new_name [id=364d2beeb13a4e77a5977e4c7d9786dc]) successfully deleted
W20240503 02:00:21.286124 9113 consensus_peers.cc:485] T f44b59277c3b47adbd4dd6aff9350af1 P 2a01ace80baf4057b6bf1995cf88468b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:21.297313 8851 consensus_peers.cc:485] T 067f54011a37432d99157884e70fc601 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:00:21.361188 6308 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
W20240503 02:00:21.365478 6785 consensus_peers.cc:485] T 46ad540deb624a25b97d3b967fe1f6f4 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:00:21.368053 6436 tablet_service.cc:1467] Processing CreateTablet for tablet d22b0105836a43e09a8de16843474dfc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1fe7d1ed4aee47f1ae55ec3c13859000]), partition=
I20240503 02:00:21.368243 9158 tablet_service.cc:1467] Processing CreateTablet for tablet d22b0105836a43e09a8de16843474dfc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1fe7d1ed4aee47f1ae55ec3c13859000]), partition=
I20240503 02:00:21.368324 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d22b0105836a43e09a8de16843474dfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:21.368542 9158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d22b0105836a43e09a8de16843474dfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:21.368649 6700 tablet_service.cc:1467] Processing CreateTablet for tablet d22b0105836a43e09a8de16843474dfc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1fe7d1ed4aee47f1ae55ec3c13859000]), partition=
I20240503 02:00:21.368901 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d22b0105836a43e09a8de16843474dfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:21.370748 14351 tablet_bootstrap.cc:492] T d22b0105836a43e09a8de16843474dfc P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:21.371016 14359 tablet_bootstrap.cc:492] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:21.371796 14351 tablet_bootstrap.cc:654] T d22b0105836a43e09a8de16843474dfc P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:21.371855 14359 tablet_bootstrap.cc:654] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:21.373106 14359 tablet_bootstrap.cc:492] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:21.373194 14359 ts_tablet_manager.cc:1367] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:21.373374 14351 tablet_bootstrap.cc:492] T d22b0105836a43e09a8de16843474dfc P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:21.373456 14351 ts_tablet_manager.cc:1367] T d22b0105836a43e09a8de16843474dfc P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:21.373632 14359 raft_consensus.cc:348] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:21.373733 14359 raft_consensus.cc:374] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:21.373762 14359 raft_consensus.cc:729] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:21.373876 14359 consensus_queue.cc:260] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:21.373905 14351 raft_consensus.cc:348] T d22b0105836a43e09a8de16843474dfc P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:21.374022 14351 raft_consensus.cc:374] T d22b0105836a43e09a8de16843474dfc P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:21.374051 14351 raft_consensus.cc:729] T d22b0105836a43e09a8de16843474dfc P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:21.374178 14351 consensus_queue.cc:260] T d22b0105836a43e09a8de16843474dfc P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:21.374251 14359 ts_tablet_manager.cc:1398] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:21.374467 14351 ts_tablet_manager.cc:1398] T d22b0105836a43e09a8de16843474dfc P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:21.375849 14374 tablet_bootstrap.cc:492] T d22b0105836a43e09a8de16843474dfc P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:21.376953 14374 tablet_bootstrap.cc:654] T d22b0105836a43e09a8de16843474dfc P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:21.380280 14374 tablet_bootstrap.cc:492] T d22b0105836a43e09a8de16843474dfc P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:21.380506 14374 ts_tablet_manager.cc:1367] T d22b0105836a43e09a8de16843474dfc P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:00:21.381341 14374 raft_consensus.cc:348] T d22b0105836a43e09a8de16843474dfc P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:21.381561 14374 raft_consensus.cc:374] T d22b0105836a43e09a8de16843474dfc P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:21.381644 14374 raft_consensus.cc:729] T d22b0105836a43e09a8de16843474dfc P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:21.381793 14374 consensus_queue.cc:260] T d22b0105836a43e09a8de16843474dfc P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:21.382915 14374 ts_tablet_manager.cc:1398] T d22b0105836a43e09a8de16843474dfc P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:21.388321 14371 raft_consensus.cc:482] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:21.388478 14371 raft_consensus.cc:504] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:21.389142 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d22b0105836a43e09a8de16843474dfc" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 02:00:21.389286 6456 raft_consensus.cc:2454] T d22b0105836a43e09a8de16843474dfc P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:21.389418 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d22b0105836a43e09a8de16843474dfc" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:21.389549 9179 raft_consensus.cc:2454] T d22b0105836a43e09a8de16843474dfc P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:21.389567 6653 leader_election.cc:302] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2ac2eb7cc264047b9ab32fe3a4fc025, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:21.389725 14284 raft_consensus.cc:2781] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:21.389743 14371 leader_election.cc:288] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:21.389814 14284 raft_consensus.cc:482] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:21.389890 14284 raft_consensus.cc:3037] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:21.390794 14284 raft_consensus.cc:504] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:21.391152 14284 leader_election.cc:288] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Requested vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:21.391534 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d22b0105836a43e09a8de16843474dfc" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 02:00:21.391657 6456 raft_consensus.cc:3037] T d22b0105836a43e09a8de16843474dfc P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:21.391646 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d22b0105836a43e09a8de16843474dfc" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:21.391758 9179 raft_consensus.cc:3037] T d22b0105836a43e09a8de16843474dfc P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:21.392642 6456 raft_consensus.cc:2454] T d22b0105836a43e09a8de16843474dfc P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:21.392747 9179 raft_consensus.cc:2454] T d22b0105836a43e09a8de16843474dfc P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:21.393132 6653 leader_election.cc:302] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:21.393349 14284 raft_consensus.cc:2781] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:21.393435 14284 raft_consensus.cc:686] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Becoming Leader. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Running, Role: LEADER
I20240503 02:00:21.393559 14284 consensus_queue.cc:237] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
W20240503 02:00:21.394825 6785 consensus_peers.cc:485] T d897c21e630d4770832bb10f43778e82 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:00:21.396116 6309 catalog_manager.cc:5484] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } }
W20240503 02:00:21.431757 6785 consensus_peers.cc:485] T 8246488dec204bf6b09e868d7951ac5a P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:21.436466 6654 consensus_peers.cc:485] T cea1ea50fa434d31b42451f84f26bc76 P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:21.449671 6523 consensus_peers.cc:485] T 4385d51d11574192883d01376e2d5620 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:21.458770 8851 consensus_peers.cc:485] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:00:21.468163 14220 consensus_queue.cc:1035] T ca907fb5b04f4d2492e7f4cea7114c95 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:21.469563 14220 consensus_queue.cc:1035] T ca907fb5b04f4d2492e7f4cea7114c95 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:21.500659 6309 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:21.505206 9179 raft_consensus.cc:1261] T d22b0105836a43e09a8de16843474dfc P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:21.505661 14284 consensus_queue.cc:1035] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:21.505792 6456 raft_consensus.cc:1261] T d22b0105836a43e09a8de16843474dfc P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:21.506172 14284 consensus_queue.cc:1035] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:00:21.508189 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:21.510370 6308 catalog_manager.cc:5576] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093) does not have the latest schema for tablet d22b0105836a43e09a8de16843474dfc (table rebalancer_dynamic_table [id=1fe7d1ed4aee47f1ae55ec3c13859000]). Expected version 1 got 0
I20240503 02:00:21.510360 14348 tablet.cc:1722] T d22b0105836a43e09a8de16843474dfc P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:21.510720 14348 tablet.cc:1620] T d22b0105836a43e09a8de16843474dfc P dec3e4a04d604f7eaaa3f077d081218a: MemRowSet was empty: no flush needed.
I20240503 02:00:21.512770 6309 catalog_manager.cc:5576] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079) does not have the latest schema for tablet d22b0105836a43e09a8de16843474dfc (table rebalancer_dynamic_table [id=1fe7d1ed4aee47f1ae55ec3c13859000]). Expected version 1 got 0
I20240503 02:00:21.514603 13968 consensus_queue.cc:1035] T 87c4b630a58749e6be9f26cceda27573 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:21.515468 6308 catalog_manager.cc:5576] TS c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655) does not have the latest schema for tablet d22b0105836a43e09a8de16843474dfc (table rebalancer_dynamic_table [id=1fe7d1ed4aee47f1ae55ec3c13859000]). Expected version 1 got 0
I20240503 02:00:21.517808 14384 tablet.cc:1722] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:21.518448 14384 tablet.cc:1620] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025: MemRowSet was empty: no flush needed.
I20240503 02:00:21.519250 13968 consensus_queue.cc:1035] T 87c4b630a58749e6be9f26cceda27573 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:21.520107 6313 catalog_manager.cc:5576] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079) does not have the latest schema for tablet d22b0105836a43e09a8de16843474dfc (table rebalancer_dynamic_table [id=1fe7d1ed4aee47f1ae55ec3c13859000]). Expected version 1 got 0
I20240503 02:00:21.520311 6314 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=1fe7d1ed4aee47f1ae55ec3c13859000] alter complete (version 1)
I20240503 02:00:21.525693 14385 tablet.cc:1722] T d22b0105836a43e09a8de16843474dfc P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:21.526095 14385 tablet.cc:1620] T d22b0105836a43e09a8de16843474dfc P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
W20240503 02:00:21.592730 8851 consensus_peers.cc:485] T f28586e70bd9473fbdc6b9126b666fa9 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:21.614953 9113 consensus_peers.cc:485] T b1740b877c1e4372a0337d99f5bcab0e P 2a01ace80baf4057b6bf1995cf88468b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:00:21.623046 6313 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_extra_table_56"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:21.624662 6309 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:21.630523 6567 tablet_service.cc:1467] Processing CreateTablet for tablet ea801f686a0a4efa9da66b24bdaa10cc (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=f0c22c72d4c14f279e3366a16e75aae5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:21.630878 6567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea801f686a0a4efa9da66b24bdaa10cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:21.630955 9028 tablet_service.cc:1467] Processing CreateTablet for tablet ea801f686a0a4efa9da66b24bdaa10cc (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=f0c22c72d4c14f279e3366a16e75aae5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:21.631186 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea801f686a0a4efa9da66b24bdaa10cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:21.632283 6569 tablet_service.cc:1467] Processing CreateTablet for tablet 98b3ee95a66e43788e5e2a428d9fa9c6 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=f0c22c72d4c14f279e3366a16e75aae5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:21.632560 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98b3ee95a66e43788e5e2a428d9fa9c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:21.632534 9027 tablet_service.cc:1467] Processing CreateTablet for tablet 8a8b7b032eea4726b24b805236601450 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=f0c22c72d4c14f279e3366a16e75aae5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:21.632797 9027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a8b7b032eea4726b24b805236601450. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:21.633651 6568 tablet_service.cc:1467] Processing CreateTablet for tablet 8a8b7b032eea4726b24b805236601450 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=f0c22c72d4c14f279e3366a16e75aae5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:21.633862 6568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a8b7b032eea4726b24b805236601450. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:21.634783 14392 tablet_bootstrap.cc:492] T ea801f686a0a4efa9da66b24bdaa10cc P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:21.634894 8897 tablet_service.cc:1467] Processing CreateTablet for tablet 8a8b7b032eea4726b24b805236601450 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=f0c22c72d4c14f279e3366a16e75aae5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:21.635157 8897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a8b7b032eea4726b24b805236601450. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:21.635911 8887 tablet_service.cc:1467] Processing CreateTablet for tablet ea801f686a0a4efa9da66b24bdaa10cc (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=f0c22c72d4c14f279e3366a16e75aae5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:21.636142 8887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea801f686a0a4efa9da66b24bdaa10cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:21.636322 14393 tablet_bootstrap.cc:492] T ea801f686a0a4efa9da66b24bdaa10cc P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:21.636713 14392 tablet_bootstrap.cc:654] T ea801f686a0a4efa9da66b24bdaa10cc P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:21.634902 8880 tablet_service.cc:1467] Processing CreateTablet for tablet 98b3ee95a66e43788e5e2a428d9fa9c6 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=f0c22c72d4c14f279e3366a16e75aae5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:21.637583 8880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98b3ee95a66e43788e5e2a428d9fa9c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:21.638765 6436 tablet_service.cc:1467] Processing CreateTablet for tablet 98b3ee95a66e43788e5e2a428d9fa9c6 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=f0c22c72d4c14f279e3366a16e75aae5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:21.639084 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98b3ee95a66e43788e5e2a428d9fa9c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:21.639767 14394 tablet_bootstrap.cc:492] T 8a8b7b032eea4726b24b805236601450 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:21.640694 14348 tablet.cc:1722] T d22b0105836a43e09a8de16843474dfc P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:21.640964 14394 tablet_bootstrap.cc:654] T 8a8b7b032eea4726b24b805236601450 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:21.641011 14348 tablet.cc:1620] T d22b0105836a43e09a8de16843474dfc P dec3e4a04d604f7eaaa3f077d081218a: MemRowSet was empty: no flush needed.
I20240503 02:00:21.642570 14394 tablet_bootstrap.cc:492] T 8a8b7b032eea4726b24b805236601450 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:21.642714 14394 ts_tablet_manager.cc:1367] T 8a8b7b032eea4726b24b805236601450 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:00:21.642678 14385 tablet.cc:1722] T d22b0105836a43e09a8de16843474dfc P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:21.643018 14385 tablet.cc:1620] T d22b0105836a43e09a8de16843474dfc P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
I20240503 02:00:21.644688 14384 tablet.cc:1722] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:21.645021 14384 tablet.cc:1620] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025: MemRowSet was empty: no flush needed.
I20240503 02:00:21.645748 14394 raft_consensus.cc:348] T 8a8b7b032eea4726b24b805236601450 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:21.645956 14394 raft_consensus.cc:374] T 8a8b7b032eea4726b24b805236601450 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:21.646013 14394 raft_consensus.cc:729] T 8a8b7b032eea4726b24b805236601450 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:21.646159 14394 consensus_queue.cc:260] T 8a8b7b032eea4726b24b805236601450 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:21.646669 14374 tablet_bootstrap.cc:492] T 98b3ee95a66e43788e5e2a428d9fa9c6 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:21.646695 14394 ts_tablet_manager.cc:1398] T 8a8b7b032eea4726b24b805236601450 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240503 02:00:21.647049 14394 tablet_bootstrap.cc:492] T ea801f686a0a4efa9da66b24bdaa10cc P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:21.648200 14374 tablet_bootstrap.cc:654] T 98b3ee95a66e43788e5e2a428d9fa9c6 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:21.652940 14374 tablet_bootstrap.cc:492] T 98b3ee95a66e43788e5e2a428d9fa9c6 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:21.653065 14374 ts_tablet_manager.cc:1367] T 98b3ee95a66e43788e5e2a428d9fa9c6 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:00:21.653553 14374 raft_consensus.cc:348] T 98b3ee95a66e43788e5e2a428d9fa9c6 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:21.653687 14374 raft_consensus.cc:374] T 98b3ee95a66e43788e5e2a428d9fa9c6 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:21.653719 14374 raft_consensus.cc:729] T 98b3ee95a66e43788e5e2a428d9fa9c6 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:21.653486 6314 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=1fe7d1ed4aee47f1ae55ec3c13859000] alter complete (version 2)
I20240503 02:00:21.653836 14374 consensus_queue.cc:260] T 98b3ee95a66e43788e5e2a428d9fa9c6 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:21.654136 14374 ts_tablet_manager.cc:1398] T 98b3ee95a66e43788e5e2a428d9fa9c6 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:21.654800 14394 tablet_bootstrap.cc:654] T ea801f686a0a4efa9da66b24bdaa10cc P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:21.656265 13900 consensus_queue.cc:1035] T 3af89caa97634222ab834577a9217ce9 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:21.658380 14393 tablet_bootstrap.cc:654] T ea801f686a0a4efa9da66b24bdaa10cc P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:21.659833 14392 tablet_bootstrap.cc:492] T ea801f686a0a4efa9da66b24bdaa10cc P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:21.659955 14392 ts_tablet_manager.cc:1367] T ea801f686a0a4efa9da66b24bdaa10cc P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.000s
I20240503 02:00:21.660526 14392 raft_consensus.cc:348] T ea801f686a0a4efa9da66b24bdaa10cc P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:21.660665 14392 raft_consensus.cc:374] T ea801f686a0a4efa9da66b24bdaa10cc P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:21.660699 14392 raft_consensus.cc:729] T ea801f686a0a4efa9da66b24bdaa10cc P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:21.660813 14392 consensus_queue.cc:260] T ea801f686a0a4efa9da66b24bdaa10cc P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:21.660953 14393 tablet_bootstrap.cc:492] T ea801f686a0a4efa9da66b24bdaa10cc P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:21.661053 14393 ts_tablet_manager.cc:1367] T ea801f686a0a4efa9da66b24bdaa10cc P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.000s
I20240503 02:00:21.661113 14392 ts_tablet_manager.cc:1398] T ea801f686a0a4efa9da66b24bdaa10cc P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:21.661288 14392 tablet_bootstrap.cc:492] T 8a8b7b032eea4726b24b805236601450 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:21.661518 14393 raft_consensus.cc:348] T ea801f686a0a4efa9da66b24bdaa10cc P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:21.661614 14393 raft_consensus.cc:374] T ea801f686a0a4efa9da66b24bdaa10cc P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:21.661643 14393 raft_consensus.cc:729] T ea801f686a0a4efa9da66b24bdaa10cc P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:21.661769 14393 consensus_queue.cc:260] T ea801f686a0a4efa9da66b24bdaa10cc P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:21.662027 14393 ts_tablet_manager.cc:1398] T ea801f686a0a4efa9da66b24bdaa10cc P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:21.662303 14392 tablet_bootstrap.cc:654] T 8a8b7b032eea4726b24b805236601450 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:21.664170 14393 tablet_bootstrap.cc:492] T 98b3ee95a66e43788e5e2a428d9fa9c6 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:21.664839 14195 consensus_queue.cc:1035] T 3af89caa97634222ab834577a9217ce9 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:21.665263 14393 tablet_bootstrap.cc:654] T 98b3ee95a66e43788e5e2a428d9fa9c6 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:21.668949 14392 tablet_bootstrap.cc:492] T 8a8b7b032eea4726b24b805236601450 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:21.669055 14392 ts_tablet_manager.cc:1367] T 8a8b7b032eea4726b24b805236601450 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 02:00:21.670464 14392 raft_consensus.cc:348] T 8a8b7b032eea4726b24b805236601450 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:21.670715 14392 raft_consensus.cc:374] T 8a8b7b032eea4726b24b805236601450 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:21.670804 14392 raft_consensus.cc:729] T 8a8b7b032eea4726b24b805236601450 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:21.671263 14392 consensus_queue.cc:260] T 8a8b7b032eea4726b24b805236601450 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:21.672621 14392 ts_tablet_manager.cc:1398] T 8a8b7b032eea4726b24b805236601450 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:21.673786 14394 tablet_bootstrap.cc:492] T ea801f686a0a4efa9da66b24bdaa10cc P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:21.673897 14394 ts_tablet_manager.cc:1367] T ea801f686a0a4efa9da66b24bdaa10cc P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.027s user 0.000s sys 0.003s
I20240503 02:00:21.674424 14394 raft_consensus.cc:348] T ea801f686a0a4efa9da66b24bdaa10cc P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:21.674659 14394 raft_consensus.cc:374] T ea801f686a0a4efa9da66b24bdaa10cc P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:21.674710 14394 raft_consensus.cc:729] T ea801f686a0a4efa9da66b24bdaa10cc P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:21.674822 14394 consensus_queue.cc:260] T ea801f686a0a4efa9da66b24bdaa10cc P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:21.675112 14394 ts_tablet_manager.cc:1398] T ea801f686a0a4efa9da66b24bdaa10cc P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:21.675383 14394 tablet_bootstrap.cc:492] T 98b3ee95a66e43788e5e2a428d9fa9c6 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:21.676635 14394 tablet_bootstrap.cc:654] T 98b3ee95a66e43788e5e2a428d9fa9c6 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:21.676753 14393 tablet_bootstrap.cc:492] T 98b3ee95a66e43788e5e2a428d9fa9c6 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:21.676857 14393 ts_tablet_manager.cc:1367] T 98b3ee95a66e43788e5e2a428d9fa9c6 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 02:00:21.677352 14393 raft_consensus.cc:348] T 98b3ee95a66e43788e5e2a428d9fa9c6 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:21.677477 14393 raft_consensus.cc:374] T 98b3ee95a66e43788e5e2a428d9fa9c6 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:21.677507 14393 raft_consensus.cc:729] T 98b3ee95a66e43788e5e2a428d9fa9c6 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:21.677632 14393 consensus_queue.cc:260] T 98b3ee95a66e43788e5e2a428d9fa9c6 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:21.677896 14393 ts_tablet_manager.cc:1398] T 98b3ee95a66e43788e5e2a428d9fa9c6 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:21.678067 14393 tablet_bootstrap.cc:492] T 8a8b7b032eea4726b24b805236601450 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:21.678112 14394 tablet_bootstrap.cc:492] T 98b3ee95a66e43788e5e2a428d9fa9c6 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:21.678198 14394 ts_tablet_manager.cc:1367] T 98b3ee95a66e43788e5e2a428d9fa9c6 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:21.678634 14394 raft_consensus.cc:348] T 98b3ee95a66e43788e5e2a428d9fa9c6 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:21.678753 14394 raft_consensus.cc:374] T 98b3ee95a66e43788e5e2a428d9fa9c6 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:21.678784 14394 raft_consensus.cc:729] T 98b3ee95a66e43788e5e2a428d9fa9c6 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:21.678941 14394 consensus_queue.cc:260] T 98b3ee95a66e43788e5e2a428d9fa9c6 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:21.679380 14394 ts_tablet_manager.cc:1398] T 98b3ee95a66e43788e5e2a428d9fa9c6 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:21.680756 14393 tablet_bootstrap.cc:654] T 8a8b7b032eea4726b24b805236601450 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:21.682283 14393 tablet_bootstrap.cc:492] T 8a8b7b032eea4726b24b805236601450 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:21.682408 14393 ts_tablet_manager.cc:1367] T 8a8b7b032eea4726b24b805236601450 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:00:21.682907 14393 raft_consensus.cc:348] T 8a8b7b032eea4726b24b805236601450 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:21.683035 14393 raft_consensus.cc:374] T 8a8b7b032eea4726b24b805236601450 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:21.683068 14393 raft_consensus.cc:729] T 8a8b7b032eea4726b24b805236601450 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:21.683197 14393 consensus_queue.cc:260] T 8a8b7b032eea4726b24b805236601450 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:21.683486 14393 ts_tablet_manager.cc:1398] T 8a8b7b032eea4726b24b805236601450 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240503 02:00:21.691885 8851 consensus_peers.cc:485] T 5f51fe5f799f4e479b62f5917a2c2606 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:21.744040 8982 consensus_peers.cc:485] T 13756a6e44814fe4b09a6b0f087546c3 P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:00:21.751487 14323 raft_consensus.cc:482] T ea801f686a0a4efa9da66b24bdaa10cc P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:21.751601 14323 raft_consensus.cc:504] T ea801f686a0a4efa9da66b24bdaa10cc P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:21.752008 14323 leader_election.cc:288] T ea801f686a0a4efa9da66b24bdaa10cc P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:21.752394 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea801f686a0a4efa9da66b24bdaa10cc" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:21.752463 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea801f686a0a4efa9da66b24bdaa10cc" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:21.752544 8916 raft_consensus.cc:2454] T ea801f686a0a4efa9da66b24bdaa10cc P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:21.752586 9048 raft_consensus.cc:2454] T ea801f686a0a4efa9da66b24bdaa10cc P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:21.752877 6525 leader_election.cc:302] T ea801f686a0a4efa9da66b24bdaa10cc P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:21.753043 14323 raft_consensus.cc:2781] T ea801f686a0a4efa9da66b24bdaa10cc P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:21.753087 14323 raft_consensus.cc:482] T ea801f686a0a4efa9da66b24bdaa10cc P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:21.753121 14323 raft_consensus.cc:3037] T ea801f686a0a4efa9da66b24bdaa10cc P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:21.754398 14323 raft_consensus.cc:504] T ea801f686a0a4efa9da66b24bdaa10cc P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:21.754741 14323 leader_election.cc:288] T ea801f686a0a4efa9da66b24bdaa10cc P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Requested vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:21.755156 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea801f686a0a4efa9da66b24bdaa10cc" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:21.755275 9048 raft_consensus.cc:3037] T ea801f686a0a4efa9da66b24bdaa10cc P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:21.756227 9048 raft_consensus.cc:2454] T ea801f686a0a4efa9da66b24bdaa10cc P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:21.756513 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea801f686a0a4efa9da66b24bdaa10cc" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:21.756635 8916 raft_consensus.cc:3037] T ea801f686a0a4efa9da66b24bdaa10cc P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:21.757608 6523 leader_election.cc:302] T ea801f686a0a4efa9da66b24bdaa10cc P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:21.757619 8916 raft_consensus.cc:2454] T ea801f686a0a4efa9da66b24bdaa10cc P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:21.757786 14323 raft_consensus.cc:2781] T ea801f686a0a4efa9da66b24bdaa10cc P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:21.757987 14323 raft_consensus.cc:686] T ea801f686a0a4efa9da66b24bdaa10cc P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Becoming Leader. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Running, Role: LEADER
I20240503 02:00:21.758141 14323 consensus_queue.cc:237] T ea801f686a0a4efa9da66b24bdaa10cc P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:21.759351 6314 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:21.760640 6309 catalog_manager.cc:5484] T ea801f686a0a4efa9da66b24bdaa10cc P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:21.765852 14384 tablet.cc:1722] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:21.766635 14348 tablet.cc:1722] T d22b0105836a43e09a8de16843474dfc P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:21.770128 14385 tablet.cc:1722] T d22b0105836a43e09a8de16843474dfc P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:21.772284 14323 raft_consensus.cc:482] T 8a8b7b032eea4726b24b805236601450 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:21.772418 14323 raft_consensus.cc:504] T 8a8b7b032eea4726b24b805236601450 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:21.772812 14323 leader_election.cc:288] T 8a8b7b032eea4726b24b805236601450 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:21.773130 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8b7b032eea4726b24b805236601450" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:21.773262 8916 raft_consensus.cc:2454] T 8a8b7b032eea4726b24b805236601450 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:21.773579 6525 leader_election.cc:302] T 8a8b7b032eea4726b24b805236601450 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:21.773747 14323 raft_consensus.cc:2781] T 8a8b7b032eea4726b24b805236601450 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:21.773793 14323 raft_consensus.cc:482] T 8a8b7b032eea4726b24b805236601450 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:21.773859 14323 raft_consensus.cc:3037] T 8a8b7b032eea4726b24b805236601450 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:21.774812 14323 raft_consensus.cc:504] T 8a8b7b032eea4726b24b805236601450 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:21.775182 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8b7b032eea4726b24b805236601450" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:21.775417 9048 raft_consensus.cc:2454] T 8a8b7b032eea4726b24b805236601450 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:21.775537 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8b7b032eea4726b24b805236601450" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:21.775656 8916 raft_consensus.cc:3037] T 8a8b7b032eea4726b24b805236601450 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:21.776762 8916 raft_consensus.cc:2454] T 8a8b7b032eea4726b24b805236601450 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:21.777174 6525 leader_election.cc:302] T 8a8b7b032eea4726b24b805236601450 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:21.777441 14323 leader_election.cc:288] T 8a8b7b032eea4726b24b805236601450 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Requested vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:21.777562 14323 raft_consensus.cc:2781] T 8a8b7b032eea4726b24b805236601450 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:21.777632 14323 raft_consensus.cc:686] T 8a8b7b032eea4726b24b805236601450 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Becoming Leader. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Running, Role: LEADER
I20240503 02:00:21.777774 14323 consensus_queue.cc:237] T 8a8b7b032eea4726b24b805236601450 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:21.777824 9047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8b7b032eea4726b24b805236601450" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:21.777922 9047 raft_consensus.cc:3037] T 8a8b7b032eea4726b24b805236601450 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:21.778028 6314 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=1fe7d1ed4aee47f1ae55ec3c13859000] alter complete (version 3)
I20240503 02:00:21.778952 9047 raft_consensus.cc:2454] T 8a8b7b032eea4726b24b805236601450 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:21.781087 6313 catalog_manager.cc:5484] T 8a8b7b032eea4726b24b805236601450 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } }
W20240503 02:00:21.818346 8851 consensus_peers.cc:485] T 7da8f614a70e4177b70e7c2f31d22a10 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:00:21.821183 14153 raft_consensus.cc:482] T 98b3ee95a66e43788e5e2a428d9fa9c6 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:21.821336 14153 raft_consensus.cc:504] T 98b3ee95a66e43788e5e2a428d9fa9c6 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:21.821748 14153 leader_election.cc:288] T 98b3ee95a66e43788e5e2a428d9fa9c6 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:21.822206 6587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98b3ee95a66e43788e5e2a428d9fa9c6" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 02:00:21.822202 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98b3ee95a66e43788e5e2a428d9fa9c6" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:21.822337 6587 raft_consensus.cc:2454] T 98b3ee95a66e43788e5e2a428d9fa9c6 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:21.822356 8916 raft_consensus.cc:2454] T 98b3ee95a66e43788e5e2a428d9fa9c6 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:21.822675 6393 leader_election.cc:302] T 98b3ee95a66e43788e5e2a428d9fa9c6 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:21.822851 14153 raft_consensus.cc:2781] T 98b3ee95a66e43788e5e2a428d9fa9c6 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:21.822921 14153 raft_consensus.cc:482] T 98b3ee95a66e43788e5e2a428d9fa9c6 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:21.822952 14153 raft_consensus.cc:3037] T 98b3ee95a66e43788e5e2a428d9fa9c6 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:21.823853 14153 raft_consensus.cc:504] T 98b3ee95a66e43788e5e2a428d9fa9c6 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:21.824236 14153 leader_election.cc:288] T 98b3ee95a66e43788e5e2a428d9fa9c6 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Requested vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:21.824640 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98b3ee95a66e43788e5e2a428d9fa9c6" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:21.824643 6587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98b3ee95a66e43788e5e2a428d9fa9c6" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 02:00:21.824784 8916 raft_consensus.cc:3037] T 98b3ee95a66e43788e5e2a428d9fa9c6 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:21.824797 6587 raft_consensus.cc:3037] T 98b3ee95a66e43788e5e2a428d9fa9c6 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:21.825796 8916 raft_consensus.cc:2454] T 98b3ee95a66e43788e5e2a428d9fa9c6 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:21.825805 6587 raft_consensus.cc:2454] T 98b3ee95a66e43788e5e2a428d9fa9c6 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:21.826107 6393 leader_election.cc:302] T 98b3ee95a66e43788e5e2a428d9fa9c6 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:21.826306 14153 raft_consensus.cc:2781] T 98b3ee95a66e43788e5e2a428d9fa9c6 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:21.826407 14153 raft_consensus.cc:686] T 98b3ee95a66e43788e5e2a428d9fa9c6 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Becoming Leader. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Running, Role: LEADER
I20240503 02:00:21.826563 14153 consensus_queue.cc:237] T 98b3ee95a66e43788e5e2a428d9fa9c6 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:21.829187 6313 catalog_manager.cc:5484] T 98b3ee95a66e43788e5e2a428d9fa9c6 P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: term changed from 0 to 1, leader changed from <none> to dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129). New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } }
W20240503 02:00:21.864775 6523 consensus_peers.cc:485] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:21.866426 6523 consensus_peers.cc:485] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:21.879271 8982 consensus_peers.cc:485] T 1a7ee79312ea4bad816264375ca5bb2d P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:21.882658 8982 consensus_peers.cc:485] T beeea71f02ae4c89a1af13d3584ef70a P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:00:21.883751 6313 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:21.883920 6313 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:21.886737 6313 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet d22b0105836a43e09a8de16843474dfc
I20240503 02:00:21.887353 6436 tablet_service.cc:1514] Processing DeleteTablet for tablet d22b0105836a43e09a8de16843474dfc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:21 UTC) from {username='slave'} at 127.0.0.1:51338
I20240503 02:00:21.887411 9158 tablet_service.cc:1514] Processing DeleteTablet for tablet d22b0105836a43e09a8de16843474dfc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:21 UTC) from {username='slave'} at 127.0.0.1:51940
I20240503 02:00:21.887691 6700 tablet_service.cc:1514] Processing DeleteTablet for tablet d22b0105836a43e09a8de16843474dfc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:21 UTC) from {username='slave'} at 127.0.0.1:40336
I20240503 02:00:21.887740 14411 tablet_replica.cc:330] T d22b0105836a43e09a8de16843474dfc P 2a01ace80baf4057b6bf1995cf88468b: stopping tablet replica
I20240503 02:00:21.887691 14410 tablet_replica.cc:330] T d22b0105836a43e09a8de16843474dfc P dec3e4a04d604f7eaaa3f077d081218a: stopping tablet replica
I20240503 02:00:21.887877 14411 raft_consensus.cc:2229] T d22b0105836a43e09a8de16843474dfc P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:21.887935 14410 raft_consensus.cc:2229] T d22b0105836a43e09a8de16843474dfc P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:21.887991 14412 tablet_replica.cc:330] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025: stopping tablet replica
I20240503 02:00:21.888013 14411 raft_consensus.cc:2258] T d22b0105836a43e09a8de16843474dfc P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:21.888084 14410 raft_consensus.cc:2258] T d22b0105836a43e09a8de16843474dfc P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:21.888653 14411 ts_tablet_manager.cc:1875] T d22b0105836a43e09a8de16843474dfc P 2a01ace80baf4057b6bf1995cf88468b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:21.889735 14412 raft_consensus.cc:2229] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:21.889941 14410 ts_tablet_manager.cc:1875] T d22b0105836a43e09a8de16843474dfc P dec3e4a04d604f7eaaa3f077d081218a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:21.889999 14412 raft_consensus.cc:2258] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:21.890507 14412 ts_tablet_manager.cc:1875] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:21.890687 14411 ts_tablet_manager.cc:1888] T d22b0105836a43e09a8de16843474dfc P 2a01ace80baf4057b6bf1995cf88468b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:21.890800 14411 log.cc:1196] T d22b0105836a43e09a8de16843474dfc P 2a01ace80baf4057b6bf1995cf88468b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/d22b0105836a43e09a8de16843474dfc
I20240503 02:00:21.891059 14411 ts_tablet_manager.cc:1909] T d22b0105836a43e09a8de16843474dfc P 2a01ace80baf4057b6bf1995cf88468b: Deleting consensus metadata
I20240503 02:00:21.891878 6300 catalog_manager.cc:4830] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079): tablet d22b0105836a43e09a8de16843474dfc (table rebalancer_dynamic_table_new_name [id=1fe7d1ed4aee47f1ae55ec3c13859000]) successfully deleted
I20240503 02:00:21.892616 14410 ts_tablet_manager.cc:1888] T d22b0105836a43e09a8de16843474dfc P dec3e4a04d604f7eaaa3f077d081218a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:21.892704 14410 log.cc:1196] T d22b0105836a43e09a8de16843474dfc P dec3e4a04d604f7eaaa3f077d081218a: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/d22b0105836a43e09a8de16843474dfc
I20240503 02:00:21.892741 14412 ts_tablet_manager.cc:1888] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:21.892812 14412 log.cc:1196] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/d22b0105836a43e09a8de16843474dfc
I20240503 02:00:21.893128 14412 ts_tablet_manager.cc:1909] T d22b0105836a43e09a8de16843474dfc P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting consensus metadata
I20240503 02:00:21.893311 14410 ts_tablet_manager.cc:1909] T d22b0105836a43e09a8de16843474dfc P dec3e4a04d604f7eaaa3f077d081218a: Deleting consensus metadata
I20240503 02:00:21.893915 6303 catalog_manager.cc:4830] TS c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655): tablet d22b0105836a43e09a8de16843474dfc (table rebalancer_dynamic_table_new_name [id=1fe7d1ed4aee47f1ae55ec3c13859000]) successfully deleted
I20240503 02:00:21.894116 6300 catalog_manager.cc:4830] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): tablet d22b0105836a43e09a8de16843474dfc (table rebalancer_dynamic_table_new_name [id=1fe7d1ed4aee47f1ae55ec3c13859000]) successfully deleted
I20240503 02:00:21.988433 6313 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:00:21.993745 9158 tablet_service.cc:1467] Processing CreateTablet for tablet 00367fac98d94b809e5f788fbb49d4db (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6f878f01c35f4ae08eb12b7b12e4b350]), partition=
I20240503 02:00:21.993788 6829 tablet_service.cc:1467] Processing CreateTablet for tablet 00367fac98d94b809e5f788fbb49d4db (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6f878f01c35f4ae08eb12b7b12e4b350]), partition=
I20240503 02:00:21.993997 6436 tablet_service.cc:1467] Processing CreateTablet for tablet 00367fac98d94b809e5f788fbb49d4db (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6f878f01c35f4ae08eb12b7b12e4b350]), partition=
I20240503 02:00:21.994051 9158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00367fac98d94b809e5f788fbb49d4db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:21.994089 6829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00367fac98d94b809e5f788fbb49d4db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:21.994235 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00367fac98d94b809e5f788fbb49d4db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:21.996644 14374 tablet_bootstrap.cc:492] T 00367fac98d94b809e5f788fbb49d4db P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:21.997570 14374 tablet_bootstrap.cc:654] T 00367fac98d94b809e5f788fbb49d4db P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:21.997367 14415 tablet_bootstrap.cc:492] T 00367fac98d94b809e5f788fbb49d4db P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:21.998245 14414 tablet_bootstrap.cc:492] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:21.998982 14415 tablet_bootstrap.cc:654] T 00367fac98d94b809e5f788fbb49d4db P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:21.999210 14414 tablet_bootstrap.cc:654] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:22.000123 14374 tablet_bootstrap.cc:492] T 00367fac98d94b809e5f788fbb49d4db P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:22.000231 14374 ts_tablet_manager.cc:1367] T 00367fac98d94b809e5f788fbb49d4db P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:22.000905 14415 tablet_bootstrap.cc:492] T 00367fac98d94b809e5f788fbb49d4db P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:22.001008 14415 ts_tablet_manager.cc:1367] T 00367fac98d94b809e5f788fbb49d4db P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 02:00:22.001313 14374 raft_consensus.cc:348] T 00367fac98d94b809e5f788fbb49d4db P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:22.001380 14415 raft_consensus.cc:348] T 00367fac98d94b809e5f788fbb49d4db P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:22.001448 14374 raft_consensus.cc:374] T 00367fac98d94b809e5f788fbb49d4db P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:22.001472 14415 raft_consensus.cc:374] T 00367fac98d94b809e5f788fbb49d4db P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:22.001488 14374 raft_consensus.cc:729] T 00367fac98d94b809e5f788fbb49d4db P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:22.001500 14415 raft_consensus.cc:729] T 00367fac98d94b809e5f788fbb49d4db P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:22.001605 14374 consensus_queue.cc:260] T 00367fac98d94b809e5f788fbb49d4db P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:22.001605 14415 consensus_queue.cc:260] T 00367fac98d94b809e5f788fbb49d4db P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:22.001879 14415 ts_tablet_manager.cc:1398] T 00367fac98d94b809e5f788fbb49d4db P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:22.001890 14374 ts_tablet_manager.cc:1398] T 00367fac98d94b809e5f788fbb49d4db P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:00:22.002411 14414 tablet_bootstrap.cc:492] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:22.002504 14414 ts_tablet_manager.cc:1367] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:22.002993 14414 raft_consensus.cc:348] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:22.003093 14414 raft_consensus.cc:374] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:22.003120 14414 raft_consensus.cc:729] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:22.003276 14414 consensus_queue.cc:260] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:22.003569 14414 ts_tablet_manager.cc:1398] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:22.011693 13968 raft_consensus.cc:482] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:22.011806 13968 raft_consensus.cc:504] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:22.012257 13968 leader_election.cc:288] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 02:00:22.012698 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00367fac98d94b809e5f788fbb49d4db" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 02:00:22.012866 6849 raft_consensus.cc:2454] T 00367fac98d94b809e5f788fbb49d4db P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:22.012837 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00367fac98d94b809e5f788fbb49d4db" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 02:00:22.012961 6456 raft_consensus.cc:2454] T 00367fac98d94b809e5f788fbb49d4db P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:22.013255 9115 leader_election.cc:302] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 492448570cc94ed1879feb2ec3b9051d; no voters:
I20240503 02:00:22.013689 13968 raft_consensus.cc:2781] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:22.013746 13968 raft_consensus.cc:482] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:22.013790 13968 raft_consensus.cc:3037] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:22.014837 13968 raft_consensus.cc:504] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:22.015197 13968 leader_election.cc:288] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Requested vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 02:00:22.015566 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00367fac98d94b809e5f788fbb49d4db" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 02:00:22.015764 6456 raft_consensus.cc:3037] T 00367fac98d94b809e5f788fbb49d4db P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:22.016510 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00367fac98d94b809e5f788fbb49d4db" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 02:00:22.016633 6849 raft_consensus.cc:3037] T 00367fac98d94b809e5f788fbb49d4db P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:22.016814 6456 raft_consensus.cc:2454] T 00367fac98d94b809e5f788fbb49d4db P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:22.017591 6849 raft_consensus.cc:2454] T 00367fac98d94b809e5f788fbb49d4db P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:22.017938 9112 leader_election.cc:302] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:22.018115 13968 raft_consensus.cc:2781] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:22.018218 13968 raft_consensus.cc:686] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Becoming Leader. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Running, Role: LEADER
I20240503 02:00:22.018354 13968 consensus_queue.cc:237] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:22.021082 6313 catalog_manager.cc:5484] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135). New cstate: current_term: 1 leader_uuid: "2a01ace80baf4057b6bf1995cf88468b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } }
W20240503 02:00:22.076439 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:22.130957 6313 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:22.135102 6849 raft_consensus.cc:1261] T 00367fac98d94b809e5f788fbb49d4db P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer 2a01ace80baf4057b6bf1995cf88468b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:22.135102 6456 raft_consensus.cc:1261] T 00367fac98d94b809e5f788fbb49d4db P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 2a01ace80baf4057b6bf1995cf88468b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:22.135854 14413 consensus_queue.cc:1035] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:22.136165 14413 consensus_queue.cc:1035] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:22.138798 14385 tablet.cc:1722] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:22.139114 6309 catalog_manager.cc:5576] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719) does not have the latest schema for tablet 00367fac98d94b809e5f788fbb49d4db (table rebalancer_dynamic_table [id=6f878f01c35f4ae08eb12b7b12e4b350]). Expected version 1 got 0
I20240503 02:00:22.139153 14385 tablet.cc:1620] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
I20240503 02:00:22.139520 6308 catalog_manager.cc:5576] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093) does not have the latest schema for tablet 00367fac98d94b809e5f788fbb49d4db (table rebalancer_dynamic_table [id=6f878f01c35f4ae08eb12b7b12e4b350]). Expected version 1 got 0
I20240503 02:00:22.141737 14348 tablet.cc:1722] T 00367fac98d94b809e5f788fbb49d4db P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:22.142107 14348 tablet.cc:1620] T 00367fac98d94b809e5f788fbb49d4db P dec3e4a04d604f7eaaa3f077d081218a: MemRowSet was empty: no flush needed.
I20240503 02:00:22.143407 6309 catalog_manager.cc:5576] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093) does not have the latest schema for tablet 00367fac98d94b809e5f788fbb49d4db (table rebalancer_dynamic_table [id=6f878f01c35f4ae08eb12b7b12e4b350]). Expected version 1 got 0
I20240503 02:00:22.145233 14422 tablet.cc:1722] T 00367fac98d94b809e5f788fbb49d4db P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:22.145464 6308 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=6f878f01c35f4ae08eb12b7b12e4b350] alter complete (version 1)
I20240503 02:00:22.145747 14422 tablet.cc:1620] T 00367fac98d94b809e5f788fbb49d4db P 492448570cc94ed1879feb2ec3b9051d: MemRowSet was empty: no flush needed.
I20240503 02:00:22.146548 6313 catalog_manager.cc:5576] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719) does not have the latest schema for tablet 00367fac98d94b809e5f788fbb49d4db (table rebalancer_dynamic_table [id=6f878f01c35f4ae08eb12b7b12e4b350]). Expected version 1 got 0
I20240503 02:00:22.220109 14349 consensus_queue.cc:1035] T 98b3ee95a66e43788e5e2a428d9fa9c6 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:22.223747 14406 consensus_queue.cc:1035] T ea801f686a0a4efa9da66b24bdaa10cc P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:22.224721 14349 consensus_queue.cc:1035] T 98b3ee95a66e43788e5e2a428d9fa9c6 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:22.229391 14323 consensus_queue.cc:1035] T ea801f686a0a4efa9da66b24bdaa10cc P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:22.251166 6313 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:22.256884 14385 tablet.cc:1722] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:22.257285 14385 tablet.cc:1620] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
I20240503 02:00:22.257476 14422 tablet.cc:1722] T 00367fac98d94b809e5f788fbb49d4db P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:22.257774 14422 tablet.cc:1620] T 00367fac98d94b809e5f788fbb49d4db P 492448570cc94ed1879feb2ec3b9051d: MemRowSet was empty: no flush needed.
I20240503 02:00:22.258503 14348 tablet.cc:1722] T 00367fac98d94b809e5f788fbb49d4db P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:22.259081 14348 tablet.cc:1620] T 00367fac98d94b809e5f788fbb49d4db P dec3e4a04d604f7eaaa3f077d081218a: MemRowSet was empty: no flush needed.
I20240503 02:00:22.264219 6309 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=6f878f01c35f4ae08eb12b7b12e4b350] alter complete (version 2)
I20240503 02:00:22.326375 14323 consensus_queue.cc:1035] T 8a8b7b032eea4726b24b805236601450 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:22.329953 14433 consensus_queue.cc:1035] T 8a8b7b032eea4726b24b805236601450 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:22.339630 6309 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_extra_table_58"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:22.346544 9158 tablet_service.cc:1467] Processing CreateTablet for tablet 703b917cf3e74caaae52326e3604fb6e (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=65ca51846bbe433c9b20578b01d413e2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:22.346879 9158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 703b917cf3e74caaae52326e3604fb6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:22.347050 6829 tablet_service.cc:1467] Processing CreateTablet for tablet dc67e2e6fcdc45f39504cb4c9e0a02e1 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=65ca51846bbe433c9b20578b01d413e2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:22.347060 6569 tablet_service.cc:1467] Processing CreateTablet for tablet dc67e2e6fcdc45f39504cb4c9e0a02e1 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=65ca51846bbe433c9b20578b01d413e2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:22.347291 6829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc67e2e6fcdc45f39504cb4c9e0a02e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:22.347304 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc67e2e6fcdc45f39504cb4c9e0a02e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:22.349148 14415 tablet_bootstrap.cc:492] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:22.349474 6700 tablet_service.cc:1467] Processing CreateTablet for tablet 703b917cf3e74caaae52326e3604fb6e (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=65ca51846bbe433c9b20578b01d413e2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:22.349737 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 703b917cf3e74caaae52326e3604fb6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:22.350126 14415 tablet_bootstrap.cc:654] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:22.350256 14435 tablet_bootstrap.cc:492] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:22.351003 6436 tablet_service.cc:1467] Processing CreateTablet for tablet da5b87a8eaf54dc9b50140491d028cf1 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=65ca51846bbe433c9b20578b01d413e2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:22.351266 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da5b87a8eaf54dc9b50140491d028cf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:22.351282 14435 tablet_bootstrap.cc:654] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:22.351245 6699 tablet_service.cc:1467] Processing CreateTablet for tablet dc67e2e6fcdc45f39504cb4c9e0a02e1 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=65ca51846bbe433c9b20578b01d413e2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:22.351461 6699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc67e2e6fcdc45f39504cb4c9e0a02e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:22.352170 6568 tablet_service.cc:1467] Processing CreateTablet for tablet 703b917cf3e74caaae52326e3604fb6e (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=65ca51846bbe433c9b20578b01d413e2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:22.352423 6568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 703b917cf3e74caaae52326e3604fb6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:22.352749 14414 tablet_bootstrap.cc:492] T 703b917cf3e74caaae52326e3604fb6e P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:22.353520 14374 tablet_bootstrap.cc:492] T da5b87a8eaf54dc9b50140491d028cf1 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:22.353760 14414 tablet_bootstrap.cc:654] T 703b917cf3e74caaae52326e3604fb6e P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:22.354422 14374 tablet_bootstrap.cc:654] T da5b87a8eaf54dc9b50140491d028cf1 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:22.354694 6829 tablet_service.cc:1467] Processing CreateTablet for tablet da5b87a8eaf54dc9b50140491d028cf1 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=65ca51846bbe433c9b20578b01d413e2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:22.354959 6829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da5b87a8eaf54dc9b50140491d028cf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:22.355824 14414 tablet_bootstrap.cc:492] T 703b917cf3e74caaae52326e3604fb6e P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:22.355909 14414 ts_tablet_manager.cc:1367] T 703b917cf3e74caaae52326e3604fb6e P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:22.355947 14374 tablet_bootstrap.cc:492] T da5b87a8eaf54dc9b50140491d028cf1 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:22.355958 14415 tablet_bootstrap.cc:492] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:22.356029 14374 ts_tablet_manager.cc:1367] T da5b87a8eaf54dc9b50140491d028cf1 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:00:22.356035 14415 ts_tablet_manager.cc:1367] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:00:22.356420 14414 raft_consensus.cc:348] T 703b917cf3e74caaae52326e3604fb6e P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:22.356469 14415 raft_consensus.cc:348] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:22.356532 14414 raft_consensus.cc:374] T 703b917cf3e74caaae52326e3604fb6e P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:22.356563 14414 raft_consensus.cc:729] T 703b917cf3e74caaae52326e3604fb6e P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:22.356566 14415 raft_consensus.cc:374] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:22.356505 14374 raft_consensus.cc:348] T da5b87a8eaf54dc9b50140491d028cf1 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:22.356597 14415 raft_consensus.cc:729] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:22.356607 14374 raft_consensus.cc:374] T da5b87a8eaf54dc9b50140491d028cf1 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:22.356637 14374 raft_consensus.cc:729] T da5b87a8eaf54dc9b50140491d028cf1 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:22.356684 14414 consensus_queue.cc:260] T 703b917cf3e74caaae52326e3604fb6e P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:22.356709 14415 consensus_queue.cc:260] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:22.356765 14374 consensus_queue.cc:260] T da5b87a8eaf54dc9b50140491d028cf1 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:22.356983 14415 ts_tablet_manager.cc:1398] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:22.357045 14374 ts_tablet_manager.cc:1398] T da5b87a8eaf54dc9b50140491d028cf1 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:22.357137 14415 tablet_bootstrap.cc:492] T da5b87a8eaf54dc9b50140491d028cf1 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:22.358006 14414 ts_tablet_manager.cc:1398] T 703b917cf3e74caaae52326e3604fb6e P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:00:22.358326 14415 tablet_bootstrap.cc:654] T da5b87a8eaf54dc9b50140491d028cf1 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:22.361593 6699 tablet_service.cc:1467] Processing CreateTablet for tablet da5b87a8eaf54dc9b50140491d028cf1 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=65ca51846bbe433c9b20578b01d413e2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:22.361917 6699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da5b87a8eaf54dc9b50140491d028cf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:22.363004 14438 tablet_bootstrap.cc:492] T 703b917cf3e74caaae52326e3604fb6e P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:22.363950 14438 tablet_bootstrap.cc:654] T 703b917cf3e74caaae52326e3604fb6e P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:22.364939 14415 tablet_bootstrap.cc:492] T da5b87a8eaf54dc9b50140491d028cf1 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:22.365026 14415 ts_tablet_manager.cc:1367] T da5b87a8eaf54dc9b50140491d028cf1 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 02:00:22.365263 14438 tablet_bootstrap.cc:492] T 703b917cf3e74caaae52326e3604fb6e P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:22.365348 14438 ts_tablet_manager.cc:1367] T 703b917cf3e74caaae52326e3604fb6e P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:00:22.365396 14415 raft_consensus.cc:348] T da5b87a8eaf54dc9b50140491d028cf1 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:22.365501 14415 raft_consensus.cc:374] T da5b87a8eaf54dc9b50140491d028cf1 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:22.365532 14415 raft_consensus.cc:729] T da5b87a8eaf54dc9b50140491d028cf1 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:22.365675 14415 consensus_queue.cc:260] T da5b87a8eaf54dc9b50140491d028cf1 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:22.365778 14438 raft_consensus.cc:348] T 703b917cf3e74caaae52326e3604fb6e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:22.365906 14438 raft_consensus.cc:374] T 703b917cf3e74caaae52326e3604fb6e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:22.365937 14438 raft_consensus.cc:729] T 703b917cf3e74caaae52326e3604fb6e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:22.366264 14415 ts_tablet_manager.cc:1398] T da5b87a8eaf54dc9b50140491d028cf1 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:22.367265 14435 tablet_bootstrap.cc:492] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:22.367393 14435 ts_tablet_manager.cc:1367] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240503 02:00:22.366045 14438 consensus_queue.cc:260] T 703b917cf3e74caaae52326e3604fb6e P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:22.367897 14435 raft_consensus.cc:348] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:22.367969 14438 ts_tablet_manager.cc:1398] T 703b917cf3e74caaae52326e3604fb6e P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 02:00:22.368026 14435 raft_consensus.cc:374] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:22.368073 14435 raft_consensus.cc:729] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:22.368134 14438 tablet_bootstrap.cc:492] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:22.368202 14435 consensus_queue.cc:260] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:22.368515 14435 ts_tablet_manager.cc:1398] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:22.369280 6314 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:22.369447 14435 tablet_bootstrap.cc:492] T 703b917cf3e74caaae52326e3604fb6e P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:22.371304 14438 tablet_bootstrap.cc:654] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:22.374513 14435 tablet_bootstrap.cc:654] T 703b917cf3e74caaae52326e3604fb6e P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:22.376799 14435 tablet_bootstrap.cc:492] T 703b917cf3e74caaae52326e3604fb6e P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:22.376901 14435 ts_tablet_manager.cc:1367] T 703b917cf3e74caaae52326e3604fb6e P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:00:22.377386 14435 raft_consensus.cc:348] T 703b917cf3e74caaae52326e3604fb6e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:22.377501 14435 raft_consensus.cc:374] T 703b917cf3e74caaae52326e3604fb6e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:22.377532 14435 raft_consensus.cc:729] T 703b917cf3e74caaae52326e3604fb6e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:22.377638 14435 consensus_queue.cc:260] T 703b917cf3e74caaae52326e3604fb6e P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:22.377943 14435 ts_tablet_manager.cc:1398] T 703b917cf3e74caaae52326e3604fb6e P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:22.380088 14385 tablet.cc:1722] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:22.380546 14422 tablet.cc:1722] T 00367fac98d94b809e5f788fbb49d4db P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:22.380970 14348 tablet.cc:1722] T 00367fac98d94b809e5f788fbb49d4db P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:22.384896 14438 tablet_bootstrap.cc:492] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:22.385026 14438 ts_tablet_manager.cc:1367] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.002s
I20240503 02:00:22.385583 14438 raft_consensus.cc:348] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:22.385789 14438 raft_consensus.cc:374] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:22.385865 14438 raft_consensus.cc:729] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:22.386049 14438 consensus_queue.cc:260] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:22.386476 14438 ts_tablet_manager.cc:1398] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:22.386838 14438 tablet_bootstrap.cc:492] T da5b87a8eaf54dc9b50140491d028cf1 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:22.388350 14438 tablet_bootstrap.cc:654] T da5b87a8eaf54dc9b50140491d028cf1 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:22.391561 6312 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=6f878f01c35f4ae08eb12b7b12e4b350] alter complete (version 3)
I20240503 02:00:22.400965 14438 tablet_bootstrap.cc:492] T da5b87a8eaf54dc9b50140491d028cf1 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:22.401237 14438 ts_tablet_manager.cc:1367] T da5b87a8eaf54dc9b50140491d028cf1 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.003s
I20240503 02:00:22.402272 14438 raft_consensus.cc:348] T da5b87a8eaf54dc9b50140491d028cf1 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:22.402410 14438 raft_consensus.cc:374] T da5b87a8eaf54dc9b50140491d028cf1 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:22.402462 14438 raft_consensus.cc:729] T da5b87a8eaf54dc9b50140491d028cf1 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:22.402613 14438 consensus_queue.cc:260] T da5b87a8eaf54dc9b50140491d028cf1 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:22.402943 14438 ts_tablet_manager.cc:1398] T da5b87a8eaf54dc9b50140491d028cf1 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:22.467536 14386 raft_consensus.cc:482] T 703b917cf3e74caaae52326e3604fb6e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:22.467670 14386 raft_consensus.cc:504] T 703b917cf3e74caaae52326e3604fb6e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:22.468082 14386 leader_election.cc:288] T 703b917cf3e74caaae52326e3604fb6e P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:22.468497 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "703b917cf3e74caaae52326e3604fb6e" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:22.468499 6587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "703b917cf3e74caaae52326e3604fb6e" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 02:00:22.468627 6587 raft_consensus.cc:2454] T 703b917cf3e74caaae52326e3604fb6e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:22.468636 9178 raft_consensus.cc:2454] T 703b917cf3e74caaae52326e3604fb6e P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:22.468921 6655 leader_election.cc:302] T 703b917cf3e74caaae52326e3604fb6e P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:22.469081 14386 raft_consensus.cc:2781] T 703b917cf3e74caaae52326e3604fb6e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:22.469147 14386 raft_consensus.cc:482] T 703b917cf3e74caaae52326e3604fb6e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:22.469215 14386 raft_consensus.cc:3037] T 703b917cf3e74caaae52326e3604fb6e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:22.470139 14386 raft_consensus.cc:504] T 703b917cf3e74caaae52326e3604fb6e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:22.470499 14386 leader_election.cc:288] T 703b917cf3e74caaae52326e3604fb6e P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Requested vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:22.470889 6587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "703b917cf3e74caaae52326e3604fb6e" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 02:00:22.471021 6587 raft_consensus.cc:3037] T 703b917cf3e74caaae52326e3604fb6e P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:22.471800 6587 raft_consensus.cc:2454] T 703b917cf3e74caaae52326e3604fb6e P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:22.472136 6655 leader_election.cc:302] T 703b917cf3e74caaae52326e3604fb6e P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:22.472211 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "703b917cf3e74caaae52326e3604fb6e" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:22.472360 14386 raft_consensus.cc:2781] T 703b917cf3e74caaae52326e3604fb6e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:22.472368 9178 raft_consensus.cc:3037] T 703b917cf3e74caaae52326e3604fb6e P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:22.472468 14386 raft_consensus.cc:686] T 703b917cf3e74caaae52326e3604fb6e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Becoming Leader. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Running, Role: LEADER
I20240503 02:00:22.472630 14386 consensus_queue.cc:237] T 703b917cf3e74caaae52326e3604fb6e P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:22.473290 9178 raft_consensus.cc:2454] T 703b917cf3e74caaae52326e3604fb6e P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:22.476938 6312 catalog_manager.cc:5484] T 703b917cf3e74caaae52326e3604fb6e P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:22.500519 6312 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:22.500679 6312 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:22.502696 6312 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet 00367fac98d94b809e5f788fbb49d4db
I20240503 02:00:22.503379 6829 tablet_service.cc:1514] Processing DeleteTablet for tablet 00367fac98d94b809e5f788fbb49d4db with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:22 UTC) from {username='slave'} at 127.0.0.1:52812
I20240503 02:00:22.503378 9158 tablet_service.cc:1514] Processing DeleteTablet for tablet 00367fac98d94b809e5f788fbb49d4db with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:22 UTC) from {username='slave'} at 127.0.0.1:51940
I20240503 02:00:22.503460 6436 tablet_service.cc:1514] Processing DeleteTablet for tablet 00367fac98d94b809e5f788fbb49d4db with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:22 UTC) from {username='slave'} at 127.0.0.1:51338
I20240503 02:00:22.503727 14451 tablet_replica.cc:330] T 00367fac98d94b809e5f788fbb49d4db P 492448570cc94ed1879feb2ec3b9051d: stopping tablet replica
I20240503 02:00:22.503773 14453 tablet_replica.cc:330] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b: stopping tablet replica
I20240503 02:00:22.503882 14451 raft_consensus.cc:2229] T 00367fac98d94b809e5f788fbb49d4db P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:22.503916 14453 raft_consensus.cc:2229] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:22.504029 14451 raft_consensus.cc:2258] T 00367fac98d94b809e5f788fbb49d4db P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:22.504127 14453 raft_consensus.cc:2258] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:22.504405 14452 tablet_replica.cc:330] T 00367fac98d94b809e5f788fbb49d4db P dec3e4a04d604f7eaaa3f077d081218a: stopping tablet replica
I20240503 02:00:22.504552 14452 raft_consensus.cc:2229] T 00367fac98d94b809e5f788fbb49d4db P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:22.504680 14452 raft_consensus.cc:2258] T 00367fac98d94b809e5f788fbb49d4db P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:22.504748 14453 ts_tablet_manager.cc:1875] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:22.504909 14451 ts_tablet_manager.cc:1875] T 00367fac98d94b809e5f788fbb49d4db P 492448570cc94ed1879feb2ec3b9051d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:22.505292 14452 ts_tablet_manager.cc:1875] T 00367fac98d94b809e5f788fbb49d4db P dec3e4a04d604f7eaaa3f077d081218a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:22.507166 14451 ts_tablet_manager.cc:1888] T 00367fac98d94b809e5f788fbb49d4db P 492448570cc94ed1879feb2ec3b9051d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:22.507238 14453 ts_tablet_manager.cc:1888] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:22.507277 14451 log.cc:1196] T 00367fac98d94b809e5f788fbb49d4db P 492448570cc94ed1879feb2ec3b9051d: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/00367fac98d94b809e5f788fbb49d4db
I20240503 02:00:22.507364 14453 log.cc:1196] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/00367fac98d94b809e5f788fbb49d4db
I20240503 02:00:22.507433 14452 ts_tablet_manager.cc:1888] T 00367fac98d94b809e5f788fbb49d4db P dec3e4a04d604f7eaaa3f077d081218a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:22.507539 14452 log.cc:1196] T 00367fac98d94b809e5f788fbb49d4db P dec3e4a04d604f7eaaa3f077d081218a: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/00367fac98d94b809e5f788fbb49d4db
I20240503 02:00:22.507601 14451 ts_tablet_manager.cc:1909] T 00367fac98d94b809e5f788fbb49d4db P 492448570cc94ed1879feb2ec3b9051d: Deleting consensus metadata
I20240503 02:00:22.507644 14453 ts_tablet_manager.cc:1909] T 00367fac98d94b809e5f788fbb49d4db P 2a01ace80baf4057b6bf1995cf88468b: Deleting consensus metadata
I20240503 02:00:22.507853 14452 ts_tablet_manager.cc:1909] T 00367fac98d94b809e5f788fbb49d4db P dec3e4a04d604f7eaaa3f077d081218a: Deleting consensus metadata
I20240503 02:00:22.508337 6303 catalog_manager.cc:4830] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719): tablet 00367fac98d94b809e5f788fbb49d4db (table rebalancer_dynamic_table_new_name [id=6f878f01c35f4ae08eb12b7b12e4b350]) successfully deleted
I20240503 02:00:22.508347 6300 catalog_manager.cc:4830] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079): tablet 00367fac98d94b809e5f788fbb49d4db (table rebalancer_dynamic_table_new_name [id=6f878f01c35f4ae08eb12b7b12e4b350]) successfully deleted
I20240503 02:00:22.508718 6300 catalog_manager.cc:4830] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): tablet 00367fac98d94b809e5f788fbb49d4db (table rebalancer_dynamic_table_new_name [id=6f878f01c35f4ae08eb12b7b12e4b350]) successfully deleted
W20240503 02:00:22.540051 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:22.556813 14442 raft_consensus.cc:482] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:22.556965 14442 raft_consensus.cc:504] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:22.557428 14442 leader_election.cc:288] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:22.557865 6587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc67e2e6fcdc45f39504cb4c9e0a02e1" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 02:00:22.558001 6587 raft_consensus.cc:2454] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 02:00:22.558276 6786 leader_election.cc:302] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 80fd18ef4d604c86b909d5c008a46b4f; no voters:
I20240503 02:00:22.558441 14442 raft_consensus.cc:2781] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:22.558490 14442 raft_consensus.cc:482] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:22.558526 14442 raft_consensus.cc:3037] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:22.559592 6720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc67e2e6fcdc45f39504cb4c9e0a02e1" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 02:00:22.559621 14442 raft_consensus.cc:504] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:22.559734 6720 raft_consensus.cc:2454] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 02:00:22.560082 14442 leader_election.cc:288] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Requested vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:22.560590 6587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc67e2e6fcdc45f39504cb4c9e0a02e1" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 02:00:22.560705 6587 raft_consensus.cc:3037] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:22.560720 6720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc67e2e6fcdc45f39504cb4c9e0a02e1" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 02:00:22.560824 6720 raft_consensus.cc:3037] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:22.561704 6720 raft_consensus.cc:2454] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 02:00:22.561695 6587 raft_consensus.cc:2454] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 02:00:22.562004 6787 leader_election.cc:302] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:22.562330 14442 raft_consensus.cc:2781] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:22.562415 14442 raft_consensus.cc:686] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Becoming Leader. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Running, Role: LEADER
I20240503 02:00:22.562546 14442 consensus_queue.cc:237] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:22.565280 6312 catalog_manager.cc:5484] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 492448570cc94ed1879feb2ec3b9051d reported cstate change: term changed from 0 to 1, leader changed from <none> to 492448570cc94ed1879feb2ec3b9051d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:22.604314 6312 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:00:22.609862 6436 tablet_service.cc:1467] Processing CreateTablet for tablet 5976a8a1983f497eb0daa9c4fa9b701e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e890cb2b2c1049fdb59177f7670af7a3]), partition=
I20240503 02:00:22.610028 8880 tablet_service.cc:1467] Processing CreateTablet for tablet 5976a8a1983f497eb0daa9c4fa9b701e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e890cb2b2c1049fdb59177f7670af7a3]), partition=
I20240503 02:00:22.610188 6829 tablet_service.cc:1467] Processing CreateTablet for tablet 5976a8a1983f497eb0daa9c4fa9b701e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e890cb2b2c1049fdb59177f7670af7a3]), partition=
I20240503 02:00:22.610219 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5976a8a1983f497eb0daa9c4fa9b701e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:22.610280 8880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5976a8a1983f497eb0daa9c4fa9b701e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:22.610399 6829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5976a8a1983f497eb0daa9c4fa9b701e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:22.611972 14374 tablet_bootstrap.cc:492] T 5976a8a1983f497eb0daa9c4fa9b701e P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:22.612444 14415 tablet_bootstrap.cc:492] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:22.612941 14374 tablet_bootstrap.cc:654] T 5976a8a1983f497eb0daa9c4fa9b701e P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:22.613296 14415 tablet_bootstrap.cc:654] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:22.614115 14374 tablet_bootstrap.cc:492] T 5976a8a1983f497eb0daa9c4fa9b701e P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:22.614192 14374 ts_tablet_manager.cc:1367] T 5976a8a1983f497eb0daa9c4fa9b701e P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:22.614342 14415 tablet_bootstrap.cc:492] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:22.614424 14415 ts_tablet_manager.cc:1367] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:00:22.614591 14374 raft_consensus.cc:348] T 5976a8a1983f497eb0daa9c4fa9b701e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:22.614704 14374 raft_consensus.cc:374] T 5976a8a1983f497eb0daa9c4fa9b701e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:22.614727 14374 raft_consensus.cc:729] T 5976a8a1983f497eb0daa9c4fa9b701e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:22.614802 14415 raft_consensus.cc:348] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:22.614853 14374 consensus_queue.cc:260] T 5976a8a1983f497eb0daa9c4fa9b701e P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:22.614908 14415 raft_consensus.cc:374] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:22.614949 14415 raft_consensus.cc:729] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:22.615077 14415 consensus_queue.cc:260] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:22.615113 14374 ts_tablet_manager.cc:1398] T 5976a8a1983f497eb0daa9c4fa9b701e P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:22.615365 14415 ts_tablet_manager.cc:1398] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:22.616904 14455 tablet_bootstrap.cc:492] T 5976a8a1983f497eb0daa9c4fa9b701e P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:22.619096 14455 tablet_bootstrap.cc:654] T 5976a8a1983f497eb0daa9c4fa9b701e P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:22.620927 14455 tablet_bootstrap.cc:492] T 5976a8a1983f497eb0daa9c4fa9b701e P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:22.621063 14455 ts_tablet_manager.cc:1367] T 5976a8a1983f497eb0daa9c4fa9b701e P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:22.621505 14455 raft_consensus.cc:348] T 5976a8a1983f497eb0daa9c4fa9b701e P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:22.621594 14455 raft_consensus.cc:374] T 5976a8a1983f497eb0daa9c4fa9b701e P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:22.621614 14455 raft_consensus.cc:729] T 5976a8a1983f497eb0daa9c4fa9b701e P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:22.621707 14455 consensus_queue.cc:260] T 5976a8a1983f497eb0daa9c4fa9b701e P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:22.621982 14455 ts_tablet_manager.cc:1398] T 5976a8a1983f497eb0daa9c4fa9b701e P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:22.745640 14349 raft_consensus.cc:482] T da5b87a8eaf54dc9b50140491d028cf1 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:22.745792 14349 raft_consensus.cc:504] T da5b87a8eaf54dc9b50140491d028cf1 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:22.746253 14349 leader_election.cc:288] T da5b87a8eaf54dc9b50140491d028cf1 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:22.746614 6719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da5b87a8eaf54dc9b50140491d028cf1" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 02:00:22.746652 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da5b87a8eaf54dc9b50140491d028cf1" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 02:00:22.746757 6719 raft_consensus.cc:2454] T da5b87a8eaf54dc9b50140491d028cf1 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:22.746820 6851 raft_consensus.cc:2454] T da5b87a8eaf54dc9b50140491d028cf1 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:22.747016 6394 leader_election.cc:302] T da5b87a8eaf54dc9b50140491d028cf1 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2ac2eb7cc264047b9ab32fe3a4fc025, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:22.747192 14349 raft_consensus.cc:2781] T da5b87a8eaf54dc9b50140491d028cf1 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:22.747262 14349 raft_consensus.cc:482] T da5b87a8eaf54dc9b50140491d028cf1 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:22.747289 14349 raft_consensus.cc:3037] T da5b87a8eaf54dc9b50140491d028cf1 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:22.748224 14349 raft_consensus.cc:504] T da5b87a8eaf54dc9b50140491d028cf1 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:22.748593 14349 leader_election.cc:288] T da5b87a8eaf54dc9b50140491d028cf1 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Requested vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:22.748905 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da5b87a8eaf54dc9b50140491d028cf1" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 02:00:22.748931 6719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da5b87a8eaf54dc9b50140491d028cf1" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 02:00:22.749032 6849 raft_consensus.cc:3037] T da5b87a8eaf54dc9b50140491d028cf1 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:22.749068 6719 raft_consensus.cc:3037] T da5b87a8eaf54dc9b50140491d028cf1 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:22.749943 6849 raft_consensus.cc:2454] T da5b87a8eaf54dc9b50140491d028cf1 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:22.749964 6719 raft_consensus.cc:2454] T da5b87a8eaf54dc9b50140491d028cf1 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:22.750236 6394 leader_election.cc:302] T da5b87a8eaf54dc9b50140491d028cf1 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:22.750407 14349 raft_consensus.cc:2781] T da5b87a8eaf54dc9b50140491d028cf1 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:22.750494 14349 raft_consensus.cc:686] T da5b87a8eaf54dc9b50140491d028cf1 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Becoming Leader. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Running, Role: LEADER
I20240503 02:00:22.750619 14349 consensus_queue.cc:237] T da5b87a8eaf54dc9b50140491d028cf1 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:22.752977 6313 catalog_manager.cc:5484] T da5b87a8eaf54dc9b50140491d028cf1 P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: term changed from 0 to 1, leader changed from <none> to dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129). New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:22.911738 14324 raft_consensus.cc:482] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:22.911873 14324 raft_consensus.cc:504] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
W20240503 02:00:22.912256 8982 consensus_peers.cc:485] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:00:22.912292 14324 leader_election.cc:288] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:22.912669 8917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5976a8a1983f497eb0daa9c4fa9b701e" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:22.912804 8917 raft_consensus.cc:2454] T 5976a8a1983f497eb0daa9c4fa9b701e P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 02:00:22.913054 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5976a8a1983f497eb0daa9c4fa9b701e" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 02:00:22.913128 6787 leader_election.cc:302] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:22.913201 6458 raft_consensus.cc:2454] T 5976a8a1983f497eb0daa9c4fa9b701e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 02:00:22.913393 14324 raft_consensus.cc:2781] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:22.913455 14324 raft_consensus.cc:482] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:22.913504 14324 raft_consensus.cc:3037] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:22.914366 14324 raft_consensus.cc:504] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:22.914702 14324 leader_election.cc:288] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Requested vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:22.915158 8917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5976a8a1983f497eb0daa9c4fa9b701e" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:22.915287 8917 raft_consensus.cc:3037] T 5976a8a1983f497eb0daa9c4fa9b701e P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:22.916222 8917 raft_consensus.cc:2454] T 5976a8a1983f497eb0daa9c4fa9b701e P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 02:00:22.916777 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5976a8a1983f497eb0daa9c4fa9b701e" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 02:00:22.916911 6458 raft_consensus.cc:3037] T 5976a8a1983f497eb0daa9c4fa9b701e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:22.916973 6787 leader_election.cc:302] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:22.917174 14324 raft_consensus.cc:2781] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:22.917362 14324 raft_consensus.cc:686] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Becoming Leader. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Running, Role: LEADER
I20240503 02:00:22.917510 14324 consensus_queue.cc:237] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:22.917987 6458 raft_consensus.cc:2454] T 5976a8a1983f497eb0daa9c4fa9b701e P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 02:00:22.920694 6313 catalog_manager.cc:5484] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d reported cstate change: term changed from 0 to 1, leader changed from <none> to 492448570cc94ed1879feb2ec3b9051d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } }
W20240503 02:00:22.982370 6785 consensus_peers.cc:485] T c3485701bf3b48f5ae2e1568afeed133 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:23.039680 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:23.047782 14386 consensus_queue.cc:1035] T 703b917cf3e74caaae52326e3604fb6e P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:23.053880 14466 consensus_queue.cc:1035] T 703b917cf3e74caaae52326e3604fb6e P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:23.079519 6313 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:23.082118 14324 consensus_queue.cc:1035] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:23.086086 8916 raft_consensus.cc:1261] T 5976a8a1983f497eb0daa9c4fa9b701e P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:23.086541 14442 consensus_queue.cc:1035] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:23.090013 14475 tablet.cc:1722] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:23.090471 14475 tablet.cc:1620] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d: MemRowSet was empty: no flush needed.
I20240503 02:00:23.090801 14476 tablet.cc:1722] T 5976a8a1983f497eb0daa9c4fa9b701e P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:23.090999 6309 catalog_manager.cc:5576] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211) does not have the latest schema for tablet 5976a8a1983f497eb0daa9c4fa9b701e (table rebalancer_dynamic_table [id=e890cb2b2c1049fdb59177f7670af7a3]). Expected version 1 got 0
I20240503 02:00:23.091100 14476 tablet.cc:1620] T 5976a8a1983f497eb0daa9c4fa9b701e P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
I20240503 02:00:23.093607 6458 raft_consensus.cc:1261] T 5976a8a1983f497eb0daa9c4fa9b701e P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:23.094045 14472 consensus_queue.cc:1035] T dc67e2e6fcdc45f39504cb4c9e0a02e1 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:23.094404 14472 consensus_queue.cc:1035] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:23.096976 14481 tablet.cc:1722] T 5976a8a1983f497eb0daa9c4fa9b701e P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:23.097414 14481 tablet.cc:1620] T 5976a8a1983f497eb0daa9c4fa9b701e P dec3e4a04d604f7eaaa3f077d081218a: MemRowSet was empty: no flush needed.
I20240503 02:00:23.098665 6314 catalog_manager.cc:5576] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093) does not have the latest schema for tablet 5976a8a1983f497eb0daa9c4fa9b701e (table rebalancer_dynamic_table [id=e890cb2b2c1049fdb59177f7670af7a3]). Expected version 1 got 0
I20240503 02:00:23.099455 6308 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=e890cb2b2c1049fdb59177f7670af7a3] alter complete (version 1)
I20240503 02:00:23.203039 6314 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:23.208948 6314 catalog_manager.cc:5576] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719) does not have the latest schema for tablet 5976a8a1983f497eb0daa9c4fa9b701e (table rebalancer_dynamic_table [id=e890cb2b2c1049fdb59177f7670af7a3]). Expected version 2 got 1
I20240503 02:00:23.208880 14475 tablet.cc:1722] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:23.209586 14475 tablet.cc:1620] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d: MemRowSet was empty: no flush needed.
I20240503 02:00:23.209535 14476 tablet.cc:1722] T 5976a8a1983f497eb0daa9c4fa9b701e P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:23.209841 14476 tablet.cc:1620] T 5976a8a1983f497eb0daa9c4fa9b701e P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
I20240503 02:00:23.213714 14481 tablet.cc:1722] T 5976a8a1983f497eb0daa9c4fa9b701e P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:23.214062 14481 tablet.cc:1620] T 5976a8a1983f497eb0daa9c4fa9b701e P dec3e4a04d604f7eaaa3f077d081218a: MemRowSet was empty: no flush needed.
I20240503 02:00:23.215735 6309 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=e890cb2b2c1049fdb59177f7670af7a3] alter complete (version 2)
I20240503 02:00:23.233090 14349 consensus_queue.cc:1035] T da5b87a8eaf54dc9b50140491d028cf1 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:23.237303 14461 consensus_queue.cc:1035] T da5b87a8eaf54dc9b50140491d028cf1 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:00:23.259267 6785 consensus_peers.cc:485] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:00:23.318502 6309 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
W20240503 02:00:23.321789 6654 consensus_peers.cc:485] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:00:23.325131 14476 tablet.cc:1722] T 5976a8a1983f497eb0daa9c4fa9b701e P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:23.325740 14481 tablet.cc:1722] T 5976a8a1983f497eb0daa9c4fa9b701e P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:23.326153 14475 tablet.cc:1722] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:23.332213 6309 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=e890cb2b2c1049fdb59177f7670af7a3] alter complete (version 3)
I20240503 02:00:23.333287 6308 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_extra_table_60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:23.339680 6699 tablet_service.cc:1467] Processing CreateTablet for tablet 5d237bb82d7c47409a8924c5ebfd194b (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=6a1debe95ca94473b630c72d639b68c5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:23.339917 9027 tablet_service.cc:1467] Processing CreateTablet for tablet 5d237bb82d7c47409a8924c5ebfd194b (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=6a1debe95ca94473b630c72d639b68c5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:23.340041 6699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d237bb82d7c47409a8924c5ebfd194b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.340209 9027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d237bb82d7c47409a8924c5ebfd194b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.340479 9025 tablet_service.cc:1467] Processing CreateTablet for tablet 93d91773da7f403da2eb85808b686354 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=6a1debe95ca94473b630c72d639b68c5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:23.340701 9025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93d91773da7f403da2eb85808b686354. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.340922 9158 tablet_service.cc:1467] Processing CreateTablet for tablet 5d237bb82d7c47409a8924c5ebfd194b (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=6a1debe95ca94473b630c72d639b68c5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:23.341164 9158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d237bb82d7c47409a8924c5ebfd194b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.342845 9159 tablet_service.cc:1467] Processing CreateTablet for tablet 93d91773da7f403da2eb85808b686354 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=6a1debe95ca94473b630c72d639b68c5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:23.342850 9028 tablet_service.cc:1467] Processing CreateTablet for tablet fe9cd379022b49f797122c6727d77ef0 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=6a1debe95ca94473b630c72d639b68c5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:23.343062 9159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93d91773da7f403da2eb85808b686354. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.343087 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe9cd379022b49f797122c6727d77ef0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.343407 14486 tablet_bootstrap.cc:492] T 5d237bb82d7c47409a8924c5ebfd194b P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:23.343578 9157 tablet_service.cc:1467] Processing CreateTablet for tablet fe9cd379022b49f797122c6727d77ef0 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=6a1debe95ca94473b630c72d639b68c5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:23.343783 9157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe9cd379022b49f797122c6727d77ef0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.344470 14486 tablet_bootstrap.cc:654] T 5d237bb82d7c47409a8924c5ebfd194b P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.345908 6436 tablet_service.cc:1467] Processing CreateTablet for tablet 93d91773da7f403da2eb85808b686354 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=6a1debe95ca94473b630c72d639b68c5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:23.346199 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93d91773da7f403da2eb85808b686354. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.346182 6568 tablet_service.cc:1467] Processing CreateTablet for tablet fe9cd379022b49f797122c6727d77ef0 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=6a1debe95ca94473b630c72d639b68c5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:23.346436 6568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe9cd379022b49f797122c6727d77ef0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.347740 14486 tablet_bootstrap.cc:492] T 5d237bb82d7c47409a8924c5ebfd194b P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:23.347833 14486 ts_tablet_manager.cc:1367] T 5d237bb82d7c47409a8924c5ebfd194b P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240503 02:00:23.348371 14486 raft_consensus.cc:348] T 5d237bb82d7c47409a8924c5ebfd194b P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:23.348526 14486 raft_consensus.cc:374] T 5d237bb82d7c47409a8924c5ebfd194b P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.348552 14486 raft_consensus.cc:729] T 5d237bb82d7c47409a8924c5ebfd194b P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.348618 14488 tablet_bootstrap.cc:492] T fe9cd379022b49f797122c6727d77ef0 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:23.348697 14486 consensus_queue.cc:260] T 5d237bb82d7c47409a8924c5ebfd194b P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:23.349010 14486 ts_tablet_manager.cc:1398] T 5d237bb82d7c47409a8924c5ebfd194b P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 02:00:23.349217 14486 tablet_bootstrap.cc:492] T 93d91773da7f403da2eb85808b686354 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:23.349751 14488 tablet_bootstrap.cc:654] T fe9cd379022b49f797122c6727d77ef0 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.350128 14486 tablet_bootstrap.cc:654] T 93d91773da7f403da2eb85808b686354 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.350478 14484 tablet_bootstrap.cc:492] T 5d237bb82d7c47409a8924c5ebfd194b P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:23.351572 14486 tablet_bootstrap.cc:492] T 93d91773da7f403da2eb85808b686354 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:23.351565 14484 tablet_bootstrap.cc:654] T 5d237bb82d7c47409a8924c5ebfd194b P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.351656 14486 ts_tablet_manager.cc:1367] T 93d91773da7f403da2eb85808b686354 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:00:23.352129 14486 raft_consensus.cc:348] T 93d91773da7f403da2eb85808b686354 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:23.352254 14486 raft_consensus.cc:374] T 93d91773da7f403da2eb85808b686354 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.352285 14486 raft_consensus.cc:729] T 93d91773da7f403da2eb85808b686354 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.352742 14486 consensus_queue.cc:260] T 93d91773da7f403da2eb85808b686354 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:23.352819 14484 tablet_bootstrap.cc:492] T 5d237bb82d7c47409a8924c5ebfd194b P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:23.352916 14484 ts_tablet_manager.cc:1367] T 5d237bb82d7c47409a8924c5ebfd194b P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:23.353052 14486 ts_tablet_manager.cc:1398] T 93d91773da7f403da2eb85808b686354 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:23.353201 14486 tablet_bootstrap.cc:492] T fe9cd379022b49f797122c6727d77ef0 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:23.353514 14484 raft_consensus.cc:348] T 5d237bb82d7c47409a8924c5ebfd194b P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:23.353628 14484 raft_consensus.cc:374] T 5d237bb82d7c47409a8924c5ebfd194b P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.353660 14484 raft_consensus.cc:729] T 5d237bb82d7c47409a8924c5ebfd194b P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.353781 14484 consensus_queue.cc:260] T 5d237bb82d7c47409a8924c5ebfd194b P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:23.354072 14484 ts_tablet_manager.cc:1398] T 5d237bb82d7c47409a8924c5ebfd194b P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:23.354140 14489 tablet_bootstrap.cc:492] T 93d91773da7f403da2eb85808b686354 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:23.354250 14484 tablet_bootstrap.cc:492] T 93d91773da7f403da2eb85808b686354 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:23.354336 14486 tablet_bootstrap.cc:654] T fe9cd379022b49f797122c6727d77ef0 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.355206 14484 tablet_bootstrap.cc:654] T 93d91773da7f403da2eb85808b686354 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.355671 14488 tablet_bootstrap.cc:492] T fe9cd379022b49f797122c6727d77ef0 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:23.355756 14488 ts_tablet_manager.cc:1367] T fe9cd379022b49f797122c6727d77ef0 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:00:23.356338 14488 raft_consensus.cc:348] T fe9cd379022b49f797122c6727d77ef0 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:23.356500 14488 raft_consensus.cc:374] T fe9cd379022b49f797122c6727d77ef0 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.356534 14488 raft_consensus.cc:729] T fe9cd379022b49f797122c6727d77ef0 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.356669 14488 consensus_queue.cc:260] T fe9cd379022b49f797122c6727d77ef0 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:23.356961 14488 ts_tablet_manager.cc:1398] T fe9cd379022b49f797122c6727d77ef0 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:23.357304 14484 tablet_bootstrap.cc:492] T 93d91773da7f403da2eb85808b686354 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:23.357461 14484 ts_tablet_manager.cc:1367] T 93d91773da7f403da2eb85808b686354 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:23.357650 14486 tablet_bootstrap.cc:492] T fe9cd379022b49f797122c6727d77ef0 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:23.357761 14486 ts_tablet_manager.cc:1367] T fe9cd379022b49f797122c6727d77ef0 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:00:23.358175 14484 raft_consensus.cc:348] T 93d91773da7f403da2eb85808b686354 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:23.358240 14486 raft_consensus.cc:348] T fe9cd379022b49f797122c6727d77ef0 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:23.358312 14484 raft_consensus.cc:374] T 93d91773da7f403da2eb85808b686354 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.358347 14484 raft_consensus.cc:729] T 93d91773da7f403da2eb85808b686354 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.358377 14486 raft_consensus.cc:374] T fe9cd379022b49f797122c6727d77ef0 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.358425 14486 raft_consensus.cc:729] T fe9cd379022b49f797122c6727d77ef0 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.358484 14484 consensus_queue.cc:260] T 93d91773da7f403da2eb85808b686354 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:23.358587 14486 consensus_queue.cc:260] T fe9cd379022b49f797122c6727d77ef0 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:23.358956 14486 ts_tablet_manager.cc:1398] T fe9cd379022b49f797122c6727d77ef0 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:23.359037 14484 ts_tablet_manager.cc:1398] T 93d91773da7f403da2eb85808b686354 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:23.359241 14484 tablet_bootstrap.cc:492] T fe9cd379022b49f797122c6727d77ef0 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:23.362273 14484 tablet_bootstrap.cc:654] T fe9cd379022b49f797122c6727d77ef0 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.363654 14484 tablet_bootstrap.cc:492] T fe9cd379022b49f797122c6727d77ef0 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:23.363755 14484 ts_tablet_manager.cc:1367] T fe9cd379022b49f797122c6727d77ef0 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:00:23.363996 14489 tablet_bootstrap.cc:654] T 93d91773da7f403da2eb85808b686354 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.364250 14484 raft_consensus.cc:348] T fe9cd379022b49f797122c6727d77ef0 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:23.364491 14484 raft_consensus.cc:374] T fe9cd379022b49f797122c6727d77ef0 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.364573 14484 raft_consensus.cc:729] T fe9cd379022b49f797122c6727d77ef0 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.364786 14484 consensus_queue.cc:260] T fe9cd379022b49f797122c6727d77ef0 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:23.365316 14485 tablet_bootstrap.cc:492] T 5d237bb82d7c47409a8924c5ebfd194b P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:23.366088 14484 ts_tablet_manager.cc:1398] T fe9cd379022b49f797122c6727d77ef0 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:23.366617 14485 tablet_bootstrap.cc:654] T 5d237bb82d7c47409a8924c5ebfd194b P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.368256 14485 tablet_bootstrap.cc:492] T 5d237bb82d7c47409a8924c5ebfd194b P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:23.368541 14485 ts_tablet_manager.cc:1367] T 5d237bb82d7c47409a8924c5ebfd194b P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:23.369058 14485 raft_consensus.cc:348] T 5d237bb82d7c47409a8924c5ebfd194b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:23.369799 14489 tablet_bootstrap.cc:492] T 93d91773da7f403da2eb85808b686354 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:23.369923 14489 ts_tablet_manager.cc:1367] T 93d91773da7f403da2eb85808b686354 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20240503 02:00:23.370432 14489 raft_consensus.cc:348] T 93d91773da7f403da2eb85808b686354 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:23.370565 14489 raft_consensus.cc:374] T 93d91773da7f403da2eb85808b686354 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.370607 14489 raft_consensus.cc:729] T 93d91773da7f403da2eb85808b686354 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.370769 14489 consensus_queue.cc:260] T 93d91773da7f403da2eb85808b686354 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:23.371244 14489 ts_tablet_manager.cc:1398] T 93d91773da7f403da2eb85808b686354 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 02:00:23.369172 14485 raft_consensus.cc:374] T 5d237bb82d7c47409a8924c5ebfd194b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.371398 14485 raft_consensus.cc:729] T 5d237bb82d7c47409a8924c5ebfd194b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.371752 14485 consensus_queue.cc:260] T 5d237bb82d7c47409a8924c5ebfd194b P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:23.372222 14485 ts_tablet_manager.cc:1398] T 5d237bb82d7c47409a8924c5ebfd194b P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 02:00:23.374706 14345 raft_consensus.cc:482] T 5d237bb82d7c47409a8924c5ebfd194b P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:23.374819 14345 raft_consensus.cc:504] T 5d237bb82d7c47409a8924c5ebfd194b P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:23.375289 14345 leader_election.cc:288] T 5d237bb82d7c47409a8924c5ebfd194b P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:23.376149 6719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d237bb82d7c47409a8924c5ebfd194b" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 02:00:23.376276 6719 raft_consensus.cc:2454] T 5d237bb82d7c47409a8924c5ebfd194b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:23.376574 9115 leader_election.cc:302] T 5d237bb82d7c47409a8924c5ebfd194b P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:23.376761 14413 raft_consensus.cc:2781] T 5d237bb82d7c47409a8924c5ebfd194b P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:23.376806 14413 raft_consensus.cc:482] T 5d237bb82d7c47409a8924c5ebfd194b P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:23.376838 14413 raft_consensus.cc:3037] T 5d237bb82d7c47409a8924c5ebfd194b P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:23.378238 14413 raft_consensus.cc:504] T 5d237bb82d7c47409a8924c5ebfd194b P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:23.378715 14413 leader_election.cc:288] T 5d237bb82d7c47409a8924c5ebfd194b P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Requested vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:23.379585 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d237bb82d7c47409a8924c5ebfd194b" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:23.379750 9048 raft_consensus.cc:3037] T 5d237bb82d7c47409a8924c5ebfd194b P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:23.379601 9047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d237bb82d7c47409a8924c5ebfd194b" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:23.380661 6719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d237bb82d7c47409a8924c5ebfd194b" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 02:00:23.380785 6719 raft_consensus.cc:3037] T 5d237bb82d7c47409a8924c5ebfd194b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:23.380895 9048 raft_consensus.cc:2454] T 5d237bb82d7c47409a8924c5ebfd194b P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:23.381425 9113 leader_election.cc:302] T 5d237bb82d7c47409a8924c5ebfd194b P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:23.381664 14413 raft_consensus.cc:2781] T 5d237bb82d7c47409a8924c5ebfd194b P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:23.381686 6719 raft_consensus.cc:2454] T 5d237bb82d7c47409a8924c5ebfd194b P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:23.381738 14413 raft_consensus.cc:686] T 5d237bb82d7c47409a8924c5ebfd194b P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Becoming Leader. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Running, Role: LEADER
I20240503 02:00:23.381861 14413 consensus_queue.cc:237] T 5d237bb82d7c47409a8924c5ebfd194b P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:23.384683 6309 catalog_manager.cc:5484] T 5d237bb82d7c47409a8924c5ebfd194b P 2a01ace80baf4057b6bf1995cf88468b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135). New cstate: current_term: 1 leader_uuid: "2a01ace80baf4057b6bf1995cf88468b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:23.434433 6309 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:23.434618 6309 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:23.436852 6309 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet 5976a8a1983f497eb0daa9c4fa9b701e
I20240503 02:00:23.437623 8880 tablet_service.cc:1514] Processing DeleteTablet for tablet 5976a8a1983f497eb0daa9c4fa9b701e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:23 UTC) from {username='slave'} at 127.0.0.1:40296
I20240503 02:00:23.437600 6436 tablet_service.cc:1514] Processing DeleteTablet for tablet 5976a8a1983f497eb0daa9c4fa9b701e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:23 UTC) from {username='slave'} at 127.0.0.1:51338
I20240503 02:00:23.438073 14500 tablet_replica.cc:330] T 5976a8a1983f497eb0daa9c4fa9b701e P dec3e4a04d604f7eaaa3f077d081218a: stopping tablet replica
I20240503 02:00:23.438199 6829 tablet_service.cc:1514] Processing DeleteTablet for tablet 5976a8a1983f497eb0daa9c4fa9b701e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:23 UTC) from {username='slave'} at 127.0.0.1:52812
I20240503 02:00:23.438493 14501 tablet_replica.cc:330] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d: stopping tablet replica
I20240503 02:00:23.438510 14500 raft_consensus.cc:2229] T 5976a8a1983f497eb0daa9c4fa9b701e P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:23.438649 14500 raft_consensus.cc:2258] T 5976a8a1983f497eb0daa9c4fa9b701e P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:23.438663 14501 raft_consensus.cc:2229] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:23.438879 14501 raft_consensus.cc:2258] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:23.439419 14500 ts_tablet_manager.cc:1875] T 5976a8a1983f497eb0daa9c4fa9b701e P dec3e4a04d604f7eaaa3f077d081218a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:23.439425 14501 ts_tablet_manager.cc:1875] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:23.441532 14500 ts_tablet_manager.cc:1888] T 5976a8a1983f497eb0daa9c4fa9b701e P dec3e4a04d604f7eaaa3f077d081218a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:23.441606 14500 log.cc:1196] T 5976a8a1983f497eb0daa9c4fa9b701e P dec3e4a04d604f7eaaa3f077d081218a: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/5976a8a1983f497eb0daa9c4fa9b701e
I20240503 02:00:23.441839 14500 ts_tablet_manager.cc:1909] T 5976a8a1983f497eb0daa9c4fa9b701e P dec3e4a04d604f7eaaa3f077d081218a: Deleting consensus metadata
I20240503 02:00:23.442047 14499 tablet_replica.cc:330] T 5976a8a1983f497eb0daa9c4fa9b701e P 9942306855404cfd891adf227dd65eb2: stopping tablet replica
I20240503 02:00:23.442209 14499 raft_consensus.cc:2229] T 5976a8a1983f497eb0daa9c4fa9b701e P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:23.442296 14501 ts_tablet_manager.cc:1888] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:23.442337 14499 raft_consensus.cc:2258] T 5976a8a1983f497eb0daa9c4fa9b701e P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:23.442379 14501 log.cc:1196] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/5976a8a1983f497eb0daa9c4fa9b701e
I20240503 02:00:23.442692 14501 ts_tablet_manager.cc:1909] T 5976a8a1983f497eb0daa9c4fa9b701e P 492448570cc94ed1879feb2ec3b9051d: Deleting consensus metadata
I20240503 02:00:23.442979 6300 catalog_manager.cc:4830] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): tablet 5976a8a1983f497eb0daa9c4fa9b701e (table rebalancer_dynamic_table_new_name [id=e890cb2b2c1049fdb59177f7670af7a3]) successfully deleted
I20240503 02:00:23.443430 14499 ts_tablet_manager.cc:1875] T 5976a8a1983f497eb0daa9c4fa9b701e P 9942306855404cfd891adf227dd65eb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:23.444291 6303 catalog_manager.cc:4830] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719): tablet 5976a8a1983f497eb0daa9c4fa9b701e (table rebalancer_dynamic_table_new_name [id=e890cb2b2c1049fdb59177f7670af7a3]) successfully deleted
I20240503 02:00:23.445396 14499 ts_tablet_manager.cc:1888] T 5976a8a1983f497eb0daa9c4fa9b701e P 9942306855404cfd891adf227dd65eb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:23.445472 14499 log.cc:1196] T 5976a8a1983f497eb0daa9c4fa9b701e P 9942306855404cfd891adf227dd65eb2: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/5976a8a1983f497eb0daa9c4fa9b701e
I20240503 02:00:23.445650 14499 ts_tablet_manager.cc:1909] T 5976a8a1983f497eb0daa9c4fa9b701e P 9942306855404cfd891adf227dd65eb2: Deleting consensus metadata
I20240503 02:00:23.446475 6303 catalog_manager.cc:4830] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211): tablet 5976a8a1983f497eb0daa9c4fa9b701e (table rebalancer_dynamic_table_new_name [id=e890cb2b2c1049fdb59177f7670af7a3]) successfully deleted
W20240503 02:00:23.476244 6654 consensus_peers.cc:485] T ba17d4d3e0754b2798eb95484d37229a P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:23.486716 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:23.539091 6309 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:00:23.544031 14462 raft_consensus.cc:482] T fe9cd379022b49f797122c6727d77ef0 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:23.544173 14462 raft_consensus.cc:504] T fe9cd379022b49f797122c6727d77ef0 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:23.544749 14462 leader_election.cc:288] T fe9cd379022b49f797122c6727d77ef0 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:23.544801 6436 tablet_service.cc:1467] Processing CreateTablet for tablet 0401cac53a9c44ec83de9643aac3e5dc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9a306ff2611d406e912ebbeffa5b62a4]), partition=
I20240503 02:00:23.544801 6568 tablet_service.cc:1467] Processing CreateTablet for tablet 0401cac53a9c44ec83de9643aac3e5dc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9a306ff2611d406e912ebbeffa5b62a4]), partition=
I20240503 02:00:23.545064 6568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0401cac53a9c44ec83de9643aac3e5dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.545078 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0401cac53a9c44ec83de9643aac3e5dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.545508 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9cd379022b49f797122c6727d77ef0" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:23.545647 9048 raft_consensus.cc:2454] T fe9cd379022b49f797122c6727d77ef0 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:23.546041 6523 leader_election.cc:302] T fe9cd379022b49f797122c6727d77ef0 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:23.546350 14462 raft_consensus.cc:2781] T fe9cd379022b49f797122c6727d77ef0 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:23.546407 14462 raft_consensus.cc:482] T fe9cd379022b49f797122c6727d77ef0 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:23.546442 14462 raft_consensus.cc:3037] T fe9cd379022b49f797122c6727d77ef0 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:23.547252 14488 tablet_bootstrap.cc:492] T 0401cac53a9c44ec83de9643aac3e5dc P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:23.547449 14489 tablet_bootstrap.cc:492] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:23.547405 14462 raft_consensus.cc:504] T fe9cd379022b49f797122c6727d77ef0 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:23.547813 14462 leader_election.cc:288] T fe9cd379022b49f797122c6727d77ef0 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Requested vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:23.547987 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9cd379022b49f797122c6727d77ef0" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:23.548121 9178 raft_consensus.cc:2454] T fe9cd379022b49f797122c6727d77ef0 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:23.548415 14489 tablet_bootstrap.cc:654] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.548414 14488 tablet_bootstrap.cc:654] T 0401cac53a9c44ec83de9643aac3e5dc P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.548460 14431 raft_consensus.cc:482] T 93d91773da7f403da2eb85808b686354 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:23.548558 14431 raft_consensus.cc:504] T 93d91773da7f403da2eb85808b686354 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:23.548763 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9cd379022b49f797122c6727d77ef0" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:23.548856 9178 raft_consensus.cc:3037] T fe9cd379022b49f797122c6727d77ef0 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:23.549084 14431 leader_election.cc:288] T 93d91773da7f403da2eb85808b686354 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 02:00:23.549547 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d91773da7f403da2eb85808b686354" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 02:00:23.549674 6458 raft_consensus.cc:2454] T 93d91773da7f403da2eb85808b686354 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:23.549861 14488 tablet_bootstrap.cc:492] T 0401cac53a9c44ec83de9643aac3e5dc P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:23.549885 9178 raft_consensus.cc:2454] T fe9cd379022b49f797122c6727d77ef0 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:23.549939 14488 ts_tablet_manager.cc:1367] T 0401cac53a9c44ec83de9643aac3e5dc P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:00:23.550065 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d91773da7f403da2eb85808b686354" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:23.550159 9179 raft_consensus.cc:2454] T 93d91773da7f403da2eb85808b686354 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:23.550422 8981 leader_election.cc:302] T 93d91773da7f403da2eb85808b686354 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:23.550397 14488 raft_consensus.cc:348] T 0401cac53a9c44ec83de9643aac3e5dc P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:23.550491 14488 raft_consensus.cc:374] T 0401cac53a9c44ec83de9643aac3e5dc P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.550506 6522 leader_election.cc:302] T fe9cd379022b49f797122c6727d77ef0 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 80fd18ef4d604c86b909d5c008a46b4f; no voters:
I20240503 02:00:23.550518 14488 raft_consensus.cc:729] T 0401cac53a9c44ec83de9643aac3e5dc P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.550674 14488 consensus_queue.cc:260] T 0401cac53a9c44ec83de9643aac3e5dc P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:23.550944 14406 raft_consensus.cc:2781] T fe9cd379022b49f797122c6727d77ef0 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:23.551012 14406 raft_consensus.cc:686] T fe9cd379022b49f797122c6727d77ef0 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Becoming Leader. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Running, Role: LEADER
I20240503 02:00:23.551041 14488 ts_tablet_manager.cc:1398] T 0401cac53a9c44ec83de9643aac3e5dc P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:23.551028 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9cd379022b49f797122c6727d77ef0" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:23.551126 9048 raft_consensus.cc:3037] T fe9cd379022b49f797122c6727d77ef0 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:23.551132 14406 consensus_queue.cc:237] T fe9cd379022b49f797122c6727d77ef0 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:23.552160 9048 raft_consensus.cc:2454] T fe9cd379022b49f797122c6727d77ef0 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:23.552503 14195 raft_consensus.cc:2781] T 93d91773da7f403da2eb85808b686354 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:23.552564 14195 raft_consensus.cc:482] T 93d91773da7f403da2eb85808b686354 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:23.552600 14195 raft_consensus.cc:3037] T 93d91773da7f403da2eb85808b686354 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:23.553265 14489 tablet_bootstrap.cc:492] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:23.553375 14489 ts_tablet_manager.cc:1367] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20240503 02:00:23.553619 14195 raft_consensus.cc:504] T 93d91773da7f403da2eb85808b686354 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:23.553905 14489 raft_consensus.cc:348] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:23.553982 14195 leader_election.cc:288] T 93d91773da7f403da2eb85808b686354 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Requested vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 02:00:23.554010 14489 raft_consensus.cc:374] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.554040 14489 raft_consensus.cc:729] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.554158 14489 consensus_queue.cc:260] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:23.554774 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d91773da7f403da2eb85808b686354" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 02:00:23.554811 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d91773da7f403da2eb85808b686354" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:23.554877 6458 raft_consensus.cc:3037] T 93d91773da7f403da2eb85808b686354 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:23.554773 6309 catalog_manager.cc:5484] T fe9cd379022b49f797122c6727d77ef0 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:23.554924 9179 raft_consensus.cc:3037] T 93d91773da7f403da2eb85808b686354 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:23.555881 6458 raft_consensus.cc:2454] T 93d91773da7f403da2eb85808b686354 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:23.555922 9179 raft_consensus.cc:2454] T 93d91773da7f403da2eb85808b686354 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:23.556036 14489 ts_tablet_manager.cc:1398] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 02:00:23.556397 8981 leader_election.cc:302] T 93d91773da7f403da2eb85808b686354 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:23.557044 14195 raft_consensus.cc:2781] T 93d91773da7f403da2eb85808b686354 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:23.557139 14195 raft_consensus.cc:686] T 93d91773da7f403da2eb85808b686354 P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Becoming Leader. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Running, Role: LEADER
I20240503 02:00:23.557297 14195 consensus_queue.cc:237] T 93d91773da7f403da2eb85808b686354 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:23.559700 8880 tablet_service.cc:1467] Processing CreateTablet for tablet 0401cac53a9c44ec83de9643aac3e5dc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9a306ff2611d406e912ebbeffa5b62a4]), partition=
I20240503 02:00:23.559989 8880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0401cac53a9c44ec83de9643aac3e5dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:23.561163 6313 catalog_manager.cc:5484] T 93d91773da7f403da2eb85808b686354 P 99ead02915954b4fab23b033ad6d546b reported cstate change: term changed from 0 to 1, leader changed from <none> to 99ead02915954b4fab23b033ad6d546b (127.0.99.134). New cstate: current_term: 1 leader_uuid: "99ead02915954b4fab23b033ad6d546b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:23.562724 14507 tablet_bootstrap.cc:492] T 0401cac53a9c44ec83de9643aac3e5dc P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:23.564548 14507 tablet_bootstrap.cc:654] T 0401cac53a9c44ec83de9643aac3e5dc P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:23.568220 14507 tablet_bootstrap.cc:492] T 0401cac53a9c44ec83de9643aac3e5dc P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:23.568337 14507 ts_tablet_manager.cc:1367] T 0401cac53a9c44ec83de9643aac3e5dc P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20240503 02:00:23.568982 14507 raft_consensus.cc:348] T 0401cac53a9c44ec83de9643aac3e5dc P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:23.569124 14507 raft_consensus.cc:374] T 0401cac53a9c44ec83de9643aac3e5dc P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:23.569165 14507 raft_consensus.cc:729] T 0401cac53a9c44ec83de9643aac3e5dc P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:23.569291 14507 consensus_queue.cc:260] T 0401cac53a9c44ec83de9643aac3e5dc P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:23.569618 14507 ts_tablet_manager.cc:1398] T 0401cac53a9c44ec83de9643aac3e5dc P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240503 02:00:23.619670 9113 consensus_peers.cc:485] T f44b59277c3b47adbd4dd6aff9350af1 P 2a01ace80baf4057b6bf1995cf88468b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:23.630760 6785 consensus_peers.cc:485] T 8246488dec204bf6b09e868d7951ac5a P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:23.675060 6523 consensus_peers.cc:485] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:23.687184 8851 consensus_peers.cc:485] T 067f54011a37432d99157884e70fc601 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:23.709378 6523 consensus_peers.cc:485] T d0d5dd42db3e4883af7fd16d8813b550 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:00:23.789397 14461 raft_consensus.cc:482] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:23.789559 14461 raft_consensus.cc:504] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:23.790028 14461 leader_election.cc:288] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:23.790549 6587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0401cac53a9c44ec83de9643aac3e5dc" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 02:00:23.790549 8917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0401cac53a9c44ec83de9643aac3e5dc" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:23.790701 8917 raft_consensus.cc:2454] T 0401cac53a9c44ec83de9643aac3e5dc P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:23.790802 6587 raft_consensus.cc:2454] T 0401cac53a9c44ec83de9643aac3e5dc P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:23.791139 6393 leader_election.cc:302] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:23.791316 14461 raft_consensus.cc:2781] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:23.791369 14461 raft_consensus.cc:482] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:23.791404 14461 raft_consensus.cc:3037] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:23.792290 14461 raft_consensus.cc:504] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:23.792698 14461 leader_election.cc:288] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Requested vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:23.793038 8917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0401cac53a9c44ec83de9643aac3e5dc" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:23.793083 6587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0401cac53a9c44ec83de9643aac3e5dc" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 02:00:23.793197 6587 raft_consensus.cc:3037] T 0401cac53a9c44ec83de9643aac3e5dc P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:23.793503 8917 raft_consensus.cc:3037] T 0401cac53a9c44ec83de9643aac3e5dc P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:00:23.794085 6785 consensus_peers.cc:485] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:00:23.794266 6587 raft_consensus.cc:2454] T 0401cac53a9c44ec83de9643aac3e5dc P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:23.794610 6393 leader_election.cc:302] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:23.794760 8917 raft_consensus.cc:2454] T 0401cac53a9c44ec83de9643aac3e5dc P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:23.794806 14461 raft_consensus.cc:2781] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:23.794879 14461 raft_consensus.cc:686] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Becoming Leader. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Running, Role: LEADER
I20240503 02:00:23.795030 14461 consensus_queue.cc:237] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:23.797606 6308 catalog_manager.cc:5484] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: term changed from 0 to 1, leader changed from <none> to dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129). New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:23.800735 14448 consensus_queue.cc:1035] T 5d237bb82d7c47409a8924c5ebfd194b P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:23.804241 14448 consensus_queue.cc:1035] T 5d237bb82d7c47409a8924c5ebfd194b P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:00:23.819082 6523 consensus_peers.cc:485] T 925612b660ec471da5b3c548802ede6e P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:23.821225 8982 consensus_peers.cc:485] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:23.838507 6785 consensus_peers.cc:485] T d897c21e630d4770832bb10f43778e82 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:23.870110 8851 consensus_peers.cc:485] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:23.896935 9113 consensus_peers.cc:485] T b1740b877c1e4372a0337d99f5bcab0e P 2a01ace80baf4057b6bf1995cf88468b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:23.900218 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:23.956315 6308 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:23.960568 8917 raft_consensus.cc:1261] T 0401cac53a9c44ec83de9643aac3e5dc P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:23.960568 6587 raft_consensus.cc:1261] T 0401cac53a9c44ec83de9643aac3e5dc P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:23.961046 14515 consensus_queue.cc:1035] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:23.961239 14461 consensus_queue.cc:1035] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:23.964609 6308 catalog_manager.cc:5576] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863) does not have the latest schema for tablet 0401cac53a9c44ec83de9643aac3e5dc (table rebalancer_dynamic_table [id=9a306ff2611d406e912ebbeffa5b62a4]). Expected version 1 got 0
I20240503 02:00:23.965258 6309 catalog_manager.cc:5576] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211) does not have the latest schema for tablet 0401cac53a9c44ec83de9643aac3e5dc (table rebalancer_dynamic_table [id=9a306ff2611d406e912ebbeffa5b62a4]). Expected version 1 got 0
I20240503 02:00:23.967518 6309 catalog_manager.cc:5576] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211) does not have the latest schema for tablet 0401cac53a9c44ec83de9643aac3e5dc (table rebalancer_dynamic_table [id=9a306ff2611d406e912ebbeffa5b62a4]). Expected version 1 got 0
I20240503 02:00:23.969583 14522 tablet.cc:1722] T 0401cac53a9c44ec83de9643aac3e5dc P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:23.969975 14522 tablet.cc:1620] T 0401cac53a9c44ec83de9643aac3e5dc P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
I20240503 02:00:23.970383 14521 tablet.cc:1722] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:23.970609 6308 catalog_manager.cc:5576] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093) does not have the latest schema for tablet 0401cac53a9c44ec83de9643aac3e5dc (table rebalancer_dynamic_table [id=9a306ff2611d406e912ebbeffa5b62a4]). Expected version 1 got 0
I20240503 02:00:23.970736 14521 tablet.cc:1620] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a: MemRowSet was empty: no flush needed.
W20240503 02:00:23.972445 6523 consensus_peers.cc:485] T 4385d51d11574192883d01376e2d5620 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:00:23.977926 6313 catalog_manager.cc:5576] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863) does not have the latest schema for tablet 0401cac53a9c44ec83de9643aac3e5dc (table rebalancer_dynamic_table [id=9a306ff2611d406e912ebbeffa5b62a4]). Expected version 1 got 0
I20240503 02:00:23.978349 6309 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=9a306ff2611d406e912ebbeffa5b62a4] alter complete (version 1)
I20240503 02:00:23.984700 14523 tablet.cc:1722] T 0401cac53a9c44ec83de9643aac3e5dc P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:23.985059 14523 tablet.cc:1620] T 0401cac53a9c44ec83de9643aac3e5dc P 80fd18ef4d604c86b909d5c008a46b4f: MemRowSet was empty: no flush needed.
W20240503 02:00:23.998529 6654 consensus_peers.cc:485] T cea1ea50fa434d31b42451f84f26bc76 P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:24.013710 6785 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111) [suppressed 62 similar messages]
I20240503 02:00:24.016319 14195 consensus_queue.cc:1035] T 93d91773da7f403da2eb85808b686354 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:24.020464 14195 consensus_queue.cc:1035] T 93d91773da7f403da2eb85808b686354 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:00:24.036712 8851 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 02:00:24.037757 8851 consensus_peers.cc:485] T f28586e70bd9473fbdc6b9126b666fa9 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:24.039000 6785 consensus_peers.cc:485] T 46ad540deb624a25b97d3b967fe1f6f4 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:00:24.051651 14462 consensus_queue.cc:1035] T fe9cd379022b49f797122c6727d77ef0 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:24.055063 14504 consensus_queue.cc:1035] T fe9cd379022b49f797122c6727d77ef0 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:24.083456 6309 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:24.088938 14521 tablet.cc:1722] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:24.089311 14521 tablet.cc:1620] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a: MemRowSet was empty: no flush needed.
I20240503 02:00:24.089442 14522 tablet.cc:1722] T 0401cac53a9c44ec83de9643aac3e5dc P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:24.089752 14522 tablet.cc:1620] T 0401cac53a9c44ec83de9643aac3e5dc P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
I20240503 02:00:24.092604 14523 tablet.cc:1722] T 0401cac53a9c44ec83de9643aac3e5dc P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:24.092957 14523 tablet.cc:1620] T 0401cac53a9c44ec83de9643aac3e5dc P 80fd18ef4d604c86b909d5c008a46b4f: MemRowSet was empty: no flush needed.
I20240503 02:00:24.096447 6309 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=9a306ff2611d406e912ebbeffa5b62a4] alter complete (version 2)
I20240503 02:00:24.106464 6313 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_extra_table_62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:24.113404 6699 tablet_service.cc:1467] Processing CreateTablet for tablet 4fb6646299c54e4da1938ea864806be4 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=aa1ce1266eff43659598655cc5166948]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:24.113644 6829 tablet_service.cc:1467] Processing CreateTablet for tablet 4fb6646299c54e4da1938ea864806be4 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=aa1ce1266eff43659598655cc5166948]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:24.113682 6700 tablet_service.cc:1467] Processing CreateTablet for tablet cdb766efec08440aa6a296edb76f84b7 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=aa1ce1266eff43659598655cc5166948]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:24.113744 6699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fb6646299c54e4da1938ea864806be4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.113907 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdb766efec08440aa6a296edb76f84b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.113934 6829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fb6646299c54e4da1938ea864806be4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.116191 14528 tablet_bootstrap.cc:492] T 4fb6646299c54e4da1938ea864806be4 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:24.116449 8880 tablet_service.cc:1467] Processing CreateTablet for tablet cdb766efec08440aa6a296edb76f84b7 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=aa1ce1266eff43659598655cc5166948]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:24.116453 9157 tablet_service.cc:1467] Processing CreateTablet for tablet 4fb6646299c54e4da1938ea864806be4 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=aa1ce1266eff43659598655cc5166948]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:24.116744 9157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fb6646299c54e4da1938ea864806be4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.116744 8880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdb766efec08440aa6a296edb76f84b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.117334 14528 tablet_bootstrap.cc:654] T 4fb6646299c54e4da1938ea864806be4 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.117558 6568 tablet_service.cc:1467] Processing CreateTablet for tablet cdb766efec08440aa6a296edb76f84b7 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=aa1ce1266eff43659598655cc5166948]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:24.117830 6568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdb766efec08440aa6a296edb76f84b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.120173 14531 tablet_bootstrap.cc:492] T cdb766efec08440aa6a296edb76f84b7 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
W20240503 02:00:24.120415 6654 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240503 02:00:24.121366 14531 tablet_bootstrap.cc:654] T cdb766efec08440aa6a296edb76f84b7 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.121922 14528 tablet_bootstrap.cc:492] T 4fb6646299c54e4da1938ea864806be4 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:24.122035 14528 ts_tablet_manager.cc:1367] T 4fb6646299c54e4da1938ea864806be4 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
W20240503 02:00:24.122462 6523 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111) [suppressed 58 similar messages]
I20240503 02:00:24.122737 14528 raft_consensus.cc:348] T 4fb6646299c54e4da1938ea864806be4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:24.122949 14528 raft_consensus.cc:374] T 4fb6646299c54e4da1938ea864806be4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.122988 14528 raft_consensus.cc:729] T 4fb6646299c54e4da1938ea864806be4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.123126 14528 consensus_queue.cc:260] T 4fb6646299c54e4da1938ea864806be4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:24.123423 14528 ts_tablet_manager.cc:1398] T 4fb6646299c54e4da1938ea864806be4 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:24.123591 14528 tablet_bootstrap.cc:492] T cdb766efec08440aa6a296edb76f84b7 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:24.124711 14531 tablet_bootstrap.cc:492] T cdb766efec08440aa6a296edb76f84b7 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:24.124897 14531 ts_tablet_manager.cc:1367] T cdb766efec08440aa6a296edb76f84b7 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20240503 02:00:24.125788 14528 tablet_bootstrap.cc:654] T cdb766efec08440aa6a296edb76f84b7 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.126482 6700 tablet_service.cc:1467] Processing CreateTablet for tablet f55cb3da9c874387a65eff5de0903331 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=aa1ce1266eff43659598655cc5166948]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:24.126756 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f55cb3da9c874387a65eff5de0903331. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.126904 14532 tablet_bootstrap.cc:492] T cdb766efec08440aa6a296edb76f84b7 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:24.127275 14528 tablet_bootstrap.cc:492] T cdb766efec08440aa6a296edb76f84b7 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:24.127344 14528 ts_tablet_manager.cc:1367] T cdb766efec08440aa6a296edb76f84b7 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:24.128365 14528 raft_consensus.cc:348] T cdb766efec08440aa6a296edb76f84b7 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:24.128499 14528 raft_consensus.cc:374] T cdb766efec08440aa6a296edb76f84b7 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.128526 14528 raft_consensus.cc:729] T cdb766efec08440aa6a296edb76f84b7 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.128643 14528 consensus_queue.cc:260] T cdb766efec08440aa6a296edb76f84b7 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:24.128943 14528 ts_tablet_manager.cc:1398] T cdb766efec08440aa6a296edb76f84b7 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:00:24.129088 14528 tablet_bootstrap.cc:492] T f55cb3da9c874387a65eff5de0903331 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:24.129560 8880 tablet_service.cc:1467] Processing CreateTablet for tablet f55cb3da9c874387a65eff5de0903331 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=aa1ce1266eff43659598655cc5166948]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:24.129820 8880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f55cb3da9c874387a65eff5de0903331. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.129990 14528 tablet_bootstrap.cc:654] T f55cb3da9c874387a65eff5de0903331 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.125470 14531 raft_consensus.cc:348] T cdb766efec08440aa6a296edb76f84b7 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:24.130043 14531 raft_consensus.cc:374] T cdb766efec08440aa6a296edb76f84b7 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.130100 14531 raft_consensus.cc:729] T cdb766efec08440aa6a296edb76f84b7 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.130254 14531 consensus_queue.cc:260] T cdb766efec08440aa6a296edb76f84b7 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:24.132155 14529 tablet_bootstrap.cc:492] T 4fb6646299c54e4da1938ea864806be4 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:24.132647 14532 tablet_bootstrap.cc:654] T cdb766efec08440aa6a296edb76f84b7 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.132755 9028 tablet_service.cc:1467] Processing CreateTablet for tablet f55cb3da9c874387a65eff5de0903331 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=aa1ce1266eff43659598655cc5166948]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:24.133037 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f55cb3da9c874387a65eff5de0903331. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.132966 14531 ts_tablet_manager.cc:1398] T cdb766efec08440aa6a296edb76f84b7 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.008s user 0.000s sys 0.001s
I20240503 02:00:24.133363 14531 tablet_bootstrap.cc:492] T f55cb3da9c874387a65eff5de0903331 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:24.134406 14534 tablet_bootstrap.cc:492] T 4fb6646299c54e4da1938ea864806be4 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:24.135797 14534 tablet_bootstrap.cc:654] T 4fb6646299c54e4da1938ea864806be4 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.135890 14531 tablet_bootstrap.cc:654] T f55cb3da9c874387a65eff5de0903331 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.137274 14529 tablet_bootstrap.cc:654] T 4fb6646299c54e4da1938ea864806be4 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.137817 14531 tablet_bootstrap.cc:492] T f55cb3da9c874387a65eff5de0903331 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:24.137956 14534 tablet_bootstrap.cc:492] T 4fb6646299c54e4da1938ea864806be4 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:24.138037 14531 ts_tablet_manager.cc:1367] T f55cb3da9c874387a65eff5de0903331 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20240503 02:00:24.138068 14534 ts_tablet_manager.cc:1367] T 4fb6646299c54e4da1938ea864806be4 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:24.138561 14534 raft_consensus.cc:348] T 4fb6646299c54e4da1938ea864806be4 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:24.138685 14534 raft_consensus.cc:374] T 4fb6646299c54e4da1938ea864806be4 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.138721 14534 raft_consensus.cc:729] T 4fb6646299c54e4da1938ea864806be4 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.138721 14531 raft_consensus.cc:348] T f55cb3da9c874387a65eff5de0903331 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:24.138841 14531 raft_consensus.cc:374] T f55cb3da9c874387a65eff5de0903331 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.138835 14534 consensus_queue.cc:260] T 4fb6646299c54e4da1938ea864806be4 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:24.138885 14531 raft_consensus.cc:729] T f55cb3da9c874387a65eff5de0903331 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.139071 14531 consensus_queue.cc:260] T f55cb3da9c874387a65eff5de0903331 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:24.139125 14534 ts_tablet_manager.cc:1398] T 4fb6646299c54e4da1938ea864806be4 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:24.142235 14531 ts_tablet_manager.cc:1398] T f55cb3da9c874387a65eff5de0903331 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240503 02:00:24.142473 14537 tablet_bootstrap.cc:492] T f55cb3da9c874387a65eff5de0903331 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:24.142402 14529 tablet_bootstrap.cc:492] T 4fb6646299c54e4da1938ea864806be4 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:24.143380 14529 ts_tablet_manager.cc:1367] T 4fb6646299c54e4da1938ea864806be4 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 02:00:24.143718 14537 tablet_bootstrap.cc:654] T f55cb3da9c874387a65eff5de0903331 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.145359 14529 raft_consensus.cc:348] T 4fb6646299c54e4da1938ea864806be4 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:24.145503 14529 raft_consensus.cc:374] T 4fb6646299c54e4da1938ea864806be4 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.145536 14529 raft_consensus.cc:729] T 4fb6646299c54e4da1938ea864806be4 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.145709 14529 consensus_queue.cc:260] T 4fb6646299c54e4da1938ea864806be4 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:24.145789 14532 tablet_bootstrap.cc:492] T cdb766efec08440aa6a296edb76f84b7 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:24.145787 14528 tablet_bootstrap.cc:492] T f55cb3da9c874387a65eff5de0903331 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:24.145879 14528 ts_tablet_manager.cc:1367] T f55cb3da9c874387a65eff5de0903331 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.002s
I20240503 02:00:24.145880 14532 ts_tablet_manager.cc:1367] T cdb766efec08440aa6a296edb76f84b7 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240503 02:00:24.146024 14529 ts_tablet_manager.cc:1398] T 4fb6646299c54e4da1938ea864806be4 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:24.146342 14528 raft_consensus.cc:348] T f55cb3da9c874387a65eff5de0903331 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:24.146448 14528 raft_consensus.cc:374] T f55cb3da9c874387a65eff5de0903331 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.146384 14532 raft_consensus.cc:348] T cdb766efec08440aa6a296edb76f84b7 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:24.146476 14528 raft_consensus.cc:729] T f55cb3da9c874387a65eff5de0903331 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.146503 14532 raft_consensus.cc:374] T cdb766efec08440aa6a296edb76f84b7 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.146528 14532 raft_consensus.cc:729] T cdb766efec08440aa6a296edb76f84b7 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.146597 14528 consensus_queue.cc:260] T f55cb3da9c874387a65eff5de0903331 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:24.146868 14528 ts_tablet_manager.cc:1398] T f55cb3da9c874387a65eff5de0903331 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:24.147204 14532 consensus_queue.cc:260] T cdb766efec08440aa6a296edb76f84b7 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:24.147773 14537 tablet_bootstrap.cc:492] T f55cb3da9c874387a65eff5de0903331 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:24.147899 14537 ts_tablet_manager.cc:1367] T f55cb3da9c874387a65eff5de0903331 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:00:24.148608 14537 raft_consensus.cc:348] T f55cb3da9c874387a65eff5de0903331 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:24.148784 14537 raft_consensus.cc:374] T f55cb3da9c874387a65eff5de0903331 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.148835 14537 raft_consensus.cc:729] T f55cb3da9c874387a65eff5de0903331 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.149012 14537 consensus_queue.cc:260] T f55cb3da9c874387a65eff5de0903331 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:24.149355 14537 ts_tablet_manager.cc:1398] T f55cb3da9c874387a65eff5de0903331 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:24.150733 14532 ts_tablet_manager.cc:1398] T cdb766efec08440aa6a296edb76f84b7 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
W20240503 02:00:24.157666 6523 consensus_peers.cc:485] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:00:24.166128 14459 raft_consensus.cc:482] T f55cb3da9c874387a65eff5de0903331 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:24.166278 14459 raft_consensus.cc:504] T f55cb3da9c874387a65eff5de0903331 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:24.166747 14459 leader_election.cc:288] T f55cb3da9c874387a65eff5de0903331 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:24.167201 6719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55cb3da9c874387a65eff5de0903331" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 02:00:24.167290 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55cb3da9c874387a65eff5de0903331" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:24.167338 6719 raft_consensus.cc:2454] T f55cb3da9c874387a65eff5de0903331 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 0.
I20240503 02:00:24.167428 9048 raft_consensus.cc:2454] T f55cb3da9c874387a65eff5de0903331 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 0.
I20240503 02:00:24.167706 8851 leader_election.cc:302] T f55cb3da9c874387a65eff5de0903331 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9942306855404cfd891adf227dd65eb2, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:24.167892 14459 raft_consensus.cc:2781] T f55cb3da9c874387a65eff5de0903331 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:24.167958 14459 raft_consensus.cc:482] T f55cb3da9c874387a65eff5de0903331 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:24.167990 14459 raft_consensus.cc:3037] T f55cb3da9c874387a65eff5de0903331 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:24.169148 14459 raft_consensus.cc:504] T f55cb3da9c874387a65eff5de0903331 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:24.169987 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55cb3da9c874387a65eff5de0903331" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:24.170068 6719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55cb3da9c874387a65eff5de0903331" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 02:00:24.170127 9048 raft_consensus.cc:3037] T f55cb3da9c874387a65eff5de0903331 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:24.170202 6719 raft_consensus.cc:3037] T f55cb3da9c874387a65eff5de0903331 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:24.171115 6719 raft_consensus.cc:2454] T f55cb3da9c874387a65eff5de0903331 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 1.
I20240503 02:00:24.171168 9048 raft_consensus.cc:2454] T f55cb3da9c874387a65eff5de0903331 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 1.
I20240503 02:00:24.169526 14459 leader_election.cc:288] T f55cb3da9c874387a65eff5de0903331 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 election: Requested vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:24.171603 8853 leader_election.cc:302] T f55cb3da9c874387a65eff5de0903331 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9942306855404cfd891adf227dd65eb2, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:24.171787 14459 raft_consensus.cc:2781] T f55cb3da9c874387a65eff5de0903331 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:24.171994 14459 raft_consensus.cc:686] T f55cb3da9c874387a65eff5de0903331 P 9942306855404cfd891adf227dd65eb2 [term 1 LEADER]: Becoming Leader. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Running, Role: LEADER
I20240503 02:00:24.172160 14459 consensus_queue.cc:237] T f55cb3da9c874387a65eff5de0903331 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:24.175002 6314 catalog_manager.cc:5484] T f55cb3da9c874387a65eff5de0903331 P 9942306855404cfd891adf227dd65eb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9942306855404cfd891adf227dd65eb2 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "9942306855404cfd891adf227dd65eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: HEALTHY } } }
W20240503 02:00:24.193240 8982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240503 02:00:24.193980 8982 consensus_peers.cc:485] T 13756a6e44814fe4b09a6b0f087546c3 P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:00:24.202240 6314 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
W20240503 02:00:24.208055 8851 consensus_peers.cc:485] T 5f51fe5f799f4e479b62f5917a2c2606 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:00:24.208961 14521 tablet.cc:1722] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:24.209756 14523 tablet.cc:1722] T 0401cac53a9c44ec83de9643aac3e5dc P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:24.213809 14522 tablet.cc:1722] T 0401cac53a9c44ec83de9643aac3e5dc P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:24.217094 6314 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=9a306ff2611d406e912ebbeffa5b62a4] alter complete (version 3)
I20240503 02:00:24.318459 6308 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:24.318615 6308 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:24.320868 6308 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet 0401cac53a9c44ec83de9643aac3e5dc
I20240503 02:00:24.321520 6436 tablet_service.cc:1514] Processing DeleteTablet for tablet 0401cac53a9c44ec83de9643aac3e5dc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:24 UTC) from {username='slave'} at 127.0.0.1:51338
I20240503 02:00:24.321599 6568 tablet_service.cc:1514] Processing DeleteTablet for tablet 0401cac53a9c44ec83de9643aac3e5dc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:24 UTC) from {username='slave'} at 127.0.0.1:59846
I20240503 02:00:24.321875 14544 tablet_replica.cc:330] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a: stopping tablet replica
I20240503 02:00:24.321897 8880 tablet_service.cc:1514] Processing DeleteTablet for tablet 0401cac53a9c44ec83de9643aac3e5dc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:24 UTC) from {username='slave'} at 127.0.0.1:40296
I20240503 02:00:24.321955 14545 tablet_replica.cc:330] T 0401cac53a9c44ec83de9643aac3e5dc P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 02:00:24.322124 14545 raft_consensus.cc:2229] T 0401cac53a9c44ec83de9643aac3e5dc P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:24.322181 14544 raft_consensus.cc:2229] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:24.322260 14546 tablet_replica.cc:330] T 0401cac53a9c44ec83de9643aac3e5dc P 9942306855404cfd891adf227dd65eb2: stopping tablet replica
I20240503 02:00:24.322299 14545 raft_consensus.cc:2258] T 0401cac53a9c44ec83de9643aac3e5dc P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:24.322415 14546 raft_consensus.cc:2229] T 0401cac53a9c44ec83de9643aac3e5dc P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:24.322486 14544 raft_consensus.cc:2258] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:24.322551 14546 raft_consensus.cc:2258] T 0401cac53a9c44ec83de9643aac3e5dc P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:24.322978 14545 ts_tablet_manager.cc:1875] T 0401cac53a9c44ec83de9643aac3e5dc P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:24.323065 14544 ts_tablet_manager.cc:1875] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:24.323340 14546 ts_tablet_manager.cc:1875] T 0401cac53a9c44ec83de9643aac3e5dc P 9942306855404cfd891adf227dd65eb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:24.325343 14544 ts_tablet_manager.cc:1888] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:24.325429 14544 log.cc:1196] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/0401cac53a9c44ec83de9643aac3e5dc
I20240503 02:00:24.325492 14545 ts_tablet_manager.cc:1888] T 0401cac53a9c44ec83de9643aac3e5dc P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:24.325587 14545 log.cc:1196] T 0401cac53a9c44ec83de9643aac3e5dc P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/0401cac53a9c44ec83de9643aac3e5dc
I20240503 02:00:24.325755 14544 ts_tablet_manager.cc:1909] T 0401cac53a9c44ec83de9643aac3e5dc P dec3e4a04d604f7eaaa3f077d081218a: Deleting consensus metadata
I20240503 02:00:24.325757 14546 ts_tablet_manager.cc:1888] T 0401cac53a9c44ec83de9643aac3e5dc P 9942306855404cfd891adf227dd65eb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:24.325839 14546 log.cc:1196] T 0401cac53a9c44ec83de9643aac3e5dc P 9942306855404cfd891adf227dd65eb2: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/0401cac53a9c44ec83de9643aac3e5dc
I20240503 02:00:24.325961 14545 ts_tablet_manager.cc:1909] T 0401cac53a9c44ec83de9643aac3e5dc P 80fd18ef4d604c86b909d5c008a46b4f: Deleting consensus metadata
I20240503 02:00:24.326115 14546 ts_tablet_manager.cc:1909] T 0401cac53a9c44ec83de9643aac3e5dc P 9942306855404cfd891adf227dd65eb2: Deleting consensus metadata
I20240503 02:00:24.326673 6300 catalog_manager.cc:4830] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): tablet 0401cac53a9c44ec83de9643aac3e5dc (table rebalancer_dynamic_table_new_name [id=9a306ff2611d406e912ebbeffa5b62a4]) successfully deleted
I20240503 02:00:24.327013 6303 catalog_manager.cc:4830] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211): tablet 0401cac53a9c44ec83de9643aac3e5dc (table rebalancer_dynamic_table_new_name [id=9a306ff2611d406e912ebbeffa5b62a4]) successfully deleted
I20240503 02:00:24.327819 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet 0401cac53a9c44ec83de9643aac3e5dc (table rebalancer_dynamic_table_new_name [id=9a306ff2611d406e912ebbeffa5b62a4]) successfully deleted
W20240503 02:00:24.344172 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
W20240503 02:00:24.386687 8851 consensus_peers.cc:485] T 7da8f614a70e4177b70e7c2f31d22a10 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:24.400113 8982 consensus_peers.cc:485] T beeea71f02ae4c89a1af13d3584ef70a P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:24.401465 8982 consensus_peers.cc:485] T 1a7ee79312ea4bad816264375ca5bb2d P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:24.422132 9113 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240503 02:00:24.423074 6308 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:00:24.428803 9157 tablet_service.cc:1467] Processing CreateTablet for tablet ecaa46386dee476492bae905d05b3dad (DEFAULT_TABLE table=rebalancer_dynamic_table [id=eb2282ddd2e94eadb12f14d01b283020]), partition=
I20240503 02:00:24.428828 9028 tablet_service.cc:1467] Processing CreateTablet for tablet ecaa46386dee476492bae905d05b3dad (DEFAULT_TABLE table=rebalancer_dynamic_table [id=eb2282ddd2e94eadb12f14d01b283020]), partition=
I20240503 02:00:24.429109 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecaa46386dee476492bae905d05b3dad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.429131 9157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecaa46386dee476492bae905d05b3dad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.430341 14472 raft_consensus.cc:482] T 4fb6646299c54e4da1938ea864806be4 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:24.430464 14472 raft_consensus.cc:504] T 4fb6646299c54e4da1938ea864806be4 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:24.431411 14472 leader_election.cc:288] T 4fb6646299c54e4da1938ea864806be4 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:24.431550 6829 tablet_service.cc:1467] Processing CreateTablet for tablet ecaa46386dee476492bae905d05b3dad (DEFAULT_TABLE table=rebalancer_dynamic_table [id=eb2282ddd2e94eadb12f14d01b283020]), partition=
I20240503 02:00:24.431777 6829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecaa46386dee476492bae905d05b3dad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:24.431951 6719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fb6646299c54e4da1938ea864806be4" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 02:00:24.432123 6719 raft_consensus.cc:2454] T 4fb6646299c54e4da1938ea864806be4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 02:00:24.432137 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fb6646299c54e4da1938ea864806be4" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:24.432245 9179 raft_consensus.cc:2454] T 4fb6646299c54e4da1938ea864806be4 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 02:00:24.432533 6784 leader_election.cc:302] T 4fb6646299c54e4da1938ea864806be4 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 492448570cc94ed1879feb2ec3b9051d; no voters:
I20240503 02:00:24.432763 14472 raft_consensus.cc:2781] T 4fb6646299c54e4da1938ea864806be4 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:24.432809 14472 raft_consensus.cc:482] T 4fb6646299c54e4da1938ea864806be4 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:24.432844 14472 raft_consensus.cc:3037] T 4fb6646299c54e4da1938ea864806be4 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:24.433926 14472 raft_consensus.cc:504] T 4fb6646299c54e4da1938ea864806be4 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:24.434245 14529 tablet_bootstrap.cc:492] T ecaa46386dee476492bae905d05b3dad P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:24.434315 14534 tablet_bootstrap.cc:492] T ecaa46386dee476492bae905d05b3dad P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:24.434809 14472 leader_election.cc:288] T 4fb6646299c54e4da1938ea864806be4 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Requested vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:24.435019 6719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fb6646299c54e4da1938ea864806be4" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 02:00:24.435148 6719 raft_consensus.cc:3037] T 4fb6646299c54e4da1938ea864806be4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:24.435281 14534 tablet_bootstrap.cc:654] T ecaa46386dee476492bae905d05b3dad P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.435387 14529 tablet_bootstrap.cc:654] T ecaa46386dee476492bae905d05b3dad P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.435881 14537 tablet_bootstrap.cc:492] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:24.436345 6719 raft_consensus.cc:2454] T 4fb6646299c54e4da1938ea864806be4 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 02:00:24.436734 6787 leader_election.cc:302] T 4fb6646299c54e4da1938ea864806be4 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:24.436726 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fb6646299c54e4da1938ea864806be4" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:24.436854 9179 raft_consensus.cc:3037] T 4fb6646299c54e4da1938ea864806be4 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:24.436904 14472 raft_consensus.cc:2781] T 4fb6646299c54e4da1938ea864806be4 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:24.436951 14537 tablet_bootstrap.cc:654] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:24.437302 14472 raft_consensus.cc:686] T 4fb6646299c54e4da1938ea864806be4 P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Becoming Leader. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Running, Role: LEADER
I20240503 02:00:24.437485 14472 consensus_queue.cc:237] T 4fb6646299c54e4da1938ea864806be4 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:24.437902 9179 raft_consensus.cc:2454] T 4fb6646299c54e4da1938ea864806be4 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 02:00:24.439076 14529 tablet_bootstrap.cc:492] T ecaa46386dee476492bae905d05b3dad P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:24.439194 14529 ts_tablet_manager.cc:1367] T ecaa46386dee476492bae905d05b3dad P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:00:24.439656 14534 tablet_bootstrap.cc:492] T ecaa46386dee476492bae905d05b3dad P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:24.439659 14529 raft_consensus.cc:348] T ecaa46386dee476492bae905d05b3dad P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:24.439751 14534 ts_tablet_manager.cc:1367] T ecaa46386dee476492bae905d05b3dad P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:00:24.439772 14529 raft_consensus.cc:374] T ecaa46386dee476492bae905d05b3dad P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.439801 14529 raft_consensus.cc:729] T ecaa46386dee476492bae905d05b3dad P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.439913 14529 consensus_queue.cc:260] T ecaa46386dee476492bae905d05b3dad P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:24.440197 14529 ts_tablet_manager.cc:1398] T ecaa46386dee476492bae905d05b3dad P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:24.440342 14534 raft_consensus.cc:348] T ecaa46386dee476492bae905d05b3dad P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:24.440467 14534 raft_consensus.cc:374] T ecaa46386dee476492bae905d05b3dad P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.440501 14534 raft_consensus.cc:729] T ecaa46386dee476492bae905d05b3dad P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.440626 14534 consensus_queue.cc:260] T ecaa46386dee476492bae905d05b3dad P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:24.440908 14534 ts_tablet_manager.cc:1398] T ecaa46386dee476492bae905d05b3dad P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:24.441504 6308 catalog_manager.cc:5484] T 4fb6646299c54e4da1938ea864806be4 P 492448570cc94ed1879feb2ec3b9051d reported cstate change: term changed from 0 to 1, leader changed from <none> to 492448570cc94ed1879feb2ec3b9051d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:24.443783 14537 tablet_bootstrap.cc:492] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:24.443893 14537 ts_tablet_manager.cc:1367] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20240503 02:00:24.444412 14537 raft_consensus.cc:348] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:24.444540 14537 raft_consensus.cc:374] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:24.444571 14537 raft_consensus.cc:729] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:24.444705 14537 consensus_queue.cc:260] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:24.444998 14537 ts_tablet_manager.cc:1398] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:24.453763 14462 raft_consensus.cc:482] T cdb766efec08440aa6a296edb76f84b7 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:24.453886 14462 raft_consensus.cc:504] T cdb766efec08440aa6a296edb76f84b7 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:24.454306 14462 leader_election.cc:288] T cdb766efec08440aa6a296edb76f84b7 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:24.454830 8917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdb766efec08440aa6a296edb76f84b7" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:24.454978 6719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdb766efec08440aa6a296edb76f84b7" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 02:00:24.454990 8917 raft_consensus.cc:2454] T cdb766efec08440aa6a296edb76f84b7 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:24.455111 6719 raft_consensus.cc:2454] T cdb766efec08440aa6a296edb76f84b7 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:24.455375 6525 leader_election.cc:302] T cdb766efec08440aa6a296edb76f84b7 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:24.455543 14462 raft_consensus.cc:2781] T cdb766efec08440aa6a296edb76f84b7 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:24.455613 14462 raft_consensus.cc:482] T cdb766efec08440aa6a296edb76f84b7 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:24.455662 14462 raft_consensus.cc:3037] T cdb766efec08440aa6a296edb76f84b7 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:24.456552 14462 raft_consensus.cc:504] T cdb766efec08440aa6a296edb76f84b7 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:24.456871 14462 leader_election.cc:288] T cdb766efec08440aa6a296edb76f84b7 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Requested vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:24.457286 6719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdb766efec08440aa6a296edb76f84b7" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 02:00:24.457269 8917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdb766efec08440aa6a296edb76f84b7" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:24.457415 8917 raft_consensus.cc:3037] T cdb766efec08440aa6a296edb76f84b7 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:24.457415 6719 raft_consensus.cc:3037] T cdb766efec08440aa6a296edb76f84b7 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:24.458410 8917 raft_consensus.cc:2454] T cdb766efec08440aa6a296edb76f84b7 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:24.458410 6719 raft_consensus.cc:2454] T cdb766efec08440aa6a296edb76f84b7 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:24.458721 6525 leader_election.cc:302] T cdb766efec08440aa6a296edb76f84b7 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:24.458902 14462 raft_consensus.cc:2781] T cdb766efec08440aa6a296edb76f84b7 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:24.458992 14462 raft_consensus.cc:686] T cdb766efec08440aa6a296edb76f84b7 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Becoming Leader. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Running, Role: LEADER
I20240503 02:00:24.459136 14462 consensus_queue.cc:237] T cdb766efec08440aa6a296edb76f84b7 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:24.461820 6308 catalog_manager.cc:5484] T cdb766efec08440aa6a296edb76f84b7 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } }
W20240503 02:00:24.494791 6523 consensus_peers.cc:485] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:00:24.502236 14552 raft_consensus.cc:482] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:24.502377 14552 raft_consensus.cc:504] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:24.502823 14552 leader_election.cc:288] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:24.503144 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecaa46386dee476492bae905d05b3dad" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:24.503173 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecaa46386dee476492bae905d05b3dad" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 02:00:24.503283 9179 raft_consensus.cc:2454] T ecaa46386dee476492bae905d05b3dad P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:24.503340 6849 raft_consensus.cc:2454] T ecaa46386dee476492bae905d05b3dad P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:24.503573 8981 leader_election.cc:302] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:24.503757 14552 raft_consensus.cc:2781] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:24.503818 14552 raft_consensus.cc:482] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:24.503845 14552 raft_consensus.cc:3037] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:24.504853 14552 raft_consensus.cc:504] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:24.505225 14552 leader_election.cc:288] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Requested vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:24.505571 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecaa46386dee476492bae905d05b3dad" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:24.505605 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecaa46386dee476492bae905d05b3dad" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 02:00:24.505690 9179 raft_consensus.cc:3037] T ecaa46386dee476492bae905d05b3dad P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:24.505720 6849 raft_consensus.cc:3037] T ecaa46386dee476492bae905d05b3dad P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:24.506547 6849 raft_consensus.cc:2454] T ecaa46386dee476492bae905d05b3dad P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:24.506549 9179 raft_consensus.cc:2454] T ecaa46386dee476492bae905d05b3dad P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:24.506875 8984 leader_election.cc:302] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:24.507053 14552 raft_consensus.cc:2781] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:24.507140 14552 raft_consensus.cc:686] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Becoming Leader. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Running, Role: LEADER
I20240503 02:00:24.507284 14552 consensus_queue.cc:237] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:24.510113 6308 catalog_manager.cc:5484] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b reported cstate change: term changed from 0 to 1, leader changed from <none> to 99ead02915954b4fab23b033ad6d546b (127.0.99.134). New cstate: current_term: 1 leader_uuid: "99ead02915954b4fab23b033ad6d546b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:24.614895 6308 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:24.619764 6849 raft_consensus.cc:1261] T ecaa46386dee476492bae905d05b3dad P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer 99ead02915954b4fab23b033ad6d546b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:24.620337 14431 consensus_queue.cc:1035] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:24.620476 9179 raft_consensus.cc:1261] T ecaa46386dee476492bae905d05b3dad P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer 99ead02915954b4fab23b033ad6d546b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:24.620970 14552 consensus_queue.cc:1035] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:24.623487 14559 tablet.cc:1722] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:24.623515 14554 mvcc.cc:205] Tried to move back new op lower bound from 7023417854437900288 to 7023417853983911936. Current Snapshot: MvccSnapshot[applied={T|T < 7023417854437900288}]
I20240503 02:00:24.624240 14559 tablet.cc:1620] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b: MemRowSet was empty: no flush needed.
I20240503 02:00:24.624585 6308 catalog_manager.cc:5576] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079) does not have the latest schema for tablet ecaa46386dee476492bae905d05b3dad (table rebalancer_dynamic_table [id=eb2282ddd2e94eadb12f14d01b283020]). Expected version 1 got 0
I20240503 02:00:24.625159 14560 tablet.cc:1722] T ecaa46386dee476492bae905d05b3dad P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:24.625535 14560 tablet.cc:1620] T ecaa46386dee476492bae905d05b3dad P 492448570cc94ed1879feb2ec3b9051d: MemRowSet was empty: no flush needed.
I20240503 02:00:24.626775 6309 catalog_manager.cc:5576] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719) does not have the latest schema for tablet ecaa46386dee476492bae905d05b3dad (table rebalancer_dynamic_table [id=eb2282ddd2e94eadb12f14d01b283020]). Expected version 1 got 0
I20240503 02:00:24.630683 6313 catalog_manager.cc:5576] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079) does not have the latest schema for tablet ecaa46386dee476492bae905d05b3dad (table rebalancer_dynamic_table [id=eb2282ddd2e94eadb12f14d01b283020]). Expected version 1 got 0
I20240503 02:00:24.631242 14561 tablet.cc:1722] T ecaa46386dee476492bae905d05b3dad P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:24.631600 14561 tablet.cc:1620] T ecaa46386dee476492bae905d05b3dad P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
I20240503 02:00:24.631685 6308 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=eb2282ddd2e94eadb12f14d01b283020] alter complete (version 1)
I20240503 02:00:24.724740 14380 consensus_queue.cc:1035] T f55cb3da9c874387a65eff5de0903331 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:24.729820 14459 consensus_queue.cc:1035] T f55cb3da9c874387a65eff5de0903331 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:24.735761 6313 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:24.741564 14559 tablet.cc:1722] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:24.741919 14559 tablet.cc:1620] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b: MemRowSet was empty: no flush needed.
I20240503 02:00:24.742115 14560 tablet.cc:1722] T ecaa46386dee476492bae905d05b3dad P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:24.742406 14560 tablet.cc:1620] T ecaa46386dee476492bae905d05b3dad P 492448570cc94ed1879feb2ec3b9051d: MemRowSet was empty: no flush needed.
I20240503 02:00:24.742700 14561 tablet.cc:1722] T ecaa46386dee476492bae905d05b3dad P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:24.742983 14561 tablet.cc:1620] T ecaa46386dee476492bae905d05b3dad P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
I20240503 02:00:24.750229 6309 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=eb2282ddd2e94eadb12f14d01b283020] alter complete (version 2)
I20240503 02:00:24.851697 6313 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:24.857203 14559 tablet.cc:1722] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:24.857982 14560 tablet.cc:1722] T ecaa46386dee476492bae905d05b3dad P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:24.859521 14561 tablet.cc:1722] T ecaa46386dee476492bae905d05b3dad P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:24.864426 6313 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=eb2282ddd2e94eadb12f14d01b283020] alter complete (version 3)
I20240503 02:00:24.897468 14562 consensus_queue.cc:1035] T cdb766efec08440aa6a296edb76f84b7 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:24.901619 14462 consensus_queue.cc:1035] T cdb766efec08440aa6a296edb76f84b7 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:24.914868 14324 consensus_queue.cc:1035] T 4fb6646299c54e4da1938ea864806be4 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:24.919044 14571 consensus_queue.cc:1035] T 4fb6646299c54e4da1938ea864806be4 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:00:24.965845 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:24.966524 6309 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:24.966676 6309 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:24.968804 6309 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet ecaa46386dee476492bae905d05b3dad
I20240503 02:00:24.969519 9028 tablet_service.cc:1514] Processing DeleteTablet for tablet ecaa46386dee476492bae905d05b3dad with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:24 UTC) from {username='slave'} at 127.0.0.1:52584
I20240503 02:00:24.969834 14573 tablet_replica.cc:330] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b: stopping tablet replica
I20240503 02:00:24.969830 9157 tablet_service.cc:1514] Processing DeleteTablet for tablet ecaa46386dee476492bae905d05b3dad with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:24 UTC) from {username='slave'} at 127.0.0.1:51940
I20240503 02:00:24.969987 14573 raft_consensus.cc:2229] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:24.970094 6829 tablet_service.cc:1514] Processing DeleteTablet for tablet ecaa46386dee476492bae905d05b3dad with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:24 UTC) from {username='slave'} at 127.0.0.1:52812
I20240503 02:00:24.970299 14573 raft_consensus.cc:2258] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:24.970415 14575 tablet_replica.cc:330] T ecaa46386dee476492bae905d05b3dad P 492448570cc94ed1879feb2ec3b9051d: stopping tablet replica
I20240503 02:00:24.970588 14575 raft_consensus.cc:2229] T ecaa46386dee476492bae905d05b3dad P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:24.970793 14575 raft_consensus.cc:2258] T ecaa46386dee476492bae905d05b3dad P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:24.971014 14574 tablet_replica.cc:330] T ecaa46386dee476492bae905d05b3dad P 2a01ace80baf4057b6bf1995cf88468b: stopping tablet replica
I20240503 02:00:24.971185 14574 raft_consensus.cc:2229] T ecaa46386dee476492bae905d05b3dad P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:24.971326 14574 raft_consensus.cc:2258] T ecaa46386dee476492bae905d05b3dad P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:24.971585 14575 ts_tablet_manager.cc:1875] T ecaa46386dee476492bae905d05b3dad P 492448570cc94ed1879feb2ec3b9051d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:24.971704 14573 ts_tablet_manager.cc:1875] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:24.971988 14574 ts_tablet_manager.cc:1875] T ecaa46386dee476492bae905d05b3dad P 2a01ace80baf4057b6bf1995cf88468b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:24.974551 14574 ts_tablet_manager.cc:1888] T ecaa46386dee476492bae905d05b3dad P 2a01ace80baf4057b6bf1995cf88468b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:24.974674 14574 log.cc:1196] T ecaa46386dee476492bae905d05b3dad P 2a01ace80baf4057b6bf1995cf88468b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/ecaa46386dee476492bae905d05b3dad
I20240503 02:00:24.974613 14575 ts_tablet_manager.cc:1888] T ecaa46386dee476492bae905d05b3dad P 492448570cc94ed1879feb2ec3b9051d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:24.974861 14575 log.cc:1196] T ecaa46386dee476492bae905d05b3dad P 492448570cc94ed1879feb2ec3b9051d: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/ecaa46386dee476492bae905d05b3dad
I20240503 02:00:24.974951 14574 ts_tablet_manager.cc:1909] T ecaa46386dee476492bae905d05b3dad P 2a01ace80baf4057b6bf1995cf88468b: Deleting consensus metadata
I20240503 02:00:24.975283 14575 ts_tablet_manager.cc:1909] T ecaa46386dee476492bae905d05b3dad P 492448570cc94ed1879feb2ec3b9051d: Deleting consensus metadata
I20240503 02:00:24.975690 6300 catalog_manager.cc:4830] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079): tablet ecaa46386dee476492bae905d05b3dad (table rebalancer_dynamic_table_new_name [id=eb2282ddd2e94eadb12f14d01b283020]) successfully deleted
I20240503 02:00:24.976487 6303 catalog_manager.cc:4830] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719): tablet ecaa46386dee476492bae905d05b3dad (table rebalancer_dynamic_table_new_name [id=eb2282ddd2e94eadb12f14d01b283020]) successfully deleted
I20240503 02:00:24.974382 14573 ts_tablet_manager.cc:1888] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:24.976727 14573 log.cc:1196] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/ecaa46386dee476492bae905d05b3dad
I20240503 02:00:24.977008 14573 ts_tablet_manager.cc:1909] T ecaa46386dee476492bae905d05b3dad P 99ead02915954b4fab23b033ad6d546b: Deleting consensus metadata
I20240503 02:00:24.977707 6301 catalog_manager.cc:4830] TS 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157): tablet ecaa46386dee476492bae905d05b3dad (table rebalancer_dynamic_table_new_name [id=eb2282ddd2e94eadb12f14d01b283020]) successfully deleted
I20240503 02:00:24.990407 6309 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_extra_table_64"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:24.999305 6568 tablet_service.cc:1467] Processing CreateTablet for tablet 898eb10f12924bd196108b9bb4ba6b77 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=4389ec3b2b7e429ca5a81a91fc4f7c6d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:24.999812 6568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 898eb10f12924bd196108b9bb4ba6b77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.000572 6829 tablet_service.cc:1467] Processing CreateTablet for tablet 898eb10f12924bd196108b9bb4ba6b77 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=4389ec3b2b7e429ca5a81a91fc4f7c6d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:25.000850 6829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 898eb10f12924bd196108b9bb4ba6b77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.000875 6436 tablet_service.cc:1467] Processing CreateTablet for tablet 5946d453a0ea4f5491d98c41cc99efd9 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=4389ec3b2b7e429ca5a81a91fc4f7c6d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:25.001128 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5946d453a0ea4f5491d98c41cc99efd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.002745 8880 tablet_service.cc:1467] Processing CreateTablet for tablet 898eb10f12924bd196108b9bb4ba6b77 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=4389ec3b2b7e429ca5a81a91fc4f7c6d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:25.002851 14576 tablet_bootstrap.cc:492] T 898eb10f12924bd196108b9bb4ba6b77 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:25.003026 8880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 898eb10f12924bd196108b9bb4ba6b77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.003374 6438 tablet_service.cc:1467] Processing CreateTablet for tablet 197ec14bb8fa494da81637271913393f (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=4389ec3b2b7e429ca5a81a91fc4f7c6d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:25.003628 6438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 197ec14bb8fa494da81637271913393f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.004067 14576 tablet_bootstrap.cc:654] T 898eb10f12924bd196108b9bb4ba6b77 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.004101 14578 tablet_bootstrap.cc:492] T 898eb10f12924bd196108b9bb4ba6b77 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:25.005100 14578 tablet_bootstrap.cc:654] T 898eb10f12924bd196108b9bb4ba6b77 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.005535 14580 tablet_bootstrap.cc:492] T 898eb10f12924bd196108b9bb4ba6b77 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:25.006572 14580 tablet_bootstrap.cc:654] T 898eb10f12924bd196108b9bb4ba6b77 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.006743 6829 tablet_service.cc:1467] Processing CreateTablet for tablet 5946d453a0ea4f5491d98c41cc99efd9 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=4389ec3b2b7e429ca5a81a91fc4f7c6d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:25.006743 8887 tablet_service.cc:1467] Processing CreateTablet for tablet 197ec14bb8fa494da81637271913393f (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=4389ec3b2b7e429ca5a81a91fc4f7c6d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:25.006996 6829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5946d453a0ea4f5491d98c41cc99efd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.006996 8887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 197ec14bb8fa494da81637271913393f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.007529 6831 tablet_service.cc:1467] Processing CreateTablet for tablet 197ec14bb8fa494da81637271913393f (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=4389ec3b2b7e429ca5a81a91fc4f7c6d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:25.007781 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 197ec14bb8fa494da81637271913393f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.009820 14576 tablet_bootstrap.cc:492] T 898eb10f12924bd196108b9bb4ba6b77 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:25.009932 14576 ts_tablet_manager.cc:1367] T 898eb10f12924bd196108b9bb4ba6b77 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20240503 02:00:25.010398 14576 raft_consensus.cc:348] T 898eb10f12924bd196108b9bb4ba6b77 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:25.010557 14576 raft_consensus.cc:374] T 898eb10f12924bd196108b9bb4ba6b77 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.010596 14576 raft_consensus.cc:729] T 898eb10f12924bd196108b9bb4ba6b77 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.010758 14576 consensus_queue.cc:260] T 898eb10f12924bd196108b9bb4ba6b77 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:25.011054 14576 ts_tablet_manager.cc:1398] T 898eb10f12924bd196108b9bb4ba6b77 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:25.011608 14580 tablet_bootstrap.cc:492] T 898eb10f12924bd196108b9bb4ba6b77 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:25.011734 14580 ts_tablet_manager.cc:1367] T 898eb10f12924bd196108b9bb4ba6b77 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:00:25.012147 8880 tablet_service.cc:1467] Processing CreateTablet for tablet 5946d453a0ea4f5491d98c41cc99efd9 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=4389ec3b2b7e429ca5a81a91fc4f7c6d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:25.012212 14580 raft_consensus.cc:348] T 898eb10f12924bd196108b9bb4ba6b77 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:25.012348 14580 raft_consensus.cc:374] T 898eb10f12924bd196108b9bb4ba6b77 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.012403 14580 raft_consensus.cc:729] T 898eb10f12924bd196108b9bb4ba6b77 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.012447 8880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5946d453a0ea4f5491d98c41cc99efd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.012537 14580 consensus_queue.cc:260] T 898eb10f12924bd196108b9bb4ba6b77 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:25.012894 14580 ts_tablet_manager.cc:1398] T 898eb10f12924bd196108b9bb4ba6b77 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:25.013875 14580 tablet_bootstrap.cc:492] T 197ec14bb8fa494da81637271913393f P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:25.014907 14580 tablet_bootstrap.cc:654] T 197ec14bb8fa494da81637271913393f P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.015424 14577 tablet_bootstrap.cc:492] T 5946d453a0ea4f5491d98c41cc99efd9 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:25.016778 14577 tablet_bootstrap.cc:654] T 5946d453a0ea4f5491d98c41cc99efd9 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.018183 14577 tablet_bootstrap.cc:492] T 5946d453a0ea4f5491d98c41cc99efd9 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:25.018364 14577 ts_tablet_manager.cc:1367] T 5946d453a0ea4f5491d98c41cc99efd9 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:25.018746 14578 tablet_bootstrap.cc:492] T 898eb10f12924bd196108b9bb4ba6b77 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:25.018986 14577 raft_consensus.cc:348] T 5946d453a0ea4f5491d98c41cc99efd9 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:25.019181 14577 raft_consensus.cc:374] T 5946d453a0ea4f5491d98c41cc99efd9 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.019256 14577 raft_consensus.cc:729] T 5946d453a0ea4f5491d98c41cc99efd9 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.019454 14577 consensus_queue.cc:260] T 5946d453a0ea4f5491d98c41cc99efd9 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:25.019832 14577 ts_tablet_manager.cc:1398] T 5946d453a0ea4f5491d98c41cc99efd9 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:25.020681 14578 ts_tablet_manager.cc:1367] T 898eb10f12924bd196108b9bb4ba6b77 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240503 02:00:25.021314 14578 raft_consensus.cc:348] T 898eb10f12924bd196108b9bb4ba6b77 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:25.021517 14577 tablet_bootstrap.cc:492] T 197ec14bb8fa494da81637271913393f P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:25.021512 14578 raft_consensus.cc:374] T 898eb10f12924bd196108b9bb4ba6b77 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.021555 14578 raft_consensus.cc:729] T 898eb10f12924bd196108b9bb4ba6b77 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.021695 14578 consensus_queue.cc:260] T 898eb10f12924bd196108b9bb4ba6b77 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:25.022059 14578 ts_tablet_manager.cc:1398] T 898eb10f12924bd196108b9bb4ba6b77 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:25.023360 14578 tablet_bootstrap.cc:492] T 5946d453a0ea4f5491d98c41cc99efd9 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:25.023846 14577 tablet_bootstrap.cc:654] T 197ec14bb8fa494da81637271913393f P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.024363 14578 tablet_bootstrap.cc:654] T 5946d453a0ea4f5491d98c41cc99efd9 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.025115 14577 tablet_bootstrap.cc:492] T 197ec14bb8fa494da81637271913393f P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:25.025207 14577 ts_tablet_manager.cc:1367] T 197ec14bb8fa494da81637271913393f P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:25.025692 14577 raft_consensus.cc:348] T 197ec14bb8fa494da81637271913393f P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:25.025806 14577 raft_consensus.cc:374] T 197ec14bb8fa494da81637271913393f P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.025832 14577 raft_consensus.cc:729] T 197ec14bb8fa494da81637271913393f P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.025955 14577 consensus_queue.cc:260] T 197ec14bb8fa494da81637271913393f P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:25.026240 14577 ts_tablet_manager.cc:1398] T 197ec14bb8fa494da81637271913393f P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:25.029258 14578 tablet_bootstrap.cc:492] T 5946d453a0ea4f5491d98c41cc99efd9 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:25.029450 14578 ts_tablet_manager.cc:1367] T 5946d453a0ea4f5491d98c41cc99efd9 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240503 02:00:25.030120 14578 raft_consensus.cc:348] T 5946d453a0ea4f5491d98c41cc99efd9 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:25.030292 14578 raft_consensus.cc:374] T 5946d453a0ea4f5491d98c41cc99efd9 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.030344 14578 raft_consensus.cc:729] T 5946d453a0ea4f5491d98c41cc99efd9 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.030498 14578 consensus_queue.cc:260] T 5946d453a0ea4f5491d98c41cc99efd9 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:25.030954 14578 ts_tablet_manager.cc:1398] T 5946d453a0ea4f5491d98c41cc99efd9 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:25.031193 14578 tablet_bootstrap.cc:492] T 197ec14bb8fa494da81637271913393f P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:25.032496 14578 tablet_bootstrap.cc:654] T 197ec14bb8fa494da81637271913393f P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.033192 14580 tablet_bootstrap.cc:492] T 197ec14bb8fa494da81637271913393f P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:25.033389 14580 ts_tablet_manager.cc:1367] T 197ec14bb8fa494da81637271913393f P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.002s
I20240503 02:00:25.034148 14580 raft_consensus.cc:348] T 197ec14bb8fa494da81637271913393f P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:25.034312 14580 raft_consensus.cc:374] T 197ec14bb8fa494da81637271913393f P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.034343 14580 raft_consensus.cc:729] T 197ec14bb8fa494da81637271913393f P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.035782 14580 consensus_queue.cc:260] T 197ec14bb8fa494da81637271913393f P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:25.036641 14580 ts_tablet_manager.cc:1398] T 197ec14bb8fa494da81637271913393f P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 02:00:25.036999 14578 tablet_bootstrap.cc:492] T 197ec14bb8fa494da81637271913393f P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:25.037376 14578 ts_tablet_manager.cc:1367] T 197ec14bb8fa494da81637271913393f P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20240503 02:00:25.037398 14580 tablet_bootstrap.cc:492] T 5946d453a0ea4f5491d98c41cc99efd9 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:25.038064 14578 raft_consensus.cc:348] T 197ec14bb8fa494da81637271913393f P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:25.038233 14578 raft_consensus.cc:374] T 197ec14bb8fa494da81637271913393f P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.038301 14578 raft_consensus.cc:729] T 197ec14bb8fa494da81637271913393f P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.038522 14578 consensus_queue.cc:260] T 197ec14bb8fa494da81637271913393f P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:25.038632 14580 tablet_bootstrap.cc:654] T 5946d453a0ea4f5491d98c41cc99efd9 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.039921 14580 tablet_bootstrap.cc:492] T 5946d453a0ea4f5491d98c41cc99efd9 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:25.039927 14578 ts_tablet_manager.cc:1398] T 197ec14bb8fa494da81637271913393f P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:00:25.040026 14580 ts_tablet_manager.cc:1367] T 5946d453a0ea4f5491d98c41cc99efd9 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:00:25.040534 14580 raft_consensus.cc:348] T 5946d453a0ea4f5491d98c41cc99efd9 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:25.040643 14580 raft_consensus.cc:374] T 5946d453a0ea4f5491d98c41cc99efd9 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.040668 14580 raft_consensus.cc:729] T 5946d453a0ea4f5491d98c41cc99efd9 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.040772 14580 consensus_queue.cc:260] T 5946d453a0ea4f5491d98c41cc99efd9 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:25.041041 14580 ts_tablet_manager.cc:1398] T 5946d453a0ea4f5491d98c41cc99efd9 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:25.050038 14472 raft_consensus.cc:482] T 898eb10f12924bd196108b9bb4ba6b77 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:25.050159 14472 raft_consensus.cc:504] T 898eb10f12924bd196108b9bb4ba6b77 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:25.050511 14472 leader_election.cc:288] T 898eb10f12924bd196108b9bb4ba6b77 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:25.050904 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "898eb10f12924bd196108b9bb4ba6b77" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:25.051131 8916 raft_consensus.cc:2454] T 898eb10f12924bd196108b9bb4ba6b77 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 02:00:25.051112 6587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "898eb10f12924bd196108b9bb4ba6b77" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 02:00:25.051252 6587 raft_consensus.cc:2454] T 898eb10f12924bd196108b9bb4ba6b77 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 02:00:25.051482 6787 leader_election.cc:302] T 898eb10f12924bd196108b9bb4ba6b77 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:25.052443 14472 raft_consensus.cc:2781] T 898eb10f12924bd196108b9bb4ba6b77 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:25.052524 14472 raft_consensus.cc:482] T 898eb10f12924bd196108b9bb4ba6b77 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:25.052567 14472 raft_consensus.cc:3037] T 898eb10f12924bd196108b9bb4ba6b77 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:25.053711 14472 raft_consensus.cc:504] T 898eb10f12924bd196108b9bb4ba6b77 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:25.054121 14472 leader_election.cc:288] T 898eb10f12924bd196108b9bb4ba6b77 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Requested vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:25.054937 6587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "898eb10f12924bd196108b9bb4ba6b77" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 02:00:25.055074 6587 raft_consensus.cc:3037] T 898eb10f12924bd196108b9bb4ba6b77 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:25.056110 6587 raft_consensus.cc:2454] T 898eb10f12924bd196108b9bb4ba6b77 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 02:00:25.056800 6786 leader_election.cc:302] T 898eb10f12924bd196108b9bb4ba6b77 P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 80fd18ef4d604c86b909d5c008a46b4f; no voters:
I20240503 02:00:25.057302 14571 raft_consensus.cc:2781] T 898eb10f12924bd196108b9bb4ba6b77 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:25.057397 14571 raft_consensus.cc:686] T 898eb10f12924bd196108b9bb4ba6b77 P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Becoming Leader. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Running, Role: LEADER
I20240503 02:00:25.057543 14571 consensus_queue.cc:237] T 898eb10f12924bd196108b9bb4ba6b77 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:25.058722 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "898eb10f12924bd196108b9bb4ba6b77" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:25.058880 8916 raft_consensus.cc:3037] T 898eb10f12924bd196108b9bb4ba6b77 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:25.059823 8916 raft_consensus.cc:2454] T 898eb10f12924bd196108b9bb4ba6b77 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 02:00:25.061959 6309 catalog_manager.cc:5484] T 898eb10f12924bd196108b9bb4ba6b77 P 492448570cc94ed1879feb2ec3b9051d reported cstate change: term changed from 0 to 1, leader changed from <none> to 492448570cc94ed1879feb2ec3b9051d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:25.071422 6309 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:00:25.078240 6700 tablet_service.cc:1467] Processing CreateTablet for tablet 65ca02bf14bf4cec8b5cd90c305e349e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e9e0570a970943cb845b71a46c761153]), partition=
I20240503 02:00:25.078368 9157 tablet_service.cc:1467] Processing CreateTablet for tablet 65ca02bf14bf4cec8b5cd90c305e349e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e9e0570a970943cb845b71a46c761153]), partition=
I20240503 02:00:25.078395 6438 tablet_service.cc:1467] Processing CreateTablet for tablet 65ca02bf14bf4cec8b5cd90c305e349e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e9e0570a970943cb845b71a46c761153]), partition=
I20240503 02:00:25.078553 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65ca02bf14bf4cec8b5cd90c305e349e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.078583 9157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65ca02bf14bf4cec8b5cd90c305e349e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.078645 6438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65ca02bf14bf4cec8b5cd90c305e349e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.080827 14577 tablet_bootstrap.cc:492] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:25.081269 14591 tablet_bootstrap.cc:492] T 65ca02bf14bf4cec8b5cd90c305e349e P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:25.082190 14577 tablet_bootstrap.cc:654] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.082271 14591 tablet_bootstrap.cc:654] T 65ca02bf14bf4cec8b5cd90c305e349e P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.083448 14577 tablet_bootstrap.cc:492] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:25.083586 14577 ts_tablet_manager.cc:1367] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240503 02:00:25.083634 14591 tablet_bootstrap.cc:492] T 65ca02bf14bf4cec8b5cd90c305e349e P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:25.083721 14591 ts_tablet_manager.cc:1367] T 65ca02bf14bf4cec8b5cd90c305e349e P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:25.084091 14577 raft_consensus.cc:348] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:25.084209 14577 raft_consensus.cc:374] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.084239 14577 raft_consensus.cc:729] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.084172 14591 raft_consensus.cc:348] T 65ca02bf14bf4cec8b5cd90c305e349e P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:25.084281 14591 raft_consensus.cc:374] T 65ca02bf14bf4cec8b5cd90c305e349e P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.084312 14591 raft_consensus.cc:729] T 65ca02bf14bf4cec8b5cd90c305e349e P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.084468 14591 consensus_queue.cc:260] T 65ca02bf14bf4cec8b5cd90c305e349e P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:25.084498 14577 consensus_queue.cc:260] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:25.084820 14591 ts_tablet_manager.cc:1398] T 65ca02bf14bf4cec8b5cd90c305e349e P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:25.084936 14577 ts_tablet_manager.cc:1398] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:25.086676 14594 tablet_bootstrap.cc:492] T 65ca02bf14bf4cec8b5cd90c305e349e P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:25.087801 14594 tablet_bootstrap.cc:654] T 65ca02bf14bf4cec8b5cd90c305e349e P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.090245 14594 tablet_bootstrap.cc:492] T 65ca02bf14bf4cec8b5cd90c305e349e P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:25.090435 14594 ts_tablet_manager.cc:1367] T 65ca02bf14bf4cec8b5cd90c305e349e P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:25.091329 14594 raft_consensus.cc:348] T 65ca02bf14bf4cec8b5cd90c305e349e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:25.091467 14594 raft_consensus.cc:374] T 65ca02bf14bf4cec8b5cd90c305e349e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.091500 14594 raft_consensus.cc:729] T 65ca02bf14bf4cec8b5cd90c305e349e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.091673 14594 consensus_queue.cc:260] T 65ca02bf14bf4cec8b5cd90c305e349e P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:25.092325 14594 ts_tablet_manager.cc:1398] T 65ca02bf14bf4cec8b5cd90c305e349e P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:25.103395 14459 raft_consensus.cc:482] T 5946d453a0ea4f5491d98c41cc99efd9 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:25.103549 14459 raft_consensus.cc:504] T 5946d453a0ea4f5491d98c41cc99efd9 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:25.104146 14459 leader_election.cc:288] T 5946d453a0ea4f5491d98c41cc99efd9 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:25.104660 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5946d453a0ea4f5491d98c41cc99efd9" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 02:00:25.104715 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5946d453a0ea4f5491d98c41cc99efd9" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 02:00:25.104866 6458 raft_consensus.cc:2454] T 5946d453a0ea4f5491d98c41cc99efd9 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 0.
I20240503 02:00:25.104882 6849 raft_consensus.cc:2454] T 5946d453a0ea4f5491d98c41cc99efd9 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 0.
I20240503 02:00:25.105317 8853 leader_election.cc:302] T 5946d453a0ea4f5491d98c41cc99efd9 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:25.105576 14459 raft_consensus.cc:2781] T 5946d453a0ea4f5491d98c41cc99efd9 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:25.105630 14459 raft_consensus.cc:482] T 5946d453a0ea4f5491d98c41cc99efd9 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:25.105665 14459 raft_consensus.cc:3037] T 5946d453a0ea4f5491d98c41cc99efd9 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:25.106946 14459 raft_consensus.cc:504] T 5946d453a0ea4f5491d98c41cc99efd9 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:25.107312 14459 leader_election.cc:288] T 5946d453a0ea4f5491d98c41cc99efd9 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 election: Requested vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:25.107796 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5946d453a0ea4f5491d98c41cc99efd9" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 02:00:25.107808 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5946d453a0ea4f5491d98c41cc99efd9" candidate_uuid: "9942306855404cfd891adf227dd65eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 02:00:25.107942 6849 raft_consensus.cc:3037] T 5946d453a0ea4f5491d98c41cc99efd9 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:25.107942 6458 raft_consensus.cc:3037] T 5946d453a0ea4f5491d98c41cc99efd9 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:25.108963 6458 raft_consensus.cc:2454] T 5946d453a0ea4f5491d98c41cc99efd9 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 1.
I20240503 02:00:25.108961 6849 raft_consensus.cc:2454] T 5946d453a0ea4f5491d98c41cc99efd9 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9942306855404cfd891adf227dd65eb2 in term 1.
I20240503 02:00:25.109354 8853 leader_election.cc:302] T 5946d453a0ea4f5491d98c41cc99efd9 P 9942306855404cfd891adf227dd65eb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:25.109558 14459 raft_consensus.cc:2781] T 5946d453a0ea4f5491d98c41cc99efd9 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:25.109648 14459 raft_consensus.cc:686] T 5946d453a0ea4f5491d98c41cc99efd9 P 9942306855404cfd891adf227dd65eb2 [term 1 LEADER]: Becoming Leader. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Running, Role: LEADER
I20240503 02:00:25.109787 14459 consensus_queue.cc:237] T 5946d453a0ea4f5491d98c41cc99efd9 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:25.112542 6314 catalog_manager.cc:5484] T 5946d453a0ea4f5491d98c41cc99efd9 P 9942306855404cfd891adf227dd65eb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9942306855404cfd891adf227dd65eb2 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "9942306855404cfd891adf227dd65eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:25.168205 14472 raft_consensus.cc:482] T 197ec14bb8fa494da81637271913393f P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:25.168342 14472 raft_consensus.cc:504] T 197ec14bb8fa494da81637271913393f P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:25.168771 14472 leader_election.cc:288] T 197ec14bb8fa494da81637271913393f P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:25.169386 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "197ec14bb8fa494da81637271913393f" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:25.169539 8916 raft_consensus.cc:2454] T 197ec14bb8fa494da81637271913393f P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 02:00:25.169744 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "197ec14bb8fa494da81637271913393f" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 02:00:25.169826 6787 leader_election.cc:302] T 197ec14bb8fa494da81637271913393f P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:25.169885 6458 raft_consensus.cc:2454] T 197ec14bb8fa494da81637271913393f P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 02:00:25.170006 14472 raft_consensus.cc:2781] T 197ec14bb8fa494da81637271913393f P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:25.170061 14472 raft_consensus.cc:482] T 197ec14bb8fa494da81637271913393f P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:25.170097 14472 raft_consensus.cc:3037] T 197ec14bb8fa494da81637271913393f P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:25.170981 14472 raft_consensus.cc:504] T 197ec14bb8fa494da81637271913393f P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:25.171329 14472 leader_election.cc:288] T 197ec14bb8fa494da81637271913393f P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Requested vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:25.171684 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "197ec14bb8fa494da81637271913393f" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:25.171703 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "197ec14bb8fa494da81637271913393f" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 02:00:25.171818 8916 raft_consensus.cc:3037] T 197ec14bb8fa494da81637271913393f P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:25.171929 6458 raft_consensus.cc:3037] T 197ec14bb8fa494da81637271913393f P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:25.172941 8916 raft_consensus.cc:2454] T 197ec14bb8fa494da81637271913393f P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 02:00:25.172941 6458 raft_consensus.cc:2454] T 197ec14bb8fa494da81637271913393f P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 02:00:25.173344 6784 leader_election.cc:302] T 197ec14bb8fa494da81637271913393f P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:25.173557 14472 raft_consensus.cc:2781] T 197ec14bb8fa494da81637271913393f P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:25.173713 14472 raft_consensus.cc:686] T 197ec14bb8fa494da81637271913393f P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Becoming Leader. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Running, Role: LEADER
I20240503 02:00:25.173864 14472 consensus_queue.cc:237] T 197ec14bb8fa494da81637271913393f P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:25.176651 6314 catalog_manager.cc:5484] T 197ec14bb8fa494da81637271913393f P 492448570cc94ed1879feb2ec3b9051d reported cstate change: term changed from 0 to 1, leader changed from <none> to 492448570cc94ed1879feb2ec3b9051d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:25.334816 14585 raft_consensus.cc:482] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:25.334954 14585 raft_consensus.cc:504] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:25.335462 14585 leader_election.cc:288] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:25.335880 6720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65ca02bf14bf4cec8b5cd90c305e349e" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 02:00:25.335906 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65ca02bf14bf4cec8b5cd90c305e349e" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:25.336017 6720 raft_consensus.cc:2454] T 65ca02bf14bf4cec8b5cd90c305e349e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:25.336052 9179 raft_consensus.cc:2454] T 65ca02bf14bf4cec8b5cd90c305e349e P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:25.336329 6394 leader_election.cc:302] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2ac2eb7cc264047b9ab32fe3a4fc025, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:25.336537 14585 raft_consensus.cc:2781] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:25.336607 14585 raft_consensus.cc:482] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:25.336652 14585 raft_consensus.cc:3037] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:25.337620 14585 raft_consensus.cc:504] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:25.338107 14585 leader_election.cc:288] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Requested vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:25.338491 6720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65ca02bf14bf4cec8b5cd90c305e349e" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 02:00:25.338491 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65ca02bf14bf4cec8b5cd90c305e349e" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:25.338641 6720 raft_consensus.cc:3037] T 65ca02bf14bf4cec8b5cd90c305e349e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:25.338796 9179 raft_consensus.cc:3037] T 65ca02bf14bf4cec8b5cd90c305e349e P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:25.339716 6720 raft_consensus.cc:2454] T 65ca02bf14bf4cec8b5cd90c305e349e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:25.339716 9179 raft_consensus.cc:2454] T 65ca02bf14bf4cec8b5cd90c305e349e P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:25.340026 6391 leader_election.cc:302] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:25.340399 14585 raft_consensus.cc:2781] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:25.340492 14585 raft_consensus.cc:686] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Becoming Leader. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Running, Role: LEADER
I20240503 02:00:25.340629 14585 consensus_queue.cc:237] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:25.342850 6314 catalog_manager.cc:5484] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: term changed from 0 to 1, leader changed from <none> to dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129). New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } }
W20240503 02:00:25.352340 6785 consensus_peers.cc:485] T c3485701bf3b48f5ae2e1568afeed133 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:25.403998 8982 consensus_peers.cc:485] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:00:25.481150 14472 consensus_queue.cc:1035] T 898eb10f12924bd196108b9bb4ba6b77 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:25.482618 6314 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:25.488212 14571 consensus_queue.cc:1035] T 898eb10f12924bd196108b9bb4ba6b77 P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:25.489696 9178 raft_consensus.cc:1261] T 65ca02bf14bf4cec8b5cd90c305e349e P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:25.490495 6720 raft_consensus.cc:1261] T 65ca02bf14bf4cec8b5cd90c305e349e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer dec3e4a04d604f7eaaa3f077d081218a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:25.491276 14153 consensus_queue.cc:1035] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:25.491592 14153 consensus_queue.cc:1035] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:25.497066 6314 catalog_manager.cc:5576] TS c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655) does not have the latest schema for tablet 65ca02bf14bf4cec8b5cd90c305e349e (table rebalancer_dynamic_table [id=e9e0570a970943cb845b71a46c761153]). Expected version 1 got 0
I20240503 02:00:25.497748 14610 tablet.cc:1722] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:25.498122 14610 tablet.cc:1620] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a: MemRowSet was empty: no flush needed.
I20240503 02:00:25.499749 6314 catalog_manager.cc:5576] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093) does not have the latest schema for tablet 65ca02bf14bf4cec8b5cd90c305e349e (table rebalancer_dynamic_table [id=e9e0570a970943cb845b71a46c761153]). Expected version 1 got 0
I20240503 02:00:25.501207 6313 catalog_manager.cc:5576] TS c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655) does not have the latest schema for tablet 65ca02bf14bf4cec8b5cd90c305e349e (table rebalancer_dynamic_table [id=e9e0570a970943cb845b71a46c761153]). Expected version 1 got 0
I20240503 02:00:25.504089 6313 catalog_manager.cc:5576] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079) does not have the latest schema for tablet 65ca02bf14bf4cec8b5cd90c305e349e (table rebalancer_dynamic_table [id=e9e0570a970943cb845b71a46c761153]). Expected version 1 got 0
I20240503 02:00:25.503880 14611 tablet.cc:1722] T 65ca02bf14bf4cec8b5cd90c305e349e P c2ac2eb7cc264047b9ab32fe3a4fc025: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:25.504458 14611 tablet.cc:1620] T 65ca02bf14bf4cec8b5cd90c305e349e P c2ac2eb7cc264047b9ab32fe3a4fc025: MemRowSet was empty: no flush needed.
I20240503 02:00:25.507692 6313 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=e9e0570a970943cb845b71a46c761153] alter complete (version 1)
I20240503 02:00:25.511083 14613 tablet.cc:1722] T 65ca02bf14bf4cec8b5cd90c305e349e P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:25.511744 14613 tablet.cc:1620] T 65ca02bf14bf4cec8b5cd90c305e349e P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
W20240503 02:00:25.557039 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:25.586073 14598 consensus_queue.cc:1035] T 5946d453a0ea4f5491d98c41cc99efd9 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:25.591269 14380 consensus_queue.cc:1035] T 5946d453a0ea4f5491d98c41cc99efd9 P 9942306855404cfd891adf227dd65eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:25.604135 14571 consensus_queue.cc:1035] T 197ec14bb8fa494da81637271913393f P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:25.608285 14571 consensus_queue.cc:1035] T 197ec14bb8fa494da81637271913393f P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:25.612740 6314 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
W20240503 02:00:25.612957 6301 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
W20240503 02:00:25.615257 6301 catalog_manager.cc:4573] TS e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3d3463f1ff2b41a396bb1b039ddf1946: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:25.619374 14611 tablet.cc:1722] T 65ca02bf14bf4cec8b5cd90c305e349e P c2ac2eb7cc264047b9ab32fe3a4fc025: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:25.619891 14611 tablet.cc:1620] T 65ca02bf14bf4cec8b5cd90c305e349e P c2ac2eb7cc264047b9ab32fe3a4fc025: MemRowSet was empty: no flush needed.
I20240503 02:00:25.619886 14610 tablet.cc:1722] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:25.620255 14613 tablet.cc:1722] T 65ca02bf14bf4cec8b5cd90c305e349e P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:25.620548 14610 tablet.cc:1620] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a: MemRowSet was empty: no flush needed.
I20240503 02:00:25.620584 14613 tablet.cc:1620] T 65ca02bf14bf4cec8b5cd90c305e349e P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
I20240503 02:00:25.626791 6313 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=e9e0570a970943cb845b71a46c761153] alter complete (version 2)
W20240503 02:00:25.665956 6785 consensus_peers.cc:485] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:25.666424 6654 consensus_peers.cc:485] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 02:00:25.707300 6309 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_extra_table_66"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:25.714625 6438 tablet_service.cc:1467] Processing CreateTablet for tablet 1dfbfcb60a3c4537a32f496e6772c916 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=45810e9ec99140aa91640e149c149851]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:25.714733 9028 tablet_service.cc:1467] Processing CreateTablet for tablet 1dfbfcb60a3c4537a32f496e6772c916 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=45810e9ec99140aa91640e149c149851]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:25.714733 6831 tablet_service.cc:1467] Processing CreateTablet for tablet 1dfbfcb60a3c4537a32f496e6772c916 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=45810e9ec99140aa91640e149c149851]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:25.714983 6438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dfbfcb60a3c4537a32f496e6772c916. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.715044 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dfbfcb60a3c4537a32f496e6772c916. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.715039 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dfbfcb60a3c4537a32f496e6772c916. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.717712 14623 tablet_bootstrap.cc:492] T 1dfbfcb60a3c4537a32f496e6772c916 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:25.717491 14621 tablet_bootstrap.cc:492] T 1dfbfcb60a3c4537a32f496e6772c916 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:25.718637 6568 tablet_service.cc:1467] Processing CreateTablet for tablet 9c00b9bef94d4890ae17b04b8ff3f517 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=45810e9ec99140aa91640e149c149851]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:25.718637 9157 tablet_service.cc:1467] Processing CreateTablet for tablet 9c00b9bef94d4890ae17b04b8ff3f517 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=45810e9ec99140aa91640e149c149851]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:25.719076 6568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c00b9bef94d4890ae17b04b8ff3f517. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.719076 9157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c00b9bef94d4890ae17b04b8ff3f517. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.719501 9159 tablet_service.cc:1467] Processing CreateTablet for tablet 41b2718b68ea4714b5d9a0deac130030 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=45810e9ec99140aa91640e149c149851]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:25.720063 14621 tablet_bootstrap.cc:654] T 1dfbfcb60a3c4537a32f496e6772c916 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.720860 14622 tablet_bootstrap.cc:492] T 1dfbfcb60a3c4537a32f496e6772c916 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:25.721030 9028 tablet_service.cc:1467] Processing CreateTablet for tablet 41b2718b68ea4714b5d9a0deac130030 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=45810e9ec99140aa91640e149c149851]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:25.721258 9159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41b2718b68ea4714b5d9a0deac130030. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.721288 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41b2718b68ea4714b5d9a0deac130030. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.722259 14623 tablet_bootstrap.cc:654] T 1dfbfcb60a3c4537a32f496e6772c916 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.723006 14624 tablet_bootstrap.cc:492] T 9c00b9bef94d4890ae17b04b8ff3f517 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:25.724332 14622 tablet_bootstrap.cc:654] T 1dfbfcb60a3c4537a32f496e6772c916 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.724318 6700 tablet_service.cc:1467] Processing CreateTablet for tablet 41b2718b68ea4714b5d9a0deac130030 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=45810e9ec99140aa91640e149c149851]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:25.724660 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41b2718b68ea4714b5d9a0deac130030. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.726016 14626 tablet_bootstrap.cc:492] T 41b2718b68ea4714b5d9a0deac130030 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:25.726555 6438 tablet_service.cc:1467] Processing CreateTablet for tablet 9c00b9bef94d4890ae17b04b8ff3f517 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=45810e9ec99140aa91640e149c149851]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:25.726822 6438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c00b9bef94d4890ae17b04b8ff3f517. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.726974 14621 tablet_bootstrap.cc:492] T 1dfbfcb60a3c4537a32f496e6772c916 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:25.727053 14621 ts_tablet_manager.cc:1367] T 1dfbfcb60a3c4537a32f496e6772c916 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.002s
I20240503 02:00:25.727607 14622 tablet_bootstrap.cc:492] T 1dfbfcb60a3c4537a32f496e6772c916 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:25.727710 14622 ts_tablet_manager.cc:1367] T 1dfbfcb60a3c4537a32f496e6772c916 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:00:25.728219 14622 raft_consensus.cc:348] T 1dfbfcb60a3c4537a32f496e6772c916 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:25.728334 14622 raft_consensus.cc:374] T 1dfbfcb60a3c4537a32f496e6772c916 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.728364 14622 raft_consensus.cc:729] T 1dfbfcb60a3c4537a32f496e6772c916 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.728890 14621 raft_consensus.cc:348] T 1dfbfcb60a3c4537a32f496e6772c916 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:25.729022 14621 raft_consensus.cc:374] T 1dfbfcb60a3c4537a32f496e6772c916 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.729060 14621 raft_consensus.cc:729] T 1dfbfcb60a3c4537a32f496e6772c916 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.729226 14621 consensus_queue.cc:260] T 1dfbfcb60a3c4537a32f496e6772c916 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:25.729651 14622 consensus_queue.cc:260] T 1dfbfcb60a3c4537a32f496e6772c916 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:25.730234 14624 tablet_bootstrap.cc:654] T 9c00b9bef94d4890ae17b04b8ff3f517 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.730945 6313 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:25.731729 14622 ts_tablet_manager.cc:1398] T 1dfbfcb60a3c4537a32f496e6772c916 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 02:00:25.731938 14622 tablet_bootstrap.cc:492] T 41b2718b68ea4714b5d9a0deac130030 P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:25.733117 14622 tablet_bootstrap.cc:654] T 41b2718b68ea4714b5d9a0deac130030 P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.733350 14621 ts_tablet_manager.cc:1398] T 1dfbfcb60a3c4537a32f496e6772c916 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240503 02:00:25.733619 14621 tablet_bootstrap.cc:492] T 9c00b9bef94d4890ae17b04b8ff3f517 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:25.735510 14624 tablet_bootstrap.cc:492] T 9c00b9bef94d4890ae17b04b8ff3f517 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:25.735639 14624 ts_tablet_manager.cc:1367] T 9c00b9bef94d4890ae17b04b8ff3f517 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 02:00:25.736166 14624 raft_consensus.cc:348] T 9c00b9bef94d4890ae17b04b8ff3f517 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:25.736321 14624 raft_consensus.cc:374] T 9c00b9bef94d4890ae17b04b8ff3f517 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.736357 14624 raft_consensus.cc:729] T 9c00b9bef94d4890ae17b04b8ff3f517 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.736508 14624 consensus_queue.cc:260] T 9c00b9bef94d4890ae17b04b8ff3f517 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:25.736828 14624 ts_tablet_manager.cc:1398] T 9c00b9bef94d4890ae17b04b8ff3f517 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:25.739120 14629 tablet_bootstrap.cc:492] T 41b2718b68ea4714b5d9a0deac130030 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:25.739740 14626 tablet_bootstrap.cc:654] T 41b2718b68ea4714b5d9a0deac130030 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.740859 14629 tablet_bootstrap.cc:654] T 41b2718b68ea4714b5d9a0deac130030 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.741042 14621 tablet_bootstrap.cc:654] T 9c00b9bef94d4890ae17b04b8ff3f517 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.742266 14623 tablet_bootstrap.cc:492] T 1dfbfcb60a3c4537a32f496e6772c916 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:25.742374 14623 ts_tablet_manager.cc:1367] T 1dfbfcb60a3c4537a32f496e6772c916 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.000s
I20240503 02:00:25.742884 14623 raft_consensus.cc:348] T 1dfbfcb60a3c4537a32f496e6772c916 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:25.743019 14623 raft_consensus.cc:374] T 1dfbfcb60a3c4537a32f496e6772c916 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.743052 14623 raft_consensus.cc:729] T 1dfbfcb60a3c4537a32f496e6772c916 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.743216 14629 tablet_bootstrap.cc:492] T 41b2718b68ea4714b5d9a0deac130030 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:25.743304 14629 ts_tablet_manager.cc:1367] T 41b2718b68ea4714b5d9a0deac130030 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:25.743515 14623 consensus_queue.cc:260] T 1dfbfcb60a3c4537a32f496e6772c916 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:25.743839 14623 ts_tablet_manager.cc:1398] T 1dfbfcb60a3c4537a32f496e6772c916 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:25.743814 14610 tablet.cc:1722] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:25.744155 14629 raft_consensus.cc:348] T 41b2718b68ea4714b5d9a0deac130030 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:25.744266 14629 raft_consensus.cc:374] T 41b2718b68ea4714b5d9a0deac130030 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.744295 14629 raft_consensus.cc:729] T 41b2718b68ea4714b5d9a0deac130030 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.744601 14629 consensus_queue.cc:260] T 41b2718b68ea4714b5d9a0deac130030 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:25.744858 14611 tablet.cc:1722] T 65ca02bf14bf4cec8b5cd90c305e349e P c2ac2eb7cc264047b9ab32fe3a4fc025: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:25.745522 14629 ts_tablet_manager.cc:1398] T 41b2718b68ea4714b5d9a0deac130030 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:25.745893 14621 tablet_bootstrap.cc:492] T 9c00b9bef94d4890ae17b04b8ff3f517 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:25.745980 14621 ts_tablet_manager.cc:1367] T 9c00b9bef94d4890ae17b04b8ff3f517 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.002s
I20240503 02:00:25.746492 14621 raft_consensus.cc:348] T 9c00b9bef94d4890ae17b04b8ff3f517 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:25.746603 14621 raft_consensus.cc:374] T 9c00b9bef94d4890ae17b04b8ff3f517 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.746631 14621 raft_consensus.cc:729] T 9c00b9bef94d4890ae17b04b8ff3f517 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.746742 14621 consensus_queue.cc:260] T 9c00b9bef94d4890ae17b04b8ff3f517 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:25.747057 14621 ts_tablet_manager.cc:1398] T 9c00b9bef94d4890ae17b04b8ff3f517 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:25.752158 14626 tablet_bootstrap.cc:492] T 41b2718b68ea4714b5d9a0deac130030 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:25.752288 14626 ts_tablet_manager.cc:1367] T 41b2718b68ea4714b5d9a0deac130030 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.000s
I20240503 02:00:25.752858 14626 raft_consensus.cc:348] T 41b2718b68ea4714b5d9a0deac130030 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:25.753001 14626 raft_consensus.cc:374] T 41b2718b68ea4714b5d9a0deac130030 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.753037 14626 raft_consensus.cc:729] T 41b2718b68ea4714b5d9a0deac130030 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.753170 14626 consensus_queue.cc:260] T 41b2718b68ea4714b5d9a0deac130030 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:25.753492 14626 ts_tablet_manager.cc:1398] T 41b2718b68ea4714b5d9a0deac130030 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:25.753659 14626 tablet_bootstrap.cc:492] T 9c00b9bef94d4890ae17b04b8ff3f517 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:25.755906 6313 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=e9e0570a970943cb845b71a46c761153] alter complete (version 3)
I20240503 02:00:25.758108 14622 tablet_bootstrap.cc:492] T 41b2718b68ea4714b5d9a0deac130030 P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:25.758168 14462 raft_consensus.cc:482] T 9c00b9bef94d4890ae17b04b8ff3f517 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:25.758231 14622 ts_tablet_manager.cc:1367] T 41b2718b68ea4714b5d9a0deac130030 P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.000s
I20240503 02:00:25.758256 14462 raft_consensus.cc:504] T 9c00b9bef94d4890ae17b04b8ff3f517 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:25.758678 14462 leader_election.cc:288] T 9c00b9bef94d4890ae17b04b8ff3f517 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:25.758869 14622 raft_consensus.cc:348] T 41b2718b68ea4714b5d9a0deac130030 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:25.758985 14626 tablet_bootstrap.cc:654] T 9c00b9bef94d4890ae17b04b8ff3f517 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.758991 14622 raft_consensus.cc:374] T 41b2718b68ea4714b5d9a0deac130030 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.759044 14622 raft_consensus.cc:729] T 41b2718b68ea4714b5d9a0deac130030 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.759171 14622 consensus_queue.cc:260] T 41b2718b68ea4714b5d9a0deac130030 P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:25.759238 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c00b9bef94d4890ae17b04b8ff3f517" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 02:00:25.759382 6456 raft_consensus.cc:2454] T 9c00b9bef94d4890ae17b04b8ff3f517 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:25.759469 14622 ts_tablet_manager.cc:1398] T 41b2718b68ea4714b5d9a0deac130030 P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:25.760177 14626 tablet_bootstrap.cc:492] T 9c00b9bef94d4890ae17b04b8ff3f517 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:25.760283 14626 ts_tablet_manager.cc:1367] T 9c00b9bef94d4890ae17b04b8ff3f517 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240503 02:00:25.760586 6522 leader_election.cc:302] T 9c00b9bef94d4890ae17b04b8ff3f517 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:25.760598 9177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c00b9bef94d4890ae17b04b8ff3f517" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:25.760675 14613 tablet.cc:1722] T 65ca02bf14bf4cec8b5cd90c305e349e P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:25.760787 14626 raft_consensus.cc:348] T 9c00b9bef94d4890ae17b04b8ff3f517 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:25.760876 14626 raft_consensus.cc:374] T 9c00b9bef94d4890ae17b04b8ff3f517 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.760905 14626 raft_consensus.cc:729] T 9c00b9bef94d4890ae17b04b8ff3f517 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.760902 14462 raft_consensus.cc:2781] T 9c00b9bef94d4890ae17b04b8ff3f517 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:25.760954 14462 raft_consensus.cc:482] T 9c00b9bef94d4890ae17b04b8ff3f517 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:25.760988 14462 raft_consensus.cc:3037] T 9c00b9bef94d4890ae17b04b8ff3f517 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:25.761037 14626 consensus_queue.cc:260] T 9c00b9bef94d4890ae17b04b8ff3f517 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:25.761318 14626 ts_tablet_manager.cc:1398] T 9c00b9bef94d4890ae17b04b8ff3f517 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:25.762126 14462 raft_consensus.cc:504] T 9c00b9bef94d4890ae17b04b8ff3f517 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:25.762487 14462 leader_election.cc:288] T 9c00b9bef94d4890ae17b04b8ff3f517 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Requested vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:25.763386 14552 raft_consensus.cc:482] T 1dfbfcb60a3c4537a32f496e6772c916 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:25.763486 14552 raft_consensus.cc:504] T 1dfbfcb60a3c4537a32f496e6772c916 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
W20240503 02:00:25.763823 6522 leader_election.cc:341] T 9c00b9bef94d4890ae17b04b8ff3f517 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:00:25.763870 14552 leader_election.cc:288] T 1dfbfcb60a3c4537a32f496e6772c916 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 02:00:25.764250 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c00b9bef94d4890ae17b04b8ff3f517" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 02:00:25.764407 6456 raft_consensus.cc:3037] T 9c00b9bef94d4890ae17b04b8ff3f517 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:25.764412 9177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c00b9bef94d4890ae17b04b8ff3f517" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:25.764537 9177 raft_consensus.cc:3037] T 9c00b9bef94d4890ae17b04b8ff3f517 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:25.764678 6458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dfbfcb60a3c4537a32f496e6772c916" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 02:00:25.764777 6458 raft_consensus.cc:2454] T 1dfbfcb60a3c4537a32f496e6772c916 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:25.765249 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dfbfcb60a3c4537a32f496e6772c916" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 02:00:25.765401 6849 raft_consensus.cc:2454] T 1dfbfcb60a3c4537a32f496e6772c916 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:25.765532 6456 raft_consensus.cc:2454] T 9c00b9bef94d4890ae17b04b8ff3f517 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:25.765605 9177 raft_consensus.cc:2454] T 9c00b9bef94d4890ae17b04b8ff3f517 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:25.766320 8981 leader_election.cc:302] T 1dfbfcb60a3c4537a32f496e6772c916 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99ead02915954b4fab23b033ad6d546b, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:25.767789 14552 raft_consensus.cc:2781] T 1dfbfcb60a3c4537a32f496e6772c916 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:25.767863 14552 raft_consensus.cc:482] T 1dfbfcb60a3c4537a32f496e6772c916 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:25.767899 14552 raft_consensus.cc:3037] T 1dfbfcb60a3c4537a32f496e6772c916 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:25.768566 6522 leader_election.cc:302] T 9c00b9bef94d4890ae17b04b8ff3f517 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 80fd18ef4d604c86b909d5c008a46b4f; no voters:
I20240503 02:00:25.768900 14462 raft_consensus.cc:2781] T 9c00b9bef94d4890ae17b04b8ff3f517 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:25.768980 14462 raft_consensus.cc:686] T 9c00b9bef94d4890ae17b04b8ff3f517 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Becoming Leader. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Running, Role: LEADER
I20240503 02:00:25.768960 14552 raft_consensus.cc:504] T 1dfbfcb60a3c4537a32f496e6772c916 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:25.769116 14462 consensus_queue.cc:237] T 9c00b9bef94d4890ae17b04b8ff3f517 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:25.769345 14552 leader_election.cc:288] T 1dfbfcb60a3c4537a32f496e6772c916 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Requested vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093)
I20240503 02:00:25.772054 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dfbfcb60a3c4537a32f496e6772c916" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 02:00:25.772194 6849 raft_consensus.cc:3037] T 1dfbfcb60a3c4537a32f496e6772c916 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:25.773339 6849 raft_consensus.cc:2454] T 1dfbfcb60a3c4537a32f496e6772c916 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:25.774602 8984 leader_election.cc:302] T 1dfbfcb60a3c4537a32f496e6772c916 P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:25.776607 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dfbfcb60a3c4537a32f496e6772c916" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 02:00:25.776746 6456 raft_consensus.cc:3037] T 1dfbfcb60a3c4537a32f496e6772c916 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:25.777216 6309 catalog_manager.cc:5484] T 9c00b9bef94d4890ae17b04b8ff3f517 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:25.777876 6456 raft_consensus.cc:2454] T 1dfbfcb60a3c4537a32f496e6772c916 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:25.782222 14195 raft_consensus.cc:2781] T 1dfbfcb60a3c4537a32f496e6772c916 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:25.782426 14195 raft_consensus.cc:686] T 1dfbfcb60a3c4537a32f496e6772c916 P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Becoming Leader. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Running, Role: LEADER
I20240503 02:00:25.782568 14195 consensus_queue.cc:237] T 1dfbfcb60a3c4537a32f496e6772c916 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:25.789299 6313 catalog_manager.cc:5484] T 1dfbfcb60a3c4537a32f496e6772c916 P 99ead02915954b4fab23b033ad6d546b reported cstate change: term changed from 0 to 1, leader changed from <none> to 99ead02915954b4fab23b033ad6d546b (127.0.99.134). New cstate: current_term: 1 leader_uuid: "99ead02915954b4fab23b033ad6d546b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:25.863520 6313 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:25.863705 6313 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:25.865805 6313 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet 65ca02bf14bf4cec8b5cd90c305e349e
I20240503 02:00:25.866464 6700 tablet_service.cc:1514] Processing DeleteTablet for tablet 65ca02bf14bf4cec8b5cd90c305e349e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:25 UTC) from {username='slave'} at 127.0.0.1:40336
I20240503 02:00:25.866519 9157 tablet_service.cc:1514] Processing DeleteTablet for tablet 65ca02bf14bf4cec8b5cd90c305e349e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:25 UTC) from {username='slave'} at 127.0.0.1:51940
I20240503 02:00:25.866524 6438 tablet_service.cc:1514] Processing DeleteTablet for tablet 65ca02bf14bf4cec8b5cd90c305e349e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:25 UTC) from {username='slave'} at 127.0.0.1:51338
I20240503 02:00:25.866865 14639 tablet_replica.cc:330] T 65ca02bf14bf4cec8b5cd90c305e349e P c2ac2eb7cc264047b9ab32fe3a4fc025: stopping tablet replica
I20240503 02:00:25.867038 14641 tablet_replica.cc:330] T 65ca02bf14bf4cec8b5cd90c305e349e P 2a01ace80baf4057b6bf1995cf88468b: stopping tablet replica
I20240503 02:00:25.867076 14639 raft_consensus.cc:2229] T 65ca02bf14bf4cec8b5cd90c305e349e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:25.867071 14640 tablet_replica.cc:330] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a: stopping tablet replica
I20240503 02:00:25.867182 14641 raft_consensus.cc:2229] T 65ca02bf14bf4cec8b5cd90c305e349e P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:25.867219 14639 raft_consensus.cc:2258] T 65ca02bf14bf4cec8b5cd90c305e349e P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:25.867331 14641 raft_consensus.cc:2258] T 65ca02bf14bf4cec8b5cd90c305e349e P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:25.867550 14640 raft_consensus.cc:2229] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:25.867821 14639 ts_tablet_manager.cc:1875] T 65ca02bf14bf4cec8b5cd90c305e349e P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:25.867832 14641 ts_tablet_manager.cc:1875] T 65ca02bf14bf4cec8b5cd90c305e349e P 2a01ace80baf4057b6bf1995cf88468b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:25.867861 14640 raft_consensus.cc:2258] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:25.868423 14640 ts_tablet_manager.cc:1875] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:25.870321 14641 ts_tablet_manager.cc:1888] T 65ca02bf14bf4cec8b5cd90c305e349e P 2a01ace80baf4057b6bf1995cf88468b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:25.870321 14639 ts_tablet_manager.cc:1888] T 65ca02bf14bf4cec8b5cd90c305e349e P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:25.870405 14639 log.cc:1196] T 65ca02bf14bf4cec8b5cd90c305e349e P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/65ca02bf14bf4cec8b5cd90c305e349e
I20240503 02:00:25.870455 14641 log.cc:1196] T 65ca02bf14bf4cec8b5cd90c305e349e P 2a01ace80baf4057b6bf1995cf88468b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/65ca02bf14bf4cec8b5cd90c305e349e
I20240503 02:00:25.870760 14641 ts_tablet_manager.cc:1909] T 65ca02bf14bf4cec8b5cd90c305e349e P 2a01ace80baf4057b6bf1995cf88468b: Deleting consensus metadata
I20240503 02:00:25.870770 14639 ts_tablet_manager.cc:1909] T 65ca02bf14bf4cec8b5cd90c305e349e P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting consensus metadata
I20240503 02:00:25.871698 14640 ts_tablet_manager.cc:1888] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:25.871829 14640 log.cc:1196] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/65ca02bf14bf4cec8b5cd90c305e349e
I20240503 02:00:25.871804 6300 catalog_manager.cc:4830] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079): tablet 65ca02bf14bf4cec8b5cd90c305e349e (table rebalancer_dynamic_table_new_name [id=e9e0570a970943cb845b71a46c761153]) successfully deleted
I20240503 02:00:25.871917 6303 catalog_manager.cc:4830] TS c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655): tablet 65ca02bf14bf4cec8b5cd90c305e349e (table rebalancer_dynamic_table_new_name [id=e9e0570a970943cb845b71a46c761153]) successfully deleted
I20240503 02:00:25.872298 14640 ts_tablet_manager.cc:1909] T 65ca02bf14bf4cec8b5cd90c305e349e P dec3e4a04d604f7eaaa3f077d081218a: Deleting consensus metadata
I20240503 02:00:25.873080 6300 catalog_manager.cc:4830] TS dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): tablet 65ca02bf14bf4cec8b5cd90c305e349e (table rebalancer_dynamic_table_new_name [id=e9e0570a970943cb845b71a46c761153]) successfully deleted
I20240503 02:00:25.968089 6313 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:00:25.974514 6700 tablet_service.cc:1467] Processing CreateTablet for tablet b73df7c0cc5643f3b77f8f276acb475f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1b4f9b18e1ba4a5c97eb865e166f296e]), partition=
I20240503 02:00:25.974854 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b73df7c0cc5643f3b77f8f276acb475f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.976835 6831 tablet_service.cc:1467] Processing CreateTablet for tablet b73df7c0cc5643f3b77f8f276acb475f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1b4f9b18e1ba4a5c97eb865e166f296e]), partition=
I20240503 02:00:25.977128 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b73df7c0cc5643f3b77f8f276acb475f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.977470 14629 tablet_bootstrap.cc:492] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:25.979085 14629 tablet_bootstrap.cc:654] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.979147 14623 tablet_bootstrap.cc:492] T b73df7c0cc5643f3b77f8f276acb475f P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:25.979887 6568 tablet_service.cc:1467] Processing CreateTablet for tablet b73df7c0cc5643f3b77f8f276acb475f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1b4f9b18e1ba4a5c97eb865e166f296e]), partition=
I20240503 02:00:25.980203 6568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b73df7c0cc5643f3b77f8f276acb475f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.980826 14623 tablet_bootstrap.cc:654] T b73df7c0cc5643f3b77f8f276acb475f P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.982398 14623 tablet_bootstrap.cc:492] T b73df7c0cc5643f3b77f8f276acb475f P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:25.982523 14623 ts_tablet_manager.cc:1367] T b73df7c0cc5643f3b77f8f276acb475f P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:25.982677 14624 tablet_bootstrap.cc:492] T b73df7c0cc5643f3b77f8f276acb475f P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:25.983074 14623 raft_consensus.cc:348] T b73df7c0cc5643f3b77f8f276acb475f P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:25.983246 14623 raft_consensus.cc:374] T b73df7c0cc5643f3b77f8f276acb475f P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.983283 14623 raft_consensus.cc:729] T b73df7c0cc5643f3b77f8f276acb475f P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.983436 14623 consensus_queue.cc:260] T b73df7c0cc5643f3b77f8f276acb475f P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:25.983695 14624 tablet_bootstrap.cc:654] T b73df7c0cc5643f3b77f8f276acb475f P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:25.983736 14623 ts_tablet_manager.cc:1398] T b73df7c0cc5643f3b77f8f276acb475f P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:25.985011 14624 tablet_bootstrap.cc:492] T b73df7c0cc5643f3b77f8f276acb475f P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:25.985113 14624 ts_tablet_manager.cc:1367] T b73df7c0cc5643f3b77f8f276acb475f P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:25.985143 14629 tablet_bootstrap.cc:492] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:25.985268 14629 ts_tablet_manager.cc:1367] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240503 02:00:25.985550 14624 raft_consensus.cc:348] T b73df7c0cc5643f3b77f8f276acb475f P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:25.985644 14624 raft_consensus.cc:374] T b73df7c0cc5643f3b77f8f276acb475f P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.985664 14624 raft_consensus.cc:729] T b73df7c0cc5643f3b77f8f276acb475f P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.985774 14624 consensus_queue.cc:260] T b73df7c0cc5643f3b77f8f276acb475f P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:25.985777 14629 raft_consensus.cc:348] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:25.985888 14629 raft_consensus.cc:374] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:25.986047 14624 ts_tablet_manager.cc:1398] T b73df7c0cc5643f3b77f8f276acb475f P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:25.985917 14629 raft_consensus.cc:729] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.987157 14629 consensus_queue.cc:260] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:25.988981 14629 ts_tablet_manager.cc:1398] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
W20240503 02:00:25.993088 6654 consensus_peers.cc:485] T ba17d4d3e0754b2798eb95484d37229a P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:25.995294 8851 consensus_peers.cc:485] T 067f54011a37432d99157884e70fc601 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:26.041015 9113 consensus_peers.cc:485] T f44b59277c3b47adbd4dd6aff9350af1 P 2a01ace80baf4057b6bf1995cf88468b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:26.102427 6523 consensus_peers.cc:485] T 4385d51d11574192883d01376e2d5620 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:26.121910 6523 consensus_peers.cc:485] T 3416b6107e1848f299ac49ccd88ae7c8 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:26.122246 8982 consensus_peers.cc:485] T 99bc944f874248b5a778971f7a2aaa0c P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:26.136173 6785 consensus_peers.cc:485] T 8246488dec204bf6b09e868d7951ac5a P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 02:00:26.144038 14386 raft_consensus.cc:482] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:26.144276 14386 raft_consensus.cc:504] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
W20240503 02:00:26.144830 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:26.145053 14386 leader_election.cc:288] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:26.145597 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b73df7c0cc5643f3b77f8f276acb475f" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 02:00:26.145624 6587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b73df7c0cc5643f3b77f8f276acb475f" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 02:00:26.145828 6587 raft_consensus.cc:2454] T b73df7c0cc5643f3b77f8f276acb475f P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:26.145835 6849 raft_consensus.cc:2454] T b73df7c0cc5643f3b77f8f276acb475f P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:26.146206 6655 leader_election.cc:302] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 80fd18ef4d604c86b909d5c008a46b4f, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:26.146863 14386 raft_consensus.cc:2781] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:26.146939 14386 raft_consensus.cc:482] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:26.147001 14386 raft_consensus.cc:3037] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.148154 14386 raft_consensus.cc:504] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:26.148572 14386 leader_election.cc:288] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Requested vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:26.149008 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b73df7c0cc5643f3b77f8f276acb475f" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 02:00:26.149160 6589 raft_consensus.cc:3037] T b73df7c0cc5643f3b77f8f276acb475f P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.149451 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b73df7c0cc5643f3b77f8f276acb475f" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 02:00:26.149585 6849 raft_consensus.cc:3037] T b73df7c0cc5643f3b77f8f276acb475f P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.150118 6589 raft_consensus.cc:2454] T b73df7c0cc5643f3b77f8f276acb475f P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:26.150494 6849 raft_consensus.cc:2454] T b73df7c0cc5643f3b77f8f276acb475f P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:26.151021 6656 leader_election.cc:302] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:26.151417 14190 raft_consensus.cc:2781] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:26.151564 14190 raft_consensus.cc:686] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Becoming Leader. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Running, Role: LEADER
I20240503 02:00:26.151719 14190 consensus_queue.cc:237] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:26.155071 6308 catalog_manager.cc:5484] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:26.163467 14386 raft_consensus.cc:482] T 41b2718b68ea4714b5d9a0deac130030 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:26.163859 14386 raft_consensus.cc:504] T 41b2718b68ea4714b5d9a0deac130030 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:26.164744 14386 leader_election.cc:288] T 41b2718b68ea4714b5d9a0deac130030 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:26.164886 14562 consensus_queue.cc:1035] T 9c00b9bef94d4890ae17b04b8ff3f517 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:26.166245 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41b2718b68ea4714b5d9a0deac130030" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:26.166393 9178 raft_consensus.cc:2454] T 41b2718b68ea4714b5d9a0deac130030 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:26.166708 6653 leader_election.cc:302] T 41b2718b68ea4714b5d9a0deac130030 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:26.169003 14386 raft_consensus.cc:2781] T 41b2718b68ea4714b5d9a0deac130030 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:26.169099 14386 raft_consensus.cc:482] T 41b2718b68ea4714b5d9a0deac130030 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:26.169023 9047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41b2718b68ea4714b5d9a0deac130030" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:26.169148 14386 raft_consensus.cc:3037] T 41b2718b68ea4714b5d9a0deac130030 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.169206 9047 raft_consensus.cc:2454] T 41b2718b68ea4714b5d9a0deac130030 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:26.171553 14462 consensus_queue.cc:1035] T 9c00b9bef94d4890ae17b04b8ff3f517 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:26.175287 14386 raft_consensus.cc:504] T 41b2718b68ea4714b5d9a0deac130030 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:26.176406 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41b2718b68ea4714b5d9a0deac130030" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:26.176541 9178 raft_consensus.cc:3037] T 41b2718b68ea4714b5d9a0deac130030 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.177424 14386 leader_election.cc:288] T 41b2718b68ea4714b5d9a0deac130030 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Requested vote from peers 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:26.177488 9047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41b2718b68ea4714b5d9a0deac130030" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:26.177548 9178 raft_consensus.cc:2454] T 41b2718b68ea4714b5d9a0deac130030 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:26.177601 9047 raft_consensus.cc:3037] T 41b2718b68ea4714b5d9a0deac130030 P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.177995 6653 leader_election.cc:302] T 41b2718b68ea4714b5d9a0deac130030 P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:26.178236 14386 raft_consensus.cc:2781] T 41b2718b68ea4714b5d9a0deac130030 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:26.178400 14386 raft_consensus.cc:686] T 41b2718b68ea4714b5d9a0deac130030 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Becoming Leader. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Running, Role: LEADER
I20240503 02:00:26.178571 14386 consensus_queue.cc:237] T 41b2718b68ea4714b5d9a0deac130030 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } }
I20240503 02:00:26.178684 9047 raft_consensus.cc:2454] T 41b2718b68ea4714b5d9a0deac130030 P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:26.181385 6309 catalog_manager.cc:5484] T 41b2718b68ea4714b5d9a0deac130030 P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } }
W20240503 02:00:26.247449 6785 consensus_peers.cc:485] T 46ad540deb624a25b97d3b967fe1f6f4 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:26.265272 6785 consensus_peers.cc:485] T b2e62c043adf4a8d9024279d786f596c P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:00:26.285496 6309 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:26.289940 6589 raft_consensus.cc:1261] T b73df7c0cc5643f3b77f8f276acb475f P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:26.290189 6849 raft_consensus.cc:1261] T b73df7c0cc5643f3b77f8f276acb475f P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer c2ac2eb7cc264047b9ab32fe3a4fc025: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:26.290637 14190 consensus_queue.cc:1035] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:26.290999 14190 consensus_queue.cc:1035] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:26.294981 14659 tablet.cc:1722] T b73df7c0cc5643f3b77f8f276acb475f P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:26.295634 14659 tablet.cc:1620] T b73df7c0cc5643f3b77f8f276acb475f P 80fd18ef4d604c86b909d5c008a46b4f: MemRowSet was empty: no flush needed.
I20240503 02:00:26.296214 6308 catalog_manager.cc:5576] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719) does not have the latest schema for tablet b73df7c0cc5643f3b77f8f276acb475f (table rebalancer_dynamic_table [id=1b4f9b18e1ba4a5c97eb865e166f296e]). Expected version 1 got 0
I20240503 02:00:26.294782 14658 tablet.cc:1722] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:26.297613 14658 tablet.cc:1620] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025: MemRowSet was empty: no flush needed.
I20240503 02:00:26.298339 6313 catalog_manager.cc:5576] TS c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655) does not have the latest schema for tablet b73df7c0cc5643f3b77f8f276acb475f (table rebalancer_dynamic_table [id=1b4f9b18e1ba4a5c97eb865e166f296e]). Expected version 1 got 0
I20240503 02:00:26.298650 6309 catalog_manager.cc:5576] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863) does not have the latest schema for tablet b73df7c0cc5643f3b77f8f276acb475f (table rebalancer_dynamic_table [id=1b4f9b18e1ba4a5c97eb865e166f296e]). Expected version 1 got 0
I20240503 02:00:26.299626 14660 tablet.cc:1722] T b73df7c0cc5643f3b77f8f276acb475f P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:26.299896 14660 tablet.cc:1620] T b73df7c0cc5643f3b77f8f276acb475f P 492448570cc94ed1879feb2ec3b9051d: MemRowSet was empty: no flush needed.
I20240503 02:00:26.301301 6313 catalog_manager.cc:5576] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719) does not have the latest schema for tablet b73df7c0cc5643f3b77f8f276acb475f (table rebalancer_dynamic_table [id=1b4f9b18e1ba4a5c97eb865e166f296e]). Expected version 1 got 0
I20240503 02:00:26.304632 6308 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=1b4f9b18e1ba4a5c97eb865e166f296e] alter complete (version 1)
W20240503 02:00:26.332499 6523 consensus_peers.cc:485] T 925612b660ec471da5b3c548802ede6e P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 02:00:26.358970 14552 consensus_queue.cc:1035] T 1dfbfcb60a3c4537a32f496e6772c916 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:26.364965 14552 consensus_queue.cc:1035] T 1dfbfcb60a3c4537a32f496e6772c916 P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:00:26.380420 9113 consensus_peers.cc:485] T b1740b877c1e4372a0337d99f5bcab0e P 2a01ace80baf4057b6bf1995cf88468b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:26.388870 6785 consensus_peers.cc:485] T d897c21e630d4770832bb10f43778e82 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:26.405189 8851 consensus_peers.cc:485] T 9f6bcf448d424d3e8bb18bf12cfb64c5 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 02:00:26.410620 6308 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:26.416625 14658 tablet.cc:1722] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:26.416998 14658 tablet.cc:1620] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025: MemRowSet was empty: no flush needed.
I20240503 02:00:26.417070 14659 tablet.cc:1722] T b73df7c0cc5643f3b77f8f276acb475f P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:26.417364 14659 tablet.cc:1620] T b73df7c0cc5643f3b77f8f276acb475f P 80fd18ef4d604c86b909d5c008a46b4f: MemRowSet was empty: no flush needed.
I20240503 02:00:26.417362 14660 tablet.cc:1722] T b73df7c0cc5643f3b77f8f276acb475f P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:26.417670 14660 tablet.cc:1620] T b73df7c0cc5643f3b77f8f276acb475f P 492448570cc94ed1879feb2ec3b9051d: MemRowSet was empty: no flush needed.
I20240503 02:00:26.423130 6308 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=1b4f9b18e1ba4a5c97eb865e166f296e] alter complete (version 2)
W20240503 02:00:26.436861 6654 consensus_peers.cc:485] T cea1ea50fa434d31b42451f84f26bc76 P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 02:00:26.526170 6309 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:26.532155 14658 tablet.cc:1722] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:26.532887 14659 tablet.cc:1722] T b73df7c0cc5643f3b77f8f276acb475f P 80fd18ef4d604c86b909d5c008a46b4f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:26.534089 14660 tablet.cc:1722] T b73df7c0cc5643f3b77f8f276acb475f P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
W20240503 02:00:26.535934 6523 consensus_peers.cc:485] T 8dfe23ff4f2a456384690fa9e48e51c1 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:00:26.538813 6314 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=1b4f9b18e1ba4a5c97eb865e166f296e] alter complete (version 3)
W20240503 02:00:26.571951 6523 consensus_peers.cc:485] T d0d5dd42db3e4883af7fd16d8813b550 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 02:00:26.642629 6308 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:26.642818 6308 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:26.645216 6308 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet b73df7c0cc5643f3b77f8f276acb475f
I20240503 02:00:26.645849 6700 tablet_service.cc:1514] Processing DeleteTablet for tablet b73df7c0cc5643f3b77f8f276acb475f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:26 UTC) from {username='slave'} at 127.0.0.1:40336
I20240503 02:00:26.645850 6568 tablet_service.cc:1514] Processing DeleteTablet for tablet b73df7c0cc5643f3b77f8f276acb475f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:26 UTC) from {username='slave'} at 127.0.0.1:59846
I20240503 02:00:26.645936 6831 tablet_service.cc:1514] Processing DeleteTablet for tablet b73df7c0cc5643f3b77f8f276acb475f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:26 UTC) from {username='slave'} at 127.0.0.1:52812
I20240503 02:00:26.646176 14666 tablet_replica.cc:330] T b73df7c0cc5643f3b77f8f276acb475f P 80fd18ef4d604c86b909d5c008a46b4f: stopping tablet replica
I20240503 02:00:26.646236 14667 tablet_replica.cc:330] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025: stopping tablet replica
I20240503 02:00:26.646329 14666 raft_consensus.cc:2229] T b73df7c0cc5643f3b77f8f276acb475f P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:26.646384 14667 raft_consensus.cc:2229] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:26.646467 14666 raft_consensus.cc:2258] T b73df7c0cc5643f3b77f8f276acb475f P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:26.646587 14667 raft_consensus.cc:2258] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:26.646827 14668 tablet_replica.cc:330] T b73df7c0cc5643f3b77f8f276acb475f P 492448570cc94ed1879feb2ec3b9051d: stopping tablet replica
I20240503 02:00:26.647049 14666 ts_tablet_manager.cc:1875] T b73df7c0cc5643f3b77f8f276acb475f P 80fd18ef4d604c86b909d5c008a46b4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:26.647085 14668 raft_consensus.cc:2229] T b73df7c0cc5643f3b77f8f276acb475f P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:26.647267 14668 raft_consensus.cc:2258] T b73df7c0cc5643f3b77f8f276acb475f P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:26.647277 14667 ts_tablet_manager.cc:1875] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:26.647832 14668 ts_tablet_manager.cc:1875] T b73df7c0cc5643f3b77f8f276acb475f P 492448570cc94ed1879feb2ec3b9051d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:26.649488 14666 ts_tablet_manager.cc:1888] T b73df7c0cc5643f3b77f8f276acb475f P 80fd18ef4d604c86b909d5c008a46b4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:26.649637 14666 log.cc:1196] T b73df7c0cc5643f3b77f8f276acb475f P 80fd18ef4d604c86b909d5c008a46b4f: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/b73df7c0cc5643f3b77f8f276acb475f
I20240503 02:00:26.650070 14666 ts_tablet_manager.cc:1909] T b73df7c0cc5643f3b77f8f276acb475f P 80fd18ef4d604c86b909d5c008a46b4f: Deleting consensus metadata
I20240503 02:00:26.650667 14668 ts_tablet_manager.cc:1888] T b73df7c0cc5643f3b77f8f276acb475f P 492448570cc94ed1879feb2ec3b9051d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:26.650774 14667 ts_tablet_manager.cc:1888] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:26.650823 14668 log.cc:1196] T b73df7c0cc5643f3b77f8f276acb475f P 492448570cc94ed1879feb2ec3b9051d: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/b73df7c0cc5643f3b77f8f276acb475f
I20240503 02:00:26.650882 14667 log.cc:1196] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/b73df7c0cc5643f3b77f8f276acb475f
I20240503 02:00:26.651006 6302 catalog_manager.cc:4830] TS 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863): tablet b73df7c0cc5643f3b77f8f276acb475f (table rebalancer_dynamic_table_new_name [id=1b4f9b18e1ba4a5c97eb865e166f296e]) successfully deleted
I20240503 02:00:26.651198 14667 ts_tablet_manager.cc:1909] T b73df7c0cc5643f3b77f8f276acb475f P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting consensus metadata
I20240503 02:00:26.651198 14668 ts_tablet_manager.cc:1909] T b73df7c0cc5643f3b77f8f276acb475f P 492448570cc94ed1879feb2ec3b9051d: Deleting consensus metadata
I20240503 02:00:26.652294 6303 catalog_manager.cc:4830] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719): tablet b73df7c0cc5643f3b77f8f276acb475f (table rebalancer_dynamic_table_new_name [id=1b4f9b18e1ba4a5c97eb865e166f296e]) successfully deleted
I20240503 02:00:26.652503 6303 catalog_manager.cc:4830] TS c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655): tablet b73df7c0cc5643f3b77f8f276acb475f (table rebalancer_dynamic_table_new_name [id=1b4f9b18e1ba4a5c97eb865e166f296e]) successfully deleted
I20240503 02:00:26.656827 14664 consensus_queue.cc:1035] T 41b2718b68ea4714b5d9a0deac130030 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:26.659860 14664 consensus_queue.cc:1035] T 41b2718b68ea4714b5d9a0deac130030 P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:00:26.664561 6301 catalog_manager.cc:4573] TS e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): CreateTablet RPC failed for tablet 56824b553788471c9a2fcd6ec82d507e: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
W20240503 02:00:26.681623 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:26.711624 6314 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_extra_table_68"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:26.718322 6700 tablet_service.cc:1467] Processing CreateTablet for tablet 3c1ce8f08de449c7b4e54aca59a187bc (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=af68bb4f304a47f9b6790678a88690e9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:26.718577 6438 tablet_service.cc:1467] Processing CreateTablet for tablet 3c1ce8f08de449c7b4e54aca59a187bc (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=af68bb4f304a47f9b6790678a88690e9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:26.718672 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c1ce8f08de449c7b4e54aca59a187bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.718842 6438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c1ce8f08de449c7b4e54aca59a187bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.720079 6436 tablet_service.cc:1467] Processing CreateTablet for tablet 81911ae22dfa45c886c1126cc77769dd (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=af68bb4f304a47f9b6790678a88690e9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:26.720351 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81911ae22dfa45c886c1126cc77769dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.720829 6437 tablet_service.cc:1467] Processing CreateTablet for tablet f6ce658b7326427583fd3d2e5d0b53a7 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=af68bb4f304a47f9b6790678a88690e9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:26.721045 6437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6ce658b7326427583fd3d2e5d0b53a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.721199 14672 tablet_bootstrap.cc:492] T 3c1ce8f08de449c7b4e54aca59a187bc P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:26.721421 8880 tablet_service.cc:1467] Processing CreateTablet for tablet 81911ae22dfa45c886c1126cc77769dd (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=af68bb4f304a47f9b6790678a88690e9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:26.721676 8880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81911ae22dfa45c886c1126cc77769dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.722435 14672 tablet_bootstrap.cc:654] T 3c1ce8f08de449c7b4e54aca59a187bc P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.722860 9157 tablet_service.cc:1467] Processing CreateTablet for tablet f6ce658b7326427583fd3d2e5d0b53a7 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=af68bb4f304a47f9b6790678a88690e9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:26.723145 9157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6ce658b7326427583fd3d2e5d0b53a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.724014 14675 tablet_bootstrap.cc:492] T 81911ae22dfa45c886c1126cc77769dd P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:26.724217 8887 tablet_service.cc:1467] Processing CreateTablet for tablet f6ce658b7326427583fd3d2e5d0b53a7 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=af68bb4f304a47f9b6790678a88690e9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:26.724299 14673 tablet_bootstrap.cc:492] T 81911ae22dfa45c886c1126cc77769dd P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:26.724498 8887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6ce658b7326427583fd3d2e5d0b53a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.725098 14675 tablet_bootstrap.cc:654] T 81911ae22dfa45c886c1126cc77769dd P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.725226 14673 tablet_bootstrap.cc:654] T 81911ae22dfa45c886c1126cc77769dd P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.725935 14678 tablet_bootstrap.cc:492] T f6ce658b7326427583fd3d2e5d0b53a7 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:26.726245 6568 tablet_service.cc:1467] Processing CreateTablet for tablet 81911ae22dfa45c886c1126cc77769dd (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=af68bb4f304a47f9b6790678a88690e9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:26.726516 6568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81911ae22dfa45c886c1126cc77769dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.726877 14678 tablet_bootstrap.cc:654] T f6ce658b7326427583fd3d2e5d0b53a7 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.728281 14678 tablet_bootstrap.cc:492] T f6ce658b7326427583fd3d2e5d0b53a7 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:26.728425 14678 ts_tablet_manager.cc:1367] T f6ce658b7326427583fd3d2e5d0b53a7 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:26.728956 14678 raft_consensus.cc:348] T f6ce658b7326427583fd3d2e5d0b53a7 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:26.729063 14678 raft_consensus.cc:374] T f6ce658b7326427583fd3d2e5d0b53a7 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.729092 14678 raft_consensus.cc:729] T f6ce658b7326427583fd3d2e5d0b53a7 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.729163 14675 tablet_bootstrap.cc:492] T 81911ae22dfa45c886c1126cc77769dd P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:26.729204 14678 consensus_queue.cc:260] T f6ce658b7326427583fd3d2e5d0b53a7 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:26.729255 14675 ts_tablet_manager.cc:1367] T 81911ae22dfa45c886c1126cc77769dd P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:00:26.729506 14678 ts_tablet_manager.cc:1398] T f6ce658b7326427583fd3d2e5d0b53a7 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:26.729718 14675 raft_consensus.cc:348] T 81911ae22dfa45c886c1126cc77769dd P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:26.729816 14675 raft_consensus.cc:374] T 81911ae22dfa45c886c1126cc77769dd P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.729846 14675 raft_consensus.cc:729] T 81911ae22dfa45c886c1126cc77769dd P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.729969 14675 consensus_queue.cc:260] T 81911ae22dfa45c886c1126cc77769dd P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:26.730242 14675 ts_tablet_manager.cc:1398] T 81911ae22dfa45c886c1126cc77769dd P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:26.730410 14675 tablet_bootstrap.cc:492] T f6ce658b7326427583fd3d2e5d0b53a7 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
W20240503 02:00:26.730767 8982 consensus_peers.cc:485] T 1a7ee79312ea4bad816264375ca5bb2d P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 02:00:26.731472 14675 tablet_bootstrap.cc:654] T f6ce658b7326427583fd3d2e5d0b53a7 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.731684 9028 tablet_service.cc:1467] Processing CreateTablet for tablet 3c1ce8f08de449c7b4e54aca59a187bc (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=af68bb4f304a47f9b6790678a88690e9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:26.731959 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c1ce8f08de449c7b4e54aca59a187bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.733464 14680 tablet_bootstrap.cc:492] T 81911ae22dfa45c886c1126cc77769dd P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:26.734112 14675 tablet_bootstrap.cc:492] T f6ce658b7326427583fd3d2e5d0b53a7 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:26.734208 14675 ts_tablet_manager.cc:1367] T f6ce658b7326427583fd3d2e5d0b53a7 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:26.735436 14675 raft_consensus.cc:348] T f6ce658b7326427583fd3d2e5d0b53a7 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:26.735894 14675 raft_consensus.cc:374] T f6ce658b7326427583fd3d2e5d0b53a7 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.735940 14675 raft_consensus.cc:729] T f6ce658b7326427583fd3d2e5d0b53a7 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.736083 14675 consensus_queue.cc:260] T f6ce658b7326427583fd3d2e5d0b53a7 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:26.736426 14675 ts_tablet_manager.cc:1398] T f6ce658b7326427583fd3d2e5d0b53a7 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:26.736845 14680 tablet_bootstrap.cc:654] T 81911ae22dfa45c886c1126cc77769dd P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.738224 14672 tablet_bootstrap.cc:492] T 3c1ce8f08de449c7b4e54aca59a187bc P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:26.738330 14672 ts_tablet_manager.cc:1367] T 3c1ce8f08de449c7b4e54aca59a187bc P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240503 02:00:26.738802 14672 raft_consensus.cc:348] T 3c1ce8f08de449c7b4e54aca59a187bc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:26.738930 14672 raft_consensus.cc:374] T 3c1ce8f08de449c7b4e54aca59a187bc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.738958 14672 raft_consensus.cc:729] T 3c1ce8f08de449c7b4e54aca59a187bc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.739053 14673 tablet_bootstrap.cc:492] T 81911ae22dfa45c886c1126cc77769dd P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:26.739063 14672 consensus_queue.cc:260] T 3c1ce8f08de449c7b4e54aca59a187bc P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:26.739271 14673 ts_tablet_manager.cc:1367] T 81911ae22dfa45c886c1126cc77769dd P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240503 02:00:26.739346 14672 ts_tablet_manager.cc:1398] T 3c1ce8f08de449c7b4e54aca59a187bc P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:26.739768 14673 raft_consensus.cc:348] T 81911ae22dfa45c886c1126cc77769dd P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:26.739856 14673 raft_consensus.cc:374] T 81911ae22dfa45c886c1126cc77769dd P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.739883 14673 raft_consensus.cc:729] T 81911ae22dfa45c886c1126cc77769dd P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.740264 14673 consensus_queue.cc:260] T 81911ae22dfa45c886c1126cc77769dd P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:26.740552 14673 ts_tablet_manager.cc:1398] T 81911ae22dfa45c886c1126cc77769dd P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:26.741261 14673 tablet_bootstrap.cc:492] T f6ce658b7326427583fd3d2e5d0b53a7 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
W20240503 02:00:26.742780 8982 consensus_peers.cc:485] T 13756a6e44814fe4b09a6b0f087546c3 P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 02:00:26.743142 14673 tablet_bootstrap.cc:654] T f6ce658b7326427583fd3d2e5d0b53a7 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.743388 14682 tablet_bootstrap.cc:492] T 3c1ce8f08de449c7b4e54aca59a187bc P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:26.745317 14680 tablet_bootstrap.cc:492] T 81911ae22dfa45c886c1126cc77769dd P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:26.745446 14680 ts_tablet_manager.cc:1367] T 81911ae22dfa45c886c1126cc77769dd P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 02:00:26.746013 14673 tablet_bootstrap.cc:492] T f6ce658b7326427583fd3d2e5d0b53a7 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:26.746114 14673 ts_tablet_manager.cc:1367] T f6ce658b7326427583fd3d2e5d0b53a7 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240503 02:00:26.746497 14680 raft_consensus.cc:348] T 81911ae22dfa45c886c1126cc77769dd P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:26.746634 14680 raft_consensus.cc:374] T 81911ae22dfa45c886c1126cc77769dd P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.746668 14680 raft_consensus.cc:729] T 81911ae22dfa45c886c1126cc77769dd P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.746606 14673 raft_consensus.cc:348] T f6ce658b7326427583fd3d2e5d0b53a7 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:26.746717 14673 raft_consensus.cc:374] T f6ce658b7326427583fd3d2e5d0b53a7 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.746753 14673 raft_consensus.cc:729] T f6ce658b7326427583fd3d2e5d0b53a7 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.746932 14682 tablet_bootstrap.cc:654] T 3c1ce8f08de449c7b4e54aca59a187bc P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.746788 14680 consensus_queue.cc:260] T 81911ae22dfa45c886c1126cc77769dd P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:26.747498 14680 ts_tablet_manager.cc:1398] T 81911ae22dfa45c886c1126cc77769dd P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:00:26.747740 14673 consensus_queue.cc:260] T f6ce658b7326427583fd3d2e5d0b53a7 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:26.748677 14673 ts_tablet_manager.cc:1398] T f6ce658b7326427583fd3d2e5d0b53a7 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:26.748924 6314 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:00:26.749248 14682 tablet_bootstrap.cc:492] T 3c1ce8f08de449c7b4e54aca59a187bc P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:26.749394 14682 ts_tablet_manager.cc:1367] T 3c1ce8f08de449c7b4e54aca59a187bc P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240503 02:00:26.752283 14682 raft_consensus.cc:348] T 3c1ce8f08de449c7b4e54aca59a187bc P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:26.752450 14682 raft_consensus.cc:374] T 3c1ce8f08de449c7b4e54aca59a187bc P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.752559 14673 tablet_bootstrap.cc:492] T 3c1ce8f08de449c7b4e54aca59a187bc P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:26.754281 14673 tablet_bootstrap.cc:654] T 3c1ce8f08de449c7b4e54aca59a187bc P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.754411 14682 raft_consensus.cc:729] T 3c1ce8f08de449c7b4e54aca59a187bc P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.754632 14682 consensus_queue.cc:260] T 3c1ce8f08de449c7b4e54aca59a187bc P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:26.754951 14682 ts_tablet_manager.cc:1398] T 3c1ce8f08de449c7b4e54aca59a187bc P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:00:26.756423 14673 tablet_bootstrap.cc:492] T 3c1ce8f08de449c7b4e54aca59a187bc P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:26.756546 14673 ts_tablet_manager.cc:1367] T 3c1ce8f08de449c7b4e54aca59a187bc P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20240503 02:00:26.757145 14673 raft_consensus.cc:348] T 3c1ce8f08de449c7b4e54aca59a187bc P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:26.757289 14673 raft_consensus.cc:374] T 3c1ce8f08de449c7b4e54aca59a187bc P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.757325 14673 raft_consensus.cc:729] T 3c1ce8f08de449c7b4e54aca59a187bc P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.757488 14673 consensus_queue.cc:260] T 3c1ce8f08de449c7b4e54aca59a187bc P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:26.757793 14673 ts_tablet_manager.cc:1398] T 3c1ce8f08de449c7b4e54aca59a187bc P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:26.761672 9028 tablet_service.cc:1467] Processing CreateTablet for tablet 61a3076a42dc47859a4b9ffbd873110c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=249868ae11634abf8527ed751495bcc6]), partition=
I20240503 02:00:26.761999 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61a3076a42dc47859a4b9ffbd873110c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.762239 9157 tablet_service.cc:1467] Processing CreateTablet for tablet 61a3076a42dc47859a4b9ffbd873110c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=249868ae11634abf8527ed751495bcc6]), partition=
I20240503 02:00:26.762482 9157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61a3076a42dc47859a4b9ffbd873110c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.764127 6831 tablet_service.cc:1467] Processing CreateTablet for tablet 61a3076a42dc47859a4b9ffbd873110c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=249868ae11634abf8527ed751495bcc6]), partition=
I20240503 02:00:26.764277 14678 tablet_bootstrap.cc:492] T 61a3076a42dc47859a4b9ffbd873110c P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:26.764348 14682 tablet_bootstrap.cc:492] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:26.764405 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61a3076a42dc47859a4b9ffbd873110c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:26.765246 14682 tablet_bootstrap.cc:654] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.765246 14678 tablet_bootstrap.cc:654] T 61a3076a42dc47859a4b9ffbd873110c P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.766904 14678 tablet_bootstrap.cc:492] T 61a3076a42dc47859a4b9ffbd873110c P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:26.766904 14682 tablet_bootstrap.cc:492] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:26.767002 14682 ts_tablet_manager.cc:1367] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:26.767002 14678 ts_tablet_manager.cc:1367] T 61a3076a42dc47859a4b9ffbd873110c P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:26.767457 14682 raft_consensus.cc:348] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:26.767491 14678 raft_consensus.cc:348] T 61a3076a42dc47859a4b9ffbd873110c P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:26.767567 14682 raft_consensus.cc:374] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.767573 14678 raft_consensus.cc:374] T 61a3076a42dc47859a4b9ffbd873110c P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.767597 14682 raft_consensus.cc:729] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.767602 14678 raft_consensus.cc:729] T 61a3076a42dc47859a4b9ffbd873110c P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.767722 14678 consensus_queue.cc:260] T 61a3076a42dc47859a4b9ffbd873110c P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:26.767741 14682 consensus_queue.cc:260] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:26.768002 14678 ts_tablet_manager.cc:1398] T 61a3076a42dc47859a4b9ffbd873110c P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:26.768013 14682 ts_tablet_manager.cc:1398] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:26.769730 14689 tablet_bootstrap.cc:492] T 61a3076a42dc47859a4b9ffbd873110c P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:26.771317 14689 tablet_bootstrap.cc:654] T 61a3076a42dc47859a4b9ffbd873110c P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:26.773133 14689 tablet_bootstrap.cc:492] T 61a3076a42dc47859a4b9ffbd873110c P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:26.773279 14689 ts_tablet_manager.cc:1367] T 61a3076a42dc47859a4b9ffbd873110c P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:00:26.773983 14689 raft_consensus.cc:348] T 61a3076a42dc47859a4b9ffbd873110c P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:26.774130 14689 raft_consensus.cc:374] T 61a3076a42dc47859a4b9ffbd873110c P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:26.774176 14689 raft_consensus.cc:729] T 61a3076a42dc47859a4b9ffbd873110c P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:26.774308 14689 consensus_queue.cc:260] T 61a3076a42dc47859a4b9ffbd873110c P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:26.774597 14689 ts_tablet_manager.cc:1398] T 61a3076a42dc47859a4b9ffbd873110c P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240503 02:00:26.776242 8851 consensus_peers.cc:485] T 5f51fe5f799f4e479b62f5917a2c2606 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 02:00:26.776906 14612 raft_consensus.cc:482] T 81911ae22dfa45c886c1126cc77769dd P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:26.777019 14612 raft_consensus.cc:504] T 81911ae22dfa45c886c1126cc77769dd P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:26.777480 14612 leader_election.cc:288] T 81911ae22dfa45c886c1126cc77769dd P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:26.777810 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81911ae22dfa45c886c1126cc77769dd" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 02:00:26.777967 6589 raft_consensus.cc:2454] T 81911ae22dfa45c886c1126cc77769dd P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:26.778061 8912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81911ae22dfa45c886c1126cc77769dd" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:26.778198 8912 raft_consensus.cc:2454] T 81911ae22dfa45c886c1126cc77769dd P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:26.778410 6393 leader_election.cc:302] T 81911ae22dfa45c886c1126cc77769dd P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:26.778600 14612 raft_consensus.cc:2781] T 81911ae22dfa45c886c1126cc77769dd P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:26.778662 14612 raft_consensus.cc:482] T 81911ae22dfa45c886c1126cc77769dd P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:26.778698 14612 raft_consensus.cc:3037] T 81911ae22dfa45c886c1126cc77769dd P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.779933 14612 raft_consensus.cc:504] T 81911ae22dfa45c886c1126cc77769dd P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:26.780370 14612 leader_election.cc:288] T 81911ae22dfa45c886c1126cc77769dd P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Requested vote from peers 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:26.781610 8912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81911ae22dfa45c886c1126cc77769dd" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:26.781746 8912 raft_consensus.cc:3037] T 81911ae22dfa45c886c1126cc77769dd P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.781769 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81911ae22dfa45c886c1126cc77769dd" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 02:00:26.781903 6589 raft_consensus.cc:3037] T 81911ae22dfa45c886c1126cc77769dd P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.782672 8912 raft_consensus.cc:2454] T 81911ae22dfa45c886c1126cc77769dd P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:26.782713 6589 raft_consensus.cc:2454] T 81911ae22dfa45c886c1126cc77769dd P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:26.782991 6394 leader_election.cc:302] T 81911ae22dfa45c886c1126cc77769dd P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9942306855404cfd891adf227dd65eb2, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:26.783200 14612 raft_consensus.cc:2781] T 81911ae22dfa45c886c1126cc77769dd P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:26.783277 14612 raft_consensus.cc:686] T 81911ae22dfa45c886c1126cc77769dd P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Becoming Leader. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Running, Role: LEADER
I20240503 02:00:26.783406 14612 consensus_queue.cc:237] T 81911ae22dfa45c886c1126cc77769dd P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } }
I20240503 02:00:26.791822 6313 catalog_manager.cc:5484] T 81911ae22dfa45c886c1126cc77769dd P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: term changed from 0 to 1, leader changed from <none> to dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129). New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } }
W20240503 02:00:26.803767 8851 consensus_peers.cc:485] T f28586e70bd9473fbdc6b9126b666fa9 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 02:00:26.840749 14612 raft_consensus.cc:482] T f6ce658b7326427583fd3d2e5d0b53a7 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:26.840893 14612 raft_consensus.cc:504] T f6ce658b7326427583fd3d2e5d0b53a7 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:26.841300 14612 leader_election.cc:288] T f6ce658b7326427583fd3d2e5d0b53a7 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:26.841652 9178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ce658b7326427583fd3d2e5d0b53a7" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:26.841722 8912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ce658b7326427583fd3d2e5d0b53a7" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:26.841784 9178 raft_consensus.cc:2454] T f6ce658b7326427583fd3d2e5d0b53a7 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:26.841871 8912 raft_consensus.cc:2454] T f6ce658b7326427583fd3d2e5d0b53a7 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 0.
I20240503 02:00:26.842105 6391 leader_election.cc:302] T f6ce658b7326427583fd3d2e5d0b53a7 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:26.842258 14612 raft_consensus.cc:2781] T f6ce658b7326427583fd3d2e5d0b53a7 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:26.842300 14612 raft_consensus.cc:482] T f6ce658b7326427583fd3d2e5d0b53a7 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:26.842332 14612 raft_consensus.cc:3037] T f6ce658b7326427583fd3d2e5d0b53a7 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.843436 14612 raft_consensus.cc:504] T f6ce658b7326427583fd3d2e5d0b53a7 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:26.843824 14612 leader_election.cc:288] T f6ce658b7326427583fd3d2e5d0b53a7 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Requested vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:26.844262 8912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ce658b7326427583fd3d2e5d0b53a7" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:26.844261 9177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ce658b7326427583fd3d2e5d0b53a7" candidate_uuid: "dec3e4a04d604f7eaaa3f077d081218a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:26.844405 9177 raft_consensus.cc:3037] T f6ce658b7326427583fd3d2e5d0b53a7 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.844417 8912 raft_consensus.cc:3037] T f6ce658b7326427583fd3d2e5d0b53a7 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.845263 9177 raft_consensus.cc:2454] T f6ce658b7326427583fd3d2e5d0b53a7 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:26.845243 8912 raft_consensus.cc:2454] T f6ce658b7326427583fd3d2e5d0b53a7 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec3e4a04d604f7eaaa3f077d081218a in term 1.
I20240503 02:00:26.845556 6391 leader_election.cc:302] T f6ce658b7326427583fd3d2e5d0b53a7 P dec3e4a04d604f7eaaa3f077d081218a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:26.845734 14612 raft_consensus.cc:2781] T f6ce658b7326427583fd3d2e5d0b53a7 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:26.845820 14612 raft_consensus.cc:686] T f6ce658b7326427583fd3d2e5d0b53a7 P dec3e4a04d604f7eaaa3f077d081218a [term 1 LEADER]: Becoming Leader. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Running, Role: LEADER
I20240503 02:00:26.845971 14612 consensus_queue.cc:237] T f6ce658b7326427583fd3d2e5d0b53a7 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } }
I20240503 02:00:26.848733 6313 catalog_manager.cc:5484] T f6ce658b7326427583fd3d2e5d0b53a7 P dec3e4a04d604f7eaaa3f077d081218a reported cstate change: term changed from 0 to 1, leader changed from <none> to dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129). New cstate: current_term: 1 leader_uuid: "dec3e4a04d604f7eaaa3f077d081218a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:26.875725 14664 raft_consensus.cc:482] T 3c1ce8f08de449c7b4e54aca59a187bc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:26.875864 14664 raft_consensus.cc:504] T 3c1ce8f08de449c7b4e54aca59a187bc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:26.876291 14664 leader_election.cc:288] T 3c1ce8f08de449c7b4e54aca59a187bc P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:26.876716 9047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1ce8f08de449c7b4e54aca59a187bc" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:26.876871 9047 raft_consensus.cc:2454] T 3c1ce8f08de449c7b4e54aca59a187bc P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:26.876978 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1ce8f08de449c7b4e54aca59a187bc" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 02:00:26.877102 6456 raft_consensus.cc:2454] T 3c1ce8f08de449c7b4e54aca59a187bc P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 0.
I20240503 02:00:26.877177 6654 leader_election.cc:302] T 3c1ce8f08de449c7b4e54aca59a187bc P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99ead02915954b4fab23b033ad6d546b, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:26.877393 14664 raft_consensus.cc:2781] T 3c1ce8f08de449c7b4e54aca59a187bc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:26.877465 14664 raft_consensus.cc:482] T 3c1ce8f08de449c7b4e54aca59a187bc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:26.877506 14664 raft_consensus.cc:3037] T 3c1ce8f08de449c7b4e54aca59a187bc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.878322 14664 raft_consensus.cc:504] T 3c1ce8f08de449c7b4e54aca59a187bc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:26.878696 14664 leader_election.cc:288] T 3c1ce8f08de449c7b4e54aca59a187bc P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Requested vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:26.879153 9047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1ce8f08de449c7b4e54aca59a187bc" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:26.879123 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1ce8f08de449c7b4e54aca59a187bc" candidate_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 02:00:26.879289 9047 raft_consensus.cc:3037] T 3c1ce8f08de449c7b4e54aca59a187bc P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.879590 6456 raft_consensus.cc:3037] T 3c1ce8f08de449c7b4e54aca59a187bc P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.880395 9047 raft_consensus.cc:2454] T 3c1ce8f08de449c7b4e54aca59a187bc P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:26.880785 6456 raft_consensus.cc:2454] T 3c1ce8f08de449c7b4e54aca59a187bc P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ac2eb7cc264047b9ab32fe3a4fc025 in term 1.
I20240503 02:00:26.880970 6654 leader_election.cc:302] T 3c1ce8f08de449c7b4e54aca59a187bc P c2ac2eb7cc264047b9ab32fe3a4fc025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99ead02915954b4fab23b033ad6d546b, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:26.881193 14664 raft_consensus.cc:2781] T 3c1ce8f08de449c7b4e54aca59a187bc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:26.881280 14664 raft_consensus.cc:686] T 3c1ce8f08de449c7b4e54aca59a187bc P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 LEADER]: Becoming Leader. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Running, Role: LEADER
I20240503 02:00:26.881423 14664 consensus_queue.cc:237] T 3c1ce8f08de449c7b4e54aca59a187bc P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:26.884482 6313 catalog_manager.cc:5484] T 3c1ce8f08de449c7b4e54aca59a187bc P c2ac2eb7cc264047b9ab32fe3a4fc025 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:26.896873 14552 raft_consensus.cc:482] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:26.897001 14552 raft_consensus.cc:504] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:26.897382 14552 leader_election.cc:288] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:26.897735 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61a3076a42dc47859a4b9ffbd873110c" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 02:00:26.897768 9177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61a3076a42dc47859a4b9ffbd873110c" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:26.897882 9177 raft_consensus.cc:2454] T 61a3076a42dc47859a4b9ffbd873110c P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:26.897882 6849 raft_consensus.cc:2454] T 61a3076a42dc47859a4b9ffbd873110c P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 0.
I20240503 02:00:26.898214 8984 leader_election.cc:302] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:26.898389 14552 raft_consensus.cc:2781] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:26.898458 14552 raft_consensus.cc:482] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:26.898496 14552 raft_consensus.cc:3037] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.899328 14552 raft_consensus.cc:504] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:26.899709 14552 leader_election.cc:288] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Requested vote from peers 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:26.900096 9177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61a3076a42dc47859a4b9ffbd873110c" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:26.900067 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61a3076a42dc47859a4b9ffbd873110c" candidate_uuid: "99ead02915954b4fab23b033ad6d546b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 02:00:26.900216 6849 raft_consensus.cc:3037] T 61a3076a42dc47859a4b9ffbd873110c P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.900216 9177 raft_consensus.cc:3037] T 61a3076a42dc47859a4b9ffbd873110c P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:26.901278 6849 raft_consensus.cc:2454] T 61a3076a42dc47859a4b9ffbd873110c P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:26.901278 9177 raft_consensus.cc:2454] T 61a3076a42dc47859a4b9ffbd873110c P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ead02915954b4fab23b033ad6d546b in term 1.
I20240503 02:00:26.901607 8981 leader_election.cc:302] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:26.901798 14552 raft_consensus.cc:2781] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:26.901909 14552 raft_consensus.cc:686] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Becoming Leader. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Running, Role: LEADER
I20240503 02:00:26.902123 14552 consensus_queue.cc:237] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:26.904633 6313 catalog_manager.cc:5484] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b reported cstate change: term changed from 0 to 1, leader changed from <none> to 99ead02915954b4fab23b033ad6d546b (127.0.99.134). New cstate: current_term: 1 leader_uuid: "99ead02915954b4fab23b033ad6d546b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } }
W20240503 02:00:26.940264 8982 consensus_peers.cc:485] T beeea71f02ae4c89a1af13d3584ef70a P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:26.950264 6523 consensus_peers.cc:485] T 915028b3478341d588ed6031e657c259 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 02:00:27.010188 6313 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:27.015036 6849 raft_consensus.cc:1261] T 61a3076a42dc47859a4b9ffbd873110c P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Refusing update from remote peer 99ead02915954b4fab23b033ad6d546b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:27.015125 9177 raft_consensus.cc:1261] T 61a3076a42dc47859a4b9ffbd873110c P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Refusing update from remote peer 99ead02915954b4fab23b033ad6d546b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:27.015540 14665 consensus_queue.cc:1035] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:27.015727 14552 consensus_queue.cc:1035] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:27.019229 6313 catalog_manager.cc:5576] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719) does not have the latest schema for tablet 61a3076a42dc47859a4b9ffbd873110c (table rebalancer_dynamic_table [id=249868ae11634abf8527ed751495bcc6]). Expected version 1 got 0
I20240503 02:00:27.019362 14700 tablet.cc:1722] T 61a3076a42dc47859a4b9ffbd873110c P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:27.019721 14700 tablet.cc:1620] T 61a3076a42dc47859a4b9ffbd873110c P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
I20240503 02:00:27.019999 14699 tablet.cc:1722] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:27.020063 6314 catalog_manager.cc:5576] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079) does not have the latest schema for tablet 61a3076a42dc47859a4b9ffbd873110c (table rebalancer_dynamic_table [id=249868ae11634abf8527ed751495bcc6]). Expected version 1 got 0
I20240503 02:00:27.020401 14699 tablet.cc:1620] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b: MemRowSet was empty: no flush needed.
I20240503 02:00:27.025668 6308 catalog_manager.cc:5576] TS 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157) does not have the latest schema for tablet 61a3076a42dc47859a4b9ffbd873110c (table rebalancer_dynamic_table [id=249868ae11634abf8527ed751495bcc6]). Expected version 1 got 0
I20240503 02:00:27.025787 14660 tablet.cc:1722] T 61a3076a42dc47859a4b9ffbd873110c P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:27.026114 14660 tablet.cc:1620] T 61a3076a42dc47859a4b9ffbd873110c P 492448570cc94ed1879feb2ec3b9051d: MemRowSet was empty: no flush needed.
I20240503 02:00:27.027678 6313 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=249868ae11634abf8527ed751495bcc6] alter complete (version 1)
W20240503 02:00:27.099258 8851 consensus_peers.cc:485] T 7da8f614a70e4177b70e7c2f31d22a10 P 9942306855404cfd891adf227dd65eb2 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 02:00:27.132761 6313 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:27.139647 14699 tablet.cc:1722] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:27.141150 14660 tablet.cc:1722] T 61a3076a42dc47859a4b9ffbd873110c P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:27.142071 14660 tablet.cc:1620] T 61a3076a42dc47859a4b9ffbd873110c P 492448570cc94ed1879feb2ec3b9051d: MemRowSet was empty: no flush needed.
I20240503 02:00:27.142074 14700 tablet.cc:1722] T 61a3076a42dc47859a4b9ffbd873110c P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:27.142127 14699 tablet.cc:1620] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b: MemRowSet was empty: no flush needed.
I20240503 02:00:27.142354 14700 tablet.cc:1620] T 61a3076a42dc47859a4b9ffbd873110c P 2a01ace80baf4057b6bf1995cf88468b: MemRowSet was empty: no flush needed.
I20240503 02:00:27.150228 6314 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=249868ae11634abf8527ed751495bcc6] alter complete (version 2)
W20240503 02:00:27.155472 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:27.226264 14693 consensus_queue.cc:1035] T 81911ae22dfa45c886c1126cc77769dd P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:27.231954 14612 consensus_queue.cc:1035] T 81911ae22dfa45c886c1126cc77769dd P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:27.254369 6313 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:27.260732 14699 tablet.cc:1722] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:27.261152 14700 tablet.cc:1722] T 61a3076a42dc47859a4b9ffbd873110c P 2a01ace80baf4057b6bf1995cf88468b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:27.263800 14660 tablet.cc:1722] T 61a3076a42dc47859a4b9ffbd873110c P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:27.267881 6313 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=249868ae11634abf8527ed751495bcc6] alter complete (version 3)
I20240503 02:00:27.295207 14612 consensus_queue.cc:1035] T f6ce658b7326427583fd3d2e5d0b53a7 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:27.299674 14693 consensus_queue.cc:1035] T f6ce658b7326427583fd3d2e5d0b53a7 P dec3e4a04d604f7eaaa3f077d081218a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:27.371126 6313 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:27.371305 6313 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:27.373970 6313 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet 61a3076a42dc47859a4b9ffbd873110c
I20240503 02:00:27.374572 9028 tablet_service.cc:1514] Processing DeleteTablet for tablet 61a3076a42dc47859a4b9ffbd873110c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:27 UTC) from {username='slave'} at 127.0.0.1:52584
I20240503 02:00:27.374660 9157 tablet_service.cc:1514] Processing DeleteTablet for tablet 61a3076a42dc47859a4b9ffbd873110c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:27 UTC) from {username='slave'} at 127.0.0.1:51940
I20240503 02:00:27.374665 6831 tablet_service.cc:1514] Processing DeleteTablet for tablet 61a3076a42dc47859a4b9ffbd873110c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:27 UTC) from {username='slave'} at 127.0.0.1:52812
I20240503 02:00:27.374971 14713 tablet_replica.cc:330] T 61a3076a42dc47859a4b9ffbd873110c P 492448570cc94ed1879feb2ec3b9051d: stopping tablet replica
I20240503 02:00:27.374994 14711 tablet_replica.cc:330] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b: stopping tablet replica
I20240503 02:00:27.375114 14711 raft_consensus.cc:2229] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:27.375118 14713 raft_consensus.cc:2229] T 61a3076a42dc47859a4b9ffbd873110c P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:27.375236 14713 raft_consensus.cc:2258] T 61a3076a42dc47859a4b9ffbd873110c P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:27.375270 14711 raft_consensus.cc:2258] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:27.375864 14711 ts_tablet_manager.cc:1875] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:27.375967 14713 ts_tablet_manager.cc:1875] T 61a3076a42dc47859a4b9ffbd873110c P 492448570cc94ed1879feb2ec3b9051d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:27.377074 14712 tablet_replica.cc:330] T 61a3076a42dc47859a4b9ffbd873110c P 2a01ace80baf4057b6bf1995cf88468b: stopping tablet replica
I20240503 02:00:27.377379 14712 raft_consensus.cc:2229] T 61a3076a42dc47859a4b9ffbd873110c P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:27.377653 14712 raft_consensus.cc:2258] T 61a3076a42dc47859a4b9ffbd873110c P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:27.378379 14713 ts_tablet_manager.cc:1888] T 61a3076a42dc47859a4b9ffbd873110c P 492448570cc94ed1879feb2ec3b9051d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:27.378453 14711 ts_tablet_manager.cc:1888] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:27.378515 14713 log.cc:1196] T 61a3076a42dc47859a4b9ffbd873110c P 492448570cc94ed1879feb2ec3b9051d: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/61a3076a42dc47859a4b9ffbd873110c
I20240503 02:00:27.378639 14711 log.cc:1196] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/61a3076a42dc47859a4b9ffbd873110c
I20240503 02:00:27.378870 14713 ts_tablet_manager.cc:1909] T 61a3076a42dc47859a4b9ffbd873110c P 492448570cc94ed1879feb2ec3b9051d: Deleting consensus metadata
I20240503 02:00:27.379061 14712 ts_tablet_manager.cc:1875] T 61a3076a42dc47859a4b9ffbd873110c P 2a01ace80baf4057b6bf1995cf88468b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:27.379258 14711 ts_tablet_manager.cc:1909] T 61a3076a42dc47859a4b9ffbd873110c P 99ead02915954b4fab23b033ad6d546b: Deleting consensus metadata
I20240503 02:00:27.379714 6303 catalog_manager.cc:4830] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719): tablet 61a3076a42dc47859a4b9ffbd873110c (table rebalancer_dynamic_table_new_name [id=249868ae11634abf8527ed751495bcc6]) successfully deleted
I20240503 02:00:27.380584 6301 catalog_manager.cc:4830] TS 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157): tablet 61a3076a42dc47859a4b9ffbd873110c (table rebalancer_dynamic_table_new_name [id=249868ae11634abf8527ed751495bcc6]) successfully deleted
I20240503 02:00:27.382052 14712 ts_tablet_manager.cc:1888] T 61a3076a42dc47859a4b9ffbd873110c P 2a01ace80baf4057b6bf1995cf88468b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:27.382123 14712 log.cc:1196] T 61a3076a42dc47859a4b9ffbd873110c P 2a01ace80baf4057b6bf1995cf88468b: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/61a3076a42dc47859a4b9ffbd873110c
I20240503 02:00:27.382339 14712 ts_tablet_manager.cc:1909] T 61a3076a42dc47859a4b9ffbd873110c P 2a01ace80baf4057b6bf1995cf88468b: Deleting consensus metadata
I20240503 02:00:27.383241 6300 catalog_manager.cc:4830] TS 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079): tablet 61a3076a42dc47859a4b9ffbd873110c (table rebalancer_dynamic_table_new_name [id=249868ae11634abf8527ed751495bcc6]) successfully deleted
I20240503 02:00:27.410357 6313 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_extra_table_70"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:27.416814 6568 tablet_service.cc:1467] Processing CreateTablet for tablet 676e4bd2e5ed4cbea577a2f4210eb781 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=c3d4f630569149c79fc93cb8000a3cd1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:27.416813 6831 tablet_service.cc:1467] Processing CreateTablet for tablet 676e4bd2e5ed4cbea577a2f4210eb781 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=c3d4f630569149c79fc93cb8000a3cd1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:27.416863 8887 tablet_service.cc:1467] Processing CreateTablet for tablet 676e4bd2e5ed4cbea577a2f4210eb781 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=c3d4f630569149c79fc93cb8000a3cd1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:27.417152 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 676e4bd2e5ed4cbea577a2f4210eb781. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:27.417156 6568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 676e4bd2e5ed4cbea577a2f4210eb781. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:27.417163 8887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 676e4bd2e5ed4cbea577a2f4210eb781. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:27.420142 14715 tablet_bootstrap.cc:492] T 676e4bd2e5ed4cbea577a2f4210eb781 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:27.420552 9157 tablet_service.cc:1467] Processing CreateTablet for tablet ab3a5e2e96c44759bca819b1a79a5dc9 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=c3d4f630569149c79fc93cb8000a3cd1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:27.420853 9157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab3a5e2e96c44759bca819b1a79a5dc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:27.421463 14715 tablet_bootstrap.cc:654] T 676e4bd2e5ed4cbea577a2f4210eb781 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:27.422132 14716 tablet_bootstrap.cc:492] T 676e4bd2e5ed4cbea577a2f4210eb781 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:27.422143 6438 tablet_service.cc:1467] Processing CreateTablet for tablet ab3a5e2e96c44759bca819b1a79a5dc9 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=c3d4f630569149c79fc93cb8000a3cd1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:27.422461 6438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab3a5e2e96c44759bca819b1a79a5dc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:27.423206 14716 tablet_bootstrap.cc:654] T 676e4bd2e5ed4cbea577a2f4210eb781 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:27.423758 14717 tablet_bootstrap.cc:492] T ab3a5e2e96c44759bca819b1a79a5dc9 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:27.423868 9028 tablet_service.cc:1467] Processing CreateTablet for tablet ec7210e0a04549fab1200b7ab6a133bf (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=c3d4f630569149c79fc93cb8000a3cd1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:27.424208 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec7210e0a04549fab1200b7ab6a133bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:27.424506 14716 tablet_bootstrap.cc:492] T 676e4bd2e5ed4cbea577a2f4210eb781 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:27.424610 14716 ts_tablet_manager.cc:1367] T 676e4bd2e5ed4cbea577a2f4210eb781 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:27.425062 14716 raft_consensus.cc:348] T 676e4bd2e5ed4cbea577a2f4210eb781 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:27.425165 14716 raft_consensus.cc:374] T 676e4bd2e5ed4cbea577a2f4210eb781 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:27.425191 14716 raft_consensus.cc:729] T 676e4bd2e5ed4cbea577a2f4210eb781 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:27.425303 14716 consensus_queue.cc:260] T 676e4bd2e5ed4cbea577a2f4210eb781 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:27.425581 14716 ts_tablet_manager.cc:1398] T 676e4bd2e5ed4cbea577a2f4210eb781 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:27.427067 6700 tablet_service.cc:1467] Processing CreateTablet for tablet ec7210e0a04549fab1200b7ab6a133bf (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=c3d4f630569149c79fc93cb8000a3cd1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:27.427470 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec7210e0a04549fab1200b7ab6a133bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:27.427595 14717 tablet_bootstrap.cc:654] T ab3a5e2e96c44759bca819b1a79a5dc9 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:27.428673 14714 tablet_bootstrap.cc:492] T 676e4bd2e5ed4cbea577a2f4210eb781 P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:27.430017 14714 tablet_bootstrap.cc:654] T 676e4bd2e5ed4cbea577a2f4210eb781 P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:27.430599 14717 tablet_bootstrap.cc:492] T ab3a5e2e96c44759bca819b1a79a5dc9 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:27.430704 14717 ts_tablet_manager.cc:1367] T ab3a5e2e96c44759bca819b1a79a5dc9 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20240503 02:00:27.430728 6568 tablet_service.cc:1467] Processing CreateTablet for tablet ab3a5e2e96c44759bca819b1a79a5dc9 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=c3d4f630569149c79fc93cb8000a3cd1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:27.431035 6568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab3a5e2e96c44759bca819b1a79a5dc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:27.431308 14722 tablet_bootstrap.cc:492] T ec7210e0a04549fab1200b7ab6a133bf P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:27.432618 14719 tablet_bootstrap.cc:492] T ab3a5e2e96c44759bca819b1a79a5dc9 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:27.432618 14722 tablet_bootstrap.cc:654] T ec7210e0a04549fab1200b7ab6a133bf P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:27.433498 14716 tablet_bootstrap.cc:492] T ab3a5e2e96c44759bca819b1a79a5dc9 P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:27.433826 14719 tablet_bootstrap.cc:654] T ab3a5e2e96c44759bca819b1a79a5dc9 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:27.434458 14716 tablet_bootstrap.cc:654] T ab3a5e2e96c44759bca819b1a79a5dc9 P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:27.434422 14717 raft_consensus.cc:348] T ab3a5e2e96c44759bca819b1a79a5dc9 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:27.434562 14717 raft_consensus.cc:374] T ab3a5e2e96c44759bca819b1a79a5dc9 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:27.434595 14717 raft_consensus.cc:729] T ab3a5e2e96c44759bca819b1a79a5dc9 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:27.434724 14717 consensus_queue.cc:260] T ab3a5e2e96c44759bca819b1a79a5dc9 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:27.430728 6569 tablet_service.cc:1467] Processing CreateTablet for tablet ec7210e0a04549fab1200b7ab6a133bf (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=c3d4f630569149c79fc93cb8000a3cd1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:27.435331 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec7210e0a04549fab1200b7ab6a133bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:27.435482 14715 tablet_bootstrap.cc:492] T 676e4bd2e5ed4cbea577a2f4210eb781 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:27.435508 14714 tablet_bootstrap.cc:492] T 676e4bd2e5ed4cbea577a2f4210eb781 P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:27.435575 14715 ts_tablet_manager.cc:1367] T 676e4bd2e5ed4cbea577a2f4210eb781 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240503 02:00:27.435582 14714 ts_tablet_manager.cc:1367] T 676e4bd2e5ed4cbea577a2f4210eb781 P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:00:27.436062 14715 raft_consensus.cc:348] T 676e4bd2e5ed4cbea577a2f4210eb781 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:27.436169 14715 raft_consensus.cc:374] T 676e4bd2e5ed4cbea577a2f4210eb781 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:27.436199 14715 raft_consensus.cc:729] T 676e4bd2e5ed4cbea577a2f4210eb781 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:27.436224 14714 raft_consensus.cc:348] T 676e4bd2e5ed4cbea577a2f4210eb781 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:27.436328 14715 consensus_queue.cc:260] T 676e4bd2e5ed4cbea577a2f4210eb781 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:27.436657 14715 ts_tablet_manager.cc:1398] T 676e4bd2e5ed4cbea577a2f4210eb781 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:27.436369 14714 raft_consensus.cc:374] T 676e4bd2e5ed4cbea577a2f4210eb781 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:27.436761 14714 raft_consensus.cc:729] T 676e4bd2e5ed4cbea577a2f4210eb781 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:27.436923 14714 consensus_queue.cc:260] T 676e4bd2e5ed4cbea577a2f4210eb781 P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:27.437345 14714 ts_tablet_manager.cc:1398] T 676e4bd2e5ed4cbea577a2f4210eb781 P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:27.438195 14716 tablet_bootstrap.cc:492] T ab3a5e2e96c44759bca819b1a79a5dc9 P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:27.438287 14716 ts_tablet_manager.cc:1367] T ab3a5e2e96c44759bca819b1a79a5dc9 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:00:27.438768 14717 ts_tablet_manager.cc:1398] T ab3a5e2e96c44759bca819b1a79a5dc9 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.008s user 0.000s sys 0.001s
I20240503 02:00:27.438831 14716 raft_consensus.cc:348] T ab3a5e2e96c44759bca819b1a79a5dc9 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:27.438918 14716 raft_consensus.cc:374] T ab3a5e2e96c44759bca819b1a79a5dc9 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:27.438941 14716 raft_consensus.cc:729] T ab3a5e2e96c44759bca819b1a79a5dc9 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:27.439067 14716 consensus_queue.cc:260] T ab3a5e2e96c44759bca819b1a79a5dc9 P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:27.439107 14724 tablet_bootstrap.cc:492] T ec7210e0a04549fab1200b7ab6a133bf P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:27.439334 14716 ts_tablet_manager.cc:1398] T ab3a5e2e96c44759bca819b1a79a5dc9 P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:27.439522 14716 tablet_bootstrap.cc:492] T ec7210e0a04549fab1200b7ab6a133bf P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:27.440441 14724 tablet_bootstrap.cc:654] T ec7210e0a04549fab1200b7ab6a133bf P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:27.441329 14716 tablet_bootstrap.cc:654] T ec7210e0a04549fab1200b7ab6a133bf P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:27.442155 14724 tablet_bootstrap.cc:492] T ec7210e0a04549fab1200b7ab6a133bf P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:27.442255 14724 ts_tablet_manager.cc:1367] T ec7210e0a04549fab1200b7ab6a133bf P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:27.442742 14724 raft_consensus.cc:348] T ec7210e0a04549fab1200b7ab6a133bf P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:27.442875 14724 raft_consensus.cc:374] T ec7210e0a04549fab1200b7ab6a133bf P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:27.442909 14724 raft_consensus.cc:729] T ec7210e0a04549fab1200b7ab6a133bf P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:27.443030 14724 consensus_queue.cc:260] T ec7210e0a04549fab1200b7ab6a133bf P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:27.443454 14724 ts_tablet_manager.cc:1398] T ec7210e0a04549fab1200b7ab6a133bf P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:27.444506 14716 tablet_bootstrap.cc:492] T ec7210e0a04549fab1200b7ab6a133bf P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:27.444749 14716 ts_tablet_manager.cc:1367] T ec7210e0a04549fab1200b7ab6a133bf P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240503 02:00:27.445312 14716 raft_consensus.cc:348] T ec7210e0a04549fab1200b7ab6a133bf P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:27.445726 14716 raft_consensus.cc:374] T ec7210e0a04549fab1200b7ab6a133bf P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:27.445808 14716 raft_consensus.cc:729] T ec7210e0a04549fab1200b7ab6a133bf P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:27.446167 14716 consensus_queue.cc:260] T ec7210e0a04549fab1200b7ab6a133bf P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:27.446548 14716 ts_tablet_manager.cc:1398] T ec7210e0a04549fab1200b7ab6a133bf P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:00:27.448444 14722 tablet_bootstrap.cc:492] T ec7210e0a04549fab1200b7ab6a133bf P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:27.448619 14722 ts_tablet_manager.cc:1367] T ec7210e0a04549fab1200b7ab6a133bf P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20240503 02:00:27.449885 14719 tablet_bootstrap.cc:492] T ab3a5e2e96c44759bca819b1a79a5dc9 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:27.450006 14719 ts_tablet_manager.cc:1367] T ab3a5e2e96c44759bca819b1a79a5dc9 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240503 02:00:27.450026 14722 raft_consensus.cc:348] T ec7210e0a04549fab1200b7ab6a133bf P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:27.450153 14722 raft_consensus.cc:374] T ec7210e0a04549fab1200b7ab6a133bf P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:27.450191 14722 raft_consensus.cc:729] T ec7210e0a04549fab1200b7ab6a133bf P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:27.450313 14722 consensus_queue.cc:260] T ec7210e0a04549fab1200b7ab6a133bf P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:27.450647 14722 ts_tablet_manager.cc:1398] T ec7210e0a04549fab1200b7ab6a133bf P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:27.450654 14719 raft_consensus.cc:348] T ab3a5e2e96c44759bca819b1a79a5dc9 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:27.450795 14719 raft_consensus.cc:374] T ab3a5e2e96c44759bca819b1a79a5dc9 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:27.450863 14719 raft_consensus.cc:729] T ab3a5e2e96c44759bca819b1a79a5dc9 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:27.450990 14719 consensus_queue.cc:260] T ab3a5e2e96c44759bca819b1a79a5dc9 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:27.451261 14719 ts_tablet_manager.cc:1398] T ab3a5e2e96c44759bca819b1a79a5dc9 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:27.477633 6314 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:00:27.483633 8887 tablet_service.cc:1467] Processing CreateTablet for tablet 3d7f87f912be4c9daf0848002c50bfeb (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ec3c6861e0924592b1687dc1dc8245cf]), partition=
I20240503 02:00:27.483711 6831 tablet_service.cc:1467] Processing CreateTablet for tablet 3d7f87f912be4c9daf0848002c50bfeb (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ec3c6861e0924592b1687dc1dc8245cf]), partition=
I20240503 02:00:27.483711 6700 tablet_service.cc:1467] Processing CreateTablet for tablet 3d7f87f912be4c9daf0848002c50bfeb (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ec3c6861e0924592b1687dc1dc8245cf]), partition=
I20240503 02:00:27.483898 8887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d7f87f912be4c9daf0848002c50bfeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:27.483968 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d7f87f912be4c9daf0848002c50bfeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:27.483984 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d7f87f912be4c9daf0848002c50bfeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:27.486099 14714 tablet_bootstrap.cc:492] T 3d7f87f912be4c9daf0848002c50bfeb P 9942306855404cfd891adf227dd65eb2: Bootstrap starting.
I20240503 02:00:27.486336 14671 raft_consensus.cc:482] T ec7210e0a04549fab1200b7ab6a133bf P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:27.486469 14671 raft_consensus.cc:504] T ec7210e0a04549fab1200b7ab6a133bf P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:27.486696 14715 tablet_bootstrap.cc:492] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:27.486958 14671 leader_election.cc:288] T ec7210e0a04549fab1200b7ab6a133bf P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:27.487257 14714 tablet_bootstrap.cc:654] T 3d7f87f912be4c9daf0848002c50bfeb P 9942306855404cfd891adf227dd65eb2: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:27.487634 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7210e0a04549fab1200b7ab6a133bf" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:27.487776 9048 raft_consensus.cc:2454] T ec7210e0a04549fab1200b7ab6a133bf P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:27.487977 14715 tablet_bootstrap.cc:654] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:27.488003 6720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7210e0a04549fab1200b7ab6a133bf" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 02:00:27.488147 6720 raft_consensus.cc:2454] T ec7210e0a04549fab1200b7ab6a133bf P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:27.488518 6523 leader_election.cc:302] T ec7210e0a04549fab1200b7ab6a133bf P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, 99ead02915954b4fab23b033ad6d546b; no voters:
I20240503 02:00:27.488687 14671 raft_consensus.cc:2781] T ec7210e0a04549fab1200b7ab6a133bf P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:27.488734 14671 raft_consensus.cc:482] T ec7210e0a04549fab1200b7ab6a133bf P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:27.488770 14671 raft_consensus.cc:3037] T ec7210e0a04549fab1200b7ab6a133bf P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:27.489890 14671 raft_consensus.cc:504] T ec7210e0a04549fab1200b7ab6a133bf P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:27.489972 14715 tablet_bootstrap.cc:492] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:27.490046 14715 ts_tablet_manager.cc:1367] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:00:27.490259 14671 leader_election.cc:288] T ec7210e0a04549fab1200b7ab6a133bf P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Requested vote from peers c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655), 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157)
I20240503 02:00:27.490578 14715 raft_consensus.cc:348] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:27.490749 14715 raft_consensus.cc:374] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:27.490805 14715 raft_consensus.cc:729] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:27.490783 6720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7210e0a04549fab1200b7ab6a133bf" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 02:00:27.490909 6720 raft_consensus.cc:3037] T ec7210e0a04549fab1200b7ab6a133bf P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:27.490950 14715 consensus_queue.cc:260] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:27.491242 14715 ts_tablet_manager.cc:1398] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:27.491771 6720 raft_consensus.cc:2454] T ec7210e0a04549fab1200b7ab6a133bf P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:27.492056 6525 leader_election.cc:302] T ec7210e0a04549fab1200b7ab6a133bf P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:27.492122 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7210e0a04549fab1200b7ab6a133bf" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:27.492215 14671 raft_consensus.cc:2781] T ec7210e0a04549fab1200b7ab6a133bf P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:27.492264 9048 raft_consensus.cc:3037] T ec7210e0a04549fab1200b7ab6a133bf P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:27.492302 14671 raft_consensus.cc:686] T ec7210e0a04549fab1200b7ab6a133bf P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Becoming Leader. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Running, Role: LEADER
I20240503 02:00:27.492478 14671 consensus_queue.cc:237] T ec7210e0a04549fab1200b7ab6a133bf P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } }
I20240503 02:00:27.493317 9048 raft_consensus.cc:2454] T ec7210e0a04549fab1200b7ab6a133bf P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:27.494212 14722 tablet_bootstrap.cc:492] T 3d7f87f912be4c9daf0848002c50bfeb P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:27.495584 14722 tablet_bootstrap.cc:654] T 3d7f87f912be4c9daf0848002c50bfeb P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:27.495976 6308 catalog_manager.cc:5484] T ec7210e0a04549fab1200b7ab6a133bf P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:27.496734 14722 tablet_bootstrap.cc:492] T 3d7f87f912be4c9daf0848002c50bfeb P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:27.496822 14722 ts_tablet_manager.cc:1367] T 3d7f87f912be4c9daf0848002c50bfeb P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:27.497267 14722 raft_consensus.cc:348] T 3d7f87f912be4c9daf0848002c50bfeb P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:27.497373 14722 raft_consensus.cc:374] T 3d7f87f912be4c9daf0848002c50bfeb P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:27.497402 14722 raft_consensus.cc:729] T 3d7f87f912be4c9daf0848002c50bfeb P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:27.497519 14722 consensus_queue.cc:260] T 3d7f87f912be4c9daf0848002c50bfeb P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:27.497783 14722 ts_tablet_manager.cc:1398] T 3d7f87f912be4c9daf0848002c50bfeb P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:27.499271 14714 tablet_bootstrap.cc:492] T 3d7f87f912be4c9daf0848002c50bfeb P 9942306855404cfd891adf227dd65eb2: No bootstrap required, opened a new log
I20240503 02:00:27.499377 14714 ts_tablet_manager.cc:1367] T 3d7f87f912be4c9daf0848002c50bfeb P 9942306855404cfd891adf227dd65eb2: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 02:00:27.499871 14714 raft_consensus.cc:348] T 3d7f87f912be4c9daf0848002c50bfeb P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:27.499990 14714 raft_consensus.cc:374] T 3d7f87f912be4c9daf0848002c50bfeb P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:27.500011 14714 raft_consensus.cc:729] T 3d7f87f912be4c9daf0848002c50bfeb P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9942306855404cfd891adf227dd65eb2, State: Initialized, Role: FOLLOWER
I20240503 02:00:27.500113 14714 consensus_queue.cc:260] T 3d7f87f912be4c9daf0848002c50bfeb P 9942306855404cfd891adf227dd65eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:27.500336 14714 ts_tablet_manager.cc:1398] T 3d7f87f912be4c9daf0848002c50bfeb P 9942306855404cfd891adf227dd65eb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:27.505743 14386 consensus_queue.cc:1035] T 3c1ce8f08de449c7b4e54aca59a187bc P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:27.506256 14386 consensus_queue.cc:1035] T 3c1ce8f08de449c7b4e54aca59a187bc P c2ac2eb7cc264047b9ab32fe3a4fc025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:27.537775 14618 raft_consensus.cc:482] T ab3a5e2e96c44759bca819b1a79a5dc9 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:27.537936 14618 raft_consensus.cc:504] T ab3a5e2e96c44759bca819b1a79a5dc9 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:27.538399 14618 leader_election.cc:288] T ab3a5e2e96c44759bca819b1a79a5dc9 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:27.538817 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab3a5e2e96c44759bca819b1a79a5dc9" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 02:00:27.539008 6456 raft_consensus.cc:2454] T ab3a5e2e96c44759bca819b1a79a5dc9 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:27.539008 6587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab3a5e2e96c44759bca819b1a79a5dc9" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f" is_pre_election: true
I20240503 02:00:27.539137 6587 raft_consensus.cc:2454] T ab3a5e2e96c44759bca819b1a79a5dc9 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:27.539347 9112 leader_election.cc:302] T ab3a5e2e96c44759bca819b1a79a5dc9 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:27.539577 14618 raft_consensus.cc:2781] T ab3a5e2e96c44759bca819b1a79a5dc9 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:27.539628 14618 raft_consensus.cc:482] T ab3a5e2e96c44759bca819b1a79a5dc9 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:27.539664 14618 raft_consensus.cc:3037] T ab3a5e2e96c44759bca819b1a79a5dc9 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:27.540565 14618 raft_consensus.cc:504] T ab3a5e2e96c44759bca819b1a79a5dc9 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:27.540953 14618 leader_election.cc:288] T ab3a5e2e96c44759bca819b1a79a5dc9 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Requested vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130:43863)
I20240503 02:00:27.541541 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab3a5e2e96c44759bca819b1a79a5dc9" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 02:00:27.541669 6456 raft_consensus.cc:3037] T ab3a5e2e96c44759bca819b1a79a5dc9 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:27.541680 6587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab3a5e2e96c44759bca819b1a79a5dc9" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd18ef4d604c86b909d5c008a46b4f"
I20240503 02:00:27.541779 6587 raft_consensus.cc:3037] T ab3a5e2e96c44759bca819b1a79a5dc9 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:27.542711 6587 raft_consensus.cc:2454] T ab3a5e2e96c44759bca819b1a79a5dc9 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:27.542711 6456 raft_consensus.cc:2454] T ab3a5e2e96c44759bca819b1a79a5dc9 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:27.543009 9114 leader_election.cc:302] T ab3a5e2e96c44759bca819b1a79a5dc9 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 80fd18ef4d604c86b909d5c008a46b4f; no voters:
I20240503 02:00:27.543197 14618 raft_consensus.cc:2781] T ab3a5e2e96c44759bca819b1a79a5dc9 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:27.543303 14618 raft_consensus.cc:686] T ab3a5e2e96c44759bca819b1a79a5dc9 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Becoming Leader. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Running, Role: LEADER
I20240503 02:00:27.543476 14618 consensus_queue.cc:237] T ab3a5e2e96c44759bca819b1a79a5dc9 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:27.546191 6314 catalog_manager.cc:5484] T ab3a5e2e96c44759bca819b1a79a5dc9 P 2a01ace80baf4057b6bf1995cf88468b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135). New cstate: current_term: 1 leader_uuid: "2a01ace80baf4057b6bf1995cf88468b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:27.561421 14671 raft_consensus.cc:482] T 676e4bd2e5ed4cbea577a2f4210eb781 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:27.561545 14671 raft_consensus.cc:504] T 676e4bd2e5ed4cbea577a2f4210eb781 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:27.561918 14671 leader_election.cc:288] T 676e4bd2e5ed4cbea577a2f4210eb781 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:27.562294 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "676e4bd2e5ed4cbea577a2f4210eb781" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 02:00:27.562426 6851 raft_consensus.cc:2454] T 676e4bd2e5ed4cbea577a2f4210eb781 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:27.562379 8912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "676e4bd2e5ed4cbea577a2f4210eb781" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:27.562520 8912 raft_consensus.cc:2454] T 676e4bd2e5ed4cbea577a2f4210eb781 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:27.562667 6525 leader_election.cc:302] T 676e4bd2e5ed4cbea577a2f4210eb781 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, 80fd18ef4d604c86b909d5c008a46b4f; no voters:
I20240503 02:00:27.562848 14671 raft_consensus.cc:2781] T 676e4bd2e5ed4cbea577a2f4210eb781 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:27.562916 14671 raft_consensus.cc:482] T 676e4bd2e5ed4cbea577a2f4210eb781 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:27.562956 14671 raft_consensus.cc:3037] T 676e4bd2e5ed4cbea577a2f4210eb781 P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:27.564059 14671 raft_consensus.cc:504] T 676e4bd2e5ed4cbea577a2f4210eb781 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:27.564464 14671 leader_election.cc:288] T 676e4bd2e5ed4cbea577a2f4210eb781 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Requested vote from peers 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719), 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211)
I20240503 02:00:27.564951 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "676e4bd2e5ed4cbea577a2f4210eb781" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 02:00:27.565074 6849 raft_consensus.cc:3037] T 676e4bd2e5ed4cbea577a2f4210eb781 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:27.565135 8912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "676e4bd2e5ed4cbea577a2f4210eb781" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:27.565261 8912 raft_consensus.cc:3037] T 676e4bd2e5ed4cbea577a2f4210eb781 P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:27.566110 6849 raft_consensus.cc:2454] T 676e4bd2e5ed4cbea577a2f4210eb781 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:27.566082 8912 raft_consensus.cc:2454] T 676e4bd2e5ed4cbea577a2f4210eb781 P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:27.566515 6525 leader_election.cc:302] T 676e4bd2e5ed4cbea577a2f4210eb781 P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fd18ef4d604c86b909d5c008a46b4f, 9942306855404cfd891adf227dd65eb2; no voters:
I20240503 02:00:27.566864 14671 raft_consensus.cc:2781] T 676e4bd2e5ed4cbea577a2f4210eb781 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:27.566967 14671 raft_consensus.cc:686] T 676e4bd2e5ed4cbea577a2f4210eb781 P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Becoming Leader. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Running, Role: LEADER
I20240503 02:00:27.567095 14671 consensus_queue.cc:237] T 676e4bd2e5ed4cbea577a2f4210eb781 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:27.569877 6314 catalog_manager.cc:5484] T 676e4bd2e5ed4cbea577a2f4210eb781 P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } }
W20240503 02:00:27.708192 8982 consensus_peers.cc:485] T 56824b553788471c9a2fcd6ec82d507e P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:27.713827 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:27.742408 14706 raft_consensus.cc:482] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:27.742544 14706 raft_consensus.cc:504] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:27.742945 14706 leader_election.cc:288] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:27.743229 8912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d7f87f912be4c9daf0848002c50bfeb" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2" is_pre_election: true
I20240503 02:00:27.743400 8912 raft_consensus.cc:2454] T 3d7f87f912be4c9daf0848002c50bfeb P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 02:00:27.743527 6720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d7f87f912be4c9daf0848002c50bfeb" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
I20240503 02:00:27.743705 6720 raft_consensus.cc:2454] T 3d7f87f912be4c9daf0848002c50bfeb P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 0.
I20240503 02:00:27.744009 6787 leader_election.cc:302] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:27.744174 14706 raft_consensus.cc:2781] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:27.744227 14706 raft_consensus.cc:482] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:27.744267 14706 raft_consensus.cc:3037] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:27.746053 14706 raft_consensus.cc:504] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:27.746441 14706 leader_election.cc:288] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Requested vote from peers 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:27.746745 8917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d7f87f912be4c9daf0848002c50bfeb" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9942306855404cfd891adf227dd65eb2"
I20240503 02:00:27.746928 6720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d7f87f912be4c9daf0848002c50bfeb" candidate_uuid: "492448570cc94ed1879feb2ec3b9051d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
I20240503 02:00:27.746974 8917 raft_consensus.cc:3037] T 3d7f87f912be4c9daf0848002c50bfeb P 9942306855404cfd891adf227dd65eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:27.747046 6720 raft_consensus.cc:3037] T 3d7f87f912be4c9daf0848002c50bfeb P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:27.747977 8917 raft_consensus.cc:2454] T 3d7f87f912be4c9daf0848002c50bfeb P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 02:00:27.747977 6720 raft_consensus.cc:2454] T 3d7f87f912be4c9daf0848002c50bfeb P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492448570cc94ed1879feb2ec3b9051d in term 1.
I20240503 02:00:27.748309 6787 leader_election.cc:302] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492448570cc94ed1879feb2ec3b9051d, c2ac2eb7cc264047b9ab32fe3a4fc025; no voters:
I20240503 02:00:27.748524 14706 raft_consensus.cc:2781] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:27.748621 14706 raft_consensus.cc:686] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Becoming Leader. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Running, Role: LEADER
I20240503 02:00:27.748826 14706 consensus_queue.cc:237] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:27.751428 6314 catalog_manager.cc:5484] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d reported cstate change: term changed from 0 to 1, leader changed from <none> to 492448570cc94ed1879feb2ec3b9051d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "492448570cc94ed1879feb2ec3b9051d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:27.886631 6314 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:27.890774 8917 raft_consensus.cc:1261] T 3d7f87f912be4c9daf0848002c50bfeb P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:27.890774 6720 raft_consensus.cc:1261] T 3d7f87f912be4c9daf0848002c50bfeb P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Refusing update from remote peer 492448570cc94ed1879feb2ec3b9051d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:27.891194 14706 consensus_queue.cc:1035] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:27.891353 14472 consensus_queue.cc:1035] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:27.894731 6314 catalog_manager.cc:5576] TS c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655) does not have the latest schema for tablet 3d7f87f912be4c9daf0848002c50bfeb (table rebalancer_dynamic_table [id=ec3c6861e0924592b1687dc1dc8245cf]). Expected version 1 got 0
I20240503 02:00:27.895094 6309 catalog_manager.cc:5576] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211) does not have the latest schema for tablet 3d7f87f912be4c9daf0848002c50bfeb (table rebalancer_dynamic_table [id=ec3c6861e0924592b1687dc1dc8245cf]). Expected version 1 got 0
I20240503 02:00:27.895969 14744 tablet.cc:1722] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:27.896399 14744 tablet.cc:1620] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d: MemRowSet was empty: no flush needed.
I20240503 02:00:27.896762 14745 tablet.cc:1722] T 3d7f87f912be4c9daf0848002c50bfeb P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:27.897197 14745 tablet.cc:1620] T 3d7f87f912be4c9daf0848002c50bfeb P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
I20240503 02:00:27.899051 14746 tablet.cc:1722] T 3d7f87f912be4c9daf0848002c50bfeb P c2ac2eb7cc264047b9ab32fe3a4fc025: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:27.899359 14746 tablet.cc:1620] T 3d7f87f912be4c9daf0848002c50bfeb P c2ac2eb7cc264047b9ab32fe3a4fc025: MemRowSet was empty: no flush needed.
I20240503 02:00:27.899910 6314 catalog_manager.cc:5576] TS c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655) does not have the latest schema for tablet 3d7f87f912be4c9daf0848002c50bfeb (table rebalancer_dynamic_table [id=ec3c6861e0924592b1687dc1dc8245cf]). Expected version 1 got 0
I20240503 02:00:27.903146 6309 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=ec3c6861e0924592b1687dc1dc8245cf] alter complete (version 1)
I20240503 02:00:27.936008 14649 consensus_queue.cc:1035] T ec7210e0a04549fab1200b7ab6a133bf P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:27.941413 14671 consensus_queue.cc:1035] T ec7210e0a04549fab1200b7ab6a133bf P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:27.983474 14618 consensus_queue.cc:1035] T ab3a5e2e96c44759bca819b1a79a5dc9 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:27.988204 14751 consensus_queue.cc:1035] T ab3a5e2e96c44759bca819b1a79a5dc9 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:28.008492 6314 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:28.014006 14744 tablet.cc:1722] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:28.014381 14744 tablet.cc:1620] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d: MemRowSet was empty: no flush needed.
I20240503 02:00:28.014743 14746 tablet.cc:1722] T 3d7f87f912be4c9daf0848002c50bfeb P c2ac2eb7cc264047b9ab32fe3a4fc025: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:28.015033 14745 tablet.cc:1722] T 3d7f87f912be4c9daf0848002c50bfeb P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:28.015110 14746 tablet.cc:1620] T 3d7f87f912be4c9daf0848002c50bfeb P c2ac2eb7cc264047b9ab32fe3a4fc025: MemRowSet was empty: no flush needed.
I20240503 02:00:28.015319 14745 tablet.cc:1620] T 3d7f87f912be4c9daf0848002c50bfeb P 9942306855404cfd891adf227dd65eb2: MemRowSet was empty: no flush needed.
I20240503 02:00:28.022046 6308 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table [id=ec3c6861e0924592b1687dc1dc8245cf] alter complete (version 2)
W20240503 02:00:28.061223 6785 consensus_peers.cc:485] T c3485701bf3b48f5ae2e1568afeed133 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 02:00:28.098096 6314 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "rebalancer_extra_table_72"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:28.105091 9028 tablet_service.cc:1467] Processing CreateTablet for tablet 2a3c9f1f41dd43d9878f97388523a2cf (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=e3918a1ea1744c97a837f7f2acb7a8d8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:28.105091 6438 tablet_service.cc:1467] Processing CreateTablet for tablet bdc99a9b4d144059bfda89af3c033803 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=e3918a1ea1744c97a837f7f2acb7a8d8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:28.105195 6831 tablet_service.cc:1467] Processing CreateTablet for tablet bdc99a9b4d144059bfda89af3c033803 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=e3918a1ea1744c97a837f7f2acb7a8d8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:28.105412 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a3c9f1f41dd43d9878f97388523a2cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:28.105412 6438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdc99a9b4d144059bfda89af3c033803. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:28.105482 6831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdc99a9b4d144059bfda89af3c033803. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:28.108029 9157 tablet_service.cc:1467] Processing CreateTablet for tablet bdc99a9b4d144059bfda89af3c033803 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=e3918a1ea1744c97a837f7f2acb7a8d8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:28.108343 9157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdc99a9b4d144059bfda89af3c033803. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:28.108748 14756 tablet_bootstrap.cc:492] T 2a3c9f1f41dd43d9878f97388523a2cf P 99ead02915954b4fab23b033ad6d546b: Bootstrap starting.
I20240503 02:00:28.109019 14755 tablet_bootstrap.cc:492] T bdc99a9b4d144059bfda89af3c033803 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:28.109191 14754 tablet_bootstrap.cc:492] T bdc99a9b4d144059bfda89af3c033803 P 492448570cc94ed1879feb2ec3b9051d: Bootstrap starting.
I20240503 02:00:28.109282 6569 tablet_service.cc:1467] Processing CreateTablet for tablet 2a3c9f1f41dd43d9878f97388523a2cf (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=e3918a1ea1744c97a837f7f2acb7a8d8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:28.109539 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a3c9f1f41dd43d9878f97388523a2cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:28.110321 14754 tablet_bootstrap.cc:654] T bdc99a9b4d144059bfda89af3c033803 P 492448570cc94ed1879feb2ec3b9051d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:28.110867 14755 tablet_bootstrap.cc:654] T bdc99a9b4d144059bfda89af3c033803 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:28.112135 14756 tablet_bootstrap.cc:654] T 2a3c9f1f41dd43d9878f97388523a2cf P 99ead02915954b4fab23b033ad6d546b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:28.112560 14759 tablet_bootstrap.cc:492] T 2a3c9f1f41dd43d9878f97388523a2cf P 80fd18ef4d604c86b909d5c008a46b4f: Bootstrap starting.
I20240503 02:00:28.112653 9157 tablet_service.cc:1467] Processing CreateTablet for tablet 2a3c9f1f41dd43d9878f97388523a2cf (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=e3918a1ea1744c97a837f7f2acb7a8d8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:28.112916 9157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a3c9f1f41dd43d9878f97388523a2cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:28.113624 14759 tablet_bootstrap.cc:654] T 2a3c9f1f41dd43d9878f97388523a2cf P 80fd18ef4d604c86b909d5c008a46b4f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:28.113817 6438 tablet_service.cc:1467] Processing CreateTablet for tablet 2e87caa8a93d43a7b61e1596b12b0906 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=e3918a1ea1744c97a837f7f2acb7a8d8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:28.114105 6438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e87caa8a93d43a7b61e1596b12b0906. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:28.114259 9159 tablet_service.cc:1467] Processing CreateTablet for tablet 2e87caa8a93d43a7b61e1596b12b0906 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=e3918a1ea1744c97a837f7f2acb7a8d8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:28.114508 9159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e87caa8a93d43a7b61e1596b12b0906. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:28.116690 6700 tablet_service.cc:1467] Processing CreateTablet for tablet 2e87caa8a93d43a7b61e1596b12b0906 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=e3918a1ea1744c97a837f7f2acb7a8d8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:28.116984 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e87caa8a93d43a7b61e1596b12b0906. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:00:28.117487 6654 consensus_peers.cc:485] T 1feaf29d8c6b4842829ce66e0f4ec23b P c2ac2eb7cc264047b9ab32fe3a4fc025 -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 02:00:28.117823 14757 tablet_bootstrap.cc:492] T bdc99a9b4d144059bfda89af3c033803 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:28.119045 14757 tablet_bootstrap.cc:654] T bdc99a9b4d144059bfda89af3c033803 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:28.120404 14762 tablet_bootstrap.cc:492] T 2e87caa8a93d43a7b61e1596b12b0906 P c2ac2eb7cc264047b9ab32fe3a4fc025: Bootstrap starting.
I20240503 02:00:28.120692 14757 tablet_bootstrap.cc:492] T bdc99a9b4d144059bfda89af3c033803 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:28.120815 14757 ts_tablet_manager.cc:1367] T bdc99a9b4d144059bfda89af3c033803 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:28.121338 14757 raft_consensus.cc:348] T bdc99a9b4d144059bfda89af3c033803 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:28.121459 14757 raft_consensus.cc:374] T bdc99a9b4d144059bfda89af3c033803 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:28.121474 14762 tablet_bootstrap.cc:654] T 2e87caa8a93d43a7b61e1596b12b0906 P c2ac2eb7cc264047b9ab32fe3a4fc025: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:28.121490 14757 raft_consensus.cc:729] T bdc99a9b4d144059bfda89af3c033803 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:28.121613 14757 consensus_queue.cc:260] T bdc99a9b4d144059bfda89af3c033803 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:28.121717 14755 tablet_bootstrap.cc:492] T bdc99a9b4d144059bfda89af3c033803 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:28.121824 14755 ts_tablet_manager.cc:1367] T bdc99a9b4d144059bfda89af3c033803 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 02:00:28.121896 14756 tablet_bootstrap.cc:492] T 2a3c9f1f41dd43d9878f97388523a2cf P 99ead02915954b4fab23b033ad6d546b: No bootstrap required, opened a new log
I20240503 02:00:28.121891 14757 ts_tablet_manager.cc:1398] T bdc99a9b4d144059bfda89af3c033803 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:28.121978 14756 ts_tablet_manager.cc:1367] T 2a3c9f1f41dd43d9878f97388523a2cf P 99ead02915954b4fab23b033ad6d546b: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.001s
I20240503 02:00:28.122064 14757 tablet_bootstrap.cc:492] T 2e87caa8a93d43a7b61e1596b12b0906 P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:28.122335 14755 raft_consensus.cc:348] T bdc99a9b4d144059bfda89af3c033803 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:28.122443 14755 raft_consensus.cc:374] T bdc99a9b4d144059bfda89af3c033803 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:28.122473 14755 raft_consensus.cc:729] T bdc99a9b4d144059bfda89af3c033803 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:28.122582 14756 raft_consensus.cc:348] T 2a3c9f1f41dd43d9878f97388523a2cf P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:28.122714 14756 raft_consensus.cc:374] T 2a3c9f1f41dd43d9878f97388523a2cf P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:28.122746 14756 raft_consensus.cc:729] T 2a3c9f1f41dd43d9878f97388523a2cf P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ead02915954b4fab23b033ad6d546b, State: Initialized, Role: FOLLOWER
I20240503 02:00:28.122736 14755 consensus_queue.cc:260] T bdc99a9b4d144059bfda89af3c033803 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:28.122990 14757 tablet_bootstrap.cc:654] T 2e87caa8a93d43a7b61e1596b12b0906 P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:28.123059 14756 consensus_queue.cc:260] T 2a3c9f1f41dd43d9878f97388523a2cf P 99ead02915954b4fab23b033ad6d546b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:28.123337 14755 ts_tablet_manager.cc:1398] T bdc99a9b4d144059bfda89af3c033803 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:28.123525 14755 tablet_bootstrap.cc:492] T 2e87caa8a93d43a7b61e1596b12b0906 P dec3e4a04d604f7eaaa3f077d081218a: Bootstrap starting.
I20240503 02:00:28.124644 14755 tablet_bootstrap.cc:654] T 2e87caa8a93d43a7b61e1596b12b0906 P dec3e4a04d604f7eaaa3f077d081218a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:28.125591 14756 ts_tablet_manager.cc:1398] T 2a3c9f1f41dd43d9878f97388523a2cf P 99ead02915954b4fab23b033ad6d546b: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240503 02:00:28.127133 14754 tablet_bootstrap.cc:492] T bdc99a9b4d144059bfda89af3c033803 P 492448570cc94ed1879feb2ec3b9051d: No bootstrap required, opened a new log
I20240503 02:00:28.127254 14754 ts_tablet_manager.cc:1367] T bdc99a9b4d144059bfda89af3c033803 P 492448570cc94ed1879feb2ec3b9051d: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240503 02:00:28.127740 14754 raft_consensus.cc:348] T bdc99a9b4d144059bfda89af3c033803 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:28.127859 14754 raft_consensus.cc:374] T bdc99a9b4d144059bfda89af3c033803 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:28.127892 14754 raft_consensus.cc:729] T bdc99a9b4d144059bfda89af3c033803 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492448570cc94ed1879feb2ec3b9051d, State: Initialized, Role: FOLLOWER
I20240503 02:00:28.127995 14754 consensus_queue.cc:260] T bdc99a9b4d144059bfda89af3c033803 P 492448570cc94ed1879feb2ec3b9051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:28.128273 14754 ts_tablet_manager.cc:1398] T bdc99a9b4d144059bfda89af3c033803 P 492448570cc94ed1879feb2ec3b9051d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:28.130029 14757 tablet_bootstrap.cc:492] T 2e87caa8a93d43a7b61e1596b12b0906 P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:28.130131 14757 ts_tablet_manager.cc:1367] T 2e87caa8a93d43a7b61e1596b12b0906 P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 02:00:28.130753 14757 raft_consensus.cc:348] T 2e87caa8a93d43a7b61e1596b12b0906 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:28.130898 14757 raft_consensus.cc:374] T 2e87caa8a93d43a7b61e1596b12b0906 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:28.130934 14757 raft_consensus.cc:729] T 2e87caa8a93d43a7b61e1596b12b0906 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:28.131052 14757 consensus_queue.cc:260] T 2e87caa8a93d43a7b61e1596b12b0906 P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:28.131345 14757 ts_tablet_manager.cc:1398] T 2e87caa8a93d43a7b61e1596b12b0906 P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:28.131562 14757 tablet_bootstrap.cc:492] T 2a3c9f1f41dd43d9878f97388523a2cf P 2a01ace80baf4057b6bf1995cf88468b: Bootstrap starting.
I20240503 02:00:28.132869 14757 tablet_bootstrap.cc:654] T 2a3c9f1f41dd43d9878f97388523a2cf P 2a01ace80baf4057b6bf1995cf88468b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:28.134801 14757 tablet_bootstrap.cc:492] T 2a3c9f1f41dd43d9878f97388523a2cf P 2a01ace80baf4057b6bf1995cf88468b: No bootstrap required, opened a new log
I20240503 02:00:28.135047 14757 ts_tablet_manager.cc:1367] T 2a3c9f1f41dd43d9878f97388523a2cf P 2a01ace80baf4057b6bf1995cf88468b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
W20240503 02:00:28.135251 8982 consensus_peers.cc:462] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111)
I20240503 02:00:28.135919 14757 raft_consensus.cc:348] T 2a3c9f1f41dd43d9878f97388523a2cf P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:28.136585 14757 raft_consensus.cc:374] T 2a3c9f1f41dd43d9878f97388523a2cf P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:28.136699 14757 raft_consensus.cc:729] T 2a3c9f1f41dd43d9878f97388523a2cf P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Initialized, Role: FOLLOWER
I20240503 02:00:28.137076 14757 consensus_queue.cc:260] T 2a3c9f1f41dd43d9878f97388523a2cf P 2a01ace80baf4057b6bf1995cf88468b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:28.137405 14755 tablet_bootstrap.cc:492] T 2e87caa8a93d43a7b61e1596b12b0906 P dec3e4a04d604f7eaaa3f077d081218a: No bootstrap required, opened a new log
I20240503 02:00:28.137660 14755 ts_tablet_manager.cc:1367] T 2e87caa8a93d43a7b61e1596b12b0906 P dec3e4a04d604f7eaaa3f077d081218a: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 02:00:28.137733 14757 ts_tablet_manager.cc:1398] T 2a3c9f1f41dd43d9878f97388523a2cf P 2a01ace80baf4057b6bf1995cf88468b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:28.137928 14671 consensus_queue.cc:1035] T 676e4bd2e5ed4cbea577a2f4210eb781 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:28.138316 14755 raft_consensus.cc:348] T 2e87caa8a93d43a7b61e1596b12b0906 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:28.138437 14755 raft_consensus.cc:374] T 2e87caa8a93d43a7b61e1596b12b0906 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:28.138471 14755 raft_consensus.cc:729] T 2e87caa8a93d43a7b61e1596b12b0906 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec3e4a04d604f7eaaa3f077d081218a, State: Initialized, Role: FOLLOWER
I20240503 02:00:28.138585 14755 consensus_queue.cc:260] T 2e87caa8a93d43a7b61e1596b12b0906 P dec3e4a04d604f7eaaa3f077d081218a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:28.138995 14755 ts_tablet_manager.cc:1398] T 2e87caa8a93d43a7b61e1596b12b0906 P dec3e4a04d604f7eaaa3f077d081218a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240503 02:00:28.139750 6785 consensus_peers.cc:485] T 1fa274c98fc147f29344ed537e2ccf17 P 492448570cc94ed1879feb2ec3b9051d -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 02:00:28.140404 6308 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:28.141166 14691 raft_consensus.cc:482] T 2e87caa8a93d43a7b61e1596b12b0906 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:28.141276 14691 raft_consensus.cc:504] T 2e87caa8a93d43a7b61e1596b12b0906 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:28.141695 14691 leader_election.cc:288] T 2e87caa8a93d43a7b61e1596b12b0906 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:28.142423 6720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e87caa8a93d43a7b61e1596b12b0906" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" is_pre_election: true
W20240503 02:00:28.142866 9115 leader_election.cc:341] T 2e87caa8a93d43a7b61e1596b12b0906 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:00:28.143303 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e87caa8a93d43a7b61e1596b12b0906" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 02:00:28.143435 6456 raft_consensus.cc:2454] T 2e87caa8a93d43a7b61e1596b12b0906 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:28.143759 9112 leader_election.cc:302] T 2e87caa8a93d43a7b61e1596b12b0906 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, dec3e4a04d604f7eaaa3f077d081218a; no voters: c2ac2eb7cc264047b9ab32fe3a4fc025
I20240503 02:00:28.143980 14691 raft_consensus.cc:2781] T 2e87caa8a93d43a7b61e1596b12b0906 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:28.144042 14691 raft_consensus.cc:482] T 2e87caa8a93d43a7b61e1596b12b0906 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:28.144080 14691 raft_consensus.cc:3037] T 2e87caa8a93d43a7b61e1596b12b0906 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:28.145161 14691 raft_consensus.cc:504] T 2e87caa8a93d43a7b61e1596b12b0906 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:28.145560 14691 leader_election.cc:288] T 2e87caa8a93d43a7b61e1596b12b0906 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Requested vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655)
I20240503 02:00:28.146287 6720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e87caa8a93d43a7b61e1596b12b0906" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025"
W20240503 02:00:28.146807 9115 leader_election.cc:341] T 2e87caa8a93d43a7b61e1596b12b0906 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:00:28.146994 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e87caa8a93d43a7b61e1596b12b0906" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 02:00:28.147114 6456 raft_consensus.cc:3037] T 2e87caa8a93d43a7b61e1596b12b0906 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:28.148177 6456 raft_consensus.cc:2454] T 2e87caa8a93d43a7b61e1596b12b0906 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:28.149161 14744 tablet.cc:1722] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:28.150575 9112 leader_election.cc:302] T 2e87caa8a93d43a7b61e1596b12b0906 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, dec3e4a04d604f7eaaa3f077d081218a; no voters: c2ac2eb7cc264047b9ab32fe3a4fc025
I20240503 02:00:28.150861 14759 tablet_bootstrap.cc:492] T 2a3c9f1f41dd43d9878f97388523a2cf P 80fd18ef4d604c86b909d5c008a46b4f: No bootstrap required, opened a new log
I20240503 02:00:28.150943 14759 ts_tablet_manager.cc:1367] T 2a3c9f1f41dd43d9878f97388523a2cf P 80fd18ef4d604c86b909d5c008a46b4f: Time spent bootstrapping tablet: real 0.038s user 0.002s sys 0.000s
I20240503 02:00:28.151384 14759 raft_consensus.cc:348] T 2a3c9f1f41dd43d9878f97388523a2cf P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:28.151484 14759 raft_consensus.cc:374] T 2a3c9f1f41dd43d9878f97388523a2cf P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:28.151504 14759 raft_consensus.cc:729] T 2a3c9f1f41dd43d9878f97388523a2cf P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Initialized, Role: FOLLOWER
I20240503 02:00:28.151592 14759 consensus_queue.cc:260] T 2a3c9f1f41dd43d9878f97388523a2cf P 80fd18ef4d604c86b909d5c008a46b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:28.151865 14759 ts_tablet_manager.cc:1398] T 2a3c9f1f41dd43d9878f97388523a2cf P 80fd18ef4d604c86b909d5c008a46b4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:28.153400 14649 consensus_queue.cc:1035] T 676e4bd2e5ed4cbea577a2f4210eb781 P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9942306855404cfd891adf227dd65eb2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:28.155495 14691 raft_consensus.cc:2781] T 2e87caa8a93d43a7b61e1596b12b0906 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:28.155611 14691 raft_consensus.cc:686] T 2e87caa8a93d43a7b61e1596b12b0906 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Becoming Leader. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Running, Role: LEADER
I20240503 02:00:28.155748 14691 consensus_queue.cc:237] T 2e87caa8a93d43a7b61e1596b12b0906 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:28.155807 14745 tablet.cc:1722] T 3d7f87f912be4c9daf0848002c50bfeb P 9942306855404cfd891adf227dd65eb2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:28.157253 14746 tablet.cc:1722] T 3d7f87f912be4c9daf0848002c50bfeb P c2ac2eb7cc264047b9ab32fe3a4fc025: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:28.158708 14762 tablet_bootstrap.cc:492] T 2e87caa8a93d43a7b61e1596b12b0906 P c2ac2eb7cc264047b9ab32fe3a4fc025: No bootstrap required, opened a new log
I20240503 02:00:28.158814 14762 ts_tablet_manager.cc:1367] T 2e87caa8a93d43a7b61e1596b12b0906 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent bootstrapping tablet: real 0.038s user 0.002s sys 0.000s
I20240503 02:00:28.162905 14762 raft_consensus.cc:348] T 2e87caa8a93d43a7b61e1596b12b0906 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:28.163057 14762 raft_consensus.cc:374] T 2e87caa8a93d43a7b61e1596b12b0906 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:28.163090 14762 raft_consensus.cc:729] T 2e87caa8a93d43a7b61e1596b12b0906 P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac2eb7cc264047b9ab32fe3a4fc025, State: Initialized, Role: FOLLOWER
I20240503 02:00:28.163223 14762 consensus_queue.cc:260] T 2e87caa8a93d43a7b61e1596b12b0906 P c2ac2eb7cc264047b9ab32fe3a4fc025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } }
I20240503 02:00:28.163547 14762 ts_tablet_manager.cc:1398] T 2e87caa8a93d43a7b61e1596b12b0906 P c2ac2eb7cc264047b9ab32fe3a4fc025: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240503 02:00:28.163930 6314 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: rebalancer_dynamic_table_new_name [id=ec3c6861e0924592b1687dc1dc8245cf] alter complete (version 3)
I20240503 02:00:28.170946 6313 catalog_manager.cc:5484] T 2e87caa8a93d43a7b61e1596b12b0906 P 2a01ace80baf4057b6bf1995cf88468b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135). New cstate: current_term: 1 leader_uuid: "2a01ace80baf4057b6bf1995cf88468b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac2eb7cc264047b9ab32fe3a4fc025" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42655 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:28.206408 14751 raft_consensus.cc:482] T bdc99a9b4d144059bfda89af3c033803 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:28.206553 14751 raft_consensus.cc:504] T bdc99a9b4d144059bfda89af3c033803 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:28.206936 14751 leader_election.cc:288] T bdc99a9b4d144059bfda89af3c033803 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:28.207228 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdc99a9b4d144059bfda89af3c033803" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a" is_pre_election: true
I20240503 02:00:28.207362 6456 raft_consensus.cc:2454] T bdc99a9b4d144059bfda89af3c033803 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:28.207520 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdc99a9b4d144059bfda89af3c033803" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d" is_pre_election: true
I20240503 02:00:28.207634 9112 leader_election.cc:302] T bdc99a9b4d144059bfda89af3c033803 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, dec3e4a04d604f7eaaa3f077d081218a; no voters:
I20240503 02:00:28.207653 6849 raft_consensus.cc:2454] T bdc99a9b4d144059bfda89af3c033803 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 0.
I20240503 02:00:28.207798 14751 raft_consensus.cc:2781] T bdc99a9b4d144059bfda89af3c033803 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:28.207847 14751 raft_consensus.cc:482] T bdc99a9b4d144059bfda89af3c033803 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:28.207882 14751 raft_consensus.cc:3037] T bdc99a9b4d144059bfda89af3c033803 P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:28.208952 14751 raft_consensus.cc:504] T bdc99a9b4d144059bfda89af3c033803 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:28.209293 14751 leader_election.cc:288] T bdc99a9b4d144059bfda89af3c033803 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Requested vote from peers dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093), 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719)
I20240503 02:00:28.210923 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdc99a9b4d144059bfda89af3c033803" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492448570cc94ed1879feb2ec3b9051d"
I20240503 02:00:28.211053 6849 raft_consensus.cc:3037] T bdc99a9b4d144059bfda89af3c033803 P 492448570cc94ed1879feb2ec3b9051d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:28.211258 6456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdc99a9b4d144059bfda89af3c033803" candidate_uuid: "2a01ace80baf4057b6bf1995cf88468b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec3e4a04d604f7eaaa3f077d081218a"
I20240503 02:00:28.211380 6456 raft_consensus.cc:3037] T bdc99a9b4d144059bfda89af3c033803 P dec3e4a04d604f7eaaa3f077d081218a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:28.212173 6456 raft_consensus.cc:2454] T bdc99a9b4d144059bfda89af3c033803 P dec3e4a04d604f7eaaa3f077d081218a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:28.212180 6849 raft_consensus.cc:2454] T bdc99a9b4d144059bfda89af3c033803 P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a01ace80baf4057b6bf1995cf88468b in term 1.
I20240503 02:00:28.212572 9115 leader_election.cc:302] T bdc99a9b4d144059bfda89af3c033803 P 2a01ace80baf4057b6bf1995cf88468b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 492448570cc94ed1879feb2ec3b9051d; no voters:
I20240503 02:00:28.212901 14751 raft_consensus.cc:2781] T bdc99a9b4d144059bfda89af3c033803 P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:28.212990 14751 raft_consensus.cc:686] T bdc99a9b4d144059bfda89af3c033803 P 2a01ace80baf4057b6bf1995cf88468b [term 1 LEADER]: Becoming Leader. State: Replica: 2a01ace80baf4057b6bf1995cf88468b, State: Running, Role: LEADER
I20240503 02:00:28.213131 14751 consensus_queue.cc:237] T bdc99a9b4d144059bfda89af3c033803 P 2a01ace80baf4057b6bf1995cf88468b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } }
I20240503 02:00:28.215942 6309 catalog_manager.cc:5484] T bdc99a9b4d144059bfda89af3c033803 P 2a01ace80baf4057b6bf1995cf88468b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135). New cstate: current_term: 1 leader_uuid: "2a01ace80baf4057b6bf1995cf88468b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec3e4a04d604f7eaaa3f077d081218a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "492448570cc94ed1879feb2ec3b9051d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42719 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:28.221616 14649 raft_consensus.cc:482] T 2a3c9f1f41dd43d9878f97388523a2cf P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:28.221733 14649 raft_consensus.cc:504] T 2a3c9f1f41dd43d9878f97388523a2cf P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:28.222119 14649 leader_election.cc:288] T 2a3c9f1f41dd43d9878f97388523a2cf P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:28.222559 9177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a3c9f1f41dd43d9878f97388523a2cf" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b" is_pre_election: true
I20240503 02:00:28.222576 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a3c9f1f41dd43d9878f97388523a2cf" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b" is_pre_election: true
I20240503 02:00:28.222710 9048 raft_consensus.cc:2454] T 2a3c9f1f41dd43d9878f97388523a2cf P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:28.222705 9177 raft_consensus.cc:2454] T 2a3c9f1f41dd43d9878f97388523a2cf P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 0.
I20240503 02:00:28.223043 6522 leader_election.cc:302] T 2a3c9f1f41dd43d9878f97388523a2cf P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 80fd18ef4d604c86b909d5c008a46b4f; no voters:
I20240503 02:00:28.223343 14649 raft_consensus.cc:2781] T 2a3c9f1f41dd43d9878f97388523a2cf P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:28.223397 14649 raft_consensus.cc:482] T 2a3c9f1f41dd43d9878f97388523a2cf P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:28.223430 14649 raft_consensus.cc:3037] T 2a3c9f1f41dd43d9878f97388523a2cf P 80fd18ef4d604c86b909d5c008a46b4f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:28.224694 14649 raft_consensus.cc:504] T 2a3c9f1f41dd43d9878f97388523a2cf P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:28.225078 14649 leader_election.cc:288] T 2a3c9f1f41dd43d9878f97388523a2cf P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Requested vote from peers 99ead02915954b4fab23b033ad6d546b (127.0.99.134:46157), 2a01ace80baf4057b6bf1995cf88468b (127.0.99.135:34079)
I20240503 02:00:28.225437 9177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a3c9f1f41dd43d9878f97388523a2cf" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a01ace80baf4057b6bf1995cf88468b"
I20240503 02:00:28.225557 9177 raft_consensus.cc:3037] T 2a3c9f1f41dd43d9878f97388523a2cf P 2a01ace80baf4057b6bf1995cf88468b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:28.225605 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a3c9f1f41dd43d9878f97388523a2cf" candidate_uuid: "80fd18ef4d604c86b909d5c008a46b4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99ead02915954b4fab23b033ad6d546b"
I20240503 02:00:28.225708 9048 raft_consensus.cc:3037] T 2a3c9f1f41dd43d9878f97388523a2cf P 99ead02915954b4fab23b033ad6d546b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:28.226581 9177 raft_consensus.cc:2454] T 2a3c9f1f41dd43d9878f97388523a2cf P 2a01ace80baf4057b6bf1995cf88468b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:28.226581 9048 raft_consensus.cc:2454] T 2a3c9f1f41dd43d9878f97388523a2cf P 99ead02915954b4fab23b033ad6d546b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fd18ef4d604c86b909d5c008a46b4f in term 1.
I20240503 02:00:28.226934 6522 leader_election.cc:302] T 2a3c9f1f41dd43d9878f97388523a2cf P 80fd18ef4d604c86b909d5c008a46b4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a01ace80baf4057b6bf1995cf88468b, 80fd18ef4d604c86b909d5c008a46b4f; no voters:
I20240503 02:00:28.227316 14671 raft_consensus.cc:2781] T 2a3c9f1f41dd43d9878f97388523a2cf P 80fd18ef4d604c86b909d5c008a46b4f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:28.227403 14671 raft_consensus.cc:686] T 2a3c9f1f41dd43d9878f97388523a2cf P 80fd18ef4d604c86b909d5c008a46b4f [term 1 LEADER]: Becoming Leader. State: Replica: 80fd18ef4d604c86b909d5c008a46b4f, State: Running, Role: LEADER
I20240503 02:00:28.227528 14671 consensus_queue.cc:237] T 2a3c9f1f41dd43d9878f97388523a2cf P 80fd18ef4d604c86b909d5c008a46b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } }
I20240503 02:00:28.231774 6309 catalog_manager.cc:5484] T 2a3c9f1f41dd43d9878f97388523a2cf P 80fd18ef4d604c86b909d5c008a46b4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 80fd18ef4d604c86b909d5c008a46b4f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "80fd18ef4d604c86b909d5c008a46b4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ead02915954b4fab23b033ad6d546b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 46157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a01ace80baf4057b6bf1995cf88468b" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fd18ef4d604c86b909d5c008a46b4f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43863 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:28.267802 6309 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:28.267961 6309 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58400:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:28.270243 6309 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 953f1c0a125d48d39bdd77e2ab9f058a: Sending DeleteTablet for 3 replicas of tablet 3d7f87f912be4c9daf0848002c50bfeb
I20240503 02:00:28.271003 6831 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d7f87f912be4c9daf0848002c50bfeb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:28 UTC) from {username='slave'} at 127.0.0.1:52812
I20240503 02:00:28.271073 8887 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d7f87f912be4c9daf0848002c50bfeb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:28 UTC) from {username='slave'} at 127.0.0.1:40296
I20240503 02:00:28.271131 6700 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d7f87f912be4c9daf0848002c50bfeb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:28 UTC) from {username='slave'} at 127.0.0.1:40336
I20240503 02:00:28.271430 14775 tablet_replica.cc:330] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d: stopping tablet replica
I20240503 02:00:28.271479 14777 tablet_replica.cc:330] T 3d7f87f912be4c9daf0848002c50bfeb P c2ac2eb7cc264047b9ab32fe3a4fc025: stopping tablet replica
I20240503 02:00:28.271764 14777 raft_consensus.cc:2229] T 3d7f87f912be4c9daf0848002c50bfeb P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:28.271765 14775 raft_consensus.cc:2229] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:28.271983 14777 raft_consensus.cc:2258] T 3d7f87f912be4c9daf0848002c50bfeb P c2ac2eb7cc264047b9ab32fe3a4fc025 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:28.272027 14775 raft_consensus.cc:2258] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:28.272293 14776 tablet_replica.cc:330] T 3d7f87f912be4c9daf0848002c50bfeb P 9942306855404cfd891adf227dd65eb2: stopping tablet replica
I20240503 02:00:28.272467 14776 raft_consensus.cc:2229] T 3d7f87f912be4c9daf0848002c50bfeb P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:28.272612 14776 raft_consensus.cc:2258] T 3d7f87f912be4c9daf0848002c50bfeb P 9942306855404cfd891adf227dd65eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:28.272954 14777 ts_tablet_manager.cc:1875] T 3d7f87f912be4c9daf0848002c50bfeb P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:28.273129 14775 ts_tablet_manager.cc:1875] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:28.274989 14776 ts_tablet_manager.cc:1875] T 3d7f87f912be4c9daf0848002c50bfeb P 9942306855404cfd891adf227dd65eb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:28.275708 14777 ts_tablet_manager.cc:1888] T 3d7f87f912be4c9daf0848002c50bfeb P c2ac2eb7cc264047b9ab32fe3a4fc025: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:28.275961 14777 log.cc:1196] T 3d7f87f912be4c9daf0848002c50bfeb P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/3d7f87f912be4c9daf0848002c50bfeb
I20240503 02:00:28.276329 14777 ts_tablet_manager.cc:1909] T 3d7f87f912be4c9daf0848002c50bfeb P c2ac2eb7cc264047b9ab32fe3a4fc025: Deleting consensus metadata
I20240503 02:00:28.277468 6303 catalog_manager.cc:4830] TS c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655): tablet 3d7f87f912be4c9daf0848002c50bfeb (table rebalancer_dynamic_table_new_name [id=ec3c6861e0924592b1687dc1dc8245cf]) successfully deleted
I20240503 02:00:28.277947 14776 ts_tablet_manager.cc:1888] T 3d7f87f912be4c9daf0848002c50bfeb P 9942306855404cfd891adf227dd65eb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:28.278038 14776 log.cc:1196] T 3d7f87f912be4c9daf0848002c50bfeb P 9942306855404cfd891adf227dd65eb2: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/3d7f87f912be4c9daf0848002c50bfeb
I20240503 02:00:28.278302 14776 ts_tablet_manager.cc:1909] T 3d7f87f912be4c9daf0848002c50bfeb P 9942306855404cfd891adf227dd65eb2: Deleting consensus metadata
I20240503 02:00:28.279558 14775 ts_tablet_manager.cc:1888] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:28.279652 14775 log.cc:1196] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d: Deleting WAL directory at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/3d7f87f912be4c9daf0848002c50bfeb
I20240503 02:00:28.279992 14775 ts_tablet_manager.cc:1909] T 3d7f87f912be4c9daf0848002c50bfeb P 492448570cc94ed1879feb2ec3b9051d: Deleting consensus metadata
I20240503 02:00:28.280754 6303 catalog_manager.cc:4830] TS 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211): tablet 3d7f87f912be4c9daf0848002c50bfeb (table rebalancer_dynamic_table_new_name [id=ec3c6861e0924592b1687dc1dc8245cf]) successfully deleted
I20240503 02:00:28.280946 6303 catalog_manager.cc:4830] TS 492448570cc94ed1879feb2ec3b9051d (127.0.99.132:42719): tablet 3d7f87f912be4c9daf0848002c50bfeb (table rebalancer_dynamic_table_new_name [id=ec3c6861e0924592b1687dc1dc8245cf]) successfully deleted
W20240503 02:00:28.330543 6523 consensus_peers.cc:485] T 4385d51d11574192883d01376e2d5620 P 80fd18ef4d604c86b909d5c008a46b4f -> Peer e1a9251ce1cf4361b8b0ede9fbfc4023 (127.0.99.136:45311): Couldn't send request to peer e1a9251ce1cf4361b8b0ede9fbfc4023. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:45311: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 02:00:28.365609 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu with pid 6376
I20240503 02:00:28.376087 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu with pid 6507
I20240503 02:00:28.386612 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu with pid 6638
W20240503 02:00:28.390647 8850 consensus_peers.cc:485] T 7da8f614a70e4177b70e7c2f31d22a10 P 9942306855404cfd891adf227dd65eb2 -> Peer dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): Couldn't send request to peer dec3e4a04d604f7eaaa3f077d081218a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33093: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:28.392160 8981 consensus_peers.cc:485] T bd8b9532bd744ecdbe8da66a1225724b P 99ead02915954b4fab23b033ad6d546b -> Peer dec3e4a04d604f7eaaa3f077d081218a (127.0.99.129:33093): Couldn't send request to peer dec3e4a04d604f7eaaa3f077d081218a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33093: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:28.397145 8853 connection.cc:535] client connection to 127.0.99.131:42655 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:00:28.397188 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu with pid 6769
W20240503 02:00:28.399657 8853 consensus_peers.cc:485] T f55cb3da9c874387a65eff5de0903331 P 9942306855404cfd891adf227dd65eb2 -> Peer c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655): Couldn't send request to peer c2ac2eb7cc264047b9ab32fe3a4fc025. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:42655: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:28.408092 8853 consensus_peers.cc:485] T 1c721b33983b4c47afd8faa8f155ca84 P 9942306855404cfd891adf227dd65eb2 -> Peer c2ac2eb7cc264047b9ab32fe3a4fc025 (127.0.99.131:42655): Couldn't send request to peer c2ac2eb7cc264047b9ab32fe3a4fc025. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:42655: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:28.409587 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu with pid 8835
W20240503 02:00:28.416628 9115 connection.cc:535] client connection to 127.0.99.133:33211 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:00:28.417423 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu with pid 8966
W20240503 02:00:28.418488 9115 consensus_peers.cc:485] T b1740b877c1e4372a0337d99f5bcab0e P 2a01ace80baf4057b6bf1995cf88468b -> Peer 9942306855404cfd891adf227dd65eb2 (127.0.99.133:33211): Couldn't send request to peer 9942306855404cfd891adf227dd65eb2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:33211: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:28.424235 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu with pid 9097
I20240503 02:00:28.431548 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu with pid 6285
2024-05-03T02:00:28Z chronyd exiting
I20240503 02:00:28.448989 398 test_util.cc:175] -----------------------------------------------
I20240503 02:00:28.449059 398 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714701555658200-398-0
[ FAILED ] DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing/1, where GetParam() = 4-byte object <01-00 00-00> (47866 ms)
[----------] 1 test from DDLDuringRebalancingTest (47866 ms total)
[----------] 1 test from RebalancingDuringElectionStormTest
[ RUN ] RebalancingDuringElectionStormTest.RoundRobin/1
I20240503 02:00:28.449743 398 test_util.cc:268] Using random seed: 360031180
I20240503 02:00:28.450536 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 02:00:28.450584 398 ts_itest-base.cc:113] --------------
I20240503 02:00:28.450594 398 ts_itest-base.cc:114] 8 tablet servers
I20240503 02:00:28.450603 398 ts_itest-base.cc:115] 3 replicas per TS
I20240503 02:00:28.450613 398 ts_itest-base.cc:116] --------------
2024-05-03T02:00:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T02:00:28Z Disabled control of system clock
I20240503 02:00:28.459030 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
/tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:38675
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:43797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:38675
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240503 02:00:28.568564 14785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:28.568851 14785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:28.568908 14785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:28.572146 14785 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:00:28.572252 14785 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 02:00:28.572283 14785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:28.572305 14785 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 02:00:28.572525 14785 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 02:00:28.576942 14785 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:38675
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:38675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.14785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:28.578008 14785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:28.579222 14785 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:28.587168 14790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:28.587327 14785 server_base.cc:1034] running on GCE node
W20240503 02:00:28.587491 14793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:28.587643 14791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:28.588218 14785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:28.589332 14785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:28.590538 14785 hybrid_clock.cc:648] HybridClock initialized: now 1714701628590514 us; error 61 us; skew 500 ppm
I20240503 02:00:28.592996 14785 webserver.cc:460] Webserver started at http://127.0.99.190:46539/ using document root <none> and password file <none>
I20240503 02:00:28.593609 14785 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:28.593670 14785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:28.593833 14785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:28.595629 14785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "284928b4d3d44522bbf4c11ce5481d5f"
format_stamp: "Formatted at 2024-05-03 02:00:28 on dist-test-slave-pz5w"
I20240503 02:00:28.596278 14785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "284928b4d3d44522bbf4c11ce5481d5f"
format_stamp: "Formatted at 2024-05-03 02:00:28 on dist-test-slave-pz5w"
I20240503 02:00:28.600126 14785 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20240503 02:00:28.602782 14799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:28.604179 14785 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240503 02:00:28.604334 14785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "284928b4d3d44522bbf4c11ce5481d5f"
format_stamp: "Formatted at 2024-05-03 02:00:28 on dist-test-slave-pz5w"
I20240503 02:00:28.604482 14785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:28.617105 14785 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:28.617995 14785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:28.618194 14785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:28.625905 14785 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:38675
I20240503 02:00:28.625926 14851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:38675 every 8 connection(s)
I20240503 02:00:28.626986 14785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 02:00:28.629899 14852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:28.635254 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu as pid 14785
I20240503 02:00:28.635391 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 02:00:28.635398 14852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 284928b4d3d44522bbf4c11ce5481d5f: Bootstrap starting.
I20240503 02:00:28.638001 14852 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 284928b4d3d44522bbf4c11ce5481d5f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:28.638773 14852 log.cc:824] T 00000000000000000000000000000000 P 284928b4d3d44522bbf4c11ce5481d5f: Log is configured to *not* fsync() on all Append() calls
I20240503 02:00:28.640897 14852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 284928b4d3d44522bbf4c11ce5481d5f: No bootstrap required, opened a new log
I20240503 02:00:28.644140 14852 raft_consensus.cc:348] T 00000000000000000000000000000000 P 284928b4d3d44522bbf4c11ce5481d5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "284928b4d3d44522bbf4c11ce5481d5f" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 38675 } }
I20240503 02:00:28.644399 14852 raft_consensus.cc:374] T 00000000000000000000000000000000 P 284928b4d3d44522bbf4c11ce5481d5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:28.644451 14852 raft_consensus.cc:729] T 00000000000000000000000000000000 P 284928b4d3d44522bbf4c11ce5481d5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 284928b4d3d44522bbf4c11ce5481d5f, State: Initialized, Role: FOLLOWER
I20240503 02:00:28.644917 14852 consensus_queue.cc:260] T 00000000000000000000000000000000 P 284928b4d3d44522bbf4c11ce5481d5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "284928b4d3d44522bbf4c11ce5481d5f" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 38675 } }
I20240503 02:00:28.645066 14852 raft_consensus.cc:388] T 00000000000000000000000000000000 P 284928b4d3d44522bbf4c11ce5481d5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:00:28.645130 14852 raft_consensus.cc:482] T 00000000000000000000000000000000 P 284928b4d3d44522bbf4c11ce5481d5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:00:28.645253 14852 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 284928b4d3d44522bbf4c11ce5481d5f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:28.646306 14852 raft_consensus.cc:504] T 00000000000000000000000000000000 P 284928b4d3d44522bbf4c11ce5481d5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "284928b4d3d44522bbf4c11ce5481d5f" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 38675 } }
I20240503 02:00:28.646674 14852 leader_election.cc:302] T 00000000000000000000000000000000 P 284928b4d3d44522bbf4c11ce5481d5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 284928b4d3d44522bbf4c11ce5481d5f; no voters:
I20240503 02:00:28.646984 14852 leader_election.cc:288] T 00000000000000000000000000000000 P 284928b4d3d44522bbf4c11ce5481d5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:00:28.647225 14857 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 284928b4d3d44522bbf4c11ce5481d5f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:28.647469 14857 raft_consensus.cc:686] T 00000000000000000000000000000000 P 284928b4d3d44522bbf4c11ce5481d5f [term 1 LEADER]: Becoming Leader. State: Replica: 284928b4d3d44522bbf4c11ce5481d5f, State: Running, Role: LEADER
I20240503 02:00:28.647769 14857 consensus_queue.cc:237] T 00000000000000000000000000000000 P 284928b4d3d44522bbf4c11ce5481d5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "284928b4d3d44522bbf4c11ce5481d5f" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 38675 } }
I20240503 02:00:28.648121 14852 sys_catalog.cc:564] T 00000000000000000000000000000000 P 284928b4d3d44522bbf4c11ce5481d5f [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:00:28.649400 14859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 284928b4d3d44522bbf4c11ce5481d5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 284928b4d3d44522bbf4c11ce5481d5f. Latest consensus state: current_term: 1 leader_uuid: "284928b4d3d44522bbf4c11ce5481d5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "284928b4d3d44522bbf4c11ce5481d5f" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 38675 } } }
I20240503 02:00:28.649413 14858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 284928b4d3d44522bbf4c11ce5481d5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "284928b4d3d44522bbf4c11ce5481d5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "284928b4d3d44522bbf4c11ce5481d5f" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 38675 } } }
I20240503 02:00:28.649581 14859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 284928b4d3d44522bbf4c11ce5481d5f [sys.catalog]: This master's current role is: LEADER
I20240503 02:00:28.649583 14858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 284928b4d3d44522bbf4c11ce5481d5f [sys.catalog]: This master's current role is: LEADER
I20240503 02:00:28.650061 14866 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:00:28.653134 14866 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:00:28.659170 14866 catalog_manager.cc:1347] Generated new cluster ID: e9b64965a1bf4c7db8db36618edb6c81
I20240503 02:00:28.659265 14866 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:00:28.675714 14866 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:00:28.676831 14866 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:00:28.690471 14866 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 284928b4d3d44522bbf4c11ce5481d5f: Generated new TSK 0
I20240503 02:00:28.691538 14866 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:00:28.701570 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
/tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38675
--builtin_ntp_servers=127.0.99.148:43797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 02:00:28.816264 14876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:28.816598 14876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:28.816675 14876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:28.819913 14876 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:00:28.820008 14876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:28.820102 14876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 02:00:28.823973 14876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38675
--never_fsync=true
--heap_profile_path=/tmp/kudu.14876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:28.824975 14876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:28.826105 14876 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:28.834599 14882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:28.834621 14884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:28.834604 14881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:28.835011 14876 server_base.cc:1034] running on GCE node
I20240503 02:00:28.835430 14876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:28.836045 14876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:28.837247 14876 hybrid_clock.cc:648] HybridClock initialized: now 1714701628837203 us; error 46 us; skew 500 ppm
I20240503 02:00:28.839419 14876 webserver.cc:460] Webserver started at http://127.0.99.129:40641/ using document root <none> and password file <none>
I20240503 02:00:28.840092 14876 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:28.840162 14876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:28.840343 14876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:28.842167 14876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "dcfdfa62edc34dab9525e2712d487582"
format_stamp: "Formatted at 2024-05-03 02:00:28 on dist-test-slave-pz5w"
I20240503 02:00:28.842659 14876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "dcfdfa62edc34dab9525e2712d487582"
format_stamp: "Formatted at 2024-05-03 02:00:28 on dist-test-slave-pz5w"
I20240503 02:00:28.846650 14876 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20240503 02:00:28.849457 14890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:28.850785 14876 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240503 02:00:28.850920 14876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "dcfdfa62edc34dab9525e2712d487582"
format_stamp: "Formatted at 2024-05-03 02:00:28 on dist-test-slave-pz5w"
I20240503 02:00:28.851032 14876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:28.876760 14876 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:28.877522 14876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:28.877694 14876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:28.878260 14876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:28.879213 14876 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:28.879262 14876 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:28.879307 14876 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:28.879334 14876 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:28.890799 14876 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:38817
I20240503 02:00:28.890856 15003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:38817 every 8 connection(s)
I20240503 02:00:28.891829 14876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 02:00:28.898105 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu as pid 14876
I20240503 02:00:28.898243 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 02:00:28.900663 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
/tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38675
--builtin_ntp_servers=127.0.99.148:43797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:00:28.902451 15004 heartbeater.cc:347] Connected to a master server at 127.0.99.190:38675
I20240503 02:00:28.902724 15004 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:28.903425 15004 heartbeater.cc:511] Master 127.0.99.190:38675 requested a full tablet report, sending...
I20240503 02:00:28.905254 14816 ts_manager.cc:194] Registered new tserver with Master: dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817)
I20240503 02:00:28.906489 14816 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:41879
W20240503 02:00:29.012943 15007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:29.013180 15007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:29.013229 15007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:29.016625 15007 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:00:29.016709 15007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:29.016782 15007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 02:00:29.020781 15007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38675
--never_fsync=true
--heap_profile_path=/tmp/kudu.15007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:29.021771 15007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:29.022946 15007 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:29.031958 15012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:29.031961 15015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:29.031965 15013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:29.032212 15007 server_base.cc:1034] running on GCE node
I20240503 02:00:29.032769 15007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:29.033445 15007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:29.034632 15007 hybrid_clock.cc:648] HybridClock initialized: now 1714701629034615 us; error 49 us; skew 500 ppm
I20240503 02:00:29.036949 15007 webserver.cc:460] Webserver started at http://127.0.99.130:40049/ using document root <none> and password file <none>
I20240503 02:00:29.037649 15007 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:29.037753 15007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:29.037961 15007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:29.040025 15007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "b602f6099cbb4758847e7e9550685372"
format_stamp: "Formatted at 2024-05-03 02:00:29 on dist-test-slave-pz5w"
I20240503 02:00:29.040625 15007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "b602f6099cbb4758847e7e9550685372"
format_stamp: "Formatted at 2024-05-03 02:00:29 on dist-test-slave-pz5w"
I20240503 02:00:29.044759 15007 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20240503 02:00:29.047286 15021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:29.048499 15007 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 02:00:29.048645 15007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "b602f6099cbb4758847e7e9550685372"
format_stamp: "Formatted at 2024-05-03 02:00:29 on dist-test-slave-pz5w"
I20240503 02:00:29.048750 15007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:29.063058 15007 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:29.063877 15007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:29.064062 15007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:29.064739 15007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:29.065768 15007 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:29.065819 15007 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:29.065876 15007 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:29.065905 15007 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:29.074414 15007 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:38207
I20240503 02:00:29.074430 15134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:38207 every 8 connection(s)
I20240503 02:00:29.075414 15007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 02:00:29.077222 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu as pid 15007
I20240503 02:00:29.077342 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 02:00:29.079375 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
/tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38675
--builtin_ntp_servers=127.0.99.148:43797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:00:29.086354 15135 heartbeater.cc:347] Connected to a master server at 127.0.99.190:38675
I20240503 02:00:29.086616 15135 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:29.087216 15135 heartbeater.cc:511] Master 127.0.99.190:38675 requested a full tablet report, sending...
I20240503 02:00:29.088349 14816 ts_manager.cc:194] Registered new tserver with Master: b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
I20240503 02:00:29.088959 14816 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:37771
W20240503 02:00:29.199410 15138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:29.199685 15138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:29.199730 15138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:29.203467 15138 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:00:29.203578 15138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:29.203651 15138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 02:00:29.208010 15138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38675
--never_fsync=true
--heap_profile_path=/tmp/kudu.15138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:29.209077 15138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:29.210286 15138 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:29.218849 15143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:29.218899 15144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:29.218807 15146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:29.219065 15138 server_base.cc:1034] running on GCE node
I20240503 02:00:29.219507 15138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:29.220173 15138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:29.221351 15138 hybrid_clock.cc:648] HybridClock initialized: now 1714701629221326 us; error 50 us; skew 500 ppm
I20240503 02:00:29.223517 15138 webserver.cc:460] Webserver started at http://127.0.99.131:36301/ using document root <none> and password file <none>
I20240503 02:00:29.224114 15138 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:29.224177 15138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:29.224359 15138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:29.226166 15138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "620596b893d84291b88b90dc9e317612"
format_stamp: "Formatted at 2024-05-03 02:00:29 on dist-test-slave-pz5w"
I20240503 02:00:29.226660 15138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "620596b893d84291b88b90dc9e317612"
format_stamp: "Formatted at 2024-05-03 02:00:29 on dist-test-slave-pz5w"
I20240503 02:00:29.230480 15138 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 02:00:29.233093 15152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:29.234418 15138 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 02:00:29.234573 15138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "620596b893d84291b88b90dc9e317612"
format_stamp: "Formatted at 2024-05-03 02:00:29 on dist-test-slave-pz5w"
I20240503 02:00:29.234686 15138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:29.261559 15138 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:29.262617 15138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:29.263163 15138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:29.264437 15138 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:29.266044 15138 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:29.266112 15138 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:29.266171 15138 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:29.266191 15138 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:29.275457 15138 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:42437
I20240503 02:00:29.275537 15265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:42437 every 8 connection(s)
I20240503 02:00:29.276890 15138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 02:00:29.286615 15266 heartbeater.cc:347] Connected to a master server at 127.0.99.190:38675
I20240503 02:00:29.286661 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu as pid 15138
I20240503 02:00:29.286753 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 02:00:29.286870 15266 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:29.287504 15266 heartbeater.cc:511] Master 127.0.99.190:38675 requested a full tablet report, sending...
I20240503 02:00:29.288587 14816 ts_manager.cc:194] Registered new tserver with Master: 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437)
I20240503 02:00:29.288878 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
/tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38675
--builtin_ntp_servers=127.0.99.148:43797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:00:29.289216 14816 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:44237
W20240503 02:00:29.404901 15269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:29.405159 15269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:29.405206 15269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:29.408483 15269 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:00:29.408573 15269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:29.408663 15269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 02:00:29.412642 15269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38675
--never_fsync=true
--heap_profile_path=/tmp/kudu.15269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:29.413659 15269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:29.414770 15269 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:29.423251 15275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:29.423251 15277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:29.423550 15274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:29.423594 15269 server_base.cc:1034] running on GCE node
I20240503 02:00:29.424021 15269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:29.424662 15269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:29.425809 15269 hybrid_clock.cc:648] HybridClock initialized: now 1714701629425790 us; error 39 us; skew 500 ppm
I20240503 02:00:29.427870 15269 webserver.cc:460] Webserver started at http://127.0.99.132:41127/ using document root <none> and password file <none>
I20240503 02:00:29.428469 15269 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:29.428531 15269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:29.428694 15269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:29.430461 15269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "c05f7620bb884392931d3243a0b47a9e"
format_stamp: "Formatted at 2024-05-03 02:00:29 on dist-test-slave-pz5w"
I20240503 02:00:29.430946 15269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "c05f7620bb884392931d3243a0b47a9e"
format_stamp: "Formatted at 2024-05-03 02:00:29 on dist-test-slave-pz5w"
I20240503 02:00:29.434413 15269 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 02:00:29.436795 15283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:29.437875 15269 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 02:00:29.437992 15269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "c05f7620bb884392931d3243a0b47a9e"
format_stamp: "Formatted at 2024-05-03 02:00:29 on dist-test-slave-pz5w"
I20240503 02:00:29.438081 15269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:29.451737 15269 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:29.452437 15269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:29.452585 15269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:29.453112 15269 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:29.454064 15269 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:29.454110 15269 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:29.454149 15269 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:29.454169 15269 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:29.462468 15269 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:43009
I20240503 02:00:29.462507 15396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:43009 every 8 connection(s)
I20240503 02:00:29.463469 15269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 02:00:29.465293 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu as pid 15269
I20240503 02:00:29.465413 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 02:00:29.467479 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
/tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38675
--builtin_ntp_servers=127.0.99.148:43797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:00:29.473315 15397 heartbeater.cc:347] Connected to a master server at 127.0.99.190:38675
I20240503 02:00:29.473565 15397 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:29.474146 15397 heartbeater.cc:511] Master 127.0.99.190:38675 requested a full tablet report, sending...
I20240503 02:00:29.475248 14816 ts_manager.cc:194] Registered new tserver with Master: c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:29.475818 14816 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:50989
W20240503 02:00:29.582863 15400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:29.583127 15400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:29.583191 15400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:29.586495 15400 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:00:29.586577 15400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:29.586668 15400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 02:00:29.590590 15400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38675
--never_fsync=true
--heap_profile_path=/tmp/kudu.15400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:29.591570 15400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:29.592795 15400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:29.601146 15405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:29.601146 15406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:29.601155 15408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:29.601469 15400 server_base.cc:1034] running on GCE node
I20240503 02:00:29.601981 15400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:29.602581 15400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:29.603770 15400 hybrid_clock.cc:648] HybridClock initialized: now 1714701629603749 us; error 42 us; skew 500 ppm
I20240503 02:00:29.605948 15400 webserver.cc:460] Webserver started at http://127.0.99.133:41511/ using document root <none> and password file <none>
I20240503 02:00:29.606531 15400 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:29.606595 15400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:29.606781 15400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:29.608649 15400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "07a1b0ea41ae45b3945dc0d17d405d61"
format_stamp: "Formatted at 2024-05-03 02:00:29 on dist-test-slave-pz5w"
I20240503 02:00:29.609162 15400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "07a1b0ea41ae45b3945dc0d17d405d61"
format_stamp: "Formatted at 2024-05-03 02:00:29 on dist-test-slave-pz5w"
I20240503 02:00:29.612782 15400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 02:00:29.615430 15414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:29.616647 15400 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 02:00:29.616791 15400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "07a1b0ea41ae45b3945dc0d17d405d61"
format_stamp: "Formatted at 2024-05-03 02:00:29 on dist-test-slave-pz5w"
I20240503 02:00:29.616894 15400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:29.644691 15400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:29.645344 15400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:29.645514 15400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:29.646070 15400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:29.646993 15400 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:29.647151 15400 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:29.647258 15400 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:29.647291 15400 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:29.657270 15400 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:43821
I20240503 02:00:29.657305 15527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:43821 every 8 connection(s)
I20240503 02:00:29.658313 15400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 02:00:29.664237 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu as pid 15400
I20240503 02:00:29.664414 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 02:00:29.666818 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
/tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:0
--local_ip_for_outbound_sockets=127.0.99.134
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38675
--builtin_ntp_servers=127.0.99.148:43797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:00:29.668205 15528 heartbeater.cc:347] Connected to a master server at 127.0.99.190:38675
I20240503 02:00:29.668524 15528 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:29.669255 15528 heartbeater.cc:511] Master 127.0.99.190:38675 requested a full tablet report, sending...
I20240503 02:00:29.670491 14816 ts_manager.cc:194] Registered new tserver with Master: 07a1b0ea41ae45b3945dc0d17d405d61 (127.0.99.133:43821)
I20240503 02:00:29.671262 14816 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:51801
W20240503 02:00:29.781126 15531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:29.781383 15531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:29.781427 15531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:29.784585 15531 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:00:29.784662 15531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:29.784732 15531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 02:00:29.788625 15531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38675
--never_fsync=true
--heap_profile_path=/tmp/kudu.15531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:29.789544 15531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:29.790647 15531 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:29.798954 15536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:29.798954 15537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:29.799011 15531 server_base.cc:1034] running on GCE node
W20240503 02:00:29.799350 15539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:29.799830 15531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:29.800501 15531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:29.801666 15531 hybrid_clock.cc:648] HybridClock initialized: now 1714701629801652 us; error 35 us; skew 500 ppm
I20240503 02:00:29.803771 15531 webserver.cc:460] Webserver started at http://127.0.99.134:33791/ using document root <none> and password file <none>
I20240503 02:00:29.804332 15531 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:29.804427 15531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:29.804622 15531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:29.806334 15531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "7262cfc0983543f7b62e31d7a00ee5b2"
format_stamp: "Formatted at 2024-05-03 02:00:29 on dist-test-slave-pz5w"
I20240503 02:00:29.806819 15531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "7262cfc0983543f7b62e31d7a00ee5b2"
format_stamp: "Formatted at 2024-05-03 02:00:29 on dist-test-slave-pz5w"
I20240503 02:00:29.810494 15531 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 02:00:29.813084 15545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:29.814291 15531 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20240503 02:00:29.814424 15531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "7262cfc0983543f7b62e31d7a00ee5b2"
format_stamp: "Formatted at 2024-05-03 02:00:29 on dist-test-slave-pz5w"
I20240503 02:00:29.814539 15531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:29.844851 15531 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:29.845448 15531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:29.845588 15531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:29.846082 15531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:29.847075 15531 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:29.847146 15531 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:29.847189 15531 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:29.847203 15531 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:29.855993 15531 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:42067
I20240503 02:00:29.856544 15658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:42067 every 8 connection(s)
I20240503 02:00:29.857033 15531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 02:00:29.863706 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu as pid 15531
I20240503 02:00:29.863853 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 02:00:29.866132 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
/tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:0
--local_ip_for_outbound_sockets=127.0.99.135
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38675
--builtin_ntp_servers=127.0.99.148:43797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:00:29.866305 15659 heartbeater.cc:347] Connected to a master server at 127.0.99.190:38675
I20240503 02:00:29.866556 15659 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:29.867252 15659 heartbeater.cc:511] Master 127.0.99.190:38675 requested a full tablet report, sending...
I20240503 02:00:29.868482 14816 ts_manager.cc:194] Registered new tserver with Master: 7262cfc0983543f7b62e31d7a00ee5b2 (127.0.99.134:42067)
I20240503 02:00:29.869189 14816 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:37773
I20240503 02:00:29.909562 15004 heartbeater.cc:503] Master 127.0.99.190:38675 was elected leader, sending a full tablet report...
W20240503 02:00:29.980203 15662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:29.980489 15662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:29.980537 15662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:29.983748 15662 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:00:29.983840 15662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:29.983915 15662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 02:00:29.987943 15662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38675
--never_fsync=true
--heap_profile_path=/tmp/kudu.15662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:29.988976 15662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:29.990103 15662 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 02:00:29.998916 15662 server_base.cc:1034] running on GCE node
W20240503 02:00:29.998863 15668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:29.998863 15667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:29.999171 15670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:29.999624 15662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:30.000264 15662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:30.001451 15662 hybrid_clock.cc:648] HybridClock initialized: now 1714701630001427 us; error 41 us; skew 500 ppm
I20240503 02:00:30.003615 15662 webserver.cc:460] Webserver started at http://127.0.99.135:42245/ using document root <none> and password file <none>
I20240503 02:00:30.004201 15662 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:30.004271 15662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:30.004494 15662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:30.006278 15662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "ce266197c0f4472ead9d54d16e706204"
format_stamp: "Formatted at 2024-05-03 02:00:30 on dist-test-slave-pz5w"
I20240503 02:00:30.006812 15662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "ce266197c0f4472ead9d54d16e706204"
format_stamp: "Formatted at 2024-05-03 02:00:30 on dist-test-slave-pz5w"
I20240503 02:00:30.010774 15662 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 02:00:30.013197 15676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:30.014310 15662 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:30.014436 15662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "ce266197c0f4472ead9d54d16e706204"
format_stamp: "Formatted at 2024-05-03 02:00:30 on dist-test-slave-pz5w"
I20240503 02:00:30.014533 15662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:30.039029 15662 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:30.039690 15662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:30.039860 15662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:30.040477 15662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:30.041416 15662 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:30.041465 15662 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:30.041522 15662 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:30.041545 15662 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:30.049981 15662 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:44193
I20240503 02:00:30.050009 15789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:44193 every 8 connection(s)
I20240503 02:00:30.051069 15662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 02:00:30.052917 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu as pid 15662
I20240503 02:00:30.053046 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 02:00:30.055497 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
/tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:0
--local_ip_for_outbound_sockets=127.0.99.136
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38675
--builtin_ntp_servers=127.0.99.148:43797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:00:30.061599 15790 heartbeater.cc:347] Connected to a master server at 127.0.99.190:38675
I20240503 02:00:30.061852 15790 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:30.062463 15790 heartbeater.cc:511] Master 127.0.99.190:38675 requested a full tablet report, sending...
I20240503 02:00:30.063556 14816 ts_manager.cc:194] Registered new tserver with Master: ce266197c0f4472ead9d54d16e706204 (127.0.99.135:44193)
I20240503 02:00:30.064103 14816 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:50373
I20240503 02:00:30.091043 15135 heartbeater.cc:503] Master 127.0.99.190:38675 was elected leader, sending a full tablet report...
W20240503 02:00:30.169270 15793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:30.169533 15793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:30.169593 15793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:30.172842 15793 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:00:30.172931 15793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:30.173024 15793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240503 02:00:30.176923 15793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38675
--never_fsync=true
--heap_profile_path=/tmp/kudu.15793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:30.177912 15793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:30.179052 15793 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:30.187296 15801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:30.187297 15798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:30.187552 15799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:30.187577 15793 server_base.cc:1034] running on GCE node
I20240503 02:00:30.188006 15793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:30.188673 15793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:30.189844 15793 hybrid_clock.cc:648] HybridClock initialized: now 1714701630189830 us; error 32 us; skew 500 ppm
I20240503 02:00:30.191879 15793 webserver.cc:460] Webserver started at http://127.0.99.136:45309/ using document root <none> and password file <none>
I20240503 02:00:30.192483 15793 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:30.192550 15793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:30.192747 15793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:30.194435 15793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "223c02bbc6d9479595889809d7384dae"
format_stamp: "Formatted at 2024-05-03 02:00:30 on dist-test-slave-pz5w"
I20240503 02:00:30.194911 15793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "223c02bbc6d9479595889809d7384dae"
format_stamp: "Formatted at 2024-05-03 02:00:30 on dist-test-slave-pz5w"
I20240503 02:00:30.198498 15793 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 02:00:30.200937 15807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:30.202095 15793 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 02:00:30.202240 15793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "223c02bbc6d9479595889809d7384dae"
format_stamp: "Formatted at 2024-05-03 02:00:30 on dist-test-slave-pz5w"
I20240503 02:00:30.202345 15793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:30.218559 15793 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:30.219224 15793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:30.219393 15793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:30.219928 15793 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:30.220893 15793 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:30.220954 15793 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:30.221006 15793 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:30.221036 15793 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:30.229184 15793 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:41611
I20240503 02:00:30.229213 15920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:41611 every 8 connection(s)
I20240503 02:00:30.230216 15793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 02:00:30.232129 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu as pid 15793
I20240503 02:00:30.232261 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 02:00:30.240454 15921 heartbeater.cc:347] Connected to a master server at 127.0.99.190:38675
I20240503 02:00:30.240756 15921 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:30.241468 15921 heartbeater.cc:511] Master 127.0.99.190:38675 requested a full tablet report, sending...
I20240503 02:00:30.242619 14816 ts_manager.cc:194] Registered new tserver with Master: 223c02bbc6d9479595889809d7384dae (127.0.99.136:41611)
I20240503 02:00:30.243204 14816 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:33925
I20240503 02:00:30.246513 398 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 02:00:30.257719 398 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 02:00:30.257813 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu with pid 15400
I20240503 02:00:30.263226 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu with pid 15531
I20240503 02:00:30.268229 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu with pid 15662
I20240503 02:00:30.273108 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu with pid 15793
I20240503 02:00:30.291718 15266 heartbeater.cc:503] Master 127.0.99.190:38675 was elected leader, sending a full tablet report...
I20240503 02:00:30.477939 15397 heartbeater.cc:503] Master 127.0.99.190:38675 was elected leader, sending a full tablet report...
I20240503 02:00:34.030820 14816 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39948:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:34.053026 14937 tablet_service.cc:1467] Processing CreateTablet for tablet 6798a9ed0cf441aca7b8ae1b05c001f3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7071a39d9779491da2d2a12d7faf7ccb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:34.054143 14937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6798a9ed0cf441aca7b8ae1b05c001f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:34.053026 14936 tablet_service.cc:1467] Processing CreateTablet for tablet 2b757c430aaa42f382d174ca57f5fcfc (DEFAULT_TABLE table=rebalance_test_table_0 [id=7071a39d9779491da2d2a12d7faf7ccb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:34.055621 14936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b757c430aaa42f382d174ca57f5fcfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:34.055316 15068 tablet_service.cc:1467] Processing CreateTablet for tablet 2b757c430aaa42f382d174ca57f5fcfc (DEFAULT_TABLE table=rebalance_test_table_0 [id=7071a39d9779491da2d2a12d7faf7ccb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:34.055316 15199 tablet_service.cc:1467] Processing CreateTablet for tablet 6798a9ed0cf441aca7b8ae1b05c001f3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7071a39d9779491da2d2a12d7faf7ccb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:34.056334 15068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b757c430aaa42f382d174ca57f5fcfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:34.056906 14938 tablet_service.cc:1467] Processing CreateTablet for tablet d805ee04df39406ba30b3b3b9e349507 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7071a39d9779491da2d2a12d7faf7ccb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:34.056334 15199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6798a9ed0cf441aca7b8ae1b05c001f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:34.057389 14938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d805ee04df39406ba30b3b3b9e349507. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:34.055316 15069 tablet_service.cc:1467] Processing CreateTablet for tablet d805ee04df39406ba30b3b3b9e349507 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7071a39d9779491da2d2a12d7faf7ccb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:34.060849 15069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d805ee04df39406ba30b3b3b9e349507. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:34.055608 15200 tablet_service.cc:1467] Processing CreateTablet for tablet 2b757c430aaa42f382d174ca57f5fcfc (DEFAULT_TABLE table=rebalance_test_table_0 [id=7071a39d9779491da2d2a12d7faf7ccb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:34.061228 15200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b757c430aaa42f382d174ca57f5fcfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:34.063661 15941 tablet_bootstrap.cc:492] T d805ee04df39406ba30b3b3b9e349507 P dcfdfa62edc34dab9525e2712d487582: Bootstrap starting.
I20240503 02:00:34.067113 15941 tablet_bootstrap.cc:654] T d805ee04df39406ba30b3b3b9e349507 P dcfdfa62edc34dab9525e2712d487582: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:34.067651 15943 tablet_bootstrap.cc:492] T 2b757c430aaa42f382d174ca57f5fcfc P 620596b893d84291b88b90dc9e317612: Bootstrap starting.
I20240503 02:00:34.068413 15941 log.cc:824] T d805ee04df39406ba30b3b3b9e349507 P dcfdfa62edc34dab9525e2712d487582: Log is configured to *not* fsync() on all Append() calls
I20240503 02:00:34.068738 15942 tablet_bootstrap.cc:492] T d805ee04df39406ba30b3b3b9e349507 P b602f6099cbb4758847e7e9550685372: Bootstrap starting.
I20240503 02:00:34.070698 15942 tablet_bootstrap.cc:654] T d805ee04df39406ba30b3b3b9e349507 P b602f6099cbb4758847e7e9550685372: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:34.070456 15331 tablet_service.cc:1467] Processing CreateTablet for tablet d805ee04df39406ba30b3b3b9e349507 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7071a39d9779491da2d2a12d7faf7ccb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:34.070456 15330 tablet_service.cc:1467] Processing CreateTablet for tablet 6798a9ed0cf441aca7b8ae1b05c001f3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7071a39d9779491da2d2a12d7faf7ccb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:34.071578 15330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6798a9ed0cf441aca7b8ae1b05c001f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:34.071523 15942 log.cc:824] T d805ee04df39406ba30b3b3b9e349507 P b602f6099cbb4758847e7e9550685372: Log is configured to *not* fsync() on all Append() calls
I20240503 02:00:34.072023 15943 tablet_bootstrap.cc:654] T 2b757c430aaa42f382d174ca57f5fcfc P 620596b893d84291b88b90dc9e317612: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:34.072685 15331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d805ee04df39406ba30b3b3b9e349507. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:34.072888 15943 log.cc:824] T 2b757c430aaa42f382d174ca57f5fcfc P 620596b893d84291b88b90dc9e317612: Log is configured to *not* fsync() on all Append() calls
I20240503 02:00:34.073599 15942 tablet_bootstrap.cc:492] T d805ee04df39406ba30b3b3b9e349507 P b602f6099cbb4758847e7e9550685372: No bootstrap required, opened a new log
I20240503 02:00:34.073801 15942 ts_tablet_manager.cc:1367] T d805ee04df39406ba30b3b3b9e349507 P b602f6099cbb4758847e7e9550685372: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 02:00:34.074627 15943 tablet_bootstrap.cc:492] T 2b757c430aaa42f382d174ca57f5fcfc P 620596b893d84291b88b90dc9e317612: No bootstrap required, opened a new log
I20240503 02:00:34.074827 15943 ts_tablet_manager.cc:1367] T 2b757c430aaa42f382d174ca57f5fcfc P 620596b893d84291b88b90dc9e317612: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240503 02:00:34.077817 15942 raft_consensus.cc:348] T d805ee04df39406ba30b3b3b9e349507 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:34.078068 15942 raft_consensus.cc:374] T d805ee04df39406ba30b3b3b9e349507 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:34.078116 15942 raft_consensus.cc:729] T d805ee04df39406ba30b3b3b9e349507 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Initialized, Role: FOLLOWER
I20240503 02:00:34.078814 15943 raft_consensus.cc:348] T 2b757c430aaa42f382d174ca57f5fcfc P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:34.079021 15943 raft_consensus.cc:374] T 2b757c430aaa42f382d174ca57f5fcfc P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:34.079077 15943 raft_consensus.cc:729] T 2b757c430aaa42f382d174ca57f5fcfc P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 620596b893d84291b88b90dc9e317612, State: Initialized, Role: FOLLOWER
I20240503 02:00:34.079430 15942 consensus_queue.cc:260] T d805ee04df39406ba30b3b3b9e349507 P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:34.079723 15943 consensus_queue.cc:260] T 2b757c430aaa42f382d174ca57f5fcfc P 620596b893d84291b88b90dc9e317612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:34.080516 15942 ts_tablet_manager.cc:1398] T d805ee04df39406ba30b3b3b9e349507 P b602f6099cbb4758847e7e9550685372: Time spent starting tablet: real 0.007s user 0.002s sys 0.005s
I20240503 02:00:34.080709 15943 ts_tablet_manager.cc:1398] T 2b757c430aaa42f382d174ca57f5fcfc P 620596b893d84291b88b90dc9e317612: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20240503 02:00:34.080878 15136 tablet.cc:2369] T d805ee04df39406ba30b3b3b9e349507 P b602f6099cbb4758847e7e9550685372: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:00:34.081003 15943 tablet_bootstrap.cc:492] T 6798a9ed0cf441aca7b8ae1b05c001f3 P 620596b893d84291b88b90dc9e317612: Bootstrap starting.
I20240503 02:00:34.082275 15943 tablet_bootstrap.cc:654] T 6798a9ed0cf441aca7b8ae1b05c001f3 P 620596b893d84291b88b90dc9e317612: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:34.082556 15942 tablet_bootstrap.cc:492] T 2b757c430aaa42f382d174ca57f5fcfc P b602f6099cbb4758847e7e9550685372: Bootstrap starting.
I20240503 02:00:34.083691 15942 tablet_bootstrap.cc:654] T 2b757c430aaa42f382d174ca57f5fcfc P b602f6099cbb4758847e7e9550685372: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:34.084089 15943 tablet_bootstrap.cc:492] T 6798a9ed0cf441aca7b8ae1b05c001f3 P 620596b893d84291b88b90dc9e317612: No bootstrap required, opened a new log
I20240503 02:00:34.084266 15943 ts_tablet_manager.cc:1367] T 6798a9ed0cf441aca7b8ae1b05c001f3 P 620596b893d84291b88b90dc9e317612: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:34.085212 15943 raft_consensus.cc:348] T 6798a9ed0cf441aca7b8ae1b05c001f3 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:34.085345 15943 raft_consensus.cc:374] T 6798a9ed0cf441aca7b8ae1b05c001f3 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:34.085386 15943 raft_consensus.cc:729] T 6798a9ed0cf441aca7b8ae1b05c001f3 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 620596b893d84291b88b90dc9e317612, State: Initialized, Role: FOLLOWER
I20240503 02:00:34.085542 15943 consensus_queue.cc:260] T 6798a9ed0cf441aca7b8ae1b05c001f3 P 620596b893d84291b88b90dc9e317612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:34.085853 15943 ts_tablet_manager.cc:1398] T 6798a9ed0cf441aca7b8ae1b05c001f3 P 620596b893d84291b88b90dc9e317612: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:34.087019 15949 raft_consensus.cc:482] T 6798a9ed0cf441aca7b8ae1b05c001f3 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:34.087179 15949 raft_consensus.cc:504] T 6798a9ed0cf441aca7b8ae1b05c001f3 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:34.087836 15942 tablet_bootstrap.cc:492] T 2b757c430aaa42f382d174ca57f5fcfc P b602f6099cbb4758847e7e9550685372: No bootstrap required, opened a new log
I20240503 02:00:34.087940 15942 ts_tablet_manager.cc:1367] T 2b757c430aaa42f382d174ca57f5fcfc P b602f6099cbb4758847e7e9550685372: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:00:34.088575 15942 raft_consensus.cc:348] T 2b757c430aaa42f382d174ca57f5fcfc P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:34.088657 15941 tablet_bootstrap.cc:492] T d805ee04df39406ba30b3b3b9e349507 P dcfdfa62edc34dab9525e2712d487582: No bootstrap required, opened a new log
I20240503 02:00:34.088711 15942 raft_consensus.cc:374] T 2b757c430aaa42f382d174ca57f5fcfc P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:34.088749 15942 raft_consensus.cc:729] T 2b757c430aaa42f382d174ca57f5fcfc P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Initialized, Role: FOLLOWER
I20240503 02:00:34.088869 15941 ts_tablet_manager.cc:1367] T d805ee04df39406ba30b3b3b9e349507 P dcfdfa62edc34dab9525e2712d487582: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.002s
I20240503 02:00:34.088892 15942 consensus_queue.cc:260] T 2b757c430aaa42f382d174ca57f5fcfc P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:34.088979 15949 leader_election.cc:288] T 6798a9ed0cf441aca7b8ae1b05c001f3 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009), dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817)
I20240503 02:00:34.089197 15942 ts_tablet_manager.cc:1398] T 2b757c430aaa42f382d174ca57f5fcfc P b602f6099cbb4758847e7e9550685372: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 02:00:34.092252 15941 raft_consensus.cc:348] T d805ee04df39406ba30b3b3b9e349507 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:34.092541 15941 raft_consensus.cc:374] T d805ee04df39406ba30b3b3b9e349507 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:34.092599 15941 raft_consensus.cc:729] T d805ee04df39406ba30b3b3b9e349507 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Initialized, Role: FOLLOWER
I20240503 02:00:34.093144 15941 consensus_queue.cc:260] T d805ee04df39406ba30b3b3b9e349507 P dcfdfa62edc34dab9525e2712d487582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:34.094178 15941 ts_tablet_manager.cc:1398] T d805ee04df39406ba30b3b3b9e349507 P dcfdfa62edc34dab9525e2712d487582: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240503 02:00:34.093350 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6798a9ed0cf441aca7b8ae1b05c001f3" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c05f7620bb884392931d3243a0b47a9e" is_pre_election: true
I20240503 02:00:34.094589 15947 tablet_bootstrap.cc:492] T d805ee04df39406ba30b3b3b9e349507 P c05f7620bb884392931d3243a0b47a9e: Bootstrap starting.
W20240503 02:00:34.095465 15155 leader_election.cc:341] T 6798a9ed0cf441aca7b8ae1b05c001f3 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:00:34.096555 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6798a9ed0cf441aca7b8ae1b05c001f3" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcfdfa62edc34dab9525e2712d487582" is_pre_election: true
I20240503 02:00:34.096669 15941 tablet_bootstrap.cc:492] T 2b757c430aaa42f382d174ca57f5fcfc P dcfdfa62edc34dab9525e2712d487582: Bootstrap starting.
I20240503 02:00:34.096892 15947 tablet_bootstrap.cc:654] T d805ee04df39406ba30b3b3b9e349507 P c05f7620bb884392931d3243a0b47a9e: Neither blocks nor log segments found. Creating new log.
W20240503 02:00:34.097330 15153 leader_election.cc:341] T 6798a9ed0cf441aca7b8ae1b05c001f3 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:00:34.097419 15153 leader_election.cc:302] T 6798a9ed0cf441aca7b8ae1b05c001f3 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 620596b893d84291b88b90dc9e317612; no voters: c05f7620bb884392931d3243a0b47a9e, dcfdfa62edc34dab9525e2712d487582
I20240503 02:00:34.097787 15949 raft_consensus.cc:2726] T 6798a9ed0cf441aca7b8ae1b05c001f3 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:00:34.098475 15941 tablet_bootstrap.cc:654] T 2b757c430aaa42f382d174ca57f5fcfc P dcfdfa62edc34dab9525e2712d487582: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:34.098598 15947 log.cc:824] T d805ee04df39406ba30b3b3b9e349507 P c05f7620bb884392931d3243a0b47a9e: Log is configured to *not* fsync() on all Append() calls
I20240503 02:00:34.099967 15941 tablet_bootstrap.cc:492] T 2b757c430aaa42f382d174ca57f5fcfc P dcfdfa62edc34dab9525e2712d487582: No bootstrap required, opened a new log
I20240503 02:00:34.100075 15941 ts_tablet_manager.cc:1367] T 2b757c430aaa42f382d174ca57f5fcfc P dcfdfa62edc34dab9525e2712d487582: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:00:34.100718 15947 tablet_bootstrap.cc:492] T d805ee04df39406ba30b3b3b9e349507 P c05f7620bb884392931d3243a0b47a9e: No bootstrap required, opened a new log
I20240503 02:00:34.100858 15941 raft_consensus.cc:348] T 2b757c430aaa42f382d174ca57f5fcfc P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:34.100946 15947 ts_tablet_manager.cc:1367] T d805ee04df39406ba30b3b3b9e349507 P c05f7620bb884392931d3243a0b47a9e: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240503 02:00:34.100984 15941 raft_consensus.cc:374] T 2b757c430aaa42f382d174ca57f5fcfc P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:34.101014 15941 raft_consensus.cc:729] T 2b757c430aaa42f382d174ca57f5fcfc P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Initialized, Role: FOLLOWER
I20240503 02:00:34.101164 15941 consensus_queue.cc:260] T 2b757c430aaa42f382d174ca57f5fcfc P dcfdfa62edc34dab9525e2712d487582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:34.101470 15941 ts_tablet_manager.cc:1398] T 2b757c430aaa42f382d174ca57f5fcfc P dcfdfa62edc34dab9525e2712d487582: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:34.101665 15941 tablet_bootstrap.cc:492] T 6798a9ed0cf441aca7b8ae1b05c001f3 P dcfdfa62edc34dab9525e2712d487582: Bootstrap starting.
I20240503 02:00:34.102931 15941 tablet_bootstrap.cc:654] T 6798a9ed0cf441aca7b8ae1b05c001f3 P dcfdfa62edc34dab9525e2712d487582: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:34.103957 15947 raft_consensus.cc:348] T d805ee04df39406ba30b3b3b9e349507 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:34.104148 15947 raft_consensus.cc:374] T d805ee04df39406ba30b3b3b9e349507 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:34.104194 15947 raft_consensus.cc:729] T d805ee04df39406ba30b3b3b9e349507 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Initialized, Role: FOLLOWER
I20240503 02:00:34.104632 15947 consensus_queue.cc:260] T d805ee04df39406ba30b3b3b9e349507 P c05f7620bb884392931d3243a0b47a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:34.105487 15947 ts_tablet_manager.cc:1398] T d805ee04df39406ba30b3b3b9e349507 P c05f7620bb884392931d3243a0b47a9e: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240503 02:00:34.105847 15947 tablet_bootstrap.cc:492] T 6798a9ed0cf441aca7b8ae1b05c001f3 P c05f7620bb884392931d3243a0b47a9e: Bootstrap starting.
I20240503 02:00:34.106251 15941 tablet_bootstrap.cc:492] T 6798a9ed0cf441aca7b8ae1b05c001f3 P dcfdfa62edc34dab9525e2712d487582: No bootstrap required, opened a new log
I20240503 02:00:34.106325 15941 ts_tablet_manager.cc:1367] T 6798a9ed0cf441aca7b8ae1b05c001f3 P dcfdfa62edc34dab9525e2712d487582: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:00:34.106786 15947 tablet_bootstrap.cc:654] T 6798a9ed0cf441aca7b8ae1b05c001f3 P c05f7620bb884392931d3243a0b47a9e: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:34.106760 15941 raft_consensus.cc:348] T 6798a9ed0cf441aca7b8ae1b05c001f3 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:34.106859 15941 raft_consensus.cc:374] T 6798a9ed0cf441aca7b8ae1b05c001f3 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:34.106904 15941 raft_consensus.cc:729] T 6798a9ed0cf441aca7b8ae1b05c001f3 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Initialized, Role: FOLLOWER
I20240503 02:00:34.107088 15941 consensus_queue.cc:260] T 6798a9ed0cf441aca7b8ae1b05c001f3 P dcfdfa62edc34dab9525e2712d487582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:34.107347 15941 ts_tablet_manager.cc:1398] T 6798a9ed0cf441aca7b8ae1b05c001f3 P dcfdfa62edc34dab9525e2712d487582: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:34.108285 15947 tablet_bootstrap.cc:492] T 6798a9ed0cf441aca7b8ae1b05c001f3 P c05f7620bb884392931d3243a0b47a9e: No bootstrap required, opened a new log
I20240503 02:00:34.108417 15947 ts_tablet_manager.cc:1367] T 6798a9ed0cf441aca7b8ae1b05c001f3 P c05f7620bb884392931d3243a0b47a9e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:34.109037 15947 raft_consensus.cc:348] T 6798a9ed0cf441aca7b8ae1b05c001f3 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:34.109165 15947 raft_consensus.cc:374] T 6798a9ed0cf441aca7b8ae1b05c001f3 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:34.109191 15947 raft_consensus.cc:729] T 6798a9ed0cf441aca7b8ae1b05c001f3 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Initialized, Role: FOLLOWER
I20240503 02:00:34.109318 15947 consensus_queue.cc:260] T 6798a9ed0cf441aca7b8ae1b05c001f3 P c05f7620bb884392931d3243a0b47a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:34.109592 15947 ts_tablet_manager.cc:1398] T 6798a9ed0cf441aca7b8ae1b05c001f3 P c05f7620bb884392931d3243a0b47a9e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20240503 02:00:34.147609 15005 tablet.cc:2369] T 2b757c430aaa42f382d174ca57f5fcfc P dcfdfa62edc34dab9525e2712d487582: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 02:00:34.218475 15398 tablet.cc:2369] T 6798a9ed0cf441aca7b8ae1b05c001f3 P c05f7620bb884392931d3243a0b47a9e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:00:34.243657 15948 raft_consensus.cc:482] T 2b757c430aaa42f382d174ca57f5fcfc P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:34.243857 15948 raft_consensus.cc:504] T 2b757c430aaa42f382d174ca57f5fcfc P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:34.244987 15948 leader_election.cc:288] T 2b757c430aaa42f382d174ca57f5fcfc P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817)
I20240503 02:00:34.249013 15220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b757c430aaa42f382d174ca57f5fcfc" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "620596b893d84291b88b90dc9e317612" is_pre_election: true
I20240503 02:00:34.249116 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b757c430aaa42f382d174ca57f5fcfc" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcfdfa62edc34dab9525e2712d487582" is_pre_election: true
I20240503 02:00:34.249279 15220 raft_consensus.cc:2454] T 2b757c430aaa42f382d174ca57f5fcfc P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 0.
I20240503 02:00:34.249478 14957 raft_consensus.cc:2454] T 2b757c430aaa42f382d174ca57f5fcfc P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 0.
I20240503 02:00:34.249723 15024 leader_election.cc:302] T 2b757c430aaa42f382d174ca57f5fcfc P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, b602f6099cbb4758847e7e9550685372; no voters:
I20240503 02:00:34.249992 15948 raft_consensus.cc:2781] T 2b757c430aaa42f382d174ca57f5fcfc P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:34.250097 15948 raft_consensus.cc:482] T 2b757c430aaa42f382d174ca57f5fcfc P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:34.250144 15948 raft_consensus.cc:3037] T 2b757c430aaa42f382d174ca57f5fcfc P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:34.251096 15948 raft_consensus.cc:504] T 2b757c430aaa42f382d174ca57f5fcfc P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:34.251482 15948 leader_election.cc:288] T 2b757c430aaa42f382d174ca57f5fcfc P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 1 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817)
I20240503 02:00:34.251860 15220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b757c430aaa42f382d174ca57f5fcfc" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:34.251888 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b757c430aaa42f382d174ca57f5fcfc" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:34.251996 15220 raft_consensus.cc:3037] T 2b757c430aaa42f382d174ca57f5fcfc P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:34.251996 14957 raft_consensus.cc:3037] T 2b757c430aaa42f382d174ca57f5fcfc P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:34.252964 14957 raft_consensus.cc:2454] T 2b757c430aaa42f382d174ca57f5fcfc P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 1.
I20240503 02:00:34.252969 15220 raft_consensus.cc:2454] T 2b757c430aaa42f382d174ca57f5fcfc P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 1.
I20240503 02:00:34.253336 15024 leader_election.cc:302] T 2b757c430aaa42f382d174ca57f5fcfc P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, b602f6099cbb4758847e7e9550685372; no voters:
I20240503 02:00:34.253525 15948 raft_consensus.cc:2781] T 2b757c430aaa42f382d174ca57f5fcfc P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:34.253723 15948 raft_consensus.cc:686] T 2b757c430aaa42f382d174ca57f5fcfc P b602f6099cbb4758847e7e9550685372 [term 1 LEADER]: Becoming Leader. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:34.254062 15948 consensus_queue.cc:237] T 2b757c430aaa42f382d174ca57f5fcfc P b602f6099cbb4758847e7e9550685372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:34.257202 14816 catalog_manager.cc:5484] T 2b757c430aaa42f382d174ca57f5fcfc P b602f6099cbb4758847e7e9550685372 reported cstate change: term changed from 0 to 1, leader changed from <none> to b602f6099cbb4758847e7e9550685372 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "b602f6099cbb4758847e7e9550685372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: HEALTHY } } }
W20240503 02:00:34.282159 15267 tablet.cc:2369] T 2b757c430aaa42f382d174ca57f5fcfc P 620596b893d84291b88b90dc9e317612: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:00:34.296574 15958 raft_consensus.cc:482] T d805ee04df39406ba30b3b3b9e349507 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:34.296795 15958 raft_consensus.cc:504] T d805ee04df39406ba30b3b3b9e349507 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:34.297861 15958 leader_election.cc:288] T d805ee04df39406ba30b3b3b9e349507 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
I20240503 02:00:34.301781 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d805ee04df39406ba30b3b3b9e349507" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcfdfa62edc34dab9525e2712d487582" is_pre_election: true
I20240503 02:00:34.301853 15089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d805ee04df39406ba30b3b3b9e349507" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b602f6099cbb4758847e7e9550685372" is_pre_election: true
I20240503 02:00:34.301977 14957 raft_consensus.cc:2454] T d805ee04df39406ba30b3b3b9e349507 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 0.
I20240503 02:00:34.302045 15089 raft_consensus.cc:2454] T d805ee04df39406ba30b3b3b9e349507 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 0.
I20240503 02:00:34.302366 15284 leader_election.cc:302] T d805ee04df39406ba30b3b3b9e349507 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c05f7620bb884392931d3243a0b47a9e, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:34.302613 15958 raft_consensus.cc:2781] T d805ee04df39406ba30b3b3b9e349507 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:34.302685 15958 raft_consensus.cc:482] T d805ee04df39406ba30b3b3b9e349507 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:34.302714 15958 raft_consensus.cc:3037] T d805ee04df39406ba30b3b3b9e349507 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:34.303817 15958 raft_consensus.cc:504] T d805ee04df39406ba30b3b3b9e349507 P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:34.304185 15958 leader_election.cc:288] T d805ee04df39406ba30b3b3b9e349507 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 1 election: Requested vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
I20240503 02:00:34.304575 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d805ee04df39406ba30b3b3b9e349507" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:34.304690 14957 raft_consensus.cc:3037] T d805ee04df39406ba30b3b3b9e349507 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:34.305439 14957 raft_consensus.cc:2454] T d805ee04df39406ba30b3b3b9e349507 P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 1.
I20240503 02:00:34.305706 15089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d805ee04df39406ba30b3b3b9e349507" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:34.305824 15089 raft_consensus.cc:3037] T d805ee04df39406ba30b3b3b9e349507 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:34.305796 15284 leader_election.cc:302] T d805ee04df39406ba30b3b3b9e349507 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c05f7620bb884392931d3243a0b47a9e, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:34.306003 15958 raft_consensus.cc:2781] T d805ee04df39406ba30b3b3b9e349507 P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:34.306262 15958 raft_consensus.cc:686] T d805ee04df39406ba30b3b3b9e349507 P c05f7620bb884392931d3243a0b47a9e [term 1 LEADER]: Becoming Leader. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:34.306655 15958 consensus_queue.cc:237] T d805ee04df39406ba30b3b3b9e349507 P c05f7620bb884392931d3243a0b47a9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:34.307291 15089 raft_consensus.cc:2454] T d805ee04df39406ba30b3b3b9e349507 P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 1.
I20240503 02:00:34.309967 14816 catalog_manager.cc:5484] T d805ee04df39406ba30b3b3b9e349507 P c05f7620bb884392931d3243a0b47a9e reported cstate change: term changed from 0 to 1, leader changed from <none> to c05f7620bb884392931d3243a0b47a9e (127.0.99.132). New cstate: current_term: 1 leader_uuid: "c05f7620bb884392931d3243a0b47a9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:34.485459 15954 raft_consensus.cc:482] T 6798a9ed0cf441aca7b8ae1b05c001f3 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:34.485612 15954 raft_consensus.cc:504] T 6798a9ed0cf441aca7b8ae1b05c001f3 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:34.486714 15954 leader_election.cc:288] T 6798a9ed0cf441aca7b8ae1b05c001f3 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:34.490489 15220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6798a9ed0cf441aca7b8ae1b05c001f3" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "620596b893d84291b88b90dc9e317612" is_pre_election: true
I20240503 02:00:34.490684 15220 raft_consensus.cc:2454] T 6798a9ed0cf441aca7b8ae1b05c001f3 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 0.
I20240503 02:00:34.490824 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6798a9ed0cf441aca7b8ae1b05c001f3" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c05f7620bb884392931d3243a0b47a9e" is_pre_election: true
I20240503 02:00:34.490991 15351 raft_consensus.cc:2454] T 6798a9ed0cf441aca7b8ae1b05c001f3 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 0.
I20240503 02:00:34.491070 14893 leader_election.cc:302] T 6798a9ed0cf441aca7b8ae1b05c001f3 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:34.491333 15954 raft_consensus.cc:2781] T 6798a9ed0cf441aca7b8ae1b05c001f3 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:34.491397 15954 raft_consensus.cc:482] T 6798a9ed0cf441aca7b8ae1b05c001f3 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:34.491425 15954 raft_consensus.cc:3037] T 6798a9ed0cf441aca7b8ae1b05c001f3 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:34.492247 15954 raft_consensus.cc:504] T 6798a9ed0cf441aca7b8ae1b05c001f3 P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:34.492653 15954 leader_election.cc:288] T 6798a9ed0cf441aca7b8ae1b05c001f3 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 1 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:34.493068 15220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6798a9ed0cf441aca7b8ae1b05c001f3" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:34.493108 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6798a9ed0cf441aca7b8ae1b05c001f3" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:34.493211 15220 raft_consensus.cc:3037] T 6798a9ed0cf441aca7b8ae1b05c001f3 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:34.493211 15351 raft_consensus.cc:3037] T 6798a9ed0cf441aca7b8ae1b05c001f3 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:34.494191 15220 raft_consensus.cc:2454] T 6798a9ed0cf441aca7b8ae1b05c001f3 P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 1.
I20240503 02:00:34.494182 15351 raft_consensus.cc:2454] T 6798a9ed0cf441aca7b8ae1b05c001f3 P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 1.
I20240503 02:00:34.494534 14893 leader_election.cc:302] T 6798a9ed0cf441aca7b8ae1b05c001f3 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c05f7620bb884392931d3243a0b47a9e, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:34.494702 15954 raft_consensus.cc:2781] T 6798a9ed0cf441aca7b8ae1b05c001f3 P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:34.494920 15954 raft_consensus.cc:686] T 6798a9ed0cf441aca7b8ae1b05c001f3 P dcfdfa62edc34dab9525e2712d487582 [term 1 LEADER]: Becoming Leader. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:34.495299 15954 consensus_queue.cc:237] T 6798a9ed0cf441aca7b8ae1b05c001f3 P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:34.498525 14816 catalog_manager.cc:5484] T 6798a9ed0cf441aca7b8ae1b05c001f3 P dcfdfa62edc34dab9525e2712d487582 reported cstate change: term changed from 0 to 1, leader changed from <none> to dcfdfa62edc34dab9525e2712d487582 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "dcfdfa62edc34dab9525e2712d487582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: HEALTHY } } }
W20240503 02:00:34.726190 15972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:34.726482 15972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:34.729699 15972 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240503 02:00:34.779363 15958 consensus_queue.cc:1035] T d805ee04df39406ba30b3b3b9e349507 P c05f7620bb884392931d3243a0b47a9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:34.795460 15995 consensus_queue.cc:1035] T 2b757c430aaa42f382d174ca57f5fcfc P b602f6099cbb4758847e7e9550685372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:34.801357 15966 consensus_queue.cc:1035] T d805ee04df39406ba30b3b3b9e349507 P c05f7620bb884392931d3243a0b47a9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:34.806830 15995 consensus_queue.cc:1035] T 2b757c430aaa42f382d174ca57f5fcfc P b602f6099cbb4758847e7e9550685372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:34.851927 15220 raft_consensus.cc:1261] T 6798a9ed0cf441aca7b8ae1b05c001f3 P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Refusing update from remote peer dcfdfa62edc34dab9525e2712d487582: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:34.852121 15351 raft_consensus.cc:1261] T 6798a9ed0cf441aca7b8ae1b05c001f3 P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Refusing update from remote peer dcfdfa62edc34dab9525e2712d487582: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:34.852674 15970 consensus_queue.cc:1035] T 6798a9ed0cf441aca7b8ae1b05c001f3 P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Connected to new peer: Peer: permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:34.855082 15970 consensus_queue.cc:1035] T 6798a9ed0cf441aca7b8ae1b05c001f3 P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:34.863379 16016 mvcc.cc:205] Tried to move back new op lower bound from 7023417896334700544 to 7023417894895775744. Current Snapshot: MvccSnapshot[applied={T|T < 7023417896334700544}]
I20240503 02:00:34.864212 15998 mvcc.cc:205] Tried to move back new op lower bound from 7023417896334700544 to 7023417894895775744. Current Snapshot: MvccSnapshot[applied={T|T < 7023417896334700544}]
I20240503 02:00:34.876631 16007 mvcc.cc:205] Tried to move back new op lower bound from 7023417896334700544 to 7023417894895775744. Current Snapshot: MvccSnapshot[applied={T|T < 7023417896334700544 or (T in {7023417896334700544})}]
INSERT report
rows total: 2000
time total: 134.844 ms
time per row: 0.0674222 ms
I20240503 02:00:34.916172 14815 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39948:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:34.922509 15331 tablet_service.cc:1467] Processing CreateTablet for tablet eade118a36da42e3ac08b96a13893a5c (DEFAULT_TABLE table=rebalance_test_table_1 [id=7d193fa391c34924a2fe2072e3e16507]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:34.922514 15068 tablet_service.cc:1467] Processing CreateTablet for tablet eade118a36da42e3ac08b96a13893a5c (DEFAULT_TABLE table=rebalance_test_table_1 [id=7d193fa391c34924a2fe2072e3e16507]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:34.922757 15330 tablet_service.cc:1467] Processing CreateTablet for tablet c74522fbb5c64d27b5c95f05896b0716 (DEFAULT_TABLE table=rebalance_test_table_1 [id=7d193fa391c34924a2fe2072e3e16507]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:34.922828 15068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eade118a36da42e3ac08b96a13893a5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:34.922844 15331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eade118a36da42e3ac08b96a13893a5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:34.923010 15330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c74522fbb5c64d27b5c95f05896b0716. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:34.923817 14937 tablet_service.cc:1467] Processing CreateTablet for tablet f85a361ce0254472b37526362d4edf0e (DEFAULT_TABLE table=rebalance_test_table_1 [id=7d193fa391c34924a2fe2072e3e16507]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:34.924029 15067 tablet_service.cc:1467] Processing CreateTablet for tablet f85a361ce0254472b37526362d4edf0e (DEFAULT_TABLE table=rebalance_test_table_1 [id=7d193fa391c34924a2fe2072e3e16507]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:34.924103 14937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f85a361ce0254472b37526362d4edf0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:34.924291 15067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f85a361ce0254472b37526362d4edf0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:34.925005 15199 tablet_service.cc:1467] Processing CreateTablet for tablet eade118a36da42e3ac08b96a13893a5c (DEFAULT_TABLE table=rebalance_test_table_1 [id=7d193fa391c34924a2fe2072e3e16507]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:34.925304 15199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eade118a36da42e3ac08b96a13893a5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:34.925402 15069 tablet_service.cc:1467] Processing CreateTablet for tablet c74522fbb5c64d27b5c95f05896b0716 (DEFAULT_TABLE table=rebalance_test_table_1 [id=7d193fa391c34924a2fe2072e3e16507]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:34.925631 15069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c74522fbb5c64d27b5c95f05896b0716. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:34.927561 16032 tablet_bootstrap.cc:492] T eade118a36da42e3ac08b96a13893a5c P 620596b893d84291b88b90dc9e317612: Bootstrap starting.
I20240503 02:00:34.927690 16030 tablet_bootstrap.cc:492] T f85a361ce0254472b37526362d4edf0e P dcfdfa62edc34dab9525e2712d487582: Bootstrap starting.
I20240503 02:00:34.927752 15200 tablet_service.cc:1467] Processing CreateTablet for tablet c74522fbb5c64d27b5c95f05896b0716 (DEFAULT_TABLE table=rebalance_test_table_1 [id=7d193fa391c34924a2fe2072e3e16507]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:34.928018 15200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c74522fbb5c64d27b5c95f05896b0716. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:34.928750 16030 tablet_bootstrap.cc:654] T f85a361ce0254472b37526362d4edf0e P dcfdfa62edc34dab9525e2712d487582: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:34.929051 16032 tablet_bootstrap.cc:654] T eade118a36da42e3ac08b96a13893a5c P 620596b893d84291b88b90dc9e317612: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:34.929729 16031 tablet_bootstrap.cc:492] T f85a361ce0254472b37526362d4edf0e P b602f6099cbb4758847e7e9550685372: Bootstrap starting.
I20240503 02:00:34.930318 16032 tablet_bootstrap.cc:492] T eade118a36da42e3ac08b96a13893a5c P 620596b893d84291b88b90dc9e317612: No bootstrap required, opened a new log
I20240503 02:00:34.930413 16032 ts_tablet_manager.cc:1367] T eade118a36da42e3ac08b96a13893a5c P 620596b893d84291b88b90dc9e317612: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:34.930797 16032 raft_consensus.cc:348] T eade118a36da42e3ac08b96a13893a5c P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:34.930866 16031 tablet_bootstrap.cc:654] T f85a361ce0254472b37526362d4edf0e P b602f6099cbb4758847e7e9550685372: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:34.930891 16032 raft_consensus.cc:374] T eade118a36da42e3ac08b96a13893a5c P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:34.930920 16032 raft_consensus.cc:729] T eade118a36da42e3ac08b96a13893a5c P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 620596b893d84291b88b90dc9e317612, State: Initialized, Role: FOLLOWER
I20240503 02:00:34.931051 16032 consensus_queue.cc:260] T eade118a36da42e3ac08b96a13893a5c P 620596b893d84291b88b90dc9e317612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:34.931334 16032 ts_tablet_manager.cc:1398] T eade118a36da42e3ac08b96a13893a5c P 620596b893d84291b88b90dc9e317612: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:34.931490 16032 tablet_bootstrap.cc:492] T c74522fbb5c64d27b5c95f05896b0716 P 620596b893d84291b88b90dc9e317612: Bootstrap starting.
I20240503 02:00:34.931452 15200 tablet_service.cc:1467] Processing CreateTablet for tablet f85a361ce0254472b37526362d4edf0e (DEFAULT_TABLE table=rebalance_test_table_1 [id=7d193fa391c34924a2fe2072e3e16507]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:34.931723 15200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f85a361ce0254472b37526362d4edf0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:34.932320 16032 tablet_bootstrap.cc:654] T c74522fbb5c64d27b5c95f05896b0716 P 620596b893d84291b88b90dc9e317612: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:34.933326 16032 tablet_bootstrap.cc:492] T c74522fbb5c64d27b5c95f05896b0716 P 620596b893d84291b88b90dc9e317612: No bootstrap required, opened a new log
I20240503 02:00:34.933405 16032 ts_tablet_manager.cc:1367] T c74522fbb5c64d27b5c95f05896b0716 P 620596b893d84291b88b90dc9e317612: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:34.933805 16032 raft_consensus.cc:348] T c74522fbb5c64d27b5c95f05896b0716 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:34.933909 16032 raft_consensus.cc:374] T c74522fbb5c64d27b5c95f05896b0716 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:34.933940 16032 raft_consensus.cc:729] T c74522fbb5c64d27b5c95f05896b0716 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 620596b893d84291b88b90dc9e317612, State: Initialized, Role: FOLLOWER
I20240503 02:00:34.934038 16029 tablet_bootstrap.cc:492] T c74522fbb5c64d27b5c95f05896b0716 P c05f7620bb884392931d3243a0b47a9e: Bootstrap starting.
I20240503 02:00:34.934092 16032 consensus_queue.cc:260] T c74522fbb5c64d27b5c95f05896b0716 P 620596b893d84291b88b90dc9e317612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:34.934373 16032 ts_tablet_manager.cc:1398] T c74522fbb5c64d27b5c95f05896b0716 P 620596b893d84291b88b90dc9e317612: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:34.934451 16030 tablet_bootstrap.cc:492] T f85a361ce0254472b37526362d4edf0e P dcfdfa62edc34dab9525e2712d487582: No bootstrap required, opened a new log
I20240503 02:00:34.934522 16032 tablet_bootstrap.cc:492] T f85a361ce0254472b37526362d4edf0e P 620596b893d84291b88b90dc9e317612: Bootstrap starting.
I20240503 02:00:34.934548 16030 ts_tablet_manager.cc:1367] T f85a361ce0254472b37526362d4edf0e P dcfdfa62edc34dab9525e2712d487582: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:00:34.935029 16030 raft_consensus.cc:348] T f85a361ce0254472b37526362d4edf0e P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:34.935132 16030 raft_consensus.cc:374] T f85a361ce0254472b37526362d4edf0e P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:34.935161 16030 raft_consensus.cc:729] T f85a361ce0254472b37526362d4edf0e P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Initialized, Role: FOLLOWER
I20240503 02:00:34.935158 16029 tablet_bootstrap.cc:654] T c74522fbb5c64d27b5c95f05896b0716 P c05f7620bb884392931d3243a0b47a9e: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:34.935338 16032 tablet_bootstrap.cc:654] T f85a361ce0254472b37526362d4edf0e P 620596b893d84291b88b90dc9e317612: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:34.935287 16030 consensus_queue.cc:260] T f85a361ce0254472b37526362d4edf0e P dcfdfa62edc34dab9525e2712d487582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:34.935454 16031 tablet_bootstrap.cc:492] T f85a361ce0254472b37526362d4edf0e P b602f6099cbb4758847e7e9550685372: No bootstrap required, opened a new log
I20240503 02:00:34.935551 16031 ts_tablet_manager.cc:1367] T f85a361ce0254472b37526362d4edf0e P b602f6099cbb4758847e7e9550685372: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:00:34.935566 16030 ts_tablet_manager.cc:1398] T f85a361ce0254472b37526362d4edf0e P dcfdfa62edc34dab9525e2712d487582: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:34.936057 16031 raft_consensus.cc:348] T f85a361ce0254472b37526362d4edf0e P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:34.936169 16031 raft_consensus.cc:374] T f85a361ce0254472b37526362d4edf0e P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:34.936204 16031 raft_consensus.cc:729] T f85a361ce0254472b37526362d4edf0e P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Initialized, Role: FOLLOWER
I20240503 02:00:34.936414 16029 tablet_bootstrap.cc:492] T c74522fbb5c64d27b5c95f05896b0716 P c05f7620bb884392931d3243a0b47a9e: No bootstrap required, opened a new log
I20240503 02:00:34.936506 16029 ts_tablet_manager.cc:1367] T c74522fbb5c64d27b5c95f05896b0716 P c05f7620bb884392931d3243a0b47a9e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:34.936892 16029 raft_consensus.cc:348] T c74522fbb5c64d27b5c95f05896b0716 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:34.937027 16029 raft_consensus.cc:374] T c74522fbb5c64d27b5c95f05896b0716 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:34.937070 16029 raft_consensus.cc:729] T c74522fbb5c64d27b5c95f05896b0716 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Initialized, Role: FOLLOWER
I20240503 02:00:34.937201 16029 consensus_queue.cc:260] T c74522fbb5c64d27b5c95f05896b0716 P c05f7620bb884392931d3243a0b47a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:34.937412 16032 tablet_bootstrap.cc:492] T f85a361ce0254472b37526362d4edf0e P 620596b893d84291b88b90dc9e317612: No bootstrap required, opened a new log
I20240503 02:00:34.937463 16029 ts_tablet_manager.cc:1398] T c74522fbb5c64d27b5c95f05896b0716 P c05f7620bb884392931d3243a0b47a9e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:34.937542 16032 ts_tablet_manager.cc:1367] T f85a361ce0254472b37526362d4edf0e P 620596b893d84291b88b90dc9e317612: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:34.937626 16029 tablet_bootstrap.cc:492] T eade118a36da42e3ac08b96a13893a5c P c05f7620bb884392931d3243a0b47a9e: Bootstrap starting.
I20240503 02:00:34.938078 16032 raft_consensus.cc:348] T f85a361ce0254472b37526362d4edf0e P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:34.936354 16031 consensus_queue.cc:260] T f85a361ce0254472b37526362d4edf0e P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:34.938218 16032 raft_consensus.cc:374] T f85a361ce0254472b37526362d4edf0e P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:34.938285 16032 raft_consensus.cc:729] T f85a361ce0254472b37526362d4edf0e P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 620596b893d84291b88b90dc9e317612, State: Initialized, Role: FOLLOWER
I20240503 02:00:34.938452 16031 ts_tablet_manager.cc:1398] T f85a361ce0254472b37526362d4edf0e P b602f6099cbb4758847e7e9550685372: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:00:34.938428 16032 consensus_queue.cc:260] T f85a361ce0254472b37526362d4edf0e P 620596b893d84291b88b90dc9e317612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:34.938578 16029 tablet_bootstrap.cc:654] T eade118a36da42e3ac08b96a13893a5c P c05f7620bb884392931d3243a0b47a9e: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:34.938657 16031 tablet_bootstrap.cc:492] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372: Bootstrap starting.
I20240503 02:00:34.938714 16032 ts_tablet_manager.cc:1398] T f85a361ce0254472b37526362d4edf0e P 620596b893d84291b88b90dc9e317612: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:34.939663 16031 tablet_bootstrap.cc:654] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:34.940423 16029 tablet_bootstrap.cc:492] T eade118a36da42e3ac08b96a13893a5c P c05f7620bb884392931d3243a0b47a9e: No bootstrap required, opened a new log
I20240503 02:00:34.940851 16029 ts_tablet_manager.cc:1367] T eade118a36da42e3ac08b96a13893a5c P c05f7620bb884392931d3243a0b47a9e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:34.940860 16031 tablet_bootstrap.cc:492] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372: No bootstrap required, opened a new log
I20240503 02:00:34.940941 16031 ts_tablet_manager.cc:1367] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:34.941388 16029 raft_consensus.cc:348] T eade118a36da42e3ac08b96a13893a5c P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:34.941376 16031 raft_consensus.cc:348] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:34.941496 16029 raft_consensus.cc:374] T eade118a36da42e3ac08b96a13893a5c P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:34.941496 16031 raft_consensus.cc:374] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:34.941538 16029 raft_consensus.cc:729] T eade118a36da42e3ac08b96a13893a5c P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Initialized, Role: FOLLOWER
I20240503 02:00:34.941538 16031 raft_consensus.cc:729] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Initialized, Role: FOLLOWER
I20240503 02:00:34.941673 16029 consensus_queue.cc:260] T eade118a36da42e3ac08b96a13893a5c P c05f7620bb884392931d3243a0b47a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:34.941673 16031 consensus_queue.cc:260] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:34.941972 16029 ts_tablet_manager.cc:1398] T eade118a36da42e3ac08b96a13893a5c P c05f7620bb884392931d3243a0b47a9e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:34.941983 16031 ts_tablet_manager.cc:1398] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:34.942162 16031 tablet_bootstrap.cc:492] T eade118a36da42e3ac08b96a13893a5c P b602f6099cbb4758847e7e9550685372: Bootstrap starting.
I20240503 02:00:34.943539 16031 tablet_bootstrap.cc:654] T eade118a36da42e3ac08b96a13893a5c P b602f6099cbb4758847e7e9550685372: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:34.944859 16031 tablet_bootstrap.cc:492] T eade118a36da42e3ac08b96a13893a5c P b602f6099cbb4758847e7e9550685372: No bootstrap required, opened a new log
I20240503 02:00:34.944967 16031 ts_tablet_manager.cc:1367] T eade118a36da42e3ac08b96a13893a5c P b602f6099cbb4758847e7e9550685372: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:34.945395 16031 raft_consensus.cc:348] T eade118a36da42e3ac08b96a13893a5c P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:34.945501 16031 raft_consensus.cc:374] T eade118a36da42e3ac08b96a13893a5c P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:34.945537 16031 raft_consensus.cc:729] T eade118a36da42e3ac08b96a13893a5c P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Initialized, Role: FOLLOWER
I20240503 02:00:34.945763 16031 consensus_queue.cc:260] T eade118a36da42e3ac08b96a13893a5c P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:34.946074 16031 ts_tablet_manager.cc:1398] T eade118a36da42e3ac08b96a13893a5c P b602f6099cbb4758847e7e9550685372: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:35.091363 15966 raft_consensus.cc:482] T eade118a36da42e3ac08b96a13893a5c P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:35.091501 15966 raft_consensus.cc:504] T eade118a36da42e3ac08b96a13893a5c P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:35.092027 15966 leader_election.cc:288] T eade118a36da42e3ac08b96a13893a5c P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207), 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437)
I20240503 02:00:35.092461 15089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eade118a36da42e3ac08b96a13893a5c" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b602f6099cbb4758847e7e9550685372" is_pre_election: true
I20240503 02:00:35.092612 15089 raft_consensus.cc:2454] T eade118a36da42e3ac08b96a13893a5c P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 0.
I20240503 02:00:35.092958 15285 leader_election.cc:302] T eade118a36da42e3ac08b96a13893a5c P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b602f6099cbb4758847e7e9550685372, c05f7620bb884392931d3243a0b47a9e; no voters:
I20240503 02:00:35.093153 15966 raft_consensus.cc:2781] T eade118a36da42e3ac08b96a13893a5c P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:35.093210 15966 raft_consensus.cc:482] T eade118a36da42e3ac08b96a13893a5c P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:35.093237 15966 raft_consensus.cc:3037] T eade118a36da42e3ac08b96a13893a5c P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:35.094637 15966 raft_consensus.cc:504] T eade118a36da42e3ac08b96a13893a5c P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:35.095013 15966 leader_election.cc:288] T eade118a36da42e3ac08b96a13893a5c P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 1 election: Requested vote from peers b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207), 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437)
I20240503 02:00:35.095532 15089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eade118a36da42e3ac08b96a13893a5c" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:35.095664 15089 raft_consensus.cc:3037] T eade118a36da42e3ac08b96a13893a5c P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:35.096463 15220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eade118a36da42e3ac08b96a13893a5c" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "620596b893d84291b88b90dc9e317612" is_pre_election: true
I20240503 02:00:35.096604 15220 raft_consensus.cc:2454] T eade118a36da42e3ac08b96a13893a5c P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 0.
I20240503 02:00:35.096647 15089 raft_consensus.cc:2454] T eade118a36da42e3ac08b96a13893a5c P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 1.
I20240503 02:00:35.096767 15219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eade118a36da42e3ac08b96a13893a5c" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:35.096894 15219 raft_consensus.cc:3037] T eade118a36da42e3ac08b96a13893a5c P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:35.097018 15285 leader_election.cc:302] T eade118a36da42e3ac08b96a13893a5c P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b602f6099cbb4758847e7e9550685372, c05f7620bb884392931d3243a0b47a9e; no voters:
I20240503 02:00:35.097209 15966 raft_consensus.cc:2781] T eade118a36da42e3ac08b96a13893a5c P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:35.097299 15966 raft_consensus.cc:686] T eade118a36da42e3ac08b96a13893a5c P c05f7620bb884392931d3243a0b47a9e [term 1 LEADER]: Becoming Leader. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:35.097445 15966 consensus_queue.cc:237] T eade118a36da42e3ac08b96a13893a5c P c05f7620bb884392931d3243a0b47a9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:35.097813 15219 raft_consensus.cc:2454] T eade118a36da42e3ac08b96a13893a5c P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 1.
I20240503 02:00:35.099325 14815 catalog_manager.cc:5484] T eade118a36da42e3ac08b96a13893a5c P c05f7620bb884392931d3243a0b47a9e reported cstate change: term changed from 0 to 1, leader changed from <none> to c05f7620bb884392931d3243a0b47a9e (127.0.99.132). New cstate: current_term: 1 leader_uuid: "c05f7620bb884392931d3243a0b47a9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:35.104651 16028 raft_consensus.cc:482] T f85a361ce0254472b37526362d4edf0e P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:35.104809 16028 raft_consensus.cc:504] T f85a361ce0254472b37526362d4edf0e P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:35.105345 16028 leader_election.cc:288] T f85a361ce0254472b37526362d4edf0e P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207), 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437)
I20240503 02:00:35.105823 15219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85a361ce0254472b37526362d4edf0e" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "620596b893d84291b88b90dc9e317612" is_pre_election: true
I20240503 02:00:35.105968 15219 raft_consensus.cc:2454] T f85a361ce0254472b37526362d4edf0e P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 0.
I20240503 02:00:35.106261 14893 leader_election.cc:302] T f85a361ce0254472b37526362d4edf0e P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:35.106436 16028 raft_consensus.cc:2781] T f85a361ce0254472b37526362d4edf0e P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:35.106494 16028 raft_consensus.cc:482] T f85a361ce0254472b37526362d4edf0e P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:35.106530 16028 raft_consensus.cc:3037] T f85a361ce0254472b37526362d4edf0e P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:35.107334 16028 raft_consensus.cc:504] T f85a361ce0254472b37526362d4edf0e P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:35.107750 16028 leader_election.cc:288] T f85a361ce0254472b37526362d4edf0e P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 1 election: Requested vote from peers b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207), 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437)
I20240503 02:00:35.108144 15219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85a361ce0254472b37526362d4edf0e" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:35.108259 15219 raft_consensus.cc:3037] T f85a361ce0254472b37526362d4edf0e P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:35.109148 15219 raft_consensus.cc:2454] T f85a361ce0254472b37526362d4edf0e P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 1.
I20240503 02:00:35.109321 15089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85a361ce0254472b37526362d4edf0e" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b602f6099cbb4758847e7e9550685372" is_pre_election: true
I20240503 02:00:35.109350 15088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85a361ce0254472b37526362d4edf0e" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:35.109478 15089 raft_consensus.cc:2454] T f85a361ce0254472b37526362d4edf0e P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 0.
I20240503 02:00:35.109571 14893 leader_election.cc:302] T f85a361ce0254472b37526362d4edf0e P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:35.109797 16028 raft_consensus.cc:2781] T f85a361ce0254472b37526362d4edf0e P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:35.109901 16028 raft_consensus.cc:686] T f85a361ce0254472b37526362d4edf0e P dcfdfa62edc34dab9525e2712d487582 [term 1 LEADER]: Becoming Leader. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:35.110054 16028 consensus_queue.cc:237] T f85a361ce0254472b37526362d4edf0e P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:35.111588 14815 catalog_manager.cc:5484] T f85a361ce0254472b37526362d4edf0e P dcfdfa62edc34dab9525e2712d487582 reported cstate change: term changed from 0 to 1, leader changed from <none> to dcfdfa62edc34dab9525e2712d487582 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "dcfdfa62edc34dab9525e2712d487582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:35.309464 16000 raft_consensus.cc:482] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:35.309619 16000 raft_consensus.cc:504] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:35.310142 16000 leader_election.cc:288] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:35.310655 15219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c74522fbb5c64d27b5c95f05896b0716" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "620596b893d84291b88b90dc9e317612" is_pre_election: true
I20240503 02:00:35.310824 15219 raft_consensus.cc:2454] T c74522fbb5c64d27b5c95f05896b0716 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 0.
I20240503 02:00:35.311246 15024 leader_election.cc:302] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, b602f6099cbb4758847e7e9550685372; no voters:
I20240503 02:00:35.311420 16000 raft_consensus.cc:2781] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:35.311481 16000 raft_consensus.cc:482] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:35.311517 16000 raft_consensus.cc:3037] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:35.312403 16000 raft_consensus.cc:504] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:35.312752 16000 leader_election.cc:288] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 1 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:35.313153 15219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c74522fbb5c64d27b5c95f05896b0716" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:35.313284 15219 raft_consensus.cc:3037] T c74522fbb5c64d27b5c95f05896b0716 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:35.314157 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c74522fbb5c64d27b5c95f05896b0716" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c05f7620bb884392931d3243a0b47a9e" is_pre_election: true
I20240503 02:00:35.314230 15350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c74522fbb5c64d27b5c95f05896b0716" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:35.314293 15351 raft_consensus.cc:2454] T c74522fbb5c64d27b5c95f05896b0716 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 0.
I20240503 02:00:35.314611 15219 raft_consensus.cc:2454] T c74522fbb5c64d27b5c95f05896b0716 P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 1.
I20240503 02:00:35.314872 15024 leader_election.cc:302] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 620596b893d84291b88b90dc9e317612, b602f6099cbb4758847e7e9550685372; no voters: c05f7620bb884392931d3243a0b47a9e
I20240503 02:00:35.315052 16000 raft_consensus.cc:2781] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:35.315141 16000 raft_consensus.cc:686] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372 [term 1 LEADER]: Becoming Leader. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:35.315275 16000 consensus_queue.cc:237] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:35.317106 14815 catalog_manager.cc:5484] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372 reported cstate change: term changed from 0 to 1, leader changed from <none> to b602f6099cbb4758847e7e9550685372 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "b602f6099cbb4758847e7e9550685372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:35.474735 15966 consensus_queue.cc:1035] T eade118a36da42e3ac08b96a13893a5c P c05f7620bb884392931d3243a0b47a9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:35.478382 15958 consensus_queue.cc:1035] T eade118a36da42e3ac08b96a13893a5c P c05f7620bb884392931d3243a0b47a9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:00:35.500654 16049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:35.500985 16049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:35.504176 16049 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240503 02:00:35.566547 16028 consensus_queue.cc:1035] T f85a361ce0254472b37526362d4edf0e P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Connected to new peer: Peer: permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:35.570499 15089 raft_consensus.cc:3037] T f85a361ce0254472b37526362d4edf0e P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:35.572060 16023 consensus_queue.cc:1035] T f85a361ce0254472b37526362d4edf0e P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:35.605955 15349 raft_consensus.cc:3037] T c74522fbb5c64d27b5c95f05896b0716 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:35.607187 15349 raft_consensus.cc:1261] T c74522fbb5c64d27b5c95f05896b0716 P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Refusing update from remote peer b602f6099cbb4758847e7e9550685372: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:35.607859 15995 consensus_queue.cc:1035] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:35.608563 15220 raft_consensus.cc:1261] T c74522fbb5c64d27b5c95f05896b0716 P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Refusing update from remote peer b602f6099cbb4758847e7e9550685372: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:35.609895 15995 consensus_queue.cc:1035] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:35.631146 16052 mvcc.cc:205] Tried to move back new op lower bound from 7023417899436150784 to 7023417898252861440. Current Snapshot: MvccSnapshot[applied={T|T < 7023417899436150784}]
INSERT report
rows total: 2000
time total: 117.229 ms
time per row: 0.0586146 ms
I20240503 02:00:35.665122 14809 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39948:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:35.671617 15068 tablet_service.cc:1467] Processing CreateTablet for tablet ca0d7d3e4d59496c9a9a59cde24b9247 (DEFAULT_TABLE table=rebalance_test_table_2 [id=6f6554bb6f36436988cf760c14b4b17c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:35.671912 15330 tablet_service.cc:1467] Processing CreateTablet for tablet 7fc3330dfd0644f6a303659b960124f4 (DEFAULT_TABLE table=rebalance_test_table_2 [id=6f6554bb6f36436988cf760c14b4b17c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:35.671980 15068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca0d7d3e4d59496c9a9a59cde24b9247. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:35.671988 15331 tablet_service.cc:1467] Processing CreateTablet for tablet ca0d7d3e4d59496c9a9a59cde24b9247 (DEFAULT_TABLE table=rebalance_test_table_2 [id=6f6554bb6f36436988cf760c14b4b17c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:35.672155 15330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fc3330dfd0644f6a303659b960124f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:35.672263 15331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca0d7d3e4d59496c9a9a59cde24b9247. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:35.672605 15069 tablet_service.cc:1467] Processing CreateTablet for tablet 6d2a1296a54a4b92a03c12954de0cc34 (DEFAULT_TABLE table=rebalance_test_table_2 [id=6f6554bb6f36436988cf760c14b4b17c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:35.672659 15329 tablet_service.cc:1467] Processing CreateTablet for tablet 6d2a1296a54a4b92a03c12954de0cc34 (DEFAULT_TABLE table=rebalance_test_table_2 [id=6f6554bb6f36436988cf760c14b4b17c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:35.672842 15069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d2a1296a54a4b92a03c12954de0cc34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:35.672971 15329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d2a1296a54a4b92a03c12954de0cc34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:35.675032 15200 tablet_service.cc:1467] Processing CreateTablet for tablet 7fc3330dfd0644f6a303659b960124f4 (DEFAULT_TABLE table=rebalance_test_table_2 [id=6f6554bb6f36436988cf760c14b4b17c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:35.675346 15200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fc3330dfd0644f6a303659b960124f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:35.675594 14937 tablet_service.cc:1467] Processing CreateTablet for tablet ca0d7d3e4d59496c9a9a59cde24b9247 (DEFAULT_TABLE table=rebalance_test_table_2 [id=6f6554bb6f36436988cf760c14b4b17c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:35.675644 14936 tablet_service.cc:1467] Processing CreateTablet for tablet 7fc3330dfd0644f6a303659b960124f4 (DEFAULT_TABLE table=rebalance_test_table_2 [id=6f6554bb6f36436988cf760c14b4b17c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:35.675868 14937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca0d7d3e4d59496c9a9a59cde24b9247. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:35.676110 16107 tablet_bootstrap.cc:492] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372: Bootstrap starting.
I20240503 02:00:35.676654 14936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fc3330dfd0644f6a303659b960124f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:35.677284 16107 tablet_bootstrap.cc:654] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:35.677390 16106 tablet_bootstrap.cc:492] T 7fc3330dfd0644f6a303659b960124f4 P c05f7620bb884392931d3243a0b47a9e: Bootstrap starting.
I20240503 02:00:35.677772 14938 tablet_service.cc:1467] Processing CreateTablet for tablet 6d2a1296a54a4b92a03c12954de0cc34 (DEFAULT_TABLE table=rebalance_test_table_2 [id=6f6554bb6f36436988cf760c14b4b17c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:35.678007 14938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d2a1296a54a4b92a03c12954de0cc34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:35.678102 16108 tablet_bootstrap.cc:492] T 7fc3330dfd0644f6a303659b960124f4 P 620596b893d84291b88b90dc9e317612: Bootstrap starting.
I20240503 02:00:35.678295 16106 tablet_bootstrap.cc:654] T 7fc3330dfd0644f6a303659b960124f4 P c05f7620bb884392931d3243a0b47a9e: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:35.678946 16108 tablet_bootstrap.cc:654] T 7fc3330dfd0644f6a303659b960124f4 P 620596b893d84291b88b90dc9e317612: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:35.680078 16108 tablet_bootstrap.cc:492] T 7fc3330dfd0644f6a303659b960124f4 P 620596b893d84291b88b90dc9e317612: No bootstrap required, opened a new log
I20240503 02:00:35.680085 16107 tablet_bootstrap.cc:492] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372: No bootstrap required, opened a new log
I20240503 02:00:35.680169 16108 ts_tablet_manager.cc:1367] T 7fc3330dfd0644f6a303659b960124f4 P 620596b893d84291b88b90dc9e317612: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:35.680171 16107 ts_tablet_manager.cc:1367] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:35.680718 16110 tablet_bootstrap.cc:492] T 7fc3330dfd0644f6a303659b960124f4 P dcfdfa62edc34dab9525e2712d487582: Bootstrap starting.
I20240503 02:00:35.680667 16107 raft_consensus.cc:348] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:35.680665 16108 raft_consensus.cc:348] T 7fc3330dfd0644f6a303659b960124f4 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:35.680774 16107 raft_consensus.cc:374] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:35.680776 16108 raft_consensus.cc:374] T 7fc3330dfd0644f6a303659b960124f4 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:35.680804 16107 raft_consensus.cc:729] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Initialized, Role: FOLLOWER
I20240503 02:00:35.680807 16108 raft_consensus.cc:729] T 7fc3330dfd0644f6a303659b960124f4 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 620596b893d84291b88b90dc9e317612, State: Initialized, Role: FOLLOWER
I20240503 02:00:35.680930 16107 consensus_queue.cc:260] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:35.680960 16108 consensus_queue.cc:260] T 7fc3330dfd0644f6a303659b960124f4 P 620596b893d84291b88b90dc9e317612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:35.681190 16107 ts_tablet_manager.cc:1398] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:35.681370 16107 tablet_bootstrap.cc:492] T ca0d7d3e4d59496c9a9a59cde24b9247 P b602f6099cbb4758847e7e9550685372: Bootstrap starting.
I20240503 02:00:35.681555 16110 tablet_bootstrap.cc:654] T 7fc3330dfd0644f6a303659b960124f4 P dcfdfa62edc34dab9525e2712d487582: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:35.682025 16108 ts_tablet_manager.cc:1398] T 7fc3330dfd0644f6a303659b960124f4 P 620596b893d84291b88b90dc9e317612: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:35.682354 16107 tablet_bootstrap.cc:654] T ca0d7d3e4d59496c9a9a59cde24b9247 P b602f6099cbb4758847e7e9550685372: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:35.683857 16110 tablet_bootstrap.cc:492] T 7fc3330dfd0644f6a303659b960124f4 P dcfdfa62edc34dab9525e2712d487582: No bootstrap required, opened a new log
I20240503 02:00:35.683967 16110 ts_tablet_manager.cc:1367] T 7fc3330dfd0644f6a303659b960124f4 P dcfdfa62edc34dab9525e2712d487582: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:00:35.683962 16107 tablet_bootstrap.cc:492] T ca0d7d3e4d59496c9a9a59cde24b9247 P b602f6099cbb4758847e7e9550685372: No bootstrap required, opened a new log
I20240503 02:00:35.684056 16107 ts_tablet_manager.cc:1367] T ca0d7d3e4d59496c9a9a59cde24b9247 P b602f6099cbb4758847e7e9550685372: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:35.684631 16106 tablet_bootstrap.cc:492] T 7fc3330dfd0644f6a303659b960124f4 P c05f7620bb884392931d3243a0b47a9e: No bootstrap required, opened a new log
I20240503 02:00:35.684648 16110 raft_consensus.cc:348] T 7fc3330dfd0644f6a303659b960124f4 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:35.684643 16107 raft_consensus.cc:348] T ca0d7d3e4d59496c9a9a59cde24b9247 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:35.684765 16110 raft_consensus.cc:374] T 7fc3330dfd0644f6a303659b960124f4 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:35.684767 16107 raft_consensus.cc:374] T ca0d7d3e4d59496c9a9a59cde24b9247 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:35.684799 16110 raft_consensus.cc:729] T 7fc3330dfd0644f6a303659b960124f4 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Initialized, Role: FOLLOWER
I20240503 02:00:35.684804 16107 raft_consensus.cc:729] T ca0d7d3e4d59496c9a9a59cde24b9247 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Initialized, Role: FOLLOWER
I20240503 02:00:35.684969 16107 consensus_queue.cc:260] T ca0d7d3e4d59496c9a9a59cde24b9247 P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:35.684981 16110 consensus_queue.cc:260] T 7fc3330dfd0644f6a303659b960124f4 P dcfdfa62edc34dab9525e2712d487582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:35.685238 16106 ts_tablet_manager.cc:1367] T 7fc3330dfd0644f6a303659b960124f4 P c05f7620bb884392931d3243a0b47a9e: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 02:00:35.685384 16107 ts_tablet_manager.cc:1398] T ca0d7d3e4d59496c9a9a59cde24b9247 P b602f6099cbb4758847e7e9550685372: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:35.685760 16110 ts_tablet_manager.cc:1398] T 7fc3330dfd0644f6a303659b960124f4 P dcfdfa62edc34dab9525e2712d487582: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:35.685966 16110 tablet_bootstrap.cc:492] T 6d2a1296a54a4b92a03c12954de0cc34 P dcfdfa62edc34dab9525e2712d487582: Bootstrap starting.
I20240503 02:00:35.685909 16106 raft_consensus.cc:348] T 7fc3330dfd0644f6a303659b960124f4 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:35.686126 16106 raft_consensus.cc:374] T 7fc3330dfd0644f6a303659b960124f4 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:35.686197 16106 raft_consensus.cc:729] T 7fc3330dfd0644f6a303659b960124f4 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Initialized, Role: FOLLOWER
I20240503 02:00:35.686386 16106 consensus_queue.cc:260] T 7fc3330dfd0644f6a303659b960124f4 P c05f7620bb884392931d3243a0b47a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:35.687057 16110 tablet_bootstrap.cc:654] T 6d2a1296a54a4b92a03c12954de0cc34 P dcfdfa62edc34dab9525e2712d487582: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:35.687268 16106 ts_tablet_manager.cc:1398] T 7fc3330dfd0644f6a303659b960124f4 P c05f7620bb884392931d3243a0b47a9e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:35.687507 16106 tablet_bootstrap.cc:492] T ca0d7d3e4d59496c9a9a59cde24b9247 P c05f7620bb884392931d3243a0b47a9e: Bootstrap starting.
I20240503 02:00:35.688535 16106 tablet_bootstrap.cc:654] T ca0d7d3e4d59496c9a9a59cde24b9247 P c05f7620bb884392931d3243a0b47a9e: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:35.689854 16110 tablet_bootstrap.cc:492] T 6d2a1296a54a4b92a03c12954de0cc34 P dcfdfa62edc34dab9525e2712d487582: No bootstrap required, opened a new log
I20240503 02:00:35.689998 16110 ts_tablet_manager.cc:1367] T 6d2a1296a54a4b92a03c12954de0cc34 P dcfdfa62edc34dab9525e2712d487582: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:35.690460 16110 raft_consensus.cc:348] T 6d2a1296a54a4b92a03c12954de0cc34 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:35.690586 16110 raft_consensus.cc:374] T 6d2a1296a54a4b92a03c12954de0cc34 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:35.690629 16110 raft_consensus.cc:729] T 6d2a1296a54a4b92a03c12954de0cc34 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Initialized, Role: FOLLOWER
I20240503 02:00:35.690764 16110 consensus_queue.cc:260] T 6d2a1296a54a4b92a03c12954de0cc34 P dcfdfa62edc34dab9525e2712d487582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:35.690913 16106 tablet_bootstrap.cc:492] T ca0d7d3e4d59496c9a9a59cde24b9247 P c05f7620bb884392931d3243a0b47a9e: No bootstrap required, opened a new log
I20240503 02:00:35.691004 16106 ts_tablet_manager.cc:1367] T ca0d7d3e4d59496c9a9a59cde24b9247 P c05f7620bb884392931d3243a0b47a9e: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 02:00:35.691299 16110 ts_tablet_manager.cc:1398] T 6d2a1296a54a4b92a03c12954de0cc34 P dcfdfa62edc34dab9525e2712d487582: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:35.691495 16110 tablet_bootstrap.cc:492] T ca0d7d3e4d59496c9a9a59cde24b9247 P dcfdfa62edc34dab9525e2712d487582: Bootstrap starting.
I20240503 02:00:35.691478 16106 raft_consensus.cc:348] T ca0d7d3e4d59496c9a9a59cde24b9247 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:35.691581 16106 raft_consensus.cc:374] T ca0d7d3e4d59496c9a9a59cde24b9247 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:35.691612 16106 raft_consensus.cc:729] T ca0d7d3e4d59496c9a9a59cde24b9247 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Initialized, Role: FOLLOWER
I20240503 02:00:35.691732 16106 consensus_queue.cc:260] T ca0d7d3e4d59496c9a9a59cde24b9247 P c05f7620bb884392931d3243a0b47a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:35.692461 16110 tablet_bootstrap.cc:654] T ca0d7d3e4d59496c9a9a59cde24b9247 P dcfdfa62edc34dab9525e2712d487582: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:35.692453 16106 ts_tablet_manager.cc:1398] T ca0d7d3e4d59496c9a9a59cde24b9247 P c05f7620bb884392931d3243a0b47a9e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:35.692631 16106 tablet_bootstrap.cc:492] T 6d2a1296a54a4b92a03c12954de0cc34 P c05f7620bb884392931d3243a0b47a9e: Bootstrap starting.
I20240503 02:00:35.694300 16110 tablet_bootstrap.cc:492] T ca0d7d3e4d59496c9a9a59cde24b9247 P dcfdfa62edc34dab9525e2712d487582: No bootstrap required, opened a new log
I20240503 02:00:35.694406 16110 ts_tablet_manager.cc:1367] T ca0d7d3e4d59496c9a9a59cde24b9247 P dcfdfa62edc34dab9525e2712d487582: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:00:35.694455 16106 tablet_bootstrap.cc:654] T 6d2a1296a54a4b92a03c12954de0cc34 P c05f7620bb884392931d3243a0b47a9e: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:35.694864 16110 raft_consensus.cc:348] T ca0d7d3e4d59496c9a9a59cde24b9247 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:35.694972 16110 raft_consensus.cc:374] T ca0d7d3e4d59496c9a9a59cde24b9247 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:35.695000 16110 raft_consensus.cc:729] T ca0d7d3e4d59496c9a9a59cde24b9247 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Initialized, Role: FOLLOWER
I20240503 02:00:35.695165 16110 consensus_queue.cc:260] T ca0d7d3e4d59496c9a9a59cde24b9247 P dcfdfa62edc34dab9525e2712d487582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:35.695436 16110 ts_tablet_manager.cc:1398] T ca0d7d3e4d59496c9a9a59cde24b9247 P dcfdfa62edc34dab9525e2712d487582: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:35.696496 16106 tablet_bootstrap.cc:492] T 6d2a1296a54a4b92a03c12954de0cc34 P c05f7620bb884392931d3243a0b47a9e: No bootstrap required, opened a new log
I20240503 02:00:35.696591 16106 ts_tablet_manager.cc:1367] T 6d2a1296a54a4b92a03c12954de0cc34 P c05f7620bb884392931d3243a0b47a9e: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240503 02:00:35.697207 16106 raft_consensus.cc:348] T 6d2a1296a54a4b92a03c12954de0cc34 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:35.697309 16106 raft_consensus.cc:374] T 6d2a1296a54a4b92a03c12954de0cc34 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:35.697337 16106 raft_consensus.cc:729] T 6d2a1296a54a4b92a03c12954de0cc34 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Initialized, Role: FOLLOWER
I20240503 02:00:35.697480 16106 consensus_queue.cc:260] T 6d2a1296a54a4b92a03c12954de0cc34 P c05f7620bb884392931d3243a0b47a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:35.697772 16106 ts_tablet_manager.cc:1398] T 6d2a1296a54a4b92a03c12954de0cc34 P c05f7620bb884392931d3243a0b47a9e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:35.744498 16000 raft_consensus.cc:482] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:35.744634 16000 raft_consensus.cc:504] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:35.745095 16000 leader_election.cc:288] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:35.745486 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2a1296a54a4b92a03c12954de0cc34" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcfdfa62edc34dab9525e2712d487582" is_pre_election: true
I20240503 02:00:35.745533 15349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2a1296a54a4b92a03c12954de0cc34" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c05f7620bb884392931d3243a0b47a9e" is_pre_election: true
I20240503 02:00:35.745627 14958 raft_consensus.cc:2454] T 6d2a1296a54a4b92a03c12954de0cc34 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 0.
I20240503 02:00:35.745714 15349 raft_consensus.cc:2454] T 6d2a1296a54a4b92a03c12954de0cc34 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 0.
I20240503 02:00:35.746037 15022 leader_election.cc:302] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b602f6099cbb4758847e7e9550685372, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:35.746243 16000 raft_consensus.cc:2781] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:35.746320 16000 raft_consensus.cc:482] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:35.746372 16000 raft_consensus.cc:3037] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:35.747239 16000 raft_consensus.cc:504] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:35.747625 16000 leader_election.cc:288] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 1 election: Requested vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:35.748049 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2a1296a54a4b92a03c12954de0cc34" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:35.748064 15349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2a1296a54a4b92a03c12954de0cc34" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:35.748188 15349 raft_consensus.cc:3037] T 6d2a1296a54a4b92a03c12954de0cc34 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:35.748188 14958 raft_consensus.cc:3037] T 6d2a1296a54a4b92a03c12954de0cc34 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:35.749244 15349 raft_consensus.cc:2454] T 6d2a1296a54a4b92a03c12954de0cc34 P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 1.
I20240503 02:00:35.749238 14958 raft_consensus.cc:2454] T 6d2a1296a54a4b92a03c12954de0cc34 P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 1.
I20240503 02:00:35.749575 15024 leader_election.cc:302] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b602f6099cbb4758847e7e9550685372, c05f7620bb884392931d3243a0b47a9e; no voters:
I20240503 02:00:35.749763 16000 raft_consensus.cc:2781] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:35.749848 16000 raft_consensus.cc:686] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372 [term 1 LEADER]: Becoming Leader. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:35.750030 16000 consensus_queue.cc:237] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:35.751740 14815 catalog_manager.cc:5484] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372 reported cstate change: term changed from 0 to 1, leader changed from <none> to b602f6099cbb4758847e7e9550685372 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "b602f6099cbb4758847e7e9550685372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:35.783190 16028 raft_consensus.cc:482] T ca0d7d3e4d59496c9a9a59cde24b9247 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:35.783365 16028 raft_consensus.cc:504] T ca0d7d3e4d59496c9a9a59cde24b9247 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:35.783772 16028 leader_election.cc:288] T ca0d7d3e4d59496c9a9a59cde24b9247 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
I20240503 02:00:35.784330 15089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca0d7d3e4d59496c9a9a59cde24b9247" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b602f6099cbb4758847e7e9550685372" is_pre_election: true
I20240503 02:00:35.784336 15349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca0d7d3e4d59496c9a9a59cde24b9247" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c05f7620bb884392931d3243a0b47a9e" is_pre_election: true
I20240503 02:00:35.784492 15349 raft_consensus.cc:2454] T ca0d7d3e4d59496c9a9a59cde24b9247 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 0.
I20240503 02:00:35.784495 15089 raft_consensus.cc:2454] T ca0d7d3e4d59496c9a9a59cde24b9247 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 0.
I20240503 02:00:35.784808 14893 leader_election.cc:302] T ca0d7d3e4d59496c9a9a59cde24b9247 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c05f7620bb884392931d3243a0b47a9e, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:35.784976 16028 raft_consensus.cc:2781] T ca0d7d3e4d59496c9a9a59cde24b9247 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:35.785044 16028 raft_consensus.cc:482] T ca0d7d3e4d59496c9a9a59cde24b9247 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:35.785089 16028 raft_consensus.cc:3037] T ca0d7d3e4d59496c9a9a59cde24b9247 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:35.785854 16028 raft_consensus.cc:504] T ca0d7d3e4d59496c9a9a59cde24b9247 P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:35.786182 16028 leader_election.cc:288] T ca0d7d3e4d59496c9a9a59cde24b9247 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 1 election: Requested vote from peers c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
I20240503 02:00:35.786525 15089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca0d7d3e4d59496c9a9a59cde24b9247" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:35.786520 15349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca0d7d3e4d59496c9a9a59cde24b9247" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:35.786629 15089 raft_consensus.cc:3037] T ca0d7d3e4d59496c9a9a59cde24b9247 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:35.786636 15349 raft_consensus.cc:3037] T ca0d7d3e4d59496c9a9a59cde24b9247 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:35.787356 15089 raft_consensus.cc:2454] T ca0d7d3e4d59496c9a9a59cde24b9247 P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 1.
I20240503 02:00:35.787396 15349 raft_consensus.cc:2454] T ca0d7d3e4d59496c9a9a59cde24b9247 P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 1.
I20240503 02:00:35.787650 14892 leader_election.cc:302] T ca0d7d3e4d59496c9a9a59cde24b9247 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b602f6099cbb4758847e7e9550685372, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:35.787823 16028 raft_consensus.cc:2781] T ca0d7d3e4d59496c9a9a59cde24b9247 P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:35.787901 16028 raft_consensus.cc:686] T ca0d7d3e4d59496c9a9a59cde24b9247 P dcfdfa62edc34dab9525e2712d487582 [term 1 LEADER]: Becoming Leader. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:35.788038 16028 consensus_queue.cc:237] T ca0d7d3e4d59496c9a9a59cde24b9247 P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:35.789763 14815 catalog_manager.cc:5484] T ca0d7d3e4d59496c9a9a59cde24b9247 P dcfdfa62edc34dab9525e2712d487582 reported cstate change: term changed from 0 to 1, leader changed from <none> to dcfdfa62edc34dab9525e2712d487582 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "dcfdfa62edc34dab9525e2712d487582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:35.817725 16053 raft_consensus.cc:482] T 7fc3330dfd0644f6a303659b960124f4 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:35.817878 16053 raft_consensus.cc:504] T 7fc3330dfd0644f6a303659b960124f4 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:35.818277 16053 leader_election.cc:288] T 7fc3330dfd0644f6a303659b960124f4 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:35.818749 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fc3330dfd0644f6a303659b960124f4" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcfdfa62edc34dab9525e2712d487582" is_pre_election: true
I20240503 02:00:35.818780 15349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fc3330dfd0644f6a303659b960124f4" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c05f7620bb884392931d3243a0b47a9e" is_pre_election: true
I20240503 02:00:35.818893 14958 raft_consensus.cc:2454] T 7fc3330dfd0644f6a303659b960124f4 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 0.
I20240503 02:00:35.818923 15349 raft_consensus.cc:2454] T 7fc3330dfd0644f6a303659b960124f4 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 0.
I20240503 02:00:35.819218 15153 leader_election.cc:302] T 7fc3330dfd0644f6a303659b960124f4 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:35.819378 16053 raft_consensus.cc:2781] T 7fc3330dfd0644f6a303659b960124f4 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:35.819437 16053 raft_consensus.cc:482] T 7fc3330dfd0644f6a303659b960124f4 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:35.819465 16053 raft_consensus.cc:3037] T 7fc3330dfd0644f6a303659b960124f4 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:35.820334 16053 raft_consensus.cc:504] T 7fc3330dfd0644f6a303659b960124f4 P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:35.820740 16053 leader_election.cc:288] T 7fc3330dfd0644f6a303659b960124f4 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 1 election: Requested vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:35.821110 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fc3330dfd0644f6a303659b960124f4" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:35.821228 14958 raft_consensus.cc:3037] T 7fc3330dfd0644f6a303659b960124f4 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:35.821198 15349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fc3330dfd0644f6a303659b960124f4" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:35.821331 15349 raft_consensus.cc:3037] T 7fc3330dfd0644f6a303659b960124f4 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:35.822021 14958 raft_consensus.cc:2454] T 7fc3330dfd0644f6a303659b960124f4 P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 1.
I20240503 02:00:35.822080 15349 raft_consensus.cc:2454] T 7fc3330dfd0644f6a303659b960124f4 P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 1.
I20240503 02:00:35.822319 15153 leader_election.cc:302] T 7fc3330dfd0644f6a303659b960124f4 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:35.822492 16053 raft_consensus.cc:2781] T 7fc3330dfd0644f6a303659b960124f4 P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:35.822751 16053 raft_consensus.cc:686] T 7fc3330dfd0644f6a303659b960124f4 P 620596b893d84291b88b90dc9e317612 [term 1 LEADER]: Becoming Leader. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:35.823084 16053 consensus_queue.cc:237] T 7fc3330dfd0644f6a303659b960124f4 P 620596b893d84291b88b90dc9e317612 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:35.825453 14815 catalog_manager.cc:5484] T 7fc3330dfd0644f6a303659b960124f4 P 620596b893d84291b88b90dc9e317612 reported cstate change: term changed from 0 to 1, leader changed from <none> to 620596b893d84291b88b90dc9e317612 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "620596b893d84291b88b90dc9e317612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } }
W20240503 02:00:35.946877 16123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:35.947155 16123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:35.950459 16123 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240503 02:00:36.043501 15349 raft_consensus.cc:1261] T 7fc3330dfd0644f6a303659b960124f4 P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Refusing update from remote peer 620596b893d84291b88b90dc9e317612: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:36.043720 15089 raft_consensus.cc:1261] T ca0d7d3e4d59496c9a9a59cde24b9247 P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Refusing update from remote peer dcfdfa62edc34dab9525e2712d487582: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:36.044121 16121 consensus_queue.cc:1035] T 7fc3330dfd0644f6a303659b960124f4 P 620596b893d84291b88b90dc9e317612 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:36.044400 14958 raft_consensus.cc:1261] T 7fc3330dfd0644f6a303659b960124f4 P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Refusing update from remote peer 620596b893d84291b88b90dc9e317612: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:36.044870 16028 consensus_queue.cc:1035] T ca0d7d3e4d59496c9a9a59cde24b9247 P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:36.044992 16154 consensus_queue.cc:1035] T 7fc3330dfd0644f6a303659b960124f4 P 620596b893d84291b88b90dc9e317612 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:36.047067 15349 raft_consensus.cc:1261] T 6d2a1296a54a4b92a03c12954de0cc34 P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Refusing update from remote peer b602f6099cbb4758847e7e9550685372: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:36.047066 14957 raft_consensus.cc:1261] T 6d2a1296a54a4b92a03c12954de0cc34 P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Refusing update from remote peer b602f6099cbb4758847e7e9550685372: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:36.047444 15995 consensus_queue.cc:1035] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:36.053011 15995 consensus_queue.cc:1035] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:36.055002 15349 raft_consensus.cc:1261] T ca0d7d3e4d59496c9a9a59cde24b9247 P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Refusing update from remote peer dcfdfa62edc34dab9525e2712d487582: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:36.059903 16028 consensus_queue.cc:1035] T ca0d7d3e4d59496c9a9a59cde24b9247 P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 122.652 ms
time per row: 0.0613262 ms
I20240503 02:00:36.116573 14810 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39948:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:36.122635 15068 tablet_service.cc:1467] Processing CreateTablet for tablet 6b225db529f14e7386f21f42bf31493b (DEFAULT_TABLE table=rebalance_test_table_3 [id=99039e7d06b541b1a19f86ba5d7e71d9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:36.122982 15068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b225db529f14e7386f21f42bf31493b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:36.123273 15199 tablet_service.cc:1467] Processing CreateTablet for tablet 5d448658ed3b460fb1725a26a7c743e8 (DEFAULT_TABLE table=rebalance_test_table_3 [id=99039e7d06b541b1a19f86ba5d7e71d9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:36.123499 15200 tablet_service.cc:1467] Processing CreateTablet for tablet 6b225db529f14e7386f21f42bf31493b (DEFAULT_TABLE table=rebalance_test_table_3 [id=99039e7d06b541b1a19f86ba5d7e71d9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:36.123570 15199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d448658ed3b460fb1725a26a7c743e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:36.123653 15069 tablet_service.cc:1467] Processing CreateTablet for tablet ebd7cdd284a74741b338e9f4778502b5 (DEFAULT_TABLE table=rebalance_test_table_3 [id=99039e7d06b541b1a19f86ba5d7e71d9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:36.123716 15200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b225db529f14e7386f21f42bf31493b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:36.123867 15069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebd7cdd284a74741b338e9f4778502b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:36.125398 15330 tablet_service.cc:1467] Processing CreateTablet for tablet ebd7cdd284a74741b338e9f4778502b5 (DEFAULT_TABLE table=rebalance_test_table_3 [id=99039e7d06b541b1a19f86ba5d7e71d9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:36.125465 15329 tablet_service.cc:1467] Processing CreateTablet for tablet 5d448658ed3b460fb1725a26a7c743e8 (DEFAULT_TABLE table=rebalance_test_table_3 [id=99039e7d06b541b1a19f86ba5d7e71d9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:36.125680 15330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebd7cdd284a74741b338e9f4778502b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:36.125759 16107 tablet_bootstrap.cc:492] T 6b225db529f14e7386f21f42bf31493b P b602f6099cbb4758847e7e9550685372: Bootstrap starting.
I20240503 02:00:36.125759 16108 tablet_bootstrap.cc:492] T 5d448658ed3b460fb1725a26a7c743e8 P 620596b893d84291b88b90dc9e317612: Bootstrap starting.
I20240503 02:00:36.126662 16107 tablet_bootstrap.cc:654] T 6b225db529f14e7386f21f42bf31493b P b602f6099cbb4758847e7e9550685372: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:36.126662 16108 tablet_bootstrap.cc:654] T 5d448658ed3b460fb1725a26a7c743e8 P 620596b893d84291b88b90dc9e317612: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:36.127449 16106 tablet_bootstrap.cc:492] T ebd7cdd284a74741b338e9f4778502b5 P c05f7620bb884392931d3243a0b47a9e: Bootstrap starting.
I20240503 02:00:36.127543 15329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d448658ed3b460fb1725a26a7c743e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:36.128067 15199 tablet_service.cc:1467] Processing CreateTablet for tablet ebd7cdd284a74741b338e9f4778502b5 (DEFAULT_TABLE table=rebalance_test_table_3 [id=99039e7d06b541b1a19f86ba5d7e71d9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:36.128342 15199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebd7cdd284a74741b338e9f4778502b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:36.128764 16106 tablet_bootstrap.cc:654] T ebd7cdd284a74741b338e9f4778502b5 P c05f7620bb884392931d3243a0b47a9e: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:36.129096 14937 tablet_service.cc:1467] Processing CreateTablet for tablet 6b225db529f14e7386f21f42bf31493b (DEFAULT_TABLE table=rebalance_test_table_3 [id=99039e7d06b541b1a19f86ba5d7e71d9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:36.129309 14937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b225db529f14e7386f21f42bf31493b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:36.129562 14938 tablet_service.cc:1467] Processing CreateTablet for tablet 5d448658ed3b460fb1725a26a7c743e8 (DEFAULT_TABLE table=rebalance_test_table_3 [id=99039e7d06b541b1a19f86ba5d7e71d9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:36.129809 14938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d448658ed3b460fb1725a26a7c743e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:36.130939 16110 tablet_bootstrap.cc:492] T 6b225db529f14e7386f21f42bf31493b P dcfdfa62edc34dab9525e2712d487582: Bootstrap starting.
I20240503 02:00:36.131798 16110 tablet_bootstrap.cc:654] T 6b225db529f14e7386f21f42bf31493b P dcfdfa62edc34dab9525e2712d487582: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:36.132995 16110 tablet_bootstrap.cc:492] T 6b225db529f14e7386f21f42bf31493b P dcfdfa62edc34dab9525e2712d487582: No bootstrap required, opened a new log
I20240503 02:00:36.132997 16107 tablet_bootstrap.cc:492] T 6b225db529f14e7386f21f42bf31493b P b602f6099cbb4758847e7e9550685372: No bootstrap required, opened a new log
I20240503 02:00:36.133087 16110 ts_tablet_manager.cc:1367] T 6b225db529f14e7386f21f42bf31493b P dcfdfa62edc34dab9525e2712d487582: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:00:36.133087 16107 ts_tablet_manager.cc:1367] T 6b225db529f14e7386f21f42bf31493b P b602f6099cbb4758847e7e9550685372: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240503 02:00:36.133574 16106 tablet_bootstrap.cc:492] T ebd7cdd284a74741b338e9f4778502b5 P c05f7620bb884392931d3243a0b47a9e: No bootstrap required, opened a new log
I20240503 02:00:36.133527 16107 raft_consensus.cc:348] T 6b225db529f14e7386f21f42bf31493b P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:36.133527 16110 raft_consensus.cc:348] T 6b225db529f14e7386f21f42bf31493b P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:36.133625 16110 raft_consensus.cc:374] T 6b225db529f14e7386f21f42bf31493b P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:36.133625 16107 raft_consensus.cc:374] T 6b225db529f14e7386f21f42bf31493b P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:36.133637 16106 ts_tablet_manager.cc:1367] T ebd7cdd284a74741b338e9f4778502b5 P c05f7620bb884392931d3243a0b47a9e: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:00:36.133656 16107 raft_consensus.cc:729] T 6b225db529f14e7386f21f42bf31493b P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Initialized, Role: FOLLOWER
I20240503 02:00:36.133656 16110 raft_consensus.cc:729] T 6b225db529f14e7386f21f42bf31493b P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Initialized, Role: FOLLOWER
I20240503 02:00:36.133773 16107 consensus_queue.cc:260] T 6b225db529f14e7386f21f42bf31493b P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:36.133773 16110 consensus_queue.cc:260] T 6b225db529f14e7386f21f42bf31493b P dcfdfa62edc34dab9525e2712d487582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:36.134032 16107 ts_tablet_manager.cc:1398] T 6b225db529f14e7386f21f42bf31493b P b602f6099cbb4758847e7e9550685372: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:36.134032 16110 ts_tablet_manager.cc:1398] T 6b225db529f14e7386f21f42bf31493b P dcfdfa62edc34dab9525e2712d487582: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:36.134061 16106 raft_consensus.cc:348] T ebd7cdd284a74741b338e9f4778502b5 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:36.134163 16106 raft_consensus.cc:374] T ebd7cdd284a74741b338e9f4778502b5 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:36.134172 16110 tablet_bootstrap.cc:492] T 5d448658ed3b460fb1725a26a7c743e8 P dcfdfa62edc34dab9525e2712d487582: Bootstrap starting.
I20240503 02:00:36.134189 16107 tablet_bootstrap.cc:492] T ebd7cdd284a74741b338e9f4778502b5 P b602f6099cbb4758847e7e9550685372: Bootstrap starting.
I20240503 02:00:36.134192 16106 raft_consensus.cc:729] T ebd7cdd284a74741b338e9f4778502b5 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Initialized, Role: FOLLOWER
I20240503 02:00:36.134306 16106 consensus_queue.cc:260] T ebd7cdd284a74741b338e9f4778502b5 P c05f7620bb884392931d3243a0b47a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:36.134549 16106 ts_tablet_manager.cc:1398] T ebd7cdd284a74741b338e9f4778502b5 P c05f7620bb884392931d3243a0b47a9e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:36.135032 16107 tablet_bootstrap.cc:654] T ebd7cdd284a74741b338e9f4778502b5 P b602f6099cbb4758847e7e9550685372: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:36.135032 16110 tablet_bootstrap.cc:654] T 5d448658ed3b460fb1725a26a7c743e8 P dcfdfa62edc34dab9525e2712d487582: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:36.138058 16110 tablet_bootstrap.cc:492] T 5d448658ed3b460fb1725a26a7c743e8 P dcfdfa62edc34dab9525e2712d487582: No bootstrap required, opened a new log
I20240503 02:00:36.138338 16107 tablet_bootstrap.cc:492] T ebd7cdd284a74741b338e9f4778502b5 P b602f6099cbb4758847e7e9550685372: No bootstrap required, opened a new log
I20240503 02:00:36.138445 16107 ts_tablet_manager.cc:1367] T ebd7cdd284a74741b338e9f4778502b5 P b602f6099cbb4758847e7e9550685372: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 02:00:36.138473 16106 tablet_bootstrap.cc:492] T 5d448658ed3b460fb1725a26a7c743e8 P c05f7620bb884392931d3243a0b47a9e: Bootstrap starting.
I20240503 02:00:36.138509 16108 tablet_bootstrap.cc:492] T 5d448658ed3b460fb1725a26a7c743e8 P 620596b893d84291b88b90dc9e317612: No bootstrap required, opened a new log
I20240503 02:00:36.138595 16108 ts_tablet_manager.cc:1367] T 5d448658ed3b460fb1725a26a7c743e8 P 620596b893d84291b88b90dc9e317612: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.000s
I20240503 02:00:36.138969 16107 raft_consensus.cc:348] T ebd7cdd284a74741b338e9f4778502b5 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:36.139093 16107 raft_consensus.cc:374] T ebd7cdd284a74741b338e9f4778502b5 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:36.139127 16107 raft_consensus.cc:729] T ebd7cdd284a74741b338e9f4778502b5 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Initialized, Role: FOLLOWER
I20240503 02:00:36.139073 16108 raft_consensus.cc:348] T 5d448658ed3b460fb1725a26a7c743e8 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:36.139180 16108 raft_consensus.cc:374] T 5d448658ed3b460fb1725a26a7c743e8 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:36.139211 16108 raft_consensus.cc:729] T 5d448658ed3b460fb1725a26a7c743e8 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 620596b893d84291b88b90dc9e317612, State: Initialized, Role: FOLLOWER
I20240503 02:00:36.139254 16107 consensus_queue.cc:260] T ebd7cdd284a74741b338e9f4778502b5 P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:36.139330 16108 consensus_queue.cc:260] T 5d448658ed3b460fb1725a26a7c743e8 P 620596b893d84291b88b90dc9e317612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:36.139551 16107 ts_tablet_manager.cc:1398] T ebd7cdd284a74741b338e9f4778502b5 P b602f6099cbb4758847e7e9550685372: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:36.139585 16108 ts_tablet_manager.cc:1398] T 5d448658ed3b460fb1725a26a7c743e8 P 620596b893d84291b88b90dc9e317612: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:36.139595 16106 tablet_bootstrap.cc:654] T 5d448658ed3b460fb1725a26a7c743e8 P c05f7620bb884392931d3243a0b47a9e: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:36.139753 16108 tablet_bootstrap.cc:492] T ebd7cdd284a74741b338e9f4778502b5 P 620596b893d84291b88b90dc9e317612: Bootstrap starting.
I20240503 02:00:36.140610 16108 tablet_bootstrap.cc:654] T ebd7cdd284a74741b338e9f4778502b5 P 620596b893d84291b88b90dc9e317612: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:36.141283 16110 ts_tablet_manager.cc:1367] T 5d448658ed3b460fb1725a26a7c743e8 P dcfdfa62edc34dab9525e2712d487582: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20240503 02:00:36.141682 16106 tablet_bootstrap.cc:492] T 5d448658ed3b460fb1725a26a7c743e8 P c05f7620bb884392931d3243a0b47a9e: No bootstrap required, opened a new log
I20240503 02:00:36.141938 16110 raft_consensus.cc:348] T 5d448658ed3b460fb1725a26a7c743e8 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:36.142057 16110 raft_consensus.cc:374] T 5d448658ed3b460fb1725a26a7c743e8 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:36.142091 16110 raft_consensus.cc:729] T 5d448658ed3b460fb1725a26a7c743e8 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Initialized, Role: FOLLOWER
I20240503 02:00:36.142217 16110 consensus_queue.cc:260] T 5d448658ed3b460fb1725a26a7c743e8 P dcfdfa62edc34dab9525e2712d487582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:36.142496 16110 ts_tablet_manager.cc:1398] T 5d448658ed3b460fb1725a26a7c743e8 P dcfdfa62edc34dab9525e2712d487582: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:36.143050 16106 ts_tablet_manager.cc:1367] T 5d448658ed3b460fb1725a26a7c743e8 P c05f7620bb884392931d3243a0b47a9e: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240503 02:00:36.143204 16108 tablet_bootstrap.cc:492] T ebd7cdd284a74741b338e9f4778502b5 P 620596b893d84291b88b90dc9e317612: No bootstrap required, opened a new log
I20240503 02:00:36.143283 16108 ts_tablet_manager.cc:1367] T ebd7cdd284a74741b338e9f4778502b5 P 620596b893d84291b88b90dc9e317612: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:36.143517 16106 raft_consensus.cc:348] T 5d448658ed3b460fb1725a26a7c743e8 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:36.143606 16106 raft_consensus.cc:374] T 5d448658ed3b460fb1725a26a7c743e8 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:36.143630 16106 raft_consensus.cc:729] T 5d448658ed3b460fb1725a26a7c743e8 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Initialized, Role: FOLLOWER
I20240503 02:00:36.143743 16106 consensus_queue.cc:260] T 5d448658ed3b460fb1725a26a7c743e8 P c05f7620bb884392931d3243a0b47a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:36.143747 16108 raft_consensus.cc:348] T ebd7cdd284a74741b338e9f4778502b5 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:36.143843 16108 raft_consensus.cc:374] T ebd7cdd284a74741b338e9f4778502b5 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:36.143870 16108 raft_consensus.cc:729] T ebd7cdd284a74741b338e9f4778502b5 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 620596b893d84291b88b90dc9e317612, State: Initialized, Role: FOLLOWER
I20240503 02:00:36.143975 16108 consensus_queue.cc:260] T ebd7cdd284a74741b338e9f4778502b5 P 620596b893d84291b88b90dc9e317612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:36.144121 16106 ts_tablet_manager.cc:1398] T 5d448658ed3b460fb1725a26a7c743e8 P c05f7620bb884392931d3243a0b47a9e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:36.144224 16108 ts_tablet_manager.cc:1398] T ebd7cdd284a74741b338e9f4778502b5 P 620596b893d84291b88b90dc9e317612: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:36.144462 16108 tablet_bootstrap.cc:492] T 6b225db529f14e7386f21f42bf31493b P 620596b893d84291b88b90dc9e317612: Bootstrap starting.
I20240503 02:00:36.145411 16108 tablet_bootstrap.cc:654] T 6b225db529f14e7386f21f42bf31493b P 620596b893d84291b88b90dc9e317612: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:36.146865 16108 tablet_bootstrap.cc:492] T 6b225db529f14e7386f21f42bf31493b P 620596b893d84291b88b90dc9e317612: No bootstrap required, opened a new log
I20240503 02:00:36.147022 16108 ts_tablet_manager.cc:1367] T 6b225db529f14e7386f21f42bf31493b P 620596b893d84291b88b90dc9e317612: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:36.147456 16108 raft_consensus.cc:348] T 6b225db529f14e7386f21f42bf31493b P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:36.147559 16108 raft_consensus.cc:374] T 6b225db529f14e7386f21f42bf31493b P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:36.147584 16108 raft_consensus.cc:729] T 6b225db529f14e7386f21f42bf31493b P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 620596b893d84291b88b90dc9e317612, State: Initialized, Role: FOLLOWER
I20240503 02:00:36.147720 16108 consensus_queue.cc:260] T 6b225db529f14e7386f21f42bf31493b P 620596b893d84291b88b90dc9e317612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:36.148000 16108 ts_tablet_manager.cc:1398] T 6b225db529f14e7386f21f42bf31493b P 620596b893d84291b88b90dc9e317612: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:36.189054 16023 raft_consensus.cc:482] T 5d448658ed3b460fb1725a26a7c743e8 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:36.189183 16023 raft_consensus.cc:504] T 5d448658ed3b460fb1725a26a7c743e8 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:36.189602 16023 leader_election.cc:288] T 5d448658ed3b460fb1725a26a7c743e8 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009), 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437)
I20240503 02:00:36.189951 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d448658ed3b460fb1725a26a7c743e8" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c05f7620bb884392931d3243a0b47a9e" is_pre_election: true
I20240503 02:00:36.190016 15220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d448658ed3b460fb1725a26a7c743e8" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "620596b893d84291b88b90dc9e317612" is_pre_election: true
I20240503 02:00:36.190073 15351 raft_consensus.cc:2454] T 5d448658ed3b460fb1725a26a7c743e8 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 0.
I20240503 02:00:36.190171 15220 raft_consensus.cc:2454] T 5d448658ed3b460fb1725a26a7c743e8 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 0.
I20240503 02:00:36.190351 14893 leader_election.cc:302] T 5d448658ed3b460fb1725a26a7c743e8 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c05f7620bb884392931d3243a0b47a9e, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:36.190538 16023 raft_consensus.cc:2781] T 5d448658ed3b460fb1725a26a7c743e8 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:36.190610 16023 raft_consensus.cc:482] T 5d448658ed3b460fb1725a26a7c743e8 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:36.190652 16023 raft_consensus.cc:3037] T 5d448658ed3b460fb1725a26a7c743e8 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:36.191504 16023 raft_consensus.cc:504] T 5d448658ed3b460fb1725a26a7c743e8 P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:36.191816 16023 leader_election.cc:288] T 5d448658ed3b460fb1725a26a7c743e8 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 1 election: Requested vote from peers c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009), 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437)
I20240503 02:00:36.192193 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d448658ed3b460fb1725a26a7c743e8" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:36.192203 15220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d448658ed3b460fb1725a26a7c743e8" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:36.192325 15351 raft_consensus.cc:3037] T 5d448658ed3b460fb1725a26a7c743e8 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:36.192325 15220 raft_consensus.cc:3037] T 5d448658ed3b460fb1725a26a7c743e8 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:36.193293 15220 raft_consensus.cc:2454] T 5d448658ed3b460fb1725a26a7c743e8 P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 1.
I20240503 02:00:36.193295 15351 raft_consensus.cc:2454] T 5d448658ed3b460fb1725a26a7c743e8 P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 1.
I20240503 02:00:36.193583 14893 leader_election.cc:302] T 5d448658ed3b460fb1725a26a7c743e8 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:36.193734 16023 raft_consensus.cc:2781] T 5d448658ed3b460fb1725a26a7c743e8 P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:36.193820 16023 raft_consensus.cc:686] T 5d448658ed3b460fb1725a26a7c743e8 P dcfdfa62edc34dab9525e2712d487582 [term 1 LEADER]: Becoming Leader. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:36.193954 16023 consensus_queue.cc:237] T 5d448658ed3b460fb1725a26a7c743e8 P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:36.195634 14813 catalog_manager.cc:5484] T 5d448658ed3b460fb1725a26a7c743e8 P dcfdfa62edc34dab9525e2712d487582 reported cstate change: term changed from 0 to 1, leader changed from <none> to dcfdfa62edc34dab9525e2712d487582 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "dcfdfa62edc34dab9525e2712d487582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:36.205868 16154 raft_consensus.cc:482] T 6b225db529f14e7386f21f42bf31493b P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:36.206010 16154 raft_consensus.cc:504] T 6b225db529f14e7386f21f42bf31493b P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:36.206502 16154 leader_election.cc:288] T 6b225db529f14e7386f21f42bf31493b P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207), dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817)
I20240503 02:00:36.207007 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b225db529f14e7386f21f42bf31493b" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcfdfa62edc34dab9525e2712d487582" is_pre_election: true
I20240503 02:00:36.207155 14957 raft_consensus.cc:2454] T 6b225db529f14e7386f21f42bf31493b P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 0.
I20240503 02:00:36.207489 15153 leader_election.cc:302] T 6b225db529f14e7386f21f42bf31493b P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:36.207651 16154 raft_consensus.cc:2781] T 6b225db529f14e7386f21f42bf31493b P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:36.207720 16154 raft_consensus.cc:482] T 6b225db529f14e7386f21f42bf31493b P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:36.207772 16154 raft_consensus.cc:3037] T 6b225db529f14e7386f21f42bf31493b P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:36.208861 16154 raft_consensus.cc:504] T 6b225db529f14e7386f21f42bf31493b P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:36.209213 16154 leader_election.cc:288] T 6b225db529f14e7386f21f42bf31493b P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 1 election: Requested vote from peers b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207), dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817)
I20240503 02:00:36.209544 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b225db529f14e7386f21f42bf31493b" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:36.209674 14957 raft_consensus.cc:3037] T 6b225db529f14e7386f21f42bf31493b P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:36.210546 14957 raft_consensus.cc:2454] T 6b225db529f14e7386f21f42bf31493b P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 1.
I20240503 02:00:36.210536 15089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b225db529f14e7386f21f42bf31493b" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b602f6099cbb4758847e7e9550685372" is_pre_election: true
I20240503 02:00:36.210561 15088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b225db529f14e7386f21f42bf31493b" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:36.210718 15088 raft_consensus.cc:3037] T 6b225db529f14e7386f21f42bf31493b P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:36.210876 15153 leader_election.cc:302] T 6b225db529f14e7386f21f42bf31493b P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:36.211067 16154 raft_consensus.cc:2781] T 6b225db529f14e7386f21f42bf31493b P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:36.211171 16154 raft_consensus.cc:686] T 6b225db529f14e7386f21f42bf31493b P 620596b893d84291b88b90dc9e317612 [term 1 LEADER]: Becoming Leader. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:36.211309 16154 consensus_queue.cc:237] T 6b225db529f14e7386f21f42bf31493b P 620596b893d84291b88b90dc9e317612 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:36.211650 15088 raft_consensus.cc:2454] T 6b225db529f14e7386f21f42bf31493b P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 1.
I20240503 02:00:36.213177 14810 catalog_manager.cc:5484] T 6b225db529f14e7386f21f42bf31493b P 620596b893d84291b88b90dc9e317612 reported cstate change: term changed from 0 to 1, leader changed from <none> to 620596b893d84291b88b90dc9e317612 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "620596b893d84291b88b90dc9e317612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:36.256477 15958 raft_consensus.cc:482] T ebd7cdd284a74741b338e9f4778502b5 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:36.256636 15958 raft_consensus.cc:504] T ebd7cdd284a74741b338e9f4778502b5 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:36.257018 15958 leader_election.cc:288] T ebd7cdd284a74741b338e9f4778502b5 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
I20240503 02:00:36.257463 15088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebd7cdd284a74741b338e9f4778502b5" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b602f6099cbb4758847e7e9550685372" is_pre_election: true
I20240503 02:00:36.257463 15220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebd7cdd284a74741b338e9f4778502b5" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "620596b893d84291b88b90dc9e317612" is_pre_election: true
I20240503 02:00:36.257613 15220 raft_consensus.cc:2454] T ebd7cdd284a74741b338e9f4778502b5 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 0.
I20240503 02:00:36.257613 15088 raft_consensus.cc:2454] T ebd7cdd284a74741b338e9f4778502b5 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 0.
I20240503 02:00:36.257916 15286 leader_election.cc:302] T ebd7cdd284a74741b338e9f4778502b5 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, c05f7620bb884392931d3243a0b47a9e; no voters:
I20240503 02:00:36.258199 15958 raft_consensus.cc:2781] T ebd7cdd284a74741b338e9f4778502b5 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:36.258280 15958 raft_consensus.cc:482] T ebd7cdd284a74741b338e9f4778502b5 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:36.258322 15958 raft_consensus.cc:3037] T ebd7cdd284a74741b338e9f4778502b5 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:36.259193 15958 raft_consensus.cc:504] T ebd7cdd284a74741b338e9f4778502b5 P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:36.259578 15958 leader_election.cc:288] T ebd7cdd284a74741b338e9f4778502b5 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 1 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
I20240503 02:00:36.260030 15088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebd7cdd284a74741b338e9f4778502b5" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:36.260043 15220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebd7cdd284a74741b338e9f4778502b5" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:36.260154 15088 raft_consensus.cc:3037] T ebd7cdd284a74741b338e9f4778502b5 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:36.260170 15220 raft_consensus.cc:3037] T ebd7cdd284a74741b338e9f4778502b5 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:36.260986 15088 raft_consensus.cc:2454] T ebd7cdd284a74741b338e9f4778502b5 P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 1.
I20240503 02:00:36.261013 15220 raft_consensus.cc:2454] T ebd7cdd284a74741b338e9f4778502b5 P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 1.
I20240503 02:00:36.261345 15286 leader_election.cc:302] T ebd7cdd284a74741b338e9f4778502b5 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, c05f7620bb884392931d3243a0b47a9e; no voters:
I20240503 02:00:36.261534 15958 raft_consensus.cc:2781] T ebd7cdd284a74741b338e9f4778502b5 P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:36.261615 15958 raft_consensus.cc:686] T ebd7cdd284a74741b338e9f4778502b5 P c05f7620bb884392931d3243a0b47a9e [term 1 LEADER]: Becoming Leader. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:36.261749 15958 consensus_queue.cc:237] T ebd7cdd284a74741b338e9f4778502b5 P c05f7620bb884392931d3243a0b47a9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:36.263566 14810 catalog_manager.cc:5484] T ebd7cdd284a74741b338e9f4778502b5 P c05f7620bb884392931d3243a0b47a9e reported cstate change: term changed from 0 to 1, leader changed from <none> to c05f7620bb884392931d3243a0b47a9e (127.0.99.132). New cstate: current_term: 1 leader_uuid: "c05f7620bb884392931d3243a0b47a9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: HEALTHY } } }
W20240503 02:00:36.403095 16181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:36.403399 16181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:36.406729 16181 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240503 02:00:36.507047 14957 raft_consensus.cc:1261] T 6b225db529f14e7386f21f42bf31493b P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Refusing update from remote peer 620596b893d84291b88b90dc9e317612: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:36.507568 15220 raft_consensus.cc:1261] T 5d448658ed3b460fb1725a26a7c743e8 P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Refusing update from remote peer dcfdfa62edc34dab9525e2712d487582: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:36.507592 15351 raft_consensus.cc:1261] T 5d448658ed3b460fb1725a26a7c743e8 P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Refusing update from remote peer dcfdfa62edc34dab9525e2712d487582: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:36.508277 16167 consensus_queue.cc:1035] T 5d448658ed3b460fb1725a26a7c743e8 P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:36.508553 16154 consensus_queue.cc:1035] T 6b225db529f14e7386f21f42bf31493b P 620596b893d84291b88b90dc9e317612 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:36.508857 16023 consensus_queue.cc:1035] T 5d448658ed3b460fb1725a26a7c743e8 P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Connected to new peer: Peer: permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:36.510039 15088 raft_consensus.cc:1261] T 6b225db529f14e7386f21f42bf31493b P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Refusing update from remote peer 620596b893d84291b88b90dc9e317612: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:36.510637 16154 consensus_queue.cc:1035] T 6b225db529f14e7386f21f42bf31493b P 620596b893d84291b88b90dc9e317612 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:36.514482 15220 raft_consensus.cc:1261] T ebd7cdd284a74741b338e9f4778502b5 P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Refusing update from remote peer c05f7620bb884392931d3243a0b47a9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:36.514859 15958 consensus_queue.cc:1035] T ebd7cdd284a74741b338e9f4778502b5 P c05f7620bb884392931d3243a0b47a9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:36.517365 15088 raft_consensus.cc:1261] T ebd7cdd284a74741b338e9f4778502b5 P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Refusing update from remote peer c05f7620bb884392931d3243a0b47a9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:36.517794 15958 consensus_queue.cc:1035] T ebd7cdd284a74741b338e9f4778502b5 P c05f7620bb884392931d3243a0b47a9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 121.784 ms
time per row: 0.0608921 ms
I20240503 02:00:36.574388 14809 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39948:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:36.579926 15069 tablet_service.cc:1467] Processing CreateTablet for tablet 42ceeb7b3c4146cd8a2e18b3e9ae987c (DEFAULT_TABLE table=rebalance_test_table_4 [id=64d27466361942ba8eb961f0f33df9f5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:36.579998 15200 tablet_service.cc:1467] Processing CreateTablet for tablet 42ceeb7b3c4146cd8a2e18b3e9ae987c (DEFAULT_TABLE table=rebalance_test_table_4 [id=64d27466361942ba8eb961f0f33df9f5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:36.579998 14938 tablet_service.cc:1467] Processing CreateTablet for tablet 42ceeb7b3c4146cd8a2e18b3e9ae987c (DEFAULT_TABLE table=rebalance_test_table_4 [id=64d27466361942ba8eb961f0f33df9f5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:36.580276 15069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42ceeb7b3c4146cd8a2e18b3e9ae987c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:36.580364 15200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42ceeb7b3c4146cd8a2e18b3e9ae987c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:36.581151 15329 tablet_service.cc:1467] Processing CreateTablet for tablet 454a24fd6d964d73ae3d5057f90fe36d (DEFAULT_TABLE table=rebalance_test_table_4 [id=64d27466361942ba8eb961f0f33df9f5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:36.581199 15330 tablet_service.cc:1467] Processing CreateTablet for tablet 11275e009edd4354b0807b7f015023be (DEFAULT_TABLE table=rebalance_test_table_4 [id=64d27466361942ba8eb961f0f33df9f5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:36.581442 15329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 454a24fd6d964d73ae3d5057f90fe36d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:36.580363 14938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42ceeb7b3c4146cd8a2e18b3e9ae987c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:36.583141 14937 tablet_service.cc:1467] Processing CreateTablet for tablet 454a24fd6d964d73ae3d5057f90fe36d (DEFAULT_TABLE table=rebalance_test_table_4 [id=64d27466361942ba8eb961f0f33df9f5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:36.583398 14937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 454a24fd6d964d73ae3d5057f90fe36d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:36.583449 15199 tablet_service.cc:1467] Processing CreateTablet for tablet 11275e009edd4354b0807b7f015023be (DEFAULT_TABLE table=rebalance_test_table_4 [id=64d27466361942ba8eb961f0f33df9f5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:36.583554 16110 tablet_bootstrap.cc:492] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P dcfdfa62edc34dab9525e2712d487582: Bootstrap starting.
I20240503 02:00:36.583684 15199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11275e009edd4354b0807b7f015023be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:36.584033 14936 tablet_service.cc:1467] Processing CreateTablet for tablet 11275e009edd4354b0807b7f015023be (DEFAULT_TABLE table=rebalance_test_table_4 [id=64d27466361942ba8eb961f0f33df9f5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:36.584235 14936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11275e009edd4354b0807b7f015023be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:36.584589 16110 tablet_bootstrap.cc:654] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P dcfdfa62edc34dab9525e2712d487582: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:36.585696 16107 tablet_bootstrap.cc:492] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P b602f6099cbb4758847e7e9550685372: Bootstrap starting.
I20240503 02:00:36.585922 15069 tablet_service.cc:1467] Processing CreateTablet for tablet 454a24fd6d964d73ae3d5057f90fe36d (DEFAULT_TABLE table=rebalance_test_table_4 [id=64d27466361942ba8eb961f0f33df9f5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:36.586163 15069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 454a24fd6d964d73ae3d5057f90fe36d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:36.586333 16108 tablet_bootstrap.cc:492] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P 620596b893d84291b88b90dc9e317612: Bootstrap starting.
I20240503 02:00:36.586534 16107 tablet_bootstrap.cc:654] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P b602f6099cbb4758847e7e9550685372: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:36.587155 16108 tablet_bootstrap.cc:654] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P 620596b893d84291b88b90dc9e317612: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:36.587603 16107 tablet_bootstrap.cc:492] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P b602f6099cbb4758847e7e9550685372: No bootstrap required, opened a new log
I20240503 02:00:36.587679 16107 ts_tablet_manager.cc:1367] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P b602f6099cbb4758847e7e9550685372: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:36.588409 16108 tablet_bootstrap.cc:492] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P 620596b893d84291b88b90dc9e317612: No bootstrap required, opened a new log
I20240503 02:00:36.588502 16108 ts_tablet_manager.cc:1367] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P 620596b893d84291b88b90dc9e317612: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:36.588699 16106 tablet_bootstrap.cc:492] T 454a24fd6d964d73ae3d5057f90fe36d P c05f7620bb884392931d3243a0b47a9e: Bootstrap starting.
I20240503 02:00:36.588955 16108 raft_consensus.cc:348] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:36.589028 16110 tablet_bootstrap.cc:492] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P dcfdfa62edc34dab9525e2712d487582: No bootstrap required, opened a new log
I20240503 02:00:36.589068 16108 raft_consensus.cc:374] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:36.589098 16108 raft_consensus.cc:729] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 620596b893d84291b88b90dc9e317612, State: Initialized, Role: FOLLOWER
I20240503 02:00:36.589116 16110 ts_tablet_manager.cc:1367] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P dcfdfa62edc34dab9525e2712d487582: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:00:36.589219 16108 consensus_queue.cc:260] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P 620596b893d84291b88b90dc9e317612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:36.589504 16108 ts_tablet_manager.cc:1398] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P 620596b893d84291b88b90dc9e317612: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:36.589563 16110 raft_consensus.cc:348] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:36.589663 16106 tablet_bootstrap.cc:654] T 454a24fd6d964d73ae3d5057f90fe36d P c05f7620bb884392931d3243a0b47a9e: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:36.589670 16110 raft_consensus.cc:374] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:36.589699 16110 raft_consensus.cc:729] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Initialized, Role: FOLLOWER
I20240503 02:00:36.589820 16110 consensus_queue.cc:260] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P dcfdfa62edc34dab9525e2712d487582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:36.590118 16110 ts_tablet_manager.cc:1398] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P dcfdfa62edc34dab9525e2712d487582: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:36.590296 16110 tablet_bootstrap.cc:492] T 454a24fd6d964d73ae3d5057f90fe36d P dcfdfa62edc34dab9525e2712d487582: Bootstrap starting.
I20240503 02:00:36.590564 15330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11275e009edd4354b0807b7f015023be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:36.590615 16108 tablet_bootstrap.cc:492] T 11275e009edd4354b0807b7f015023be P 620596b893d84291b88b90dc9e317612: Bootstrap starting.
I20240503 02:00:36.591217 16110 tablet_bootstrap.cc:654] T 454a24fd6d964d73ae3d5057f90fe36d P dcfdfa62edc34dab9525e2712d487582: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:36.591884 16108 tablet_bootstrap.cc:654] T 11275e009edd4354b0807b7f015023be P 620596b893d84291b88b90dc9e317612: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:36.593137 16107 raft_consensus.cc:348] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:36.593264 16107 raft_consensus.cc:374] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:36.593298 16107 raft_consensus.cc:729] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Initialized, Role: FOLLOWER
I20240503 02:00:36.593293 16108 tablet_bootstrap.cc:492] T 11275e009edd4354b0807b7f015023be P 620596b893d84291b88b90dc9e317612: No bootstrap required, opened a new log
I20240503 02:00:36.593379 16108 ts_tablet_manager.cc:1367] T 11275e009edd4354b0807b7f015023be P 620596b893d84291b88b90dc9e317612: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:36.593426 16107 consensus_queue.cc:260] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:36.593717 16107 ts_tablet_manager.cc:1398] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P b602f6099cbb4758847e7e9550685372: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:00:36.593804 16108 raft_consensus.cc:348] T 11275e009edd4354b0807b7f015023be P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:36.593894 16108 raft_consensus.cc:374] T 11275e009edd4354b0807b7f015023be P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:36.593902 16107 tablet_bootstrap.cc:492] T 454a24fd6d964d73ae3d5057f90fe36d P b602f6099cbb4758847e7e9550685372: Bootstrap starting.
I20240503 02:00:36.593932 16108 raft_consensus.cc:729] T 11275e009edd4354b0807b7f015023be P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 620596b893d84291b88b90dc9e317612, State: Initialized, Role: FOLLOWER
I20240503 02:00:36.594039 16108 consensus_queue.cc:260] T 11275e009edd4354b0807b7f015023be P 620596b893d84291b88b90dc9e317612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:36.594324 16108 ts_tablet_manager.cc:1398] T 11275e009edd4354b0807b7f015023be P 620596b893d84291b88b90dc9e317612: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:36.594826 16106 tablet_bootstrap.cc:492] T 454a24fd6d964d73ae3d5057f90fe36d P c05f7620bb884392931d3243a0b47a9e: No bootstrap required, opened a new log
I20240503 02:00:36.594913 16107 tablet_bootstrap.cc:654] T 454a24fd6d964d73ae3d5057f90fe36d P b602f6099cbb4758847e7e9550685372: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:36.594916 16106 ts_tablet_manager.cc:1367] T 454a24fd6d964d73ae3d5057f90fe36d P c05f7620bb884392931d3243a0b47a9e: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20240503 02:00:36.594961 16110 tablet_bootstrap.cc:492] T 454a24fd6d964d73ae3d5057f90fe36d P dcfdfa62edc34dab9525e2712d487582: No bootstrap required, opened a new log
I20240503 02:00:36.595042 16110 ts_tablet_manager.cc:1367] T 454a24fd6d964d73ae3d5057f90fe36d P dcfdfa62edc34dab9525e2712d487582: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:00:36.595378 16106 raft_consensus.cc:348] T 454a24fd6d964d73ae3d5057f90fe36d P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:36.595476 16106 raft_consensus.cc:374] T 454a24fd6d964d73ae3d5057f90fe36d P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:36.595443 16110 raft_consensus.cc:348] T 454a24fd6d964d73ae3d5057f90fe36d P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:36.595503 16106 raft_consensus.cc:729] T 454a24fd6d964d73ae3d5057f90fe36d P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Initialized, Role: FOLLOWER
I20240503 02:00:36.595527 16110 raft_consensus.cc:374] T 454a24fd6d964d73ae3d5057f90fe36d P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:36.595556 16110 raft_consensus.cc:729] T 454a24fd6d964d73ae3d5057f90fe36d P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Initialized, Role: FOLLOWER
I20240503 02:00:36.595626 16106 consensus_queue.cc:260] T 454a24fd6d964d73ae3d5057f90fe36d P c05f7620bb884392931d3243a0b47a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:36.595657 16110 consensus_queue.cc:260] T 454a24fd6d964d73ae3d5057f90fe36d P dcfdfa62edc34dab9525e2712d487582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:36.595861 16107 tablet_bootstrap.cc:492] T 454a24fd6d964d73ae3d5057f90fe36d P b602f6099cbb4758847e7e9550685372: No bootstrap required, opened a new log
I20240503 02:00:36.595872 16106 ts_tablet_manager.cc:1398] T 454a24fd6d964d73ae3d5057f90fe36d P c05f7620bb884392931d3243a0b47a9e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:36.595896 16110 ts_tablet_manager.cc:1398] T 454a24fd6d964d73ae3d5057f90fe36d P dcfdfa62edc34dab9525e2712d487582: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:36.595939 16107 ts_tablet_manager.cc:1367] T 454a24fd6d964d73ae3d5057f90fe36d P b602f6099cbb4758847e7e9550685372: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:36.596040 16106 tablet_bootstrap.cc:492] T 11275e009edd4354b0807b7f015023be P c05f7620bb884392931d3243a0b47a9e: Bootstrap starting.
I20240503 02:00:36.596050 16110 tablet_bootstrap.cc:492] T 11275e009edd4354b0807b7f015023be P dcfdfa62edc34dab9525e2712d487582: Bootstrap starting.
I20240503 02:00:36.596328 16107 raft_consensus.cc:348] T 454a24fd6d964d73ae3d5057f90fe36d P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:36.596493 16107 raft_consensus.cc:374] T 454a24fd6d964d73ae3d5057f90fe36d P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:36.596529 16107 raft_consensus.cc:729] T 454a24fd6d964d73ae3d5057f90fe36d P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Initialized, Role: FOLLOWER
I20240503 02:00:36.596637 16107 consensus_queue.cc:260] T 454a24fd6d964d73ae3d5057f90fe36d P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:36.596918 16107 ts_tablet_manager.cc:1398] T 454a24fd6d964d73ae3d5057f90fe36d P b602f6099cbb4758847e7e9550685372: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:36.597193 16110 tablet_bootstrap.cc:654] T 11275e009edd4354b0807b7f015023be P dcfdfa62edc34dab9525e2712d487582: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:36.597342 16106 tablet_bootstrap.cc:654] T 11275e009edd4354b0807b7f015023be P c05f7620bb884392931d3243a0b47a9e: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:36.598527 16110 tablet_bootstrap.cc:492] T 11275e009edd4354b0807b7f015023be P dcfdfa62edc34dab9525e2712d487582: No bootstrap required, opened a new log
I20240503 02:00:36.598623 16110 ts_tablet_manager.cc:1367] T 11275e009edd4354b0807b7f015023be P dcfdfa62edc34dab9525e2712d487582: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:36.598532 16106 tablet_bootstrap.cc:492] T 11275e009edd4354b0807b7f015023be P c05f7620bb884392931d3243a0b47a9e: No bootstrap required, opened a new log
I20240503 02:00:36.598888 16106 ts_tablet_manager.cc:1367] T 11275e009edd4354b0807b7f015023be P c05f7620bb884392931d3243a0b47a9e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:00:36.599278 16110 raft_consensus.cc:348] T 11275e009edd4354b0807b7f015023be P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:36.599448 16110 raft_consensus.cc:374] T 11275e009edd4354b0807b7f015023be P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:36.599498 16110 raft_consensus.cc:729] T 11275e009edd4354b0807b7f015023be P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Initialized, Role: FOLLOWER
I20240503 02:00:36.600054 16106 raft_consensus.cc:348] T 11275e009edd4354b0807b7f015023be P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:36.600219 16106 raft_consensus.cc:374] T 11275e009edd4354b0807b7f015023be P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:36.600271 16106 raft_consensus.cc:729] T 11275e009edd4354b0807b7f015023be P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Initialized, Role: FOLLOWER
I20240503 02:00:36.600545 16106 consensus_queue.cc:260] T 11275e009edd4354b0807b7f015023be P c05f7620bb884392931d3243a0b47a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:36.601109 16106 ts_tablet_manager.cc:1398] T 11275e009edd4354b0807b7f015023be P c05f7620bb884392931d3243a0b47a9e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:36.601069 16110 consensus_queue.cc:260] T 11275e009edd4354b0807b7f015023be P dcfdfa62edc34dab9525e2712d487582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:36.601411 16110 ts_tablet_manager.cc:1398] T 11275e009edd4354b0807b7f015023be P dcfdfa62edc34dab9525e2712d487582: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:00:36.653719 16154 raft_consensus.cc:482] T 11275e009edd4354b0807b7f015023be P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:36.653856 16154 raft_consensus.cc:504] T 11275e009edd4354b0807b7f015023be P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:36.654321 16154 leader_election.cc:288] T 11275e009edd4354b0807b7f015023be P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009), dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817)
I20240503 02:00:36.654726 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11275e009edd4354b0807b7f015023be" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c05f7620bb884392931d3243a0b47a9e" is_pre_election: true
I20240503 02:00:36.654875 15351 raft_consensus.cc:2454] T 11275e009edd4354b0807b7f015023be P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 0.
I20240503 02:00:36.654856 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11275e009edd4354b0807b7f015023be" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcfdfa62edc34dab9525e2712d487582" is_pre_election: true
I20240503 02:00:36.654986 14957 raft_consensus.cc:2454] T 11275e009edd4354b0807b7f015023be P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 0.
I20240503 02:00:36.655208 15155 leader_election.cc:302] T 11275e009edd4354b0807b7f015023be P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, c05f7620bb884392931d3243a0b47a9e; no voters:
I20240503 02:00:36.655391 16154 raft_consensus.cc:2781] T 11275e009edd4354b0807b7f015023be P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:36.655439 16154 raft_consensus.cc:482] T 11275e009edd4354b0807b7f015023be P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:36.655483 16154 raft_consensus.cc:3037] T 11275e009edd4354b0807b7f015023be P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:36.656298 16154 raft_consensus.cc:504] T 11275e009edd4354b0807b7f015023be P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:36.656699 16154 leader_election.cc:288] T 11275e009edd4354b0807b7f015023be P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 1 election: Requested vote from peers c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009), dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817)
I20240503 02:00:36.657035 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11275e009edd4354b0807b7f015023be" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:36.657090 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11275e009edd4354b0807b7f015023be" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:36.657146 14957 raft_consensus.cc:3037] T 11275e009edd4354b0807b7f015023be P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:36.657234 15351 raft_consensus.cc:3037] T 11275e009edd4354b0807b7f015023be P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:36.658088 14957 raft_consensus.cc:2454] T 11275e009edd4354b0807b7f015023be P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 1.
I20240503 02:00:36.658092 15351 raft_consensus.cc:2454] T 11275e009edd4354b0807b7f015023be P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 1.
I20240503 02:00:36.658403 15155 leader_election.cc:302] T 11275e009edd4354b0807b7f015023be P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, c05f7620bb884392931d3243a0b47a9e; no voters:
I20240503 02:00:36.658567 16154 raft_consensus.cc:2781] T 11275e009edd4354b0807b7f015023be P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:36.658638 16154 raft_consensus.cc:686] T 11275e009edd4354b0807b7f015023be P 620596b893d84291b88b90dc9e317612 [term 1 LEADER]: Becoming Leader. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:36.658767 16154 consensus_queue.cc:237] T 11275e009edd4354b0807b7f015023be P 620596b893d84291b88b90dc9e317612 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:36.660482 14816 catalog_manager.cc:5484] T 11275e009edd4354b0807b7f015023be P 620596b893d84291b88b90dc9e317612 reported cstate change: term changed from 0 to 1, leader changed from <none> to 620596b893d84291b88b90dc9e317612 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "620596b893d84291b88b90dc9e317612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:36.715212 16154 raft_consensus.cc:482] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:36.715353 16154 raft_consensus.cc:504] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:36.715773 16154 leader_election.cc:288] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207), dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817)
I20240503 02:00:36.716353 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ceeb7b3c4146cd8a2e18b3e9ae987c" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcfdfa62edc34dab9525e2712d487582" is_pre_election: true
I20240503 02:00:36.716353 15088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ceeb7b3c4146cd8a2e18b3e9ae987c" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b602f6099cbb4758847e7e9550685372" is_pre_election: true
I20240503 02:00:36.716531 14957 raft_consensus.cc:2454] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 0.
I20240503 02:00:36.716531 15088 raft_consensus.cc:2454] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 0.
I20240503 02:00:36.716866 15154 leader_election.cc:302] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, b602f6099cbb4758847e7e9550685372, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:36.717056 16154 raft_consensus.cc:2781] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:36.717113 16154 raft_consensus.cc:482] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:36.717137 16154 raft_consensus.cc:3037] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:36.717964 16154 raft_consensus.cc:504] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:36.718307 16154 leader_election.cc:288] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 1 election: Requested vote from peers b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207), dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817)
I20240503 02:00:36.718695 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ceeb7b3c4146cd8a2e18b3e9ae987c" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:36.718744 15088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ceeb7b3c4146cd8a2e18b3e9ae987c" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:36.718811 14957 raft_consensus.cc:3037] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:36.718849 15088 raft_consensus.cc:3037] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:36.719538 14957 raft_consensus.cc:2454] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 1.
I20240503 02:00:36.719553 15088 raft_consensus.cc:2454] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 1.
I20240503 02:00:36.719831 15154 leader_election.cc:302] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, b602f6099cbb4758847e7e9550685372; no voters:
I20240503 02:00:36.719995 16154 raft_consensus.cc:2781] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:36.720086 16154 raft_consensus.cc:686] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P 620596b893d84291b88b90dc9e317612 [term 1 LEADER]: Becoming Leader. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:36.720218 16154 consensus_queue.cc:237] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P 620596b893d84291b88b90dc9e317612 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:36.721920 14816 catalog_manager.cc:5484] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P 620596b893d84291b88b90dc9e317612 reported cstate change: term changed from 0 to 1, leader changed from <none> to 620596b893d84291b88b90dc9e317612 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "620596b893d84291b88b90dc9e317612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:36.723662 15966 raft_consensus.cc:482] T 454a24fd6d964d73ae3d5057f90fe36d P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:36.723785 15966 raft_consensus.cc:504] T 454a24fd6d964d73ae3d5057f90fe36d P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:36.724184 15966 leader_election.cc:288] T 454a24fd6d964d73ae3d5057f90fe36d P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207), dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817)
I20240503 02:00:36.724645 15088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "454a24fd6d964d73ae3d5057f90fe36d" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b602f6099cbb4758847e7e9550685372" is_pre_election: true
I20240503 02:00:36.724673 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "454a24fd6d964d73ae3d5057f90fe36d" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcfdfa62edc34dab9525e2712d487582" is_pre_election: true
I20240503 02:00:36.724782 14957 raft_consensus.cc:2454] T 454a24fd6d964d73ae3d5057f90fe36d P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 0.
I20240503 02:00:36.724787 15088 raft_consensus.cc:2454] T 454a24fd6d964d73ae3d5057f90fe36d P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 0.
I20240503 02:00:36.725075 15285 leader_election.cc:302] T 454a24fd6d964d73ae3d5057f90fe36d P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b602f6099cbb4758847e7e9550685372, c05f7620bb884392931d3243a0b47a9e; no voters:
I20240503 02:00:36.725239 15966 raft_consensus.cc:2781] T 454a24fd6d964d73ae3d5057f90fe36d P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:36.725297 15966 raft_consensus.cc:482] T 454a24fd6d964d73ae3d5057f90fe36d P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:36.725334 15966 raft_consensus.cc:3037] T 454a24fd6d964d73ae3d5057f90fe36d P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:36.726120 15966 raft_consensus.cc:504] T 454a24fd6d964d73ae3d5057f90fe36d P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:36.726473 15966 leader_election.cc:288] T 454a24fd6d964d73ae3d5057f90fe36d P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 1 election: Requested vote from peers b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207), dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817)
I20240503 02:00:36.726842 15088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "454a24fd6d964d73ae3d5057f90fe36d" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:36.726856 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "454a24fd6d964d73ae3d5057f90fe36d" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:36.726958 15088 raft_consensus.cc:3037] T 454a24fd6d964d73ae3d5057f90fe36d P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:36.726958 14957 raft_consensus.cc:3037] T 454a24fd6d964d73ae3d5057f90fe36d P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:36.727694 15088 raft_consensus.cc:2454] T 454a24fd6d964d73ae3d5057f90fe36d P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 1.
I20240503 02:00:36.727983 15285 leader_election.cc:302] T 454a24fd6d964d73ae3d5057f90fe36d P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b602f6099cbb4758847e7e9550685372, c05f7620bb884392931d3243a0b47a9e; no voters:
I20240503 02:00:36.728140 15966 raft_consensus.cc:2781] T 454a24fd6d964d73ae3d5057f90fe36d P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:36.728204 15966 raft_consensus.cc:686] T 454a24fd6d964d73ae3d5057f90fe36d P c05f7620bb884392931d3243a0b47a9e [term 1 LEADER]: Becoming Leader. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:36.728332 15966 consensus_queue.cc:237] T 454a24fd6d964d73ae3d5057f90fe36d P c05f7620bb884392931d3243a0b47a9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:36.728444 14957 raft_consensus.cc:2454] T 454a24fd6d964d73ae3d5057f90fe36d P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 1.
I20240503 02:00:36.730087 14816 catalog_manager.cc:5484] T 454a24fd6d964d73ae3d5057f90fe36d P c05f7620bb884392931d3243a0b47a9e reported cstate change: term changed from 0 to 1, leader changed from <none> to c05f7620bb884392931d3243a0b47a9e (127.0.99.132). New cstate: current_term: 1 leader_uuid: "c05f7620bb884392931d3243a0b47a9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } }
W20240503 02:00:36.850587 16232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:36.850869 16232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:36.854300 16232 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240503 02:00:36.966987 15088 raft_consensus.cc:1261] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Refusing update from remote peer 620596b893d84291b88b90dc9e317612: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:36.967660 16053 consensus_queue.cc:1035] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P 620596b893d84291b88b90dc9e317612 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:36.968703 14957 raft_consensus.cc:1261] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Refusing update from remote peer 620596b893d84291b88b90dc9e317612: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:36.969303 15351 raft_consensus.cc:1261] T 11275e009edd4354b0807b7f015023be P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Refusing update from remote peer 620596b893d84291b88b90dc9e317612: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:36.969455 14958 raft_consensus.cc:1261] T 11275e009edd4354b0807b7f015023be P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Refusing update from remote peer 620596b893d84291b88b90dc9e317612: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:36.970203 15088 raft_consensus.cc:1261] T 454a24fd6d964d73ae3d5057f90fe36d P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Refusing update from remote peer c05f7620bb884392931d3243a0b47a9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:36.970224 14958 raft_consensus.cc:1261] T 454a24fd6d964d73ae3d5057f90fe36d P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Refusing update from remote peer c05f7620bb884392931d3243a0b47a9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:36.970614 16088 consensus_queue.cc:1035] T 454a24fd6d964d73ae3d5057f90fe36d P c05f7620bb884392931d3243a0b47a9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:36.971426 16088 consensus_queue.cc:1035] T 454a24fd6d964d73ae3d5057f90fe36d P c05f7620bb884392931d3243a0b47a9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:36.973320 16154 consensus_queue.cc:1035] T 11275e009edd4354b0807b7f015023be P 620596b893d84291b88b90dc9e317612 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:36.973733 16154 consensus_queue.cc:1035] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P 620596b893d84291b88b90dc9e317612 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:36.974680 16154 consensus_queue.cc:1035] T 11275e009edd4354b0807b7f015023be P 620596b893d84291b88b90dc9e317612 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 133.46 ms
time per row: 0.0667301 ms
I20240503 02:00:37.037024 14815 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39948:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:37.043005 15199 tablet_service.cc:1467] Processing CreateTablet for tablet 5da91b7af68e4cf597dc89a017ee0197 (DEFAULT_TABLE table=rebalance_test_table_5 [id=34bd3308977d471ca541e993ff99f49b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:37.043083 15069 tablet_service.cc:1467] Processing CreateTablet for tablet 5da91b7af68e4cf597dc89a017ee0197 (DEFAULT_TABLE table=rebalance_test_table_5 [id=34bd3308977d471ca541e993ff99f49b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:37.043344 15199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5da91b7af68e4cf597dc89a017ee0197. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:37.043359 15069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5da91b7af68e4cf597dc89a017ee0197. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:37.043284 15330 tablet_service.cc:1467] Processing CreateTablet for tablet 5da91b7af68e4cf597dc89a017ee0197 (DEFAULT_TABLE table=rebalance_test_table_5 [id=34bd3308977d471ca541e993ff99f49b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:37.043536 15330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5da91b7af68e4cf597dc89a017ee0197. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:37.045295 16106 tablet_bootstrap.cc:492] T 5da91b7af68e4cf597dc89a017ee0197 P c05f7620bb884392931d3243a0b47a9e: Bootstrap starting.
I20240503 02:00:37.045490 16108 tablet_bootstrap.cc:492] T 5da91b7af68e4cf597dc89a017ee0197 P 620596b893d84291b88b90dc9e317612: Bootstrap starting.
I20240503 02:00:37.045518 16107 tablet_bootstrap.cc:492] T 5da91b7af68e4cf597dc89a017ee0197 P b602f6099cbb4758847e7e9550685372: Bootstrap starting.
I20240503 02:00:37.046229 16106 tablet_bootstrap.cc:654] T 5da91b7af68e4cf597dc89a017ee0197 P c05f7620bb884392931d3243a0b47a9e: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:37.046238 16108 tablet_bootstrap.cc:654] T 5da91b7af68e4cf597dc89a017ee0197 P 620596b893d84291b88b90dc9e317612: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:37.046245 16107 tablet_bootstrap.cc:654] T 5da91b7af68e4cf597dc89a017ee0197 P b602f6099cbb4758847e7e9550685372: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:37.047497 15199 tablet_service.cc:1467] Processing CreateTablet for tablet 5736c284a85540a1bbab2be6df672fc1 (DEFAULT_TABLE table=rebalance_test_table_5 [id=34bd3308977d471ca541e993ff99f49b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:37.047535 14936 tablet_service.cc:1467] Processing CreateTablet for tablet 6b0c8b72e1884529a8a9b83d2a734d71 (DEFAULT_TABLE table=rebalance_test_table_5 [id=34bd3308977d471ca541e993ff99f49b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:37.047761 15199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5736c284a85540a1bbab2be6df672fc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:37.047789 14936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b0c8b72e1884529a8a9b83d2a734d71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:37.048105 16106 tablet_bootstrap.cc:492] T 5da91b7af68e4cf597dc89a017ee0197 P c05f7620bb884392931d3243a0b47a9e: No bootstrap required, opened a new log
I20240503 02:00:37.048228 16106 ts_tablet_manager.cc:1367] T 5da91b7af68e4cf597dc89a017ee0197 P c05f7620bb884392931d3243a0b47a9e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:00:37.048656 15200 tablet_service.cc:1467] Processing CreateTablet for tablet 6b0c8b72e1884529a8a9b83d2a734d71 (DEFAULT_TABLE table=rebalance_test_table_5 [id=34bd3308977d471ca541e993ff99f49b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:37.048751 16106 raft_consensus.cc:348] T 5da91b7af68e4cf597dc89a017ee0197 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:37.048856 16106 raft_consensus.cc:374] T 5da91b7af68e4cf597dc89a017ee0197 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:37.048888 15200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b0c8b72e1884529a8a9b83d2a734d71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:37.048885 16106 raft_consensus.cc:729] T 5da91b7af68e4cf597dc89a017ee0197 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Initialized, Role: FOLLOWER
I20240503 02:00:37.049038 16106 consensus_queue.cc:260] T 5da91b7af68e4cf597dc89a017ee0197 P c05f7620bb884392931d3243a0b47a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:37.049350 16106 ts_tablet_manager.cc:1398] T 5da91b7af68e4cf597dc89a017ee0197 P c05f7620bb884392931d3243a0b47a9e: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 02:00:37.049991 15068 tablet_service.cc:1467] Processing CreateTablet for tablet 6b0c8b72e1884529a8a9b83d2a734d71 (DEFAULT_TABLE table=rebalance_test_table_5 [id=34bd3308977d471ca541e993ff99f49b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:37.050226 15068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b0c8b72e1884529a8a9b83d2a734d71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:37.051009 15330 tablet_service.cc:1467] Processing CreateTablet for tablet 5736c284a85540a1bbab2be6df672fc1 (DEFAULT_TABLE table=rebalance_test_table_5 [id=34bd3308977d471ca541e993ff99f49b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:37.051259 15330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5736c284a85540a1bbab2be6df672fc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:37.052215 16110 tablet_bootstrap.cc:492] T 6b0c8b72e1884529a8a9b83d2a734d71 P dcfdfa62edc34dab9525e2712d487582: Bootstrap starting.
I20240503 02:00:37.053177 16110 tablet_bootstrap.cc:654] T 6b0c8b72e1884529a8a9b83d2a734d71 P dcfdfa62edc34dab9525e2712d487582: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:37.053176 16108 tablet_bootstrap.cc:492] T 5da91b7af68e4cf597dc89a017ee0197 P 620596b893d84291b88b90dc9e317612: No bootstrap required, opened a new log
I20240503 02:00:37.053269 16108 ts_tablet_manager.cc:1367] T 5da91b7af68e4cf597dc89a017ee0197 P 620596b893d84291b88b90dc9e317612: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240503 02:00:37.053444 16106 tablet_bootstrap.cc:492] T 5736c284a85540a1bbab2be6df672fc1 P c05f7620bb884392931d3243a0b47a9e: Bootstrap starting.
I20240503 02:00:37.053735 16108 raft_consensus.cc:348] T 5da91b7af68e4cf597dc89a017ee0197 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:37.053850 16108 raft_consensus.cc:374] T 5da91b7af68e4cf597dc89a017ee0197 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:37.053879 16108 raft_consensus.cc:729] T 5da91b7af68e4cf597dc89a017ee0197 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 620596b893d84291b88b90dc9e317612, State: Initialized, Role: FOLLOWER
I20240503 02:00:37.054000 16108 consensus_queue.cc:260] T 5da91b7af68e4cf597dc89a017ee0197 P 620596b893d84291b88b90dc9e317612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:37.054283 16108 ts_tablet_manager.cc:1398] T 5da91b7af68e4cf597dc89a017ee0197 P 620596b893d84291b88b90dc9e317612: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:37.054387 16106 tablet_bootstrap.cc:654] T 5736c284a85540a1bbab2be6df672fc1 P c05f7620bb884392931d3243a0b47a9e: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:37.054457 16108 tablet_bootstrap.cc:492] T 6b0c8b72e1884529a8a9b83d2a734d71 P 620596b893d84291b88b90dc9e317612: Bootstrap starting.
I20240503 02:00:37.054823 16107 tablet_bootstrap.cc:492] T 5da91b7af68e4cf597dc89a017ee0197 P b602f6099cbb4758847e7e9550685372: No bootstrap required, opened a new log
I20240503 02:00:37.054914 16107 ts_tablet_manager.cc:1367] T 5da91b7af68e4cf597dc89a017ee0197 P b602f6099cbb4758847e7e9550685372: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240503 02:00:37.055400 16107 raft_consensus.cc:348] T 5da91b7af68e4cf597dc89a017ee0197 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:37.055510 16107 raft_consensus.cc:374] T 5da91b7af68e4cf597dc89a017ee0197 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:37.055546 16107 raft_consensus.cc:729] T 5da91b7af68e4cf597dc89a017ee0197 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Initialized, Role: FOLLOWER
I20240503 02:00:37.055662 16107 consensus_queue.cc:260] T 5da91b7af68e4cf597dc89a017ee0197 P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:37.055708 16108 tablet_bootstrap.cc:654] T 6b0c8b72e1884529a8a9b83d2a734d71 P 620596b893d84291b88b90dc9e317612: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:37.055929 16107 ts_tablet_manager.cc:1398] T 5da91b7af68e4cf597dc89a017ee0197 P b602f6099cbb4758847e7e9550685372: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:37.056509 16106 tablet_bootstrap.cc:492] T 5736c284a85540a1bbab2be6df672fc1 P c05f7620bb884392931d3243a0b47a9e: No bootstrap required, opened a new log
I20240503 02:00:37.056598 16106 ts_tablet_manager.cc:1367] T 5736c284a85540a1bbab2be6df672fc1 P c05f7620bb884392931d3243a0b47a9e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:37.056557 15069 tablet_service.cc:1467] Processing CreateTablet for tablet 5736c284a85540a1bbab2be6df672fc1 (DEFAULT_TABLE table=rebalance_test_table_5 [id=34bd3308977d471ca541e993ff99f49b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:37.056797 15069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5736c284a85540a1bbab2be6df672fc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:37.057044 16106 raft_consensus.cc:348] T 5736c284a85540a1bbab2be6df672fc1 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:37.057152 16106 raft_consensus.cc:374] T 5736c284a85540a1bbab2be6df672fc1 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:37.057180 16106 raft_consensus.cc:729] T 5736c284a85540a1bbab2be6df672fc1 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Initialized, Role: FOLLOWER
I20240503 02:00:37.057301 16106 consensus_queue.cc:260] T 5736c284a85540a1bbab2be6df672fc1 P c05f7620bb884392931d3243a0b47a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:37.057576 16106 ts_tablet_manager.cc:1398] T 5736c284a85540a1bbab2be6df672fc1 P c05f7620bb884392931d3243a0b47a9e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:37.057739 16107 tablet_bootstrap.cc:492] T 6b0c8b72e1884529a8a9b83d2a734d71 P b602f6099cbb4758847e7e9550685372: Bootstrap starting.
I20240503 02:00:37.058809 16107 tablet_bootstrap.cc:654] T 6b0c8b72e1884529a8a9b83d2a734d71 P b602f6099cbb4758847e7e9550685372: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:37.059834 16108 tablet_bootstrap.cc:492] T 6b0c8b72e1884529a8a9b83d2a734d71 P 620596b893d84291b88b90dc9e317612: No bootstrap required, opened a new log
I20240503 02:00:37.059902 16107 tablet_bootstrap.cc:492] T 6b0c8b72e1884529a8a9b83d2a734d71 P b602f6099cbb4758847e7e9550685372: No bootstrap required, opened a new log
I20240503 02:00:37.059937 16110 tablet_bootstrap.cc:492] T 6b0c8b72e1884529a8a9b83d2a734d71 P dcfdfa62edc34dab9525e2712d487582: No bootstrap required, opened a new log
I20240503 02:00:37.059979 16107 ts_tablet_manager.cc:1367] T 6b0c8b72e1884529a8a9b83d2a734d71 P b602f6099cbb4758847e7e9550685372: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:37.060014 16110 ts_tablet_manager.cc:1367] T 6b0c8b72e1884529a8a9b83d2a734d71 P dcfdfa62edc34dab9525e2712d487582: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 02:00:37.060163 16108 ts_tablet_manager.cc:1367] T 6b0c8b72e1884529a8a9b83d2a734d71 P 620596b893d84291b88b90dc9e317612: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:00:37.060467 16107 raft_consensus.cc:348] T 6b0c8b72e1884529a8a9b83d2a734d71 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:37.060475 16110 raft_consensus.cc:348] T 6b0c8b72e1884529a8a9b83d2a734d71 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:37.060570 16107 raft_consensus.cc:374] T 6b0c8b72e1884529a8a9b83d2a734d71 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:37.060570 16110 raft_consensus.cc:374] T 6b0c8b72e1884529a8a9b83d2a734d71 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:37.060599 16110 raft_consensus.cc:729] T 6b0c8b72e1884529a8a9b83d2a734d71 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Initialized, Role: FOLLOWER
I20240503 02:00:37.060599 16107 raft_consensus.cc:729] T 6b0c8b72e1884529a8a9b83d2a734d71 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Initialized, Role: FOLLOWER
I20240503 02:00:37.060652 16108 raft_consensus.cc:348] T 6b0c8b72e1884529a8a9b83d2a734d71 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:37.060719 16110 consensus_queue.cc:260] T 6b0c8b72e1884529a8a9b83d2a734d71 P dcfdfa62edc34dab9525e2712d487582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:37.060770 16108 raft_consensus.cc:374] T 6b0c8b72e1884529a8a9b83d2a734d71 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:37.060802 16108 raft_consensus.cc:729] T 6b0c8b72e1884529a8a9b83d2a734d71 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 620596b893d84291b88b90dc9e317612, State: Initialized, Role: FOLLOWER
I20240503 02:00:37.060794 16107 consensus_queue.cc:260] T 6b0c8b72e1884529a8a9b83d2a734d71 P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:37.060942 16108 consensus_queue.cc:260] T 6b0c8b72e1884529a8a9b83d2a734d71 P 620596b893d84291b88b90dc9e317612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:37.060989 16110 ts_tablet_manager.cc:1398] T 6b0c8b72e1884529a8a9b83d2a734d71 P dcfdfa62edc34dab9525e2712d487582: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:37.061156 16107 ts_tablet_manager.cc:1398] T 6b0c8b72e1884529a8a9b83d2a734d71 P b602f6099cbb4758847e7e9550685372: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:37.061307 16107 tablet_bootstrap.cc:492] T 5736c284a85540a1bbab2be6df672fc1 P b602f6099cbb4758847e7e9550685372: Bootstrap starting.
I20240503 02:00:37.061313 16108 ts_tablet_manager.cc:1398] T 6b0c8b72e1884529a8a9b83d2a734d71 P 620596b893d84291b88b90dc9e317612: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:37.061479 16108 tablet_bootstrap.cc:492] T 5736c284a85540a1bbab2be6df672fc1 P 620596b893d84291b88b90dc9e317612: Bootstrap starting.
I20240503 02:00:37.062242 16107 tablet_bootstrap.cc:654] T 5736c284a85540a1bbab2be6df672fc1 P b602f6099cbb4758847e7e9550685372: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:37.062350 16108 tablet_bootstrap.cc:654] T 5736c284a85540a1bbab2be6df672fc1 P 620596b893d84291b88b90dc9e317612: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:37.063413 16107 tablet_bootstrap.cc:492] T 5736c284a85540a1bbab2be6df672fc1 P b602f6099cbb4758847e7e9550685372: No bootstrap required, opened a new log
I20240503 02:00:37.063498 16107 ts_tablet_manager.cc:1367] T 5736c284a85540a1bbab2be6df672fc1 P b602f6099cbb4758847e7e9550685372: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:37.063660 16108 tablet_bootstrap.cc:492] T 5736c284a85540a1bbab2be6df672fc1 P 620596b893d84291b88b90dc9e317612: No bootstrap required, opened a new log
I20240503 02:00:37.063727 16108 ts_tablet_manager.cc:1367] T 5736c284a85540a1bbab2be6df672fc1 P 620596b893d84291b88b90dc9e317612: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:37.063947 16107 raft_consensus.cc:348] T 5736c284a85540a1bbab2be6df672fc1 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:37.064065 16107 raft_consensus.cc:374] T 5736c284a85540a1bbab2be6df672fc1 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:37.064096 16107 raft_consensus.cc:729] T 5736c284a85540a1bbab2be6df672fc1 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Initialized, Role: FOLLOWER
I20240503 02:00:37.064131 16108 raft_consensus.cc:348] T 5736c284a85540a1bbab2be6df672fc1 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:37.064215 16108 raft_consensus.cc:374] T 5736c284a85540a1bbab2be6df672fc1 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:37.064244 16108 raft_consensus.cc:729] T 5736c284a85540a1bbab2be6df672fc1 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 620596b893d84291b88b90dc9e317612, State: Initialized, Role: FOLLOWER
I20240503 02:00:37.064220 16107 consensus_queue.cc:260] T 5736c284a85540a1bbab2be6df672fc1 P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:37.064360 16108 consensus_queue.cc:260] T 5736c284a85540a1bbab2be6df672fc1 P 620596b893d84291b88b90dc9e317612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:37.064575 16107 ts_tablet_manager.cc:1398] T 5736c284a85540a1bbab2be6df672fc1 P b602f6099cbb4758847e7e9550685372: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:37.064728 16108 ts_tablet_manager.cc:1398] T 5736c284a85540a1bbab2be6df672fc1 P 620596b893d84291b88b90dc9e317612: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:37.068230 16264 raft_consensus.cc:482] T 6b0c8b72e1884529a8a9b83d2a734d71 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:37.068346 16264 raft_consensus.cc:504] T 6b0c8b72e1884529a8a9b83d2a734d71 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:37.068761 16264 leader_election.cc:288] T 6b0c8b72e1884529a8a9b83d2a734d71 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
I20240503 02:00:37.069226 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b0c8b72e1884529a8a9b83d2a734d71" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcfdfa62edc34dab9525e2712d487582" is_pre_election: true
I20240503 02:00:37.069219 15088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b0c8b72e1884529a8a9b83d2a734d71" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b602f6099cbb4758847e7e9550685372" is_pre_election: true
I20240503 02:00:37.069376 15088 raft_consensus.cc:2454] T 6b0c8b72e1884529a8a9b83d2a734d71 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 0.
I20240503 02:00:37.069376 14957 raft_consensus.cc:2454] T 6b0c8b72e1884529a8a9b83d2a734d71 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 0.
I20240503 02:00:37.069711 15154 leader_election.cc:302] T 6b0c8b72e1884529a8a9b83d2a734d71 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, b602f6099cbb4758847e7e9550685372; no voters:
I20240503 02:00:37.069921 16264 raft_consensus.cc:2781] T 6b0c8b72e1884529a8a9b83d2a734d71 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:37.069979 16264 raft_consensus.cc:482] T 6b0c8b72e1884529a8a9b83d2a734d71 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:37.070004 16264 raft_consensus.cc:3037] T 6b0c8b72e1884529a8a9b83d2a734d71 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:37.070798 16264 raft_consensus.cc:504] T 6b0c8b72e1884529a8a9b83d2a734d71 P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:37.071162 16264 leader_election.cc:288] T 6b0c8b72e1884529a8a9b83d2a734d71 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 1 election: Requested vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
I20240503 02:00:37.071627 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b0c8b72e1884529a8a9b83d2a734d71" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:37.071627 15088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b0c8b72e1884529a8a9b83d2a734d71" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:37.071758 14957 raft_consensus.cc:3037] T 6b0c8b72e1884529a8a9b83d2a734d71 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:37.071758 15088 raft_consensus.cc:3037] T 6b0c8b72e1884529a8a9b83d2a734d71 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:37.072796 14957 raft_consensus.cc:2454] T 6b0c8b72e1884529a8a9b83d2a734d71 P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 1.
I20240503 02:00:37.072796 15088 raft_consensus.cc:2454] T 6b0c8b72e1884529a8a9b83d2a734d71 P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 1.
I20240503 02:00:37.073164 15153 leader_election.cc:302] T 6b0c8b72e1884529a8a9b83d2a734d71 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:37.073385 16264 raft_consensus.cc:2781] T 6b0c8b72e1884529a8a9b83d2a734d71 P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:37.073470 16264 raft_consensus.cc:686] T 6b0c8b72e1884529a8a9b83d2a734d71 P 620596b893d84291b88b90dc9e317612 [term 1 LEADER]: Becoming Leader. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:37.073607 16264 consensus_queue.cc:237] T 6b0c8b72e1884529a8a9b83d2a734d71 P 620596b893d84291b88b90dc9e317612 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:37.075350 14811 catalog_manager.cc:5484] T 6b0c8b72e1884529a8a9b83d2a734d71 P 620596b893d84291b88b90dc9e317612 reported cstate change: term changed from 0 to 1, leader changed from <none> to 620596b893d84291b88b90dc9e317612 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "620596b893d84291b88b90dc9e317612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:37.105468 16088 raft_consensus.cc:482] T 5da91b7af68e4cf597dc89a017ee0197 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:37.105597 16088 raft_consensus.cc:504] T 5da91b7af68e4cf597dc89a017ee0197 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:37.105986 16088 leader_election.cc:288] T 5da91b7af68e4cf597dc89a017ee0197 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
I20240503 02:00:37.106408 15220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5da91b7af68e4cf597dc89a017ee0197" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "620596b893d84291b88b90dc9e317612" is_pre_election: true
I20240503 02:00:37.106551 15220 raft_consensus.cc:2454] T 5da91b7af68e4cf597dc89a017ee0197 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 0.
I20240503 02:00:37.106526 15088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5da91b7af68e4cf597dc89a017ee0197" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b602f6099cbb4758847e7e9550685372" is_pre_election: true
I20240503 02:00:37.106710 15088 raft_consensus.cc:2454] T 5da91b7af68e4cf597dc89a017ee0197 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 0.
I20240503 02:00:37.107017 15285 leader_election.cc:302] T 5da91b7af68e4cf597dc89a017ee0197 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b602f6099cbb4758847e7e9550685372, c05f7620bb884392931d3243a0b47a9e; no voters:
I20240503 02:00:37.107187 16088 raft_consensus.cc:2781] T 5da91b7af68e4cf597dc89a017ee0197 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:37.107277 16088 raft_consensus.cc:482] T 5da91b7af68e4cf597dc89a017ee0197 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:37.107306 16088 raft_consensus.cc:3037] T 5da91b7af68e4cf597dc89a017ee0197 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:37.108140 16088 raft_consensus.cc:504] T 5da91b7af68e4cf597dc89a017ee0197 P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:37.108584 16088 leader_election.cc:288] T 5da91b7af68e4cf597dc89a017ee0197 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 1 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
I20240503 02:00:37.108989 15088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5da91b7af68e4cf597dc89a017ee0197" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:37.109009 15220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5da91b7af68e4cf597dc89a017ee0197" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:37.109127 15088 raft_consensus.cc:3037] T 5da91b7af68e4cf597dc89a017ee0197 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:37.109126 15220 raft_consensus.cc:3037] T 5da91b7af68e4cf597dc89a017ee0197 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:37.109910 15088 raft_consensus.cc:2454] T 5da91b7af68e4cf597dc89a017ee0197 P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 1.
I20240503 02:00:37.110265 15285 leader_election.cc:302] T 5da91b7af68e4cf597dc89a017ee0197 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b602f6099cbb4758847e7e9550685372, c05f7620bb884392931d3243a0b47a9e; no voters:
I20240503 02:00:37.110469 16088 raft_consensus.cc:2781] T 5da91b7af68e4cf597dc89a017ee0197 P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:37.110548 16088 raft_consensus.cc:686] T 5da91b7af68e4cf597dc89a017ee0197 P c05f7620bb884392931d3243a0b47a9e [term 1 LEADER]: Becoming Leader. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:37.110693 16088 consensus_queue.cc:237] T 5da91b7af68e4cf597dc89a017ee0197 P c05f7620bb884392931d3243a0b47a9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:37.112526 14811 catalog_manager.cc:5484] T 5da91b7af68e4cf597dc89a017ee0197 P c05f7620bb884392931d3243a0b47a9e reported cstate change: term changed from 0 to 1, leader changed from <none> to c05f7620bb884392931d3243a0b47a9e (127.0.99.132). New cstate: current_term: 1 leader_uuid: "c05f7620bb884392931d3243a0b47a9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:37.129220 15220 raft_consensus.cc:2454] T 5da91b7af68e4cf597dc89a017ee0197 P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 1.
I20240503 02:00:37.287463 16088 raft_consensus.cc:482] T 5736c284a85540a1bbab2be6df672fc1 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:37.287623 16088 raft_consensus.cc:504] T 5736c284a85540a1bbab2be6df672fc1 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:37.288080 16088 leader_election.cc:288] T 5736c284a85540a1bbab2be6df672fc1 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207), 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437)
I20240503 02:00:37.288520 15088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5736c284a85540a1bbab2be6df672fc1" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b602f6099cbb4758847e7e9550685372" is_pre_election: true
I20240503 02:00:37.288537 15220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5736c284a85540a1bbab2be6df672fc1" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "620596b893d84291b88b90dc9e317612" is_pre_election: true
I20240503 02:00:37.288666 15088 raft_consensus.cc:2454] T 5736c284a85540a1bbab2be6df672fc1 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 0.
I20240503 02:00:37.288683 15220 raft_consensus.cc:2454] T 5736c284a85540a1bbab2be6df672fc1 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 0.
I20240503 02:00:37.288980 15286 leader_election.cc:302] T 5736c284a85540a1bbab2be6df672fc1 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, c05f7620bb884392931d3243a0b47a9e; no voters:
I20240503 02:00:37.289184 16088 raft_consensus.cc:2781] T 5736c284a85540a1bbab2be6df672fc1 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:37.289250 16088 raft_consensus.cc:482] T 5736c284a85540a1bbab2be6df672fc1 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:37.289289 16088 raft_consensus.cc:3037] T 5736c284a85540a1bbab2be6df672fc1 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:37.290328 16088 raft_consensus.cc:504] T 5736c284a85540a1bbab2be6df672fc1 P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:37.291002 16088 leader_election.cc:288] T 5736c284a85540a1bbab2be6df672fc1 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 1 election: Requested vote from peers b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207), 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437)
I20240503 02:00:37.291050 15220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5736c284a85540a1bbab2be6df672fc1" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:37.291060 15088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5736c284a85540a1bbab2be6df672fc1" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:37.291185 15088 raft_consensus.cc:3037] T 5736c284a85540a1bbab2be6df672fc1 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:37.291260 15220 raft_consensus.cc:3037] T 5736c284a85540a1bbab2be6df672fc1 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:37.292145 15088 raft_consensus.cc:2454] T 5736c284a85540a1bbab2be6df672fc1 P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 1.
I20240503 02:00:37.292145 15220 raft_consensus.cc:2454] T 5736c284a85540a1bbab2be6df672fc1 P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 1.
I20240503 02:00:37.292502 15286 leader_election.cc:302] T 5736c284a85540a1bbab2be6df672fc1 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, c05f7620bb884392931d3243a0b47a9e; no voters:
I20240503 02:00:37.292737 16088 raft_consensus.cc:2781] T 5736c284a85540a1bbab2be6df672fc1 P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:37.292829 16088 raft_consensus.cc:686] T 5736c284a85540a1bbab2be6df672fc1 P c05f7620bb884392931d3243a0b47a9e [term 1 LEADER]: Becoming Leader. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:37.292975 16088 consensus_queue.cc:237] T 5736c284a85540a1bbab2be6df672fc1 P c05f7620bb884392931d3243a0b47a9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:37.294803 14811 catalog_manager.cc:5484] T 5736c284a85540a1bbab2be6df672fc1 P c05f7620bb884392931d3243a0b47a9e reported cstate change: term changed from 0 to 1, leader changed from <none> to c05f7620bb884392931d3243a0b47a9e (127.0.99.132). New cstate: current_term: 1 leader_uuid: "c05f7620bb884392931d3243a0b47a9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: HEALTHY } } }
W20240503 02:00:37.472501 16279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:37.472843 16279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:37.476534 16279 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240503 02:00:37.512671 16264 consensus_queue.cc:1035] T 6b0c8b72e1884529a8a9b83d2a734d71 P 620596b893d84291b88b90dc9e317612 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:37.526839 16304 consensus_queue.cc:1035] T 6b0c8b72e1884529a8a9b83d2a734d71 P 620596b893d84291b88b90dc9e317612 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:37.531474 16088 consensus_queue.cc:1035] T 5da91b7af68e4cf597dc89a017ee0197 P c05f7620bb884392931d3243a0b47a9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:37.552582 15958 consensus_queue.cc:1035] T 5da91b7af68e4cf597dc89a017ee0197 P c05f7620bb884392931d3243a0b47a9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:37.572029 15088 raft_consensus.cc:1261] T 5736c284a85540a1bbab2be6df672fc1 P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Refusing update from remote peer c05f7620bb884392931d3243a0b47a9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:37.572492 15220 raft_consensus.cc:1261] T 5736c284a85540a1bbab2be6df672fc1 P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Refusing update from remote peer c05f7620bb884392931d3243a0b47a9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:37.572552 16088 consensus_queue.cc:1035] T 5736c284a85540a1bbab2be6df672fc1 P c05f7620bb884392931d3243a0b47a9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:37.572942 16088 consensus_queue.cc:1035] T 5736c284a85540a1bbab2be6df672fc1 P c05f7620bb884392931d3243a0b47a9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 116.601 ms
time per row: 0.0583004 ms
I20240503 02:00:37.641253 14810 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39948:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:37.646973 14936 tablet_service.cc:1467] Processing CreateTablet for tablet 0989d98050744df6a1c699cb86ada505 (DEFAULT_TABLE table=rebalance_test_table_6 [id=aeab937d32b44418989332625d6cc6a8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:37.647022 15330 tablet_service.cc:1467] Processing CreateTablet for tablet 0989d98050744df6a1c699cb86ada505 (DEFAULT_TABLE table=rebalance_test_table_6 [id=aeab937d32b44418989332625d6cc6a8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:37.647310 15330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0989d98050744df6a1c699cb86ada505. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:37.647310 14936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0989d98050744df6a1c699cb86ada505. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:37.648005 15069 tablet_service.cc:1467] Processing CreateTablet for tablet 0989d98050744df6a1c699cb86ada505 (DEFAULT_TABLE table=rebalance_test_table_6 [id=aeab937d32b44418989332625d6cc6a8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:37.648085 15329 tablet_service.cc:1467] Processing CreateTablet for tablet 9a74cca2988a4f219f6fd9b07ade42d8 (DEFAULT_TABLE table=rebalance_test_table_6 [id=aeab937d32b44418989332625d6cc6a8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:37.648294 15069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0989d98050744df6a1c699cb86ada505. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:37.648332 15329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a74cca2988a4f219f6fd9b07ade42d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:37.649685 16338 tablet_bootstrap.cc:492] T 0989d98050744df6a1c699cb86ada505 P c05f7620bb884392931d3243a0b47a9e: Bootstrap starting.
I20240503 02:00:37.650229 14937 tablet_service.cc:1467] Processing CreateTablet for tablet 9a74cca2988a4f219f6fd9b07ade42d8 (DEFAULT_TABLE table=rebalance_test_table_6 [id=aeab937d32b44418989332625d6cc6a8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:37.650347 15199 tablet_service.cc:1467] Processing CreateTablet for tablet 9a74cca2988a4f219f6fd9b07ade42d8 (DEFAULT_TABLE table=rebalance_test_table_6 [id=aeab937d32b44418989332625d6cc6a8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:37.650487 14937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a74cca2988a4f219f6fd9b07ade42d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:37.650607 15199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a74cca2988a4f219f6fd9b07ade42d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:37.650758 15068 tablet_service.cc:1467] Processing CreateTablet for tablet 0e3bb299e61148fcb355641e0134b3df (DEFAULT_TABLE table=rebalance_test_table_6 [id=aeab937d32b44418989332625d6cc6a8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:37.650892 16338 tablet_bootstrap.cc:654] T 0989d98050744df6a1c699cb86ada505 P c05f7620bb884392931d3243a0b47a9e: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:37.650983 15068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e3bb299e61148fcb355641e0134b3df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:37.650938 14938 tablet_service.cc:1467] Processing CreateTablet for tablet 0e3bb299e61148fcb355641e0134b3df (DEFAULT_TABLE table=rebalance_test_table_6 [id=aeab937d32b44418989332625d6cc6a8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:37.651124 14938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e3bb299e61148fcb355641e0134b3df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:37.652071 16339 tablet_bootstrap.cc:492] T 0989d98050744df6a1c699cb86ada505 P b602f6099cbb4758847e7e9550685372: Bootstrap starting.
I20240503 02:00:37.652839 15331 tablet_service.cc:1467] Processing CreateTablet for tablet 0e3bb299e61148fcb355641e0134b3df (DEFAULT_TABLE table=rebalance_test_table_6 [id=aeab937d32b44418989332625d6cc6a8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:37.653002 16339 tablet_bootstrap.cc:654] T 0989d98050744df6a1c699cb86ada505 P b602f6099cbb4758847e7e9550685372: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:37.653038 15331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e3bb299e61148fcb355641e0134b3df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:37.653432 16341 tablet_bootstrap.cc:492] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612: Bootstrap starting.
I20240503 02:00:37.654214 16339 tablet_bootstrap.cc:492] T 0989d98050744df6a1c699cb86ada505 P b602f6099cbb4758847e7e9550685372: No bootstrap required, opened a new log
I20240503 02:00:37.654295 16339 ts_tablet_manager.cc:1367] T 0989d98050744df6a1c699cb86ada505 P b602f6099cbb4758847e7e9550685372: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:37.654331 16341 tablet_bootstrap.cc:654] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:37.654762 16339 raft_consensus.cc:348] T 0989d98050744df6a1c699cb86ada505 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:37.655154 16339 raft_consensus.cc:374] T 0989d98050744df6a1c699cb86ada505 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:37.655200 16339 raft_consensus.cc:729] T 0989d98050744df6a1c699cb86ada505 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Initialized, Role: FOLLOWER
I20240503 02:00:37.655356 16339 consensus_queue.cc:260] T 0989d98050744df6a1c699cb86ada505 P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:37.655799 16338 tablet_bootstrap.cc:492] T 0989d98050744df6a1c699cb86ada505 P c05f7620bb884392931d3243a0b47a9e: No bootstrap required, opened a new log
I20240503 02:00:37.655898 16338 ts_tablet_manager.cc:1367] T 0989d98050744df6a1c699cb86ada505 P c05f7620bb884392931d3243a0b47a9e: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:00:37.656119 16339 ts_tablet_manager.cc:1398] T 0989d98050744df6a1c699cb86ada505 P b602f6099cbb4758847e7e9550685372: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:37.656296 16339 tablet_bootstrap.cc:492] T 0e3bb299e61148fcb355641e0134b3df P b602f6099cbb4758847e7e9550685372: Bootstrap starting.
I20240503 02:00:37.656667 16338 raft_consensus.cc:348] T 0989d98050744df6a1c699cb86ada505 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:37.656797 16338 raft_consensus.cc:374] T 0989d98050744df6a1c699cb86ada505 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:37.656828 16338 raft_consensus.cc:729] T 0989d98050744df6a1c699cb86ada505 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Initialized, Role: FOLLOWER
I20240503 02:00:37.656957 16338 consensus_queue.cc:260] T 0989d98050744df6a1c699cb86ada505 P c05f7620bb884392931d3243a0b47a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:37.657239 16338 ts_tablet_manager.cc:1398] T 0989d98050744df6a1c699cb86ada505 P c05f7620bb884392931d3243a0b47a9e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:37.657279 16339 tablet_bootstrap.cc:654] T 0e3bb299e61148fcb355641e0134b3df P b602f6099cbb4758847e7e9550685372: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:37.657415 16338 tablet_bootstrap.cc:492] T 9a74cca2988a4f219f6fd9b07ade42d8 P c05f7620bb884392931d3243a0b47a9e: Bootstrap starting.
I20240503 02:00:37.658282 16338 tablet_bootstrap.cc:654] T 9a74cca2988a4f219f6fd9b07ade42d8 P c05f7620bb884392931d3243a0b47a9e: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:37.658648 16339 tablet_bootstrap.cc:492] T 0e3bb299e61148fcb355641e0134b3df P b602f6099cbb4758847e7e9550685372: No bootstrap required, opened a new log
I20240503 02:00:37.658726 16339 ts_tablet_manager.cc:1367] T 0e3bb299e61148fcb355641e0134b3df P b602f6099cbb4758847e7e9550685372: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:37.659173 16339 raft_consensus.cc:348] T 0e3bb299e61148fcb355641e0134b3df P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:37.659282 16339 raft_consensus.cc:374] T 0e3bb299e61148fcb355641e0134b3df P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:37.659319 16339 raft_consensus.cc:729] T 0e3bb299e61148fcb355641e0134b3df P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Initialized, Role: FOLLOWER
I20240503 02:00:37.659529 16339 consensus_queue.cc:260] T 0e3bb299e61148fcb355641e0134b3df P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:37.659801 16338 tablet_bootstrap.cc:492] T 9a74cca2988a4f219f6fd9b07ade42d8 P c05f7620bb884392931d3243a0b47a9e: No bootstrap required, opened a new log
I20240503 02:00:37.659893 16338 ts_tablet_manager.cc:1367] T 9a74cca2988a4f219f6fd9b07ade42d8 P c05f7620bb884392931d3243a0b47a9e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:00:37.660312 16338 raft_consensus.cc:348] T 9a74cca2988a4f219f6fd9b07ade42d8 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:37.660431 16338 raft_consensus.cc:374] T 9a74cca2988a4f219f6fd9b07ade42d8 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:37.660462 16338 raft_consensus.cc:729] T 9a74cca2988a4f219f6fd9b07ade42d8 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Initialized, Role: FOLLOWER
I20240503 02:00:37.660265 16339 ts_tablet_manager.cc:1398] T 0e3bb299e61148fcb355641e0134b3df P b602f6099cbb4758847e7e9550685372: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:37.660591 16338 consensus_queue.cc:260] T 9a74cca2988a4f219f6fd9b07ade42d8 P c05f7620bb884392931d3243a0b47a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:37.660904 16338 ts_tablet_manager.cc:1398] T 9a74cca2988a4f219f6fd9b07ade42d8 P c05f7620bb884392931d3243a0b47a9e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:37.661103 16338 tablet_bootstrap.cc:492] T 0e3bb299e61148fcb355641e0134b3df P c05f7620bb884392931d3243a0b47a9e: Bootstrap starting.
I20240503 02:00:37.661330 16341 tablet_bootstrap.cc:492] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612: No bootstrap required, opened a new log
I20240503 02:00:37.661464 16341 ts_tablet_manager.cc:1367] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 02:00:37.661974 16341 raft_consensus.cc:348] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:37.662127 16341 raft_consensus.cc:374] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:37.662190 16341 raft_consensus.cc:729] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 620596b893d84291b88b90dc9e317612, State: Initialized, Role: FOLLOWER
I20240503 02:00:37.662214 16338 tablet_bootstrap.cc:654] T 0e3bb299e61148fcb355641e0134b3df P c05f7620bb884392931d3243a0b47a9e: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:37.662382 16341 consensus_queue.cc:260] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:37.662745 16341 ts_tablet_manager.cc:1398] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:37.663379 16338 tablet_bootstrap.cc:492] T 0e3bb299e61148fcb355641e0134b3df P c05f7620bb884392931d3243a0b47a9e: No bootstrap required, opened a new log
I20240503 02:00:37.663458 16338 ts_tablet_manager.cc:1367] T 0e3bb299e61148fcb355641e0134b3df P c05f7620bb884392931d3243a0b47a9e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:37.663884 16338 raft_consensus.cc:348] T 0e3bb299e61148fcb355641e0134b3df P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:37.663981 16338 raft_consensus.cc:374] T 0e3bb299e61148fcb355641e0134b3df P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:37.664009 16338 raft_consensus.cc:729] T 0e3bb299e61148fcb355641e0134b3df P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Initialized, Role: FOLLOWER
I20240503 02:00:37.664120 16338 consensus_queue.cc:260] T 0e3bb299e61148fcb355641e0134b3df P c05f7620bb884392931d3243a0b47a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:37.664426 16338 ts_tablet_manager.cc:1398] T 0e3bb299e61148fcb355641e0134b3df P c05f7620bb884392931d3243a0b47a9e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:37.670387 16347 tablet_bootstrap.cc:492] T 0989d98050744df6a1c699cb86ada505 P dcfdfa62edc34dab9525e2712d487582: Bootstrap starting.
I20240503 02:00:37.670408 16121 raft_consensus.cc:482] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:37.670517 16121 raft_consensus.cc:504] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:37.670948 16121 leader_election.cc:288] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:37.671272 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a74cca2988a4f219f6fd9b07ade42d8" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcfdfa62edc34dab9525e2712d487582" is_pre_election: true
I20240503 02:00:37.671586 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a74cca2988a4f219f6fd9b07ade42d8" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c05f7620bb884392931d3243a0b47a9e" is_pre_election: true
I20240503 02:00:37.671710 15351 raft_consensus.cc:2454] T 9a74cca2988a4f219f6fd9b07ade42d8 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 0.
I20240503 02:00:37.671919 16347 tablet_bootstrap.cc:654] T 0989d98050744df6a1c699cb86ada505 P dcfdfa62edc34dab9525e2712d487582: Neither blocks nor log segments found. Creating new log.
W20240503 02:00:37.672020 15153 leader_election.cc:341] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:00:37.672174 15155 leader_election.cc:302] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 620596b893d84291b88b90dc9e317612, c05f7620bb884392931d3243a0b47a9e; no voters: dcfdfa62edc34dab9525e2712d487582
I20240503 02:00:37.672323 16121 raft_consensus.cc:2781] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:37.672405 16121 raft_consensus.cc:482] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:37.672452 16121 raft_consensus.cc:3037] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:37.673032 16347 tablet_bootstrap.cc:492] T 0989d98050744df6a1c699cb86ada505 P dcfdfa62edc34dab9525e2712d487582: No bootstrap required, opened a new log
I20240503 02:00:37.673103 16347 ts_tablet_manager.cc:1367] T 0989d98050744df6a1c699cb86ada505 P dcfdfa62edc34dab9525e2712d487582: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:37.673182 16121 raft_consensus.cc:504] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:37.673429 16347 raft_consensus.cc:348] T 0989d98050744df6a1c699cb86ada505 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:37.673520 16347 raft_consensus.cc:374] T 0989d98050744df6a1c699cb86ada505 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:37.673519 16121 leader_election.cc:288] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 1 election: Requested vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:37.673550 16347 raft_consensus.cc:729] T 0989d98050744df6a1c699cb86ada505 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Initialized, Role: FOLLOWER
I20240503 02:00:37.673887 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a74cca2988a4f219f6fd9b07ade42d8" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:37.673985 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a74cca2988a4f219f6fd9b07ade42d8" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:37.674116 15351 raft_consensus.cc:3037] T 9a74cca2988a4f219f6fd9b07ade42d8 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:00:37.674198 15153 leader_election.cc:341] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:00:37.674369 16347 consensus_queue.cc:260] T 0989d98050744df6a1c699cb86ada505 P dcfdfa62edc34dab9525e2712d487582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:37.674669 16347 ts_tablet_manager.cc:1398] T 0989d98050744df6a1c699cb86ada505 P dcfdfa62edc34dab9525e2712d487582: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:37.674831 16347 tablet_bootstrap.cc:492] T 0e3bb299e61148fcb355641e0134b3df P dcfdfa62edc34dab9525e2712d487582: Bootstrap starting.
I20240503 02:00:37.675105 15351 raft_consensus.cc:2454] T 9a74cca2988a4f219f6fd9b07ade42d8 P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 1.
I20240503 02:00:37.675480 15155 leader_election.cc:302] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 620596b893d84291b88b90dc9e317612, c05f7620bb884392931d3243a0b47a9e; no voters: dcfdfa62edc34dab9525e2712d487582
I20240503 02:00:37.675645 16121 raft_consensus.cc:2781] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:37.675719 16121 raft_consensus.cc:686] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612 [term 1 LEADER]: Becoming Leader. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:37.675734 16347 tablet_bootstrap.cc:654] T 0e3bb299e61148fcb355641e0134b3df P dcfdfa62edc34dab9525e2712d487582: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:37.675843 16121 consensus_queue.cc:237] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:37.678066 16347 tablet_bootstrap.cc:492] T 0e3bb299e61148fcb355641e0134b3df P dcfdfa62edc34dab9525e2712d487582: No bootstrap required, opened a new log
I20240503 02:00:37.678193 16347 ts_tablet_manager.cc:1367] T 0e3bb299e61148fcb355641e0134b3df P dcfdfa62edc34dab9525e2712d487582: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:00:37.678274 14811 catalog_manager.cc:5484] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612 reported cstate change: term changed from 0 to 1, leader changed from <none> to 620596b893d84291b88b90dc9e317612 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "620596b893d84291b88b90dc9e317612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:37.678627 16347 raft_consensus.cc:348] T 0e3bb299e61148fcb355641e0134b3df P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:37.678735 16347 raft_consensus.cc:374] T 0e3bb299e61148fcb355641e0134b3df P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:37.678764 16347 raft_consensus.cc:729] T 0e3bb299e61148fcb355641e0134b3df P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Initialized, Role: FOLLOWER
I20240503 02:00:37.679388 16347 consensus_queue.cc:260] T 0e3bb299e61148fcb355641e0134b3df P dcfdfa62edc34dab9525e2712d487582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:37.679693 16347 ts_tablet_manager.cc:1398] T 0e3bb299e61148fcb355641e0134b3df P dcfdfa62edc34dab9525e2712d487582: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:37.679876 16347 tablet_bootstrap.cc:492] T 9a74cca2988a4f219f6fd9b07ade42d8 P dcfdfa62edc34dab9525e2712d487582: Bootstrap starting.
I20240503 02:00:37.680763 16347 tablet_bootstrap.cc:654] T 9a74cca2988a4f219f6fd9b07ade42d8 P dcfdfa62edc34dab9525e2712d487582: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:37.681802 16347 tablet_bootstrap.cc:492] T 9a74cca2988a4f219f6fd9b07ade42d8 P dcfdfa62edc34dab9525e2712d487582: No bootstrap required, opened a new log
I20240503 02:00:37.681890 16347 ts_tablet_manager.cc:1367] T 9a74cca2988a4f219f6fd9b07ade42d8 P dcfdfa62edc34dab9525e2712d487582: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:37.682258 16347 raft_consensus.cc:348] T 9a74cca2988a4f219f6fd9b07ade42d8 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:37.682350 16347 raft_consensus.cc:374] T 9a74cca2988a4f219f6fd9b07ade42d8 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:37.682379 16347 raft_consensus.cc:729] T 9a74cca2988a4f219f6fd9b07ade42d8 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Initialized, Role: FOLLOWER
I20240503 02:00:37.682498 16347 consensus_queue.cc:260] T 9a74cca2988a4f219f6fd9b07ade42d8 P dcfdfa62edc34dab9525e2712d487582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:37.682734 16347 ts_tablet_manager.cc:1398] T 9a74cca2988a4f219f6fd9b07ade42d8 P dcfdfa62edc34dab9525e2712d487582: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:37.705691 16254 raft_consensus.cc:482] T 0989d98050744df6a1c699cb86ada505 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:37.705861 16254 raft_consensus.cc:504] T 0989d98050744df6a1c699cb86ada505 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:37.706334 16254 leader_election.cc:288] T 0989d98050744df6a1c699cb86ada505 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:37.706816 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0989d98050744df6a1c699cb86ada505" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcfdfa62edc34dab9525e2712d487582" is_pre_election: true
I20240503 02:00:37.706816 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0989d98050744df6a1c699cb86ada505" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c05f7620bb884392931d3243a0b47a9e" is_pre_election: true
I20240503 02:00:37.706966 15351 raft_consensus.cc:2454] T 0989d98050744df6a1c699cb86ada505 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 0.
I20240503 02:00:37.706965 14957 raft_consensus.cc:2454] T 0989d98050744df6a1c699cb86ada505 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 0.
I20240503 02:00:37.707391 15024 leader_election.cc:302] T 0989d98050744df6a1c699cb86ada505 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b602f6099cbb4758847e7e9550685372, c05f7620bb884392931d3243a0b47a9e; no voters:
I20240503 02:00:37.707706 16254 raft_consensus.cc:2781] T 0989d98050744df6a1c699cb86ada505 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:37.707789 16254 raft_consensus.cc:482] T 0989d98050744df6a1c699cb86ada505 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:37.707834 16254 raft_consensus.cc:3037] T 0989d98050744df6a1c699cb86ada505 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:37.709031 16254 raft_consensus.cc:504] T 0989d98050744df6a1c699cb86ada505 P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:37.709487 16254 leader_election.cc:288] T 0989d98050744df6a1c699cb86ada505 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 1 election: Requested vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:37.709925 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0989d98050744df6a1c699cb86ada505" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:37.709976 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0989d98050744df6a1c699cb86ada505" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:37.710088 15351 raft_consensus.cc:3037] T 0989d98050744df6a1c699cb86ada505 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:37.710121 14957 raft_consensus.cc:3037] T 0989d98050744df6a1c699cb86ada505 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:37.711062 15351 raft_consensus.cc:2454] T 0989d98050744df6a1c699cb86ada505 P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 1.
I20240503 02:00:37.711062 14957 raft_consensus.cc:2454] T 0989d98050744df6a1c699cb86ada505 P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 1.
I20240503 02:00:37.711431 15024 leader_election.cc:302] T 0989d98050744df6a1c699cb86ada505 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b602f6099cbb4758847e7e9550685372, c05f7620bb884392931d3243a0b47a9e; no voters:
I20240503 02:00:37.711647 16254 raft_consensus.cc:2781] T 0989d98050744df6a1c699cb86ada505 P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:37.711766 16254 raft_consensus.cc:686] T 0989d98050744df6a1c699cb86ada505 P b602f6099cbb4758847e7e9550685372 [term 1 LEADER]: Becoming Leader. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:37.711916 16254 consensus_queue.cc:237] T 0989d98050744df6a1c699cb86ada505 P b602f6099cbb4758847e7e9550685372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:37.714165 14811 catalog_manager.cc:5484] T 0989d98050744df6a1c699cb86ada505 P b602f6099cbb4758847e7e9550685372 reported cstate change: term changed from 0 to 1, leader changed from <none> to b602f6099cbb4758847e7e9550685372 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "b602f6099cbb4758847e7e9550685372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:37.866266 16088 raft_consensus.cc:482] T 0e3bb299e61148fcb355641e0134b3df P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:37.866395 16088 raft_consensus.cc:504] T 0e3bb299e61148fcb355641e0134b3df P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:37.866768 16088 leader_election.cc:288] T 0e3bb299e61148fcb355641e0134b3df P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207), dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817)
I20240503 02:00:37.867235 15088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e3bb299e61148fcb355641e0134b3df" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b602f6099cbb4758847e7e9550685372" is_pre_election: true
I20240503 02:00:37.867241 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e3bb299e61148fcb355641e0134b3df" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcfdfa62edc34dab9525e2712d487582" is_pre_election: true
I20240503 02:00:37.867377 15088 raft_consensus.cc:2454] T 0e3bb299e61148fcb355641e0134b3df P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 0.
I20240503 02:00:37.867389 14957 raft_consensus.cc:2454] T 0e3bb299e61148fcb355641e0134b3df P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 0.
I20240503 02:00:37.867694 15285 leader_election.cc:302] T 0e3bb299e61148fcb355641e0134b3df P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b602f6099cbb4758847e7e9550685372, c05f7620bb884392931d3243a0b47a9e; no voters:
I20240503 02:00:37.867883 16088 raft_consensus.cc:2781] T 0e3bb299e61148fcb355641e0134b3df P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:37.867946 16088 raft_consensus.cc:482] T 0e3bb299e61148fcb355641e0134b3df P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:37.867985 16088 raft_consensus.cc:3037] T 0e3bb299e61148fcb355641e0134b3df P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:37.868800 16088 raft_consensus.cc:504] T 0e3bb299e61148fcb355641e0134b3df P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:37.869168 16088 leader_election.cc:288] T 0e3bb299e61148fcb355641e0134b3df P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 1 election: Requested vote from peers b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207), dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817)
I20240503 02:00:37.869530 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e3bb299e61148fcb355641e0134b3df" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:37.869531 15088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e3bb299e61148fcb355641e0134b3df" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:37.869652 15088 raft_consensus.cc:3037] T 0e3bb299e61148fcb355641e0134b3df P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:37.869668 14957 raft_consensus.cc:3037] T 0e3bb299e61148fcb355641e0134b3df P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:37.870415 14957 raft_consensus.cc:2454] T 0e3bb299e61148fcb355641e0134b3df P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 1.
I20240503 02:00:37.870427 15088 raft_consensus.cc:2454] T 0e3bb299e61148fcb355641e0134b3df P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 1.
I20240503 02:00:37.870740 15284 leader_election.cc:302] T 0e3bb299e61148fcb355641e0134b3df P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c05f7620bb884392931d3243a0b47a9e, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:37.871068 16088 raft_consensus.cc:2781] T 0e3bb299e61148fcb355641e0134b3df P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:37.871155 16088 raft_consensus.cc:686] T 0e3bb299e61148fcb355641e0134b3df P c05f7620bb884392931d3243a0b47a9e [term 1 LEADER]: Becoming Leader. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:37.871296 16088 consensus_queue.cc:237] T 0e3bb299e61148fcb355641e0134b3df P c05f7620bb884392931d3243a0b47a9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:37.873296 14811 catalog_manager.cc:5484] T 0e3bb299e61148fcb355641e0134b3df P c05f7620bb884392931d3243a0b47a9e reported cstate change: term changed from 0 to 1, leader changed from <none> to c05f7620bb884392931d3243a0b47a9e (127.0.99.132). New cstate: current_term: 1 leader_uuid: "c05f7620bb884392931d3243a0b47a9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } }
W20240503 02:00:38.007946 16354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:38.008239 16354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:38.011617 16354 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240503 02:00:38.077658 16167 raft_consensus.cc:482] T 9a74cca2988a4f219f6fd9b07ade42d8 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:38.077801 16167 raft_consensus.cc:504] T 9a74cca2988a4f219f6fd9b07ade42d8 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:38.078250 16167 leader_election.cc:288] T 9a74cca2988a4f219f6fd9b07ade42d8 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:38.078814 15220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a74cca2988a4f219f6fd9b07ade42d8" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "620596b893d84291b88b90dc9e317612" is_pre_election: true
I20240503 02:00:38.078814 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a74cca2988a4f219f6fd9b07ade42d8" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c05f7620bb884392931d3243a0b47a9e" is_pre_election: true
I20240503 02:00:38.079098 15351 raft_consensus.cc:2379] T 9a74cca2988a4f219f6fd9b07ade42d8 P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dcfdfa62edc34dab9525e2712d487582 in current term 1: Already voted for candidate 620596b893d84291b88b90dc9e317612 in this term.
I20240503 02:00:38.079499 14893 leader_election.cc:302] T 9a74cca2988a4f219f6fd9b07ade42d8 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dcfdfa62edc34dab9525e2712d487582; no voters: 620596b893d84291b88b90dc9e317612, c05f7620bb884392931d3243a0b47a9e
I20240503 02:00:38.079748 16167 raft_consensus.cc:3037] T 9a74cca2988a4f219f6fd9b07ade42d8 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:38.080855 16167 raft_consensus.cc:2726] T 9a74cca2988a4f219f6fd9b07ade42d8 P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:00:38.112452 15351 raft_consensus.cc:1261] T 0989d98050744df6a1c699cb86ada505 P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Refusing update from remote peer b602f6099cbb4758847e7e9550685372: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:38.113085 16254 consensus_queue.cc:1035] T 0989d98050744df6a1c699cb86ada505 P b602f6099cbb4758847e7e9550685372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:38.113907 14957 raft_consensus.cc:1261] T 0989d98050744df6a1c699cb86ada505 P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Refusing update from remote peer b602f6099cbb4758847e7e9550685372: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:38.114569 16166 consensus_queue.cc:1035] T 0989d98050744df6a1c699cb86ada505 P b602f6099cbb4758847e7e9550685372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:38.130282 14957 raft_consensus.cc:1261] T 0e3bb299e61148fcb355641e0134b3df P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Refusing update from remote peer c05f7620bb884392931d3243a0b47a9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:38.130398 15088 raft_consensus.cc:1261] T 0e3bb299e61148fcb355641e0134b3df P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Refusing update from remote peer c05f7620bb884392931d3243a0b47a9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:38.130704 16088 consensus_queue.cc:1035] T 0e3bb299e61148fcb355641e0134b3df P c05f7620bb884392931d3243a0b47a9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:38.130877 16322 consensus_queue.cc:1035] T 0e3bb299e61148fcb355641e0134b3df P c05f7620bb884392931d3243a0b47a9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:38.138708 15349 raft_consensus.cc:1261] T 9a74cca2988a4f219f6fd9b07ade42d8 P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Refusing update from remote peer 620596b893d84291b88b90dc9e317612: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:38.139850 16264 consensus_queue.cc:1035] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:38.140774 14956 raft_consensus.cc:1261] T 9a74cca2988a4f219f6fd9b07ade42d8 P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Refusing update from remote peer 620596b893d84291b88b90dc9e317612: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:38.141542 16264 consensus_queue.cc:1035] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:38.167372 16394 mvcc.cc:205] Tried to move back new op lower bound from 7023417909783449600 to 7023417907921600512. Current Snapshot: MvccSnapshot[applied={T|T < 7023417909783449600}]
INSERT report
rows total: 2000
time total: 129.302 ms
time per row: 0.0646508 ms
I20240503 02:00:38.187806 14815 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39948:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:38.193994 15199 tablet_service.cc:1467] Processing CreateTablet for tablet 6d98ba8805024eb79bf5df7a8225d784 (DEFAULT_TABLE table=rebalance_test_table_7 [id=2176cab2c69b4568ad8667bd78019b5b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:38.194005 14938 tablet_service.cc:1467] Processing CreateTablet for tablet 6d98ba8805024eb79bf5df7a8225d784 (DEFAULT_TABLE table=rebalance_test_table_7 [id=2176cab2c69b4568ad8667bd78019b5b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:38.194327 15199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d98ba8805024eb79bf5df7a8225d784. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:38.194337 14938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d98ba8805024eb79bf5df7a8225d784. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:38.194708 15331 tablet_service.cc:1467] Processing CreateTablet for tablet 50c5abd860ae4b09bb289e9c7550c84e (DEFAULT_TABLE table=rebalance_test_table_7 [id=2176cab2c69b4568ad8667bd78019b5b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:38.194773 15329 tablet_service.cc:1467] Processing CreateTablet for tablet 9e9899666b15498791f62dab6d9dda16 (DEFAULT_TABLE table=rebalance_test_table_7 [id=2176cab2c69b4568ad8667bd78019b5b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:38.194965 15331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50c5abd860ae4b09bb289e9c7550c84e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:38.195705 15329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e9899666b15498791f62dab6d9dda16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:38.197307 16407 tablet_bootstrap.cc:492] T 6d98ba8805024eb79bf5df7a8225d784 P dcfdfa62edc34dab9525e2712d487582: Bootstrap starting.
I20240503 02:00:38.197402 16409 tablet_bootstrap.cc:492] T 50c5abd860ae4b09bb289e9c7550c84e P c05f7620bb884392931d3243a0b47a9e: Bootstrap starting.
I20240503 02:00:38.197921 15200 tablet_service.cc:1467] Processing CreateTablet for tablet 50c5abd860ae4b09bb289e9c7550c84e (DEFAULT_TABLE table=rebalance_test_table_7 [id=2176cab2c69b4568ad8667bd78019b5b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:38.198207 15200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50c5abd860ae4b09bb289e9c7550c84e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:38.198251 16407 tablet_bootstrap.cc:654] T 6d98ba8805024eb79bf5df7a8225d784 P dcfdfa62edc34dab9525e2712d487582: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:38.198371 14938 tablet_service.cc:1467] Processing CreateTablet for tablet 50c5abd860ae4b09bb289e9c7550c84e (DEFAULT_TABLE table=rebalance_test_table_7 [id=2176cab2c69b4568ad8667bd78019b5b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:38.198613 14938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50c5abd860ae4b09bb289e9c7550c84e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:38.199100 16409 tablet_bootstrap.cc:654] T 50c5abd860ae4b09bb289e9c7550c84e P c05f7620bb884392931d3243a0b47a9e: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:38.200412 16407 tablet_bootstrap.cc:492] T 6d98ba8805024eb79bf5df7a8225d784 P dcfdfa62edc34dab9525e2712d487582: No bootstrap required, opened a new log
I20240503 02:00:38.200345 15198 tablet_service.cc:1467] Processing CreateTablet for tablet 9e9899666b15498791f62dab6d9dda16 (DEFAULT_TABLE table=rebalance_test_table_7 [id=2176cab2c69b4568ad8667bd78019b5b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:38.200510 16407 ts_tablet_manager.cc:1367] T 6d98ba8805024eb79bf5df7a8225d784 P dcfdfa62edc34dab9525e2712d487582: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:38.200626 15198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e9899666b15498791f62dab6d9dda16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:38.200971 16407 raft_consensus.cc:348] T 6d98ba8805024eb79bf5df7a8225d784 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:38.201138 16407 raft_consensus.cc:374] T 6d98ba8805024eb79bf5df7a8225d784 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:38.201182 16407 raft_consensus.cc:729] T 6d98ba8805024eb79bf5df7a8225d784 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Initialized, Role: FOLLOWER
I20240503 02:00:38.201319 16407 consensus_queue.cc:260] T 6d98ba8805024eb79bf5df7a8225d784 P dcfdfa62edc34dab9525e2712d487582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:38.201694 16407 ts_tablet_manager.cc:1398] T 6d98ba8805024eb79bf5df7a8225d784 P dcfdfa62edc34dab9525e2712d487582: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:38.201870 16407 tablet_bootstrap.cc:492] T 50c5abd860ae4b09bb289e9c7550c84e P dcfdfa62edc34dab9525e2712d487582: Bootstrap starting.
I20240503 02:00:38.202832 16407 tablet_bootstrap.cc:654] T 50c5abd860ae4b09bb289e9c7550c84e P dcfdfa62edc34dab9525e2712d487582: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:38.202948 15068 tablet_service.cc:1467] Processing CreateTablet for tablet 6d98ba8805024eb79bf5df7a8225d784 (DEFAULT_TABLE table=rebalance_test_table_7 [id=2176cab2c69b4568ad8667bd78019b5b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:38.203258 15068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d98ba8805024eb79bf5df7a8225d784. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:38.204164 16409 tablet_bootstrap.cc:492] T 50c5abd860ae4b09bb289e9c7550c84e P c05f7620bb884392931d3243a0b47a9e: No bootstrap required, opened a new log
I20240503 02:00:38.204288 16409 ts_tablet_manager.cc:1367] T 50c5abd860ae4b09bb289e9c7550c84e P c05f7620bb884392931d3243a0b47a9e: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:00:38.204829 16409 raft_consensus.cc:348] T 50c5abd860ae4b09bb289e9c7550c84e P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:38.204937 16409 raft_consensus.cc:374] T 50c5abd860ae4b09bb289e9c7550c84e P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:38.204967 16409 raft_consensus.cc:729] T 50c5abd860ae4b09bb289e9c7550c84e P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Initialized, Role: FOLLOWER
I20240503 02:00:38.205106 16409 consensus_queue.cc:260] T 50c5abd860ae4b09bb289e9c7550c84e P c05f7620bb884392931d3243a0b47a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:38.205418 16409 ts_tablet_manager.cc:1398] T 50c5abd860ae4b09bb289e9c7550c84e P c05f7620bb884392931d3243a0b47a9e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:38.205624 16413 tablet_bootstrap.cc:492] T 6d98ba8805024eb79bf5df7a8225d784 P b602f6099cbb4758847e7e9550685372: Bootstrap starting.
I20240503 02:00:38.205605 16409 tablet_bootstrap.cc:492] T 9e9899666b15498791f62dab6d9dda16 P c05f7620bb884392931d3243a0b47a9e: Bootstrap starting.
I20240503 02:00:38.205834 15069 tablet_service.cc:1467] Processing CreateTablet for tablet 9e9899666b15498791f62dab6d9dda16 (DEFAULT_TABLE table=rebalance_test_table_7 [id=2176cab2c69b4568ad8667bd78019b5b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:38.206089 15069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e9899666b15498791f62dab6d9dda16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:38.206282 16407 tablet_bootstrap.cc:492] T 50c5abd860ae4b09bb289e9c7550c84e P dcfdfa62edc34dab9525e2712d487582: No bootstrap required, opened a new log
I20240503 02:00:38.206373 16407 ts_tablet_manager.cc:1367] T 50c5abd860ae4b09bb289e9c7550c84e P dcfdfa62edc34dab9525e2712d487582: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:00:38.206805 16413 tablet_bootstrap.cc:654] T 6d98ba8805024eb79bf5df7a8225d784 P b602f6099cbb4758847e7e9550685372: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:38.206851 16407 raft_consensus.cc:348] T 50c5abd860ae4b09bb289e9c7550c84e P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:38.207077 16407 raft_consensus.cc:374] T 50c5abd860ae4b09bb289e9c7550c84e P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:38.207142 16407 raft_consensus.cc:729] T 50c5abd860ae4b09bb289e9c7550c84e P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Initialized, Role: FOLLOWER
I20240503 02:00:38.207285 16407 consensus_queue.cc:260] T 50c5abd860ae4b09bb289e9c7550c84e P dcfdfa62edc34dab9525e2712d487582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:38.207579 16407 ts_tablet_manager.cc:1398] T 50c5abd860ae4b09bb289e9c7550c84e P dcfdfa62edc34dab9525e2712d487582: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 02:00:38.207626 16409 tablet_bootstrap.cc:654] T 9e9899666b15498791f62dab6d9dda16 P c05f7620bb884392931d3243a0b47a9e: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:38.209403 16409 tablet_bootstrap.cc:492] T 9e9899666b15498791f62dab6d9dda16 P c05f7620bb884392931d3243a0b47a9e: No bootstrap required, opened a new log
I20240503 02:00:38.209458 16413 tablet_bootstrap.cc:492] T 6d98ba8805024eb79bf5df7a8225d784 P b602f6099cbb4758847e7e9550685372: No bootstrap required, opened a new log
I20240503 02:00:38.209509 16409 ts_tablet_manager.cc:1367] T 9e9899666b15498791f62dab6d9dda16 P c05f7620bb884392931d3243a0b47a9e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:38.209571 16413 ts_tablet_manager.cc:1367] T 6d98ba8805024eb79bf5df7a8225d784 P b602f6099cbb4758847e7e9550685372: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:38.210131 16413 raft_consensus.cc:348] T 6d98ba8805024eb79bf5df7a8225d784 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:38.210263 16413 raft_consensus.cc:374] T 6d98ba8805024eb79bf5df7a8225d784 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:38.210258 16408 tablet_bootstrap.cc:492] T 6d98ba8805024eb79bf5df7a8225d784 P 620596b893d84291b88b90dc9e317612: Bootstrap starting.
I20240503 02:00:38.210297 16413 raft_consensus.cc:729] T 6d98ba8805024eb79bf5df7a8225d784 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Initialized, Role: FOLLOWER
I20240503 02:00:38.210320 16409 raft_consensus.cc:348] T 9e9899666b15498791f62dab6d9dda16 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:38.210434 16409 raft_consensus.cc:374] T 9e9899666b15498791f62dab6d9dda16 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:38.210462 16409 raft_consensus.cc:729] T 9e9899666b15498791f62dab6d9dda16 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Initialized, Role: FOLLOWER
I20240503 02:00:38.210443 16413 consensus_queue.cc:260] T 6d98ba8805024eb79bf5df7a8225d784 P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:38.210561 16409 consensus_queue.cc:260] T 9e9899666b15498791f62dab6d9dda16 P c05f7620bb884392931d3243a0b47a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:38.210757 16413 ts_tablet_manager.cc:1398] T 6d98ba8805024eb79bf5df7a8225d784 P b602f6099cbb4758847e7e9550685372: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:38.210844 16409 ts_tablet_manager.cc:1398] T 9e9899666b15498791f62dab6d9dda16 P c05f7620bb884392931d3243a0b47a9e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:38.210963 16413 tablet_bootstrap.cc:492] T 9e9899666b15498791f62dab6d9dda16 P b602f6099cbb4758847e7e9550685372: Bootstrap starting.
I20240503 02:00:38.211762 16408 tablet_bootstrap.cc:654] T 6d98ba8805024eb79bf5df7a8225d784 P 620596b893d84291b88b90dc9e317612: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:38.211834 16413 tablet_bootstrap.cc:654] T 9e9899666b15498791f62dab6d9dda16 P b602f6099cbb4758847e7e9550685372: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:38.212878 16408 tablet_bootstrap.cc:492] T 6d98ba8805024eb79bf5df7a8225d784 P 620596b893d84291b88b90dc9e317612: No bootstrap required, opened a new log
I20240503 02:00:38.212966 16408 ts_tablet_manager.cc:1367] T 6d98ba8805024eb79bf5df7a8225d784 P 620596b893d84291b88b90dc9e317612: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:38.213405 16408 raft_consensus.cc:348] T 6d98ba8805024eb79bf5df7a8225d784 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:38.213516 16408 raft_consensus.cc:374] T 6d98ba8805024eb79bf5df7a8225d784 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:38.213542 16408 raft_consensus.cc:729] T 6d98ba8805024eb79bf5df7a8225d784 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 620596b893d84291b88b90dc9e317612, State: Initialized, Role: FOLLOWER
I20240503 02:00:38.213663 16408 consensus_queue.cc:260] T 6d98ba8805024eb79bf5df7a8225d784 P 620596b893d84291b88b90dc9e317612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:38.213927 16408 ts_tablet_manager.cc:1398] T 6d98ba8805024eb79bf5df7a8225d784 P 620596b893d84291b88b90dc9e317612: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:38.214084 16408 tablet_bootstrap.cc:492] T 50c5abd860ae4b09bb289e9c7550c84e P 620596b893d84291b88b90dc9e317612: Bootstrap starting.
I20240503 02:00:38.215556 16413 tablet_bootstrap.cc:492] T 9e9899666b15498791f62dab6d9dda16 P b602f6099cbb4758847e7e9550685372: No bootstrap required, opened a new log
I20240503 02:00:38.215692 16408 tablet_bootstrap.cc:654] T 50c5abd860ae4b09bb289e9c7550c84e P 620596b893d84291b88b90dc9e317612: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:38.216272 16413 ts_tablet_manager.cc:1367] T 9e9899666b15498791f62dab6d9dda16 P b602f6099cbb4758847e7e9550685372: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:00:38.218117 16413 raft_consensus.cc:348] T 9e9899666b15498791f62dab6d9dda16 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:38.218372 16413 raft_consensus.cc:374] T 9e9899666b15498791f62dab6d9dda16 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:38.218429 16413 raft_consensus.cc:729] T 9e9899666b15498791f62dab6d9dda16 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Initialized, Role: FOLLOWER
I20240503 02:00:38.218597 16413 consensus_queue.cc:260] T 9e9899666b15498791f62dab6d9dda16 P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:38.218901 16413 ts_tablet_manager.cc:1398] T 9e9899666b15498791f62dab6d9dda16 P b602f6099cbb4758847e7e9550685372: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:38.219659 16408 tablet_bootstrap.cc:492] T 50c5abd860ae4b09bb289e9c7550c84e P 620596b893d84291b88b90dc9e317612: No bootstrap required, opened a new log
I20240503 02:00:38.219764 16408 ts_tablet_manager.cc:1367] T 50c5abd860ae4b09bb289e9c7550c84e P 620596b893d84291b88b90dc9e317612: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:00:38.220266 16408 raft_consensus.cc:348] T 50c5abd860ae4b09bb289e9c7550c84e P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:38.220449 16408 raft_consensus.cc:374] T 50c5abd860ae4b09bb289e9c7550c84e P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:38.220495 16408 raft_consensus.cc:729] T 50c5abd860ae4b09bb289e9c7550c84e P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 620596b893d84291b88b90dc9e317612, State: Initialized, Role: FOLLOWER
I20240503 02:00:38.220613 16408 consensus_queue.cc:260] T 50c5abd860ae4b09bb289e9c7550c84e P 620596b893d84291b88b90dc9e317612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:38.220901 16408 ts_tablet_manager.cc:1398] T 50c5abd860ae4b09bb289e9c7550c84e P 620596b893d84291b88b90dc9e317612: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:38.221097 16408 tablet_bootstrap.cc:492] T 9e9899666b15498791f62dab6d9dda16 P 620596b893d84291b88b90dc9e317612: Bootstrap starting.
I20240503 02:00:38.221980 16408 tablet_bootstrap.cc:654] T 9e9899666b15498791f62dab6d9dda16 P 620596b893d84291b88b90dc9e317612: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:38.223453 16408 tablet_bootstrap.cc:492] T 9e9899666b15498791f62dab6d9dda16 P 620596b893d84291b88b90dc9e317612: No bootstrap required, opened a new log
I20240503 02:00:38.223531 16408 ts_tablet_manager.cc:1367] T 9e9899666b15498791f62dab6d9dda16 P 620596b893d84291b88b90dc9e317612: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:38.223922 16408 raft_consensus.cc:348] T 9e9899666b15498791f62dab6d9dda16 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:38.224038 16408 raft_consensus.cc:374] T 9e9899666b15498791f62dab6d9dda16 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:38.224063 16408 raft_consensus.cc:729] T 9e9899666b15498791f62dab6d9dda16 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 620596b893d84291b88b90dc9e317612, State: Initialized, Role: FOLLOWER
I20240503 02:00:38.224169 16408 consensus_queue.cc:260] T 9e9899666b15498791f62dab6d9dda16 P 620596b893d84291b88b90dc9e317612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:38.224475 16408 ts_tablet_manager.cc:1398] T 9e9899666b15498791f62dab6d9dda16 P 620596b893d84291b88b90dc9e317612: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:38.263721 16264 raft_consensus.cc:482] T 9e9899666b15498791f62dab6d9dda16 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:38.263860 16264 raft_consensus.cc:504] T 9e9899666b15498791f62dab6d9dda16 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:38.264364 16264 leader_election.cc:288] T 9e9899666b15498791f62dab6d9dda16 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:38.264786 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e9899666b15498791f62dab6d9dda16" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c05f7620bb884392931d3243a0b47a9e" is_pre_election: true
I20240503 02:00:38.264792 15088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e9899666b15498791f62dab6d9dda16" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b602f6099cbb4758847e7e9550685372" is_pre_election: true
I20240503 02:00:38.264958 15351 raft_consensus.cc:2454] T 9e9899666b15498791f62dab6d9dda16 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 0.
I20240503 02:00:38.264957 15088 raft_consensus.cc:2454] T 9e9899666b15498791f62dab6d9dda16 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 0.
I20240503 02:00:38.265287 15154 leader_election.cc:302] T 9e9899666b15498791f62dab6d9dda16 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, b602f6099cbb4758847e7e9550685372, c05f7620bb884392931d3243a0b47a9e; no voters:
I20240503 02:00:38.265471 16264 raft_consensus.cc:2781] T 9e9899666b15498791f62dab6d9dda16 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:38.265532 16264 raft_consensus.cc:482] T 9e9899666b15498791f62dab6d9dda16 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:38.265556 16264 raft_consensus.cc:3037] T 9e9899666b15498791f62dab6d9dda16 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:38.266368 16264 raft_consensus.cc:504] T 9e9899666b15498791f62dab6d9dda16 P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:38.266741 16264 leader_election.cc:288] T 9e9899666b15498791f62dab6d9dda16 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 1 election: Requested vote from peers b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:38.267285 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e9899666b15498791f62dab6d9dda16" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:38.267277 15088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e9899666b15498791f62dab6d9dda16" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:38.267428 15088 raft_consensus.cc:3037] T 9e9899666b15498791f62dab6d9dda16 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:38.267443 15351 raft_consensus.cc:3037] T 9e9899666b15498791f62dab6d9dda16 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:38.268437 15088 raft_consensus.cc:2454] T 9e9899666b15498791f62dab6d9dda16 P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 1.
I20240503 02:00:38.268437 15351 raft_consensus.cc:2454] T 9e9899666b15498791f62dab6d9dda16 P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 1.
I20240503 02:00:38.268774 15154 leader_election.cc:302] T 9e9899666b15498791f62dab6d9dda16 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, b602f6099cbb4758847e7e9550685372; no voters:
I20240503 02:00:38.268976 16264 raft_consensus.cc:2781] T 9e9899666b15498791f62dab6d9dda16 P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:38.269078 16264 raft_consensus.cc:686] T 9e9899666b15498791f62dab6d9dda16 P 620596b893d84291b88b90dc9e317612 [term 1 LEADER]: Becoming Leader. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:38.269227 16264 consensus_queue.cc:237] T 9e9899666b15498791f62dab6d9dda16 P 620596b893d84291b88b90dc9e317612 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:38.271486 14812 catalog_manager.cc:5484] T 9e9899666b15498791f62dab6d9dda16 P 620596b893d84291b88b90dc9e317612 reported cstate change: term changed from 0 to 1, leader changed from <none> to 620596b893d84291b88b90dc9e317612 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "620596b893d84291b88b90dc9e317612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:38.314611 16264 raft_consensus.cc:482] T 6d98ba8805024eb79bf5df7a8225d784 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:38.314754 16264 raft_consensus.cc:504] T 6d98ba8805024eb79bf5df7a8225d784 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:38.315358 16264 leader_election.cc:288] T 6d98ba8805024eb79bf5df7a8225d784 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
I20240503 02:00:38.315876 14956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d98ba8805024eb79bf5df7a8225d784" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcfdfa62edc34dab9525e2712d487582" is_pre_election: true
I20240503 02:00:38.315835 15088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d98ba8805024eb79bf5df7a8225d784" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b602f6099cbb4758847e7e9550685372" is_pre_election: true
I20240503 02:00:38.316037 14956 raft_consensus.cc:2454] T 6d98ba8805024eb79bf5df7a8225d784 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 0.
I20240503 02:00:38.316053 15088 raft_consensus.cc:2454] T 6d98ba8805024eb79bf5df7a8225d784 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 0.
I20240503 02:00:38.316438 15153 leader_election.cc:302] T 6d98ba8805024eb79bf5df7a8225d784 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:38.316643 16264 raft_consensus.cc:2781] T 6d98ba8805024eb79bf5df7a8225d784 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:38.316704 16264 raft_consensus.cc:482] T 6d98ba8805024eb79bf5df7a8225d784 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:38.316749 16264 raft_consensus.cc:3037] T 6d98ba8805024eb79bf5df7a8225d784 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:38.317579 16264 raft_consensus.cc:504] T 6d98ba8805024eb79bf5df7a8225d784 P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:38.317991 16264 leader_election.cc:288] T 6d98ba8805024eb79bf5df7a8225d784 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 1 election: Requested vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
I20240503 02:00:38.318449 14956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d98ba8805024eb79bf5df7a8225d784" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:38.318544 15088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d98ba8805024eb79bf5df7a8225d784" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:38.318683 15088 raft_consensus.cc:3037] T 6d98ba8805024eb79bf5df7a8225d784 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:38.318665 14956 raft_consensus.cc:3037] T 6d98ba8805024eb79bf5df7a8225d784 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:38.319501 15088 raft_consensus.cc:2454] T 6d98ba8805024eb79bf5df7a8225d784 P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 1.
I20240503 02:00:38.319576 14956 raft_consensus.cc:2454] T 6d98ba8805024eb79bf5df7a8225d784 P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 1.
I20240503 02:00:38.319913 15153 leader_election.cc:302] T 6d98ba8805024eb79bf5df7a8225d784 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:38.320130 16264 raft_consensus.cc:2781] T 6d98ba8805024eb79bf5df7a8225d784 P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:38.320240 16264 raft_consensus.cc:686] T 6d98ba8805024eb79bf5df7a8225d784 P 620596b893d84291b88b90dc9e317612 [term 1 LEADER]: Becoming Leader. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:38.320436 16264 consensus_queue.cc:237] T 6d98ba8805024eb79bf5df7a8225d784 P 620596b893d84291b88b90dc9e317612 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:38.322407 14812 catalog_manager.cc:5484] T 6d98ba8805024eb79bf5df7a8225d784 P 620596b893d84291b88b90dc9e317612 reported cstate change: term changed from 0 to 1, leader changed from <none> to 620596b893d84291b88b90dc9e317612 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "620596b893d84291b88b90dc9e317612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:38.500509 16322 raft_consensus.cc:482] T 50c5abd860ae4b09bb289e9c7550c84e P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:38.500656 16322 raft_consensus.cc:504] T 50c5abd860ae4b09bb289e9c7550c84e P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:38.501194 16322 leader_election.cc:288] T 50c5abd860ae4b09bb289e9c7550c84e P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437)
I20240503 02:00:38.501587 14956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50c5abd860ae4b09bb289e9c7550c84e" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcfdfa62edc34dab9525e2712d487582" is_pre_election: true
I20240503 02:00:38.501749 14956 raft_consensus.cc:2454] T 50c5abd860ae4b09bb289e9c7550c84e P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 0.
I20240503 02:00:38.501754 15220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50c5abd860ae4b09bb289e9c7550c84e" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "620596b893d84291b88b90dc9e317612" is_pre_election: true
I20240503 02:00:38.501871 15220 raft_consensus.cc:2454] T 50c5abd860ae4b09bb289e9c7550c84e P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 0.
I20240503 02:00:38.502051 15284 leader_election.cc:302] T 50c5abd860ae4b09bb289e9c7550c84e P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c05f7620bb884392931d3243a0b47a9e, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:38.502259 16322 raft_consensus.cc:2781] T 50c5abd860ae4b09bb289e9c7550c84e P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:38.502318 16322 raft_consensus.cc:482] T 50c5abd860ae4b09bb289e9c7550c84e P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:38.502357 16322 raft_consensus.cc:3037] T 50c5abd860ae4b09bb289e9c7550c84e P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:38.503242 16322 raft_consensus.cc:504] T 50c5abd860ae4b09bb289e9c7550c84e P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:38.503618 16322 leader_election.cc:288] T 50c5abd860ae4b09bb289e9c7550c84e P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 1 election: Requested vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437)
I20240503 02:00:38.504074 15220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50c5abd860ae4b09bb289e9c7550c84e" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:38.504091 14956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50c5abd860ae4b09bb289e9c7550c84e" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:38.504218 15220 raft_consensus.cc:3037] T 50c5abd860ae4b09bb289e9c7550c84e P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:38.504218 14956 raft_consensus.cc:3037] T 50c5abd860ae4b09bb289e9c7550c84e P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:38.505208 15220 raft_consensus.cc:2454] T 50c5abd860ae4b09bb289e9c7550c84e P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 1.
I20240503 02:00:38.505208 14956 raft_consensus.cc:2454] T 50c5abd860ae4b09bb289e9c7550c84e P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 1.
I20240503 02:00:38.505597 15284 leader_election.cc:302] T 50c5abd860ae4b09bb289e9c7550c84e P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c05f7620bb884392931d3243a0b47a9e, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:38.505782 16322 raft_consensus.cc:2781] T 50c5abd860ae4b09bb289e9c7550c84e P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:38.505911 16322 raft_consensus.cc:686] T 50c5abd860ae4b09bb289e9c7550c84e P c05f7620bb884392931d3243a0b47a9e [term 1 LEADER]: Becoming Leader. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:38.506060 16322 consensus_queue.cc:237] T 50c5abd860ae4b09bb289e9c7550c84e P c05f7620bb884392931d3243a0b47a9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:38.508004 14812 catalog_manager.cc:5484] T 50c5abd860ae4b09bb289e9c7550c84e P c05f7620bb884392931d3243a0b47a9e reported cstate change: term changed from 0 to 1, leader changed from <none> to c05f7620bb884392931d3243a0b47a9e (127.0.99.132). New cstate: current_term: 1 leader_uuid: "c05f7620bb884392931d3243a0b47a9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: HEALTHY } } }
W20240503 02:00:38.691138 16423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:38.691435 16423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:38.694928 16423 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240503 02:00:38.731491 16264 consensus_queue.cc:1035] T 6d98ba8805024eb79bf5df7a8225d784 P 620596b893d84291b88b90dc9e317612 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:38.734926 16264 consensus_queue.cc:1035] T 6d98ba8805024eb79bf5df7a8225d784 P 620596b893d84291b88b90dc9e317612 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:38.799441 15351 raft_consensus.cc:1261] T 9e9899666b15498791f62dab6d9dda16 P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Refusing update from remote peer 620596b893d84291b88b90dc9e317612: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:38.799873 15088 raft_consensus.cc:1261] T 9e9899666b15498791f62dab6d9dda16 P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Refusing update from remote peer 620596b893d84291b88b90dc9e317612: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:38.803012 16264 consensus_queue.cc:1035] T 9e9899666b15498791f62dab6d9dda16 P 620596b893d84291b88b90dc9e317612 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:38.803228 15220 raft_consensus.cc:1261] T 50c5abd860ae4b09bb289e9c7550c84e P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Refusing update from remote peer c05f7620bb884392931d3243a0b47a9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:38.803573 16264 consensus_queue.cc:1035] T 9e9899666b15498791f62dab6d9dda16 P 620596b893d84291b88b90dc9e317612 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:38.803689 16088 consensus_queue.cc:1035] T 50c5abd860ae4b09bb289e9c7550c84e P c05f7620bb884392931d3243a0b47a9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:38.805433 14956 raft_consensus.cc:1261] T 50c5abd860ae4b09bb289e9c7550c84e P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Refusing update from remote peer c05f7620bb884392931d3243a0b47a9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:38.809950 16322 consensus_queue.cc:1035] T 50c5abd860ae4b09bb289e9c7550c84e P c05f7620bb884392931d3243a0b47a9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:38.834417 16450 mvcc.cc:205] Tried to move back new op lower bound from 7023417912516603904 to 7023417910352453632. Current Snapshot: MvccSnapshot[applied={T|T < 7023417912516603904}]
I20240503 02:00:38.836468 16456 mvcc.cc:205] Tried to move back new op lower bound from 7023417912528773120 to 7023417911322361856. Current Snapshot: MvccSnapshot[applied={T|T < 7023417912528773120}]
INSERT report
rows total: 2000
time total: 134.488 ms
time per row: 0.0672441 ms
I20240503 02:00:38.878540 14810 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39948:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:38.884418 15198 tablet_service.cc:1467] Processing CreateTablet for tablet 830da5cb4972453cb9bf2145a0839ced (DEFAULT_TABLE table=rebalance_test_table_8 [id=a1002cdd5dbc4efd9e0c9206ca076748]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:38.884420 14938 tablet_service.cc:1467] Processing CreateTablet for tablet 830da5cb4972453cb9bf2145a0839ced (DEFAULT_TABLE table=rebalance_test_table_8 [id=a1002cdd5dbc4efd9e0c9206ca076748]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:38.884760 15198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 830da5cb4972453cb9bf2145a0839ced. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:38.884774 14938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 830da5cb4972453cb9bf2145a0839ced. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:38.885196 14936 tablet_service.cc:1467] Processing CreateTablet for tablet 21d92d412a5247e8907c5c41d253ad2e (DEFAULT_TABLE table=rebalance_test_table_8 [id=a1002cdd5dbc4efd9e0c9206ca076748]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:38.885411 14936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21d92d412a5247e8907c5c41d253ad2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:38.885387 15069 tablet_service.cc:1467] Processing CreateTablet for tablet 830da5cb4972453cb9bf2145a0839ced (DEFAULT_TABLE table=rebalance_test_table_8 [id=a1002cdd5dbc4efd9e0c9206ca076748]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:38.885640 15069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 830da5cb4972453cb9bf2145a0839ced. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:38.884771 14937 tablet_service.cc:1467] Processing CreateTablet for tablet 590736f5f1ca4703821c409d882c2f77 (DEFAULT_TABLE table=rebalance_test_table_8 [id=a1002cdd5dbc4efd9e0c9206ca076748]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:38.887631 14937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 590736f5f1ca4703821c409d882c2f77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:38.888139 15331 tablet_service.cc:1467] Processing CreateTablet for tablet 21d92d412a5247e8907c5c41d253ad2e (DEFAULT_TABLE table=rebalance_test_table_8 [id=a1002cdd5dbc4efd9e0c9206ca076748]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:38.888139 15329 tablet_service.cc:1467] Processing CreateTablet for tablet 590736f5f1ca4703821c409d882c2f77 (DEFAULT_TABLE table=rebalance_test_table_8 [id=a1002cdd5dbc4efd9e0c9206ca076748]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:38.888484 15331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21d92d412a5247e8907c5c41d253ad2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:38.889227 15329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 590736f5f1ca4703821c409d882c2f77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:38.889174 15069 tablet_service.cc:1467] Processing CreateTablet for tablet 590736f5f1ca4703821c409d882c2f77 (DEFAULT_TABLE table=rebalance_test_table_8 [id=a1002cdd5dbc4efd9e0c9206ca076748]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:38.889415 15069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 590736f5f1ca4703821c409d882c2f77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:38.890161 16480 tablet_bootstrap.cc:492] T 830da5cb4972453cb9bf2145a0839ced P dcfdfa62edc34dab9525e2712d487582: Bootstrap starting.
I20240503 02:00:38.890831 15198 tablet_service.cc:1467] Processing CreateTablet for tablet 21d92d412a5247e8907c5c41d253ad2e (DEFAULT_TABLE table=rebalance_test_table_8 [id=a1002cdd5dbc4efd9e0c9206ca076748]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:38.891110 15198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21d92d412a5247e8907c5c41d253ad2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:38.891398 16480 tablet_bootstrap.cc:654] T 830da5cb4972453cb9bf2145a0839ced P dcfdfa62edc34dab9525e2712d487582: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:38.891791 16481 tablet_bootstrap.cc:492] T 830da5cb4972453cb9bf2145a0839ced P 620596b893d84291b88b90dc9e317612: Bootstrap starting.
I20240503 02:00:38.892036 16482 tablet_bootstrap.cc:492] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372: Bootstrap starting.
I20240503 02:00:38.892810 16480 tablet_bootstrap.cc:492] T 830da5cb4972453cb9bf2145a0839ced P dcfdfa62edc34dab9525e2712d487582: No bootstrap required, opened a new log
I20240503 02:00:38.892917 16480 ts_tablet_manager.cc:1367] T 830da5cb4972453cb9bf2145a0839ced P dcfdfa62edc34dab9525e2712d487582: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:38.893412 16480 raft_consensus.cc:348] T 830da5cb4972453cb9bf2145a0839ced P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:38.893539 16480 raft_consensus.cc:374] T 830da5cb4972453cb9bf2145a0839ced P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:38.893580 16480 raft_consensus.cc:729] T 830da5cb4972453cb9bf2145a0839ced P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Initialized, Role: FOLLOWER
I20240503 02:00:38.893726 16480 consensus_queue.cc:260] T 830da5cb4972453cb9bf2145a0839ced P dcfdfa62edc34dab9525e2712d487582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:38.894011 16480 ts_tablet_manager.cc:1398] T 830da5cb4972453cb9bf2145a0839ced P dcfdfa62edc34dab9525e2712d487582: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:38.894202 16480 tablet_bootstrap.cc:492] T 21d92d412a5247e8907c5c41d253ad2e P dcfdfa62edc34dab9525e2712d487582: Bootstrap starting.
I20240503 02:00:38.894805 16481 tablet_bootstrap.cc:654] T 830da5cb4972453cb9bf2145a0839ced P 620596b893d84291b88b90dc9e317612: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:38.894048 16482 tablet_bootstrap.cc:654] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:38.895116 16480 tablet_bootstrap.cc:654] T 21d92d412a5247e8907c5c41d253ad2e P dcfdfa62edc34dab9525e2712d487582: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:38.896811 16482 tablet_bootstrap.cc:492] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372: No bootstrap required, opened a new log
I20240503 02:00:38.896914 16482 ts_tablet_manager.cc:1367] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:00:38.896939 16480 tablet_bootstrap.cc:492] T 21d92d412a5247e8907c5c41d253ad2e P dcfdfa62edc34dab9525e2712d487582: No bootstrap required, opened a new log
I20240503 02:00:38.897028 16480 ts_tablet_manager.cc:1367] T 21d92d412a5247e8907c5c41d253ad2e P dcfdfa62edc34dab9525e2712d487582: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:38.897331 16482 raft_consensus.cc:348] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:38.897439 16482 raft_consensus.cc:374] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:38.897468 16482 raft_consensus.cc:729] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Initialized, Role: FOLLOWER
I20240503 02:00:38.897461 16480 raft_consensus.cc:348] T 21d92d412a5247e8907c5c41d253ad2e P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:38.897573 16480 raft_consensus.cc:374] T 21d92d412a5247e8907c5c41d253ad2e P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:38.897609 16480 raft_consensus.cc:729] T 21d92d412a5247e8907c5c41d253ad2e P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Initialized, Role: FOLLOWER
I20240503 02:00:38.897712 16481 tablet_bootstrap.cc:492] T 830da5cb4972453cb9bf2145a0839ced P 620596b893d84291b88b90dc9e317612: No bootstrap required, opened a new log
I20240503 02:00:38.897753 16480 consensus_queue.cc:260] T 21d92d412a5247e8907c5c41d253ad2e P dcfdfa62edc34dab9525e2712d487582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:38.897819 16481 ts_tablet_manager.cc:1367] T 830da5cb4972453cb9bf2145a0839ced P 620596b893d84291b88b90dc9e317612: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:00:38.898007 16480 ts_tablet_manager.cc:1398] T 21d92d412a5247e8907c5c41d253ad2e P dcfdfa62edc34dab9525e2712d487582: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:38.898177 16480 tablet_bootstrap.cc:492] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582: Bootstrap starting.
I20240503 02:00:38.898293 16481 raft_consensus.cc:348] T 830da5cb4972453cb9bf2145a0839ced P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:38.898393 16481 raft_consensus.cc:374] T 830da5cb4972453cb9bf2145a0839ced P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:38.898427 16481 raft_consensus.cc:729] T 830da5cb4972453cb9bf2145a0839ced P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 620596b893d84291b88b90dc9e317612, State: Initialized, Role: FOLLOWER
I20240503 02:00:38.898550 16481 consensus_queue.cc:260] T 830da5cb4972453cb9bf2145a0839ced P 620596b893d84291b88b90dc9e317612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:38.898835 16481 ts_tablet_manager.cc:1398] T 830da5cb4972453cb9bf2145a0839ced P 620596b893d84291b88b90dc9e317612: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:38.898990 16481 tablet_bootstrap.cc:492] T 21d92d412a5247e8907c5c41d253ad2e P 620596b893d84291b88b90dc9e317612: Bootstrap starting.
I20240503 02:00:38.899152 16480 tablet_bootstrap.cc:654] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:38.899772 16481 tablet_bootstrap.cc:654] T 21d92d412a5247e8907c5c41d253ad2e P 620596b893d84291b88b90dc9e317612: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:38.900552 16482 consensus_queue.cc:260] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:38.900915 16482 ts_tablet_manager.cc:1398] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 02:00:38.901239 16480 tablet_bootstrap.cc:492] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582: No bootstrap required, opened a new log
I20240503 02:00:38.901326 16481 tablet_bootstrap.cc:492] T 21d92d412a5247e8907c5c41d253ad2e P 620596b893d84291b88b90dc9e317612: No bootstrap required, opened a new log
I20240503 02:00:38.901332 16480 ts_tablet_manager.cc:1367] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:38.901407 16481 ts_tablet_manager.cc:1367] T 21d92d412a5247e8907c5c41d253ad2e P 620596b893d84291b88b90dc9e317612: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:00:38.901795 16481 raft_consensus.cc:348] T 21d92d412a5247e8907c5c41d253ad2e P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:38.901901 16481 raft_consensus.cc:374] T 21d92d412a5247e8907c5c41d253ad2e P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:38.901932 16481 raft_consensus.cc:729] T 21d92d412a5247e8907c5c41d253ad2e P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 620596b893d84291b88b90dc9e317612, State: Initialized, Role: FOLLOWER
I20240503 02:00:38.901796 16480 raft_consensus.cc:348] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:38.901984 16480 raft_consensus.cc:374] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:38.902037 16480 raft_consensus.cc:729] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Initialized, Role: FOLLOWER
I20240503 02:00:38.902176 16480 consensus_queue.cc:260] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:38.902621 16480 ts_tablet_manager.cc:1398] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:38.902830 16482 tablet_bootstrap.cc:492] T 590736f5f1ca4703821c409d882c2f77 P b602f6099cbb4758847e7e9550685372: Bootstrap starting.
I20240503 02:00:38.904572 16482 tablet_bootstrap.cc:654] T 590736f5f1ca4703821c409d882c2f77 P b602f6099cbb4758847e7e9550685372: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:38.902046 16481 consensus_queue.cc:260] T 21d92d412a5247e8907c5c41d253ad2e P 620596b893d84291b88b90dc9e317612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:38.905565 16481 ts_tablet_manager.cc:1398] T 21d92d412a5247e8907c5c41d253ad2e P 620596b893d84291b88b90dc9e317612: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 02:00:38.906383 16482 tablet_bootstrap.cc:492] T 590736f5f1ca4703821c409d882c2f77 P b602f6099cbb4758847e7e9550685372: No bootstrap required, opened a new log
I20240503 02:00:38.906483 16482 ts_tablet_manager.cc:1367] T 590736f5f1ca4703821c409d882c2f77 P b602f6099cbb4758847e7e9550685372: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 02:00:38.906956 16482 raft_consensus.cc:348] T 590736f5f1ca4703821c409d882c2f77 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:38.907074 16482 raft_consensus.cc:374] T 590736f5f1ca4703821c409d882c2f77 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:38.907114 16482 raft_consensus.cc:729] T 590736f5f1ca4703821c409d882c2f77 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Initialized, Role: FOLLOWER
I20240503 02:00:38.907254 16482 consensus_queue.cc:260] T 590736f5f1ca4703821c409d882c2f77 P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:38.907538 16482 ts_tablet_manager.cc:1398] T 590736f5f1ca4703821c409d882c2f77 P b602f6099cbb4758847e7e9550685372: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:38.912719 16483 tablet_bootstrap.cc:492] T 21d92d412a5247e8907c5c41d253ad2e P c05f7620bb884392931d3243a0b47a9e: Bootstrap starting.
I20240503 02:00:38.913769 16483 tablet_bootstrap.cc:654] T 21d92d412a5247e8907c5c41d253ad2e P c05f7620bb884392931d3243a0b47a9e: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:38.914840 16483 tablet_bootstrap.cc:492] T 21d92d412a5247e8907c5c41d253ad2e P c05f7620bb884392931d3243a0b47a9e: No bootstrap required, opened a new log
I20240503 02:00:38.914933 16483 ts_tablet_manager.cc:1367] T 21d92d412a5247e8907c5c41d253ad2e P c05f7620bb884392931d3243a0b47a9e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:38.915344 16483 raft_consensus.cc:348] T 21d92d412a5247e8907c5c41d253ad2e P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:38.915445 16483 raft_consensus.cc:374] T 21d92d412a5247e8907c5c41d253ad2e P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:38.915467 16483 raft_consensus.cc:729] T 21d92d412a5247e8907c5c41d253ad2e P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Initialized, Role: FOLLOWER
I20240503 02:00:38.915596 16483 consensus_queue.cc:260] T 21d92d412a5247e8907c5c41d253ad2e P c05f7620bb884392931d3243a0b47a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:38.915877 16483 ts_tablet_manager.cc:1398] T 21d92d412a5247e8907c5c41d253ad2e P c05f7620bb884392931d3243a0b47a9e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:38.916045 16483 tablet_bootstrap.cc:492] T 590736f5f1ca4703821c409d882c2f77 P c05f7620bb884392931d3243a0b47a9e: Bootstrap starting.
I20240503 02:00:38.917011 16483 tablet_bootstrap.cc:654] T 590736f5f1ca4703821c409d882c2f77 P c05f7620bb884392931d3243a0b47a9e: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:38.919451 16490 raft_consensus.cc:482] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:38.919567 16490 raft_consensus.cc:504] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:38.920007 16490 leader_election.cc:288] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
I20240503 02:00:38.920436 15088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "590736f5f1ca4703821c409d882c2f77" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b602f6099cbb4758847e7e9550685372" is_pre_election: true
I20240503 02:00:38.920418 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "590736f5f1ca4703821c409d882c2f77" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c05f7620bb884392931d3243a0b47a9e" is_pre_election: true
I20240503 02:00:38.920570 15088 raft_consensus.cc:2454] T 590736f5f1ca4703821c409d882c2f77 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 0.
I20240503 02:00:38.920820 14892 leader_election.cc:302] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b602f6099cbb4758847e7e9550685372, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:38.921000 16490 raft_consensus.cc:2781] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20240503 02:00:38.920946 14893 leader_election.cc:341] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:00:38.921087 16490 raft_consensus.cc:482] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:38.921176 16490 raft_consensus.cc:3037] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:38.922042 16490 raft_consensus.cc:504] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:38.922406 16490 leader_election.cc:288] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 1 election: Requested vote from peers c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
I20240503 02:00:38.922756 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "590736f5f1ca4703821c409d882c2f77" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:38.922772 15088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "590736f5f1ca4703821c409d882c2f77" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:38.922880 15088 raft_consensus.cc:3037] T 590736f5f1ca4703821c409d882c2f77 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:00:38.923120 14893 leader_election.cc:341] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:00:38.923719 15088 raft_consensus.cc:2454] T 590736f5f1ca4703821c409d882c2f77 P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 1.
I20240503 02:00:38.923995 14892 leader_election.cc:302] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b602f6099cbb4758847e7e9550685372, dcfdfa62edc34dab9525e2712d487582; no voters: c05f7620bb884392931d3243a0b47a9e
I20240503 02:00:38.924171 16490 raft_consensus.cc:2781] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:38.924266 16490 raft_consensus.cc:686] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582 [term 1 LEADER]: Becoming Leader. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:38.924430 16490 consensus_queue.cc:237] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:38.926334 16483 tablet_bootstrap.cc:492] T 590736f5f1ca4703821c409d882c2f77 P c05f7620bb884392931d3243a0b47a9e: No bootstrap required, opened a new log
I20240503 02:00:38.926443 16483 ts_tablet_manager.cc:1367] T 590736f5f1ca4703821c409d882c2f77 P c05f7620bb884392931d3243a0b47a9e: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 02:00:38.926738 14815 catalog_manager.cc:5484] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582 reported cstate change: term changed from 0 to 1, leader changed from <none> to dcfdfa62edc34dab9525e2712d487582 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "dcfdfa62edc34dab9525e2712d487582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:38.926863 16483 raft_consensus.cc:348] T 590736f5f1ca4703821c409d882c2f77 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:38.926966 16483 raft_consensus.cc:374] T 590736f5f1ca4703821c409d882c2f77 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:38.926993 16483 raft_consensus.cc:729] T 590736f5f1ca4703821c409d882c2f77 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Initialized, Role: FOLLOWER
I20240503 02:00:38.927109 16483 consensus_queue.cc:260] T 590736f5f1ca4703821c409d882c2f77 P c05f7620bb884392931d3243a0b47a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:38.927459 16483 ts_tablet_manager.cc:1398] T 590736f5f1ca4703821c409d882c2f77 P c05f7620bb884392931d3243a0b47a9e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:38.952056 16490 raft_consensus.cc:482] T 21d92d412a5247e8907c5c41d253ad2e P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:38.952225 16490 raft_consensus.cc:504] T 21d92d412a5247e8907c5c41d253ad2e P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:38.952752 16490 leader_election.cc:288] T 21d92d412a5247e8907c5c41d253ad2e P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:38.953222 15220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21d92d412a5247e8907c5c41d253ad2e" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "620596b893d84291b88b90dc9e317612" is_pre_election: true
I20240503 02:00:38.953279 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21d92d412a5247e8907c5c41d253ad2e" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c05f7620bb884392931d3243a0b47a9e" is_pre_election: true
I20240503 02:00:38.953364 15220 raft_consensus.cc:2454] T 21d92d412a5247e8907c5c41d253ad2e P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 0.
I20240503 02:00:38.953423 15351 raft_consensus.cc:2454] T 21d92d412a5247e8907c5c41d253ad2e P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 0.
I20240503 02:00:38.953684 14893 leader_election.cc:302] T 21d92d412a5247e8907c5c41d253ad2e P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:38.953938 16490 raft_consensus.cc:2781] T 21d92d412a5247e8907c5c41d253ad2e P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:38.953996 16490 raft_consensus.cc:482] T 21d92d412a5247e8907c5c41d253ad2e P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:38.954033 16490 raft_consensus.cc:3037] T 21d92d412a5247e8907c5c41d253ad2e P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:38.954957 16490 raft_consensus.cc:504] T 21d92d412a5247e8907c5c41d253ad2e P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:38.955333 16490 leader_election.cc:288] T 21d92d412a5247e8907c5c41d253ad2e P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 1 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:38.955660 15220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21d92d412a5247e8907c5c41d253ad2e" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:38.955694 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21d92d412a5247e8907c5c41d253ad2e" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:38.955794 15220 raft_consensus.cc:3037] T 21d92d412a5247e8907c5c41d253ad2e P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:38.955794 15351 raft_consensus.cc:3037] T 21d92d412a5247e8907c5c41d253ad2e P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:38.956919 15220 raft_consensus.cc:2454] T 21d92d412a5247e8907c5c41d253ad2e P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 1.
I20240503 02:00:38.956919 15351 raft_consensus.cc:2454] T 21d92d412a5247e8907c5c41d253ad2e P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 1.
I20240503 02:00:38.957244 14893 leader_election.cc:302] T 21d92d412a5247e8907c5c41d253ad2e P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:38.957437 16490 raft_consensus.cc:2781] T 21d92d412a5247e8907c5c41d253ad2e P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:38.957523 16490 raft_consensus.cc:686] T 21d92d412a5247e8907c5c41d253ad2e P dcfdfa62edc34dab9525e2712d487582 [term 1 LEADER]: Becoming Leader. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:38.957670 16490 consensus_queue.cc:237] T 21d92d412a5247e8907c5c41d253ad2e P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:38.959653 14811 catalog_manager.cc:5484] T 21d92d412a5247e8907c5c41d253ad2e P dcfdfa62edc34dab9525e2712d487582 reported cstate change: term changed from 0 to 1, leader changed from <none> to dcfdfa62edc34dab9525e2712d487582 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "dcfdfa62edc34dab9525e2712d487582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:39.005713 16397 raft_consensus.cc:482] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:39.005980 16397 raft_consensus.cc:504] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:39.006618 16397 leader_election.cc:288] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817)
I20240503 02:00:39.007115 14956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "830da5cb4972453cb9bf2145a0839ced" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcfdfa62edc34dab9525e2712d487582" is_pre_election: true
I20240503 02:00:39.007169 15220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "830da5cb4972453cb9bf2145a0839ced" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "620596b893d84291b88b90dc9e317612" is_pre_election: true
I20240503 02:00:39.007282 14956 raft_consensus.cc:2454] T 830da5cb4972453cb9bf2145a0839ced P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 0.
I20240503 02:00:39.007304 15220 raft_consensus.cc:2454] T 830da5cb4972453cb9bf2145a0839ced P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 0.
I20240503 02:00:39.007609 15022 leader_election.cc:302] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b602f6099cbb4758847e7e9550685372, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:39.007867 16397 raft_consensus.cc:2781] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:39.007927 16397 raft_consensus.cc:482] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:39.007961 16397 raft_consensus.cc:3037] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:39.009126 16397 raft_consensus.cc:504] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:39.009538 16397 leader_election.cc:288] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 1 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817)
I20240503 02:00:39.009946 14956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "830da5cb4972453cb9bf2145a0839ced" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:39.009964 15220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "830da5cb4972453cb9bf2145a0839ced" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:39.010088 15220 raft_consensus.cc:3037] T 830da5cb4972453cb9bf2145a0839ced P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:39.010088 14956 raft_consensus.cc:3037] T 830da5cb4972453cb9bf2145a0839ced P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:39.011051 15220 raft_consensus.cc:2454] T 830da5cb4972453cb9bf2145a0839ced P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 1.
I20240503 02:00:39.011051 14956 raft_consensus.cc:2454] T 830da5cb4972453cb9bf2145a0839ced P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 1.
I20240503 02:00:39.011353 15024 leader_election.cc:302] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, b602f6099cbb4758847e7e9550685372; no voters:
I20240503 02:00:39.011829 16496 raft_consensus.cc:2781] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:39.012017 16496 raft_consensus.cc:686] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372 [term 1 LEADER]: Becoming Leader. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:39.012197 16496 consensus_queue.cc:237] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:39.014017 14811 catalog_manager.cc:5484] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372 reported cstate change: term changed from 0 to 1, leader changed from <none> to b602f6099cbb4758847e7e9550685372 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "b602f6099cbb4758847e7e9550685372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:39.047504 16474 raft_consensus.cc:482] T 590736f5f1ca4703821c409d882c2f77 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:39.047664 16474 raft_consensus.cc:504] T 590736f5f1ca4703821c409d882c2f77 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:39.048168 16474 leader_election.cc:288] T 590736f5f1ca4703821c409d882c2f77 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
I20240503 02:00:39.048672 15088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "590736f5f1ca4703821c409d882c2f77" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b602f6099cbb4758847e7e9550685372" is_pre_election: true
I20240503 02:00:39.048696 14956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "590736f5f1ca4703821c409d882c2f77" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcfdfa62edc34dab9525e2712d487582" is_pre_election: true
I20240503 02:00:39.048856 15088 raft_consensus.cc:2379] T 590736f5f1ca4703821c409d882c2f77 P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c05f7620bb884392931d3243a0b47a9e in current term 1: Already voted for candidate dcfdfa62edc34dab9525e2712d487582 in this term.
I20240503 02:00:39.049268 15284 leader_election.cc:302] T 590736f5f1ca4703821c409d882c2f77 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c05f7620bb884392931d3243a0b47a9e; no voters: b602f6099cbb4758847e7e9550685372, dcfdfa62edc34dab9525e2712d487582
I20240503 02:00:39.049461 16474 raft_consensus.cc:3037] T 590736f5f1ca4703821c409d882c2f77 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:39.052675 16474 raft_consensus.cc:2726] T 590736f5f1ca4703821c409d882c2f77 P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:00:39.145124 16500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:39.145409 16500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:39.148849 16500 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240503 02:00:39.235399 15088 raft_consensus.cc:1261] T 590736f5f1ca4703821c409d882c2f77 P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Refusing update from remote peer dcfdfa62edc34dab9525e2712d487582: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:39.235499 15349 raft_consensus.cc:1261] T 590736f5f1ca4703821c409d882c2f77 P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Refusing update from remote peer dcfdfa62edc34dab9525e2712d487582: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:39.236461 15219 raft_consensus.cc:1261] T 21d92d412a5247e8907c5c41d253ad2e P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Refusing update from remote peer dcfdfa62edc34dab9525e2712d487582: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:39.236763 16490 consensus_queue.cc:1035] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:39.237520 16490 consensus_queue.cc:1035] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:39.239157 15351 raft_consensus.cc:1261] T 21d92d412a5247e8907c5c41d253ad2e P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Refusing update from remote peer dcfdfa62edc34dab9525e2712d487582: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:39.239607 16167 consensus_queue.cc:1035] T 21d92d412a5247e8907c5c41d253ad2e P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Connected to new peer: Peer: permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:39.240485 16531 consensus_queue.cc:1035] T 21d92d412a5247e8907c5c41d253ad2e P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:39.241511 14956 raft_consensus.cc:1261] T 830da5cb4972453cb9bf2145a0839ced P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Refusing update from remote peer b602f6099cbb4758847e7e9550685372: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:39.246690 16496 consensus_queue.cc:1035] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:39.255591 15220 raft_consensus.cc:1261] T 830da5cb4972453cb9bf2145a0839ced P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Refusing update from remote peer b602f6099cbb4758847e7e9550685372: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:39.257823 16496 consensus_queue.cc:1035] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:39.267482 16445 mvcc.cc:205] Tried to move back new op lower bound from 7023417914324209664 to 7023417913172070400. Current Snapshot: MvccSnapshot[applied={T|T < 7023417914301485056}]
INSERT report
rows total: 2000
time total: 110.435 ms
time per row: 0.0552173 ms
I20240503 02:00:39.309720 14808 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39948:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:39.317544 15198 tablet_service.cc:1467] Processing CreateTablet for tablet de805ef7511d4ee9b9bbed67e0a4186c (DEFAULT_TABLE table=rebalance_test_table_9 [id=615bc73a81674a899a85198265830843]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:39.317564 15329 tablet_service.cc:1467] Processing CreateTablet for tablet de805ef7511d4ee9b9bbed67e0a4186c (DEFAULT_TABLE table=rebalance_test_table_9 [id=615bc73a81674a899a85198265830843]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:39.317893 15329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de805ef7511d4ee9b9bbed67e0a4186c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:39.317891 15198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de805ef7511d4ee9b9bbed67e0a4186c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:39.317871 15069 tablet_service.cc:1467] Processing CreateTablet for tablet de805ef7511d4ee9b9bbed67e0a4186c (DEFAULT_TABLE table=rebalance_test_table_9 [id=615bc73a81674a899a85198265830843]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:39.317950 15068 tablet_service.cc:1467] Processing CreateTablet for tablet 4cdcdde158054cdaabc1856a90fe1302 (DEFAULT_TABLE table=rebalance_test_table_9 [id=615bc73a81674a899a85198265830843]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:39.318316 15068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cdcdde158054cdaabc1856a90fe1302. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:39.319085 14937 tablet_service.cc:1467] Processing CreateTablet for tablet 7b4f203f4048490ba599353a4cd92748 (DEFAULT_TABLE table=rebalance_test_table_9 [id=615bc73a81674a899a85198265830843]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:39.319357 14937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b4f203f4048490ba599353a4cd92748. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:39.320554 15067 tablet_service.cc:1467] Processing CreateTablet for tablet 7b4f203f4048490ba599353a4cd92748 (DEFAULT_TABLE table=rebalance_test_table_9 [id=615bc73a81674a899a85198265830843]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:39.320560 15200 tablet_service.cc:1467] Processing CreateTablet for tablet 4cdcdde158054cdaabc1856a90fe1302 (DEFAULT_TABLE table=rebalance_test_table_9 [id=615bc73a81674a899a85198265830843]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:39.320806 15200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cdcdde158054cdaabc1856a90fe1302. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:39.320807 15067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b4f203f4048490ba599353a4cd92748. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:39.321369 16482 tablet_bootstrap.cc:492] T 4cdcdde158054cdaabc1856a90fe1302 P b602f6099cbb4758847e7e9550685372: Bootstrap starting.
I20240503 02:00:39.321859 16480 tablet_bootstrap.cc:492] T 7b4f203f4048490ba599353a4cd92748 P dcfdfa62edc34dab9525e2712d487582: Bootstrap starting.
I20240503 02:00:39.322341 16482 tablet_bootstrap.cc:654] T 4cdcdde158054cdaabc1856a90fe1302 P b602f6099cbb4758847e7e9550685372: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:39.322705 16480 tablet_bootstrap.cc:654] T 7b4f203f4048490ba599353a4cd92748 P dcfdfa62edc34dab9525e2712d487582: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:39.322990 16481 tablet_bootstrap.cc:492] T de805ef7511d4ee9b9bbed67e0a4186c P 620596b893d84291b88b90dc9e317612: Bootstrap starting.
I20240503 02:00:39.323871 16483 tablet_bootstrap.cc:492] T de805ef7511d4ee9b9bbed67e0a4186c P c05f7620bb884392931d3243a0b47a9e: Bootstrap starting.
I20240503 02:00:39.323890 16481 tablet_bootstrap.cc:654] T de805ef7511d4ee9b9bbed67e0a4186c P 620596b893d84291b88b90dc9e317612: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:39.324278 16482 tablet_bootstrap.cc:492] T 4cdcdde158054cdaabc1856a90fe1302 P b602f6099cbb4758847e7e9550685372: No bootstrap required, opened a new log
I20240503 02:00:39.324362 16482 ts_tablet_manager.cc:1367] T 4cdcdde158054cdaabc1856a90fe1302 P b602f6099cbb4758847e7e9550685372: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:00:39.324831 15069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de805ef7511d4ee9b9bbed67e0a4186c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:39.324865 16483 tablet_bootstrap.cc:654] T de805ef7511d4ee9b9bbed67e0a4186c P c05f7620bb884392931d3243a0b47a9e: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:39.324865 16482 raft_consensus.cc:348] T 4cdcdde158054cdaabc1856a90fe1302 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:39.324970 16482 raft_consensus.cc:374] T 4cdcdde158054cdaabc1856a90fe1302 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:39.324999 16482 raft_consensus.cc:729] T 4cdcdde158054cdaabc1856a90fe1302 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Initialized, Role: FOLLOWER
I20240503 02:00:39.325142 16482 consensus_queue.cc:260] T 4cdcdde158054cdaabc1856a90fe1302 P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:39.325434 16482 ts_tablet_manager.cc:1398] T 4cdcdde158054cdaabc1856a90fe1302 P b602f6099cbb4758847e7e9550685372: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:39.325608 16482 tablet_bootstrap.cc:492] T 7b4f203f4048490ba599353a4cd92748 P b602f6099cbb4758847e7e9550685372: Bootstrap starting.
I20240503 02:00:39.326509 16482 tablet_bootstrap.cc:654] T 7b4f203f4048490ba599353a4cd92748 P b602f6099cbb4758847e7e9550685372: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:39.327510 16480 tablet_bootstrap.cc:492] T 7b4f203f4048490ba599353a4cd92748 P dcfdfa62edc34dab9525e2712d487582: No bootstrap required, opened a new log
I20240503 02:00:39.327651 16480 ts_tablet_manager.cc:1367] T 7b4f203f4048490ba599353a4cd92748 P dcfdfa62edc34dab9525e2712d487582: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 02:00:39.328184 16480 raft_consensus.cc:348] T 7b4f203f4048490ba599353a4cd92748 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:39.328317 16480 raft_consensus.cc:374] T 7b4f203f4048490ba599353a4cd92748 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:39.328346 16480 raft_consensus.cc:729] T 7b4f203f4048490ba599353a4cd92748 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Initialized, Role: FOLLOWER
I20240503 02:00:39.328285 15329 tablet_service.cc:1467] Processing CreateTablet for tablet 4cdcdde158054cdaabc1856a90fe1302 (DEFAULT_TABLE table=rebalance_test_table_9 [id=615bc73a81674a899a85198265830843]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:39.328508 16480 consensus_queue.cc:260] T 7b4f203f4048490ba599353a4cd92748 P dcfdfa62edc34dab9525e2712d487582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:39.328656 15329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cdcdde158054cdaabc1856a90fe1302. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:39.328928 16480 ts_tablet_manager.cc:1398] T 7b4f203f4048490ba599353a4cd92748 P dcfdfa62edc34dab9525e2712d487582: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 02:00:39.329263 16481 tablet_bootstrap.cc:492] T de805ef7511d4ee9b9bbed67e0a4186c P 620596b893d84291b88b90dc9e317612: No bootstrap required, opened a new log
I20240503 02:00:39.329363 16481 ts_tablet_manager.cc:1367] T de805ef7511d4ee9b9bbed67e0a4186c P 620596b893d84291b88b90dc9e317612: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:00:39.329804 16481 raft_consensus.cc:348] T de805ef7511d4ee9b9bbed67e0a4186c P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:39.329906 16481 raft_consensus.cc:374] T de805ef7511d4ee9b9bbed67e0a4186c P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:39.329934 16481 raft_consensus.cc:729] T de805ef7511d4ee9b9bbed67e0a4186c P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 620596b893d84291b88b90dc9e317612, State: Initialized, Role: FOLLOWER
I20240503 02:00:39.330063 16481 consensus_queue.cc:260] T de805ef7511d4ee9b9bbed67e0a4186c P 620596b893d84291b88b90dc9e317612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:39.330315 16481 ts_tablet_manager.cc:1398] T de805ef7511d4ee9b9bbed67e0a4186c P 620596b893d84291b88b90dc9e317612: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:39.330485 16481 tablet_bootstrap.cc:492] T 4cdcdde158054cdaabc1856a90fe1302 P 620596b893d84291b88b90dc9e317612: Bootstrap starting.
I20240503 02:00:39.331048 15200 tablet_service.cc:1467] Processing CreateTablet for tablet 7b4f203f4048490ba599353a4cd92748 (DEFAULT_TABLE table=rebalance_test_table_9 [id=615bc73a81674a899a85198265830843]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:39.331298 15200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b4f203f4048490ba599353a4cd92748. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:39.331347 16481 tablet_bootstrap.cc:654] T 4cdcdde158054cdaabc1856a90fe1302 P 620596b893d84291b88b90dc9e317612: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:39.332654 16481 tablet_bootstrap.cc:492] T 4cdcdde158054cdaabc1856a90fe1302 P 620596b893d84291b88b90dc9e317612: No bootstrap required, opened a new log
I20240503 02:00:39.332762 16481 ts_tablet_manager.cc:1367] T 4cdcdde158054cdaabc1856a90fe1302 P 620596b893d84291b88b90dc9e317612: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:39.332877 16483 tablet_bootstrap.cc:492] T de805ef7511d4ee9b9bbed67e0a4186c P c05f7620bb884392931d3243a0b47a9e: No bootstrap required, opened a new log
I20240503 02:00:39.332985 16483 ts_tablet_manager.cc:1367] T de805ef7511d4ee9b9bbed67e0a4186c P c05f7620bb884392931d3243a0b47a9e: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240503 02:00:39.333209 16482 tablet_bootstrap.cc:492] T 7b4f203f4048490ba599353a4cd92748 P b602f6099cbb4758847e7e9550685372: No bootstrap required, opened a new log
I20240503 02:00:39.333315 16482 ts_tablet_manager.cc:1367] T 7b4f203f4048490ba599353a4cd92748 P b602f6099cbb4758847e7e9550685372: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 02:00:39.333263 16481 raft_consensus.cc:348] T 4cdcdde158054cdaabc1856a90fe1302 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:39.333361 16481 raft_consensus.cc:374] T 4cdcdde158054cdaabc1856a90fe1302 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:39.333421 16481 raft_consensus.cc:729] T 4cdcdde158054cdaabc1856a90fe1302 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 620596b893d84291b88b90dc9e317612, State: Initialized, Role: FOLLOWER
I20240503 02:00:39.333431 16483 raft_consensus.cc:348] T de805ef7511d4ee9b9bbed67e0a4186c P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:39.333554 16483 raft_consensus.cc:374] T de805ef7511d4ee9b9bbed67e0a4186c P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:39.333587 16483 raft_consensus.cc:729] T de805ef7511d4ee9b9bbed67e0a4186c P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Initialized, Role: FOLLOWER
I20240503 02:00:39.333719 16483 consensus_queue.cc:260] T de805ef7511d4ee9b9bbed67e0a4186c P c05f7620bb884392931d3243a0b47a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:39.333762 16482 raft_consensus.cc:348] T 7b4f203f4048490ba599353a4cd92748 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:39.333873 16482 raft_consensus.cc:374] T 7b4f203f4048490ba599353a4cd92748 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:39.333916 16482 raft_consensus.cc:729] T 7b4f203f4048490ba599353a4cd92748 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Initialized, Role: FOLLOWER
I20240503 02:00:39.334002 16483 ts_tablet_manager.cc:1398] T de805ef7511d4ee9b9bbed67e0a4186c P c05f7620bb884392931d3243a0b47a9e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:39.334044 16482 consensus_queue.cc:260] T 7b4f203f4048490ba599353a4cd92748 P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:39.334190 16483 tablet_bootstrap.cc:492] T 4cdcdde158054cdaabc1856a90fe1302 P c05f7620bb884392931d3243a0b47a9e: Bootstrap starting.
I20240503 02:00:39.334414 16482 ts_tablet_manager.cc:1398] T 7b4f203f4048490ba599353a4cd92748 P b602f6099cbb4758847e7e9550685372: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:39.335016 16481 consensus_queue.cc:260] T 4cdcdde158054cdaabc1856a90fe1302 P 620596b893d84291b88b90dc9e317612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:39.335165 16483 tablet_bootstrap.cc:654] T 4cdcdde158054cdaabc1856a90fe1302 P c05f7620bb884392931d3243a0b47a9e: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:39.335217 16482 tablet_bootstrap.cc:492] T de805ef7511d4ee9b9bbed67e0a4186c P b602f6099cbb4758847e7e9550685372: Bootstrap starting.
I20240503 02:00:39.335441 16481 ts_tablet_manager.cc:1398] T 4cdcdde158054cdaabc1856a90fe1302 P 620596b893d84291b88b90dc9e317612: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:00:39.336254 16482 tablet_bootstrap.cc:654] T de805ef7511d4ee9b9bbed67e0a4186c P b602f6099cbb4758847e7e9550685372: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:39.338254 16482 tablet_bootstrap.cc:492] T de805ef7511d4ee9b9bbed67e0a4186c P b602f6099cbb4758847e7e9550685372: No bootstrap required, opened a new log
I20240503 02:00:39.338392 16482 ts_tablet_manager.cc:1367] T de805ef7511d4ee9b9bbed67e0a4186c P b602f6099cbb4758847e7e9550685372: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:39.338609 16481 tablet_bootstrap.cc:492] T 7b4f203f4048490ba599353a4cd92748 P 620596b893d84291b88b90dc9e317612: Bootstrap starting.
I20240503 02:00:39.338891 16482 raft_consensus.cc:348] T de805ef7511d4ee9b9bbed67e0a4186c P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:39.339008 16482 raft_consensus.cc:374] T de805ef7511d4ee9b9bbed67e0a4186c P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:39.339051 16482 raft_consensus.cc:729] T de805ef7511d4ee9b9bbed67e0a4186c P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Initialized, Role: FOLLOWER
I20240503 02:00:39.339174 16482 consensus_queue.cc:260] T de805ef7511d4ee9b9bbed67e0a4186c P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:39.339462 16482 ts_tablet_manager.cc:1398] T de805ef7511d4ee9b9bbed67e0a4186c P b602f6099cbb4758847e7e9550685372: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 02:00:39.339666 16481 tablet_bootstrap.cc:654] T 7b4f203f4048490ba599353a4cd92748 P 620596b893d84291b88b90dc9e317612: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:39.340849 16483 tablet_bootstrap.cc:492] T 4cdcdde158054cdaabc1856a90fe1302 P c05f7620bb884392931d3243a0b47a9e: No bootstrap required, opened a new log
I20240503 02:00:39.340981 16483 ts_tablet_manager.cc:1367] T 4cdcdde158054cdaabc1856a90fe1302 P c05f7620bb884392931d3243a0b47a9e: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:00:39.341658 16481 tablet_bootstrap.cc:492] T 7b4f203f4048490ba599353a4cd92748 P 620596b893d84291b88b90dc9e317612: No bootstrap required, opened a new log
I20240503 02:00:39.341779 16481 ts_tablet_manager.cc:1367] T 7b4f203f4048490ba599353a4cd92748 P 620596b893d84291b88b90dc9e317612: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:39.342272 16481 raft_consensus.cc:348] T 7b4f203f4048490ba599353a4cd92748 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:39.342415 16481 raft_consensus.cc:374] T 7b4f203f4048490ba599353a4cd92748 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:39.342450 16481 raft_consensus.cc:729] T 7b4f203f4048490ba599353a4cd92748 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 620596b893d84291b88b90dc9e317612, State: Initialized, Role: FOLLOWER
I20240503 02:00:39.342401 16483 raft_consensus.cc:348] T 4cdcdde158054cdaabc1856a90fe1302 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:39.342545 16483 raft_consensus.cc:374] T 4cdcdde158054cdaabc1856a90fe1302 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:39.342584 16483 raft_consensus.cc:729] T 4cdcdde158054cdaabc1856a90fe1302 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Initialized, Role: FOLLOWER
I20240503 02:00:39.342579 16481 consensus_queue.cc:260] T 7b4f203f4048490ba599353a4cd92748 P 620596b893d84291b88b90dc9e317612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:39.342870 16481 ts_tablet_manager.cc:1398] T 7b4f203f4048490ba599353a4cd92748 P 620596b893d84291b88b90dc9e317612: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:39.343446 16483 consensus_queue.cc:260] T 4cdcdde158054cdaabc1856a90fe1302 P c05f7620bb884392931d3243a0b47a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:39.343823 16483 ts_tablet_manager.cc:1398] T 4cdcdde158054cdaabc1856a90fe1302 P c05f7620bb884392931d3243a0b47a9e: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:00:39.432026 16548 raft_consensus.cc:482] T 7b4f203f4048490ba599353a4cd92748 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:39.432168 16548 raft_consensus.cc:504] T 7b4f203f4048490ba599353a4cd92748 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:39.432615 16548 leader_election.cc:288] T 7b4f203f4048490ba599353a4cd92748 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207), dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817)
I20240503 02:00:39.433053 15088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4f203f4048490ba599353a4cd92748" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b602f6099cbb4758847e7e9550685372" is_pre_election: true
I20240503 02:00:39.433120 14956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4f203f4048490ba599353a4cd92748" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcfdfa62edc34dab9525e2712d487582" is_pre_election: true
I20240503 02:00:39.433199 15088 raft_consensus.cc:2454] T 7b4f203f4048490ba599353a4cd92748 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 0.
I20240503 02:00:39.433252 14956 raft_consensus.cc:2454] T 7b4f203f4048490ba599353a4cd92748 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 0.
I20240503 02:00:39.433499 15154 leader_election.cc:302] T 7b4f203f4048490ba599353a4cd92748 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, b602f6099cbb4758847e7e9550685372; no voters:
I20240503 02:00:39.433682 16548 raft_consensus.cc:2781] T 7b4f203f4048490ba599353a4cd92748 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:39.433737 16548 raft_consensus.cc:482] T 7b4f203f4048490ba599353a4cd92748 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:39.433768 16548 raft_consensus.cc:3037] T 7b4f203f4048490ba599353a4cd92748 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:39.434834 16548 raft_consensus.cc:504] T 7b4f203f4048490ba599353a4cd92748 P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:39.435212 16548 leader_election.cc:288] T 7b4f203f4048490ba599353a4cd92748 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 1 election: Requested vote from peers b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207), dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817)
I20240503 02:00:39.435607 14956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4f203f4048490ba599353a4cd92748" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:39.435592 15088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4f203f4048490ba599353a4cd92748" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:39.435719 14956 raft_consensus.cc:3037] T 7b4f203f4048490ba599353a4cd92748 P dcfdfa62edc34dab9525e2712d487582 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:39.435737 15088 raft_consensus.cc:3037] T 7b4f203f4048490ba599353a4cd92748 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:39.436527 15088 raft_consensus.cc:2454] T 7b4f203f4048490ba599353a4cd92748 P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 1.
I20240503 02:00:39.436532 14956 raft_consensus.cc:2454] T 7b4f203f4048490ba599353a4cd92748 P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 1.
I20240503 02:00:39.436842 15153 leader_election.cc:302] T 7b4f203f4048490ba599353a4cd92748 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:39.437037 16548 raft_consensus.cc:2781] T 7b4f203f4048490ba599353a4cd92748 P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:39.437124 16548 raft_consensus.cc:686] T 7b4f203f4048490ba599353a4cd92748 P 620596b893d84291b88b90dc9e317612 [term 1 LEADER]: Becoming Leader. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:39.437264 16548 consensus_queue.cc:237] T 7b4f203f4048490ba599353a4cd92748 P 620596b893d84291b88b90dc9e317612 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:39.439028 14815 catalog_manager.cc:5484] T 7b4f203f4048490ba599353a4cd92748 P 620596b893d84291b88b90dc9e317612 reported cstate change: term changed from 0 to 1, leader changed from <none> to 620596b893d84291b88b90dc9e317612 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "620596b893d84291b88b90dc9e317612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:39.444540 16548 raft_consensus.cc:482] T de805ef7511d4ee9b9bbed67e0a4186c P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:39.444689 16548 raft_consensus.cc:504] T de805ef7511d4ee9b9bbed67e0a4186c P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:39.445080 16548 leader_election.cc:288] T de805ef7511d4ee9b9bbed67e0a4186c P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
I20240503 02:00:39.445564 15088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de805ef7511d4ee9b9bbed67e0a4186c" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b602f6099cbb4758847e7e9550685372" is_pre_election: true
I20240503 02:00:39.445570 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de805ef7511d4ee9b9bbed67e0a4186c" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c05f7620bb884392931d3243a0b47a9e" is_pre_election: true
I20240503 02:00:39.445698 15088 raft_consensus.cc:2454] T de805ef7511d4ee9b9bbed67e0a4186c P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 0.
I20240503 02:00:39.445732 15351 raft_consensus.cc:2454] T de805ef7511d4ee9b9bbed67e0a4186c P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 0.
I20240503 02:00:39.446025 15154 leader_election.cc:302] T de805ef7511d4ee9b9bbed67e0a4186c P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, b602f6099cbb4758847e7e9550685372; no voters:
I20240503 02:00:39.446206 16548 raft_consensus.cc:2781] T de805ef7511d4ee9b9bbed67e0a4186c P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:39.446278 16548 raft_consensus.cc:482] T de805ef7511d4ee9b9bbed67e0a4186c P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:39.446310 16548 raft_consensus.cc:3037] T de805ef7511d4ee9b9bbed67e0a4186c P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:39.447185 16548 raft_consensus.cc:504] T de805ef7511d4ee9b9bbed67e0a4186c P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:39.447610 16548 leader_election.cc:288] T de805ef7511d4ee9b9bbed67e0a4186c P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 1 election: Requested vote from peers c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
I20240503 02:00:39.447983 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de805ef7511d4ee9b9bbed67e0a4186c" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:39.448119 15351 raft_consensus.cc:3037] T de805ef7511d4ee9b9bbed67e0a4186c P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:39.448077 15088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de805ef7511d4ee9b9bbed67e0a4186c" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:39.448204 15088 raft_consensus.cc:3037] T de805ef7511d4ee9b9bbed67e0a4186c P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:39.449060 15351 raft_consensus.cc:2454] T de805ef7511d4ee9b9bbed67e0a4186c P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 1.
I20240503 02:00:39.449095 15088 raft_consensus.cc:2454] T de805ef7511d4ee9b9bbed67e0a4186c P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 1.
I20240503 02:00:39.449443 15154 leader_election.cc:302] T de805ef7511d4ee9b9bbed67e0a4186c P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, b602f6099cbb4758847e7e9550685372; no voters:
I20240503 02:00:39.449673 16548 raft_consensus.cc:2781] T de805ef7511d4ee9b9bbed67e0a4186c P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:39.449792 16548 raft_consensus.cc:686] T de805ef7511d4ee9b9bbed67e0a4186c P 620596b893d84291b88b90dc9e317612 [term 1 LEADER]: Becoming Leader. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:39.449968 16548 consensus_queue.cc:237] T de805ef7511d4ee9b9bbed67e0a4186c P 620596b893d84291b88b90dc9e317612 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:39.451938 14815 catalog_manager.cc:5484] T de805ef7511d4ee9b9bbed67e0a4186c P 620596b893d84291b88b90dc9e317612 reported cstate change: term changed from 0 to 1, leader changed from <none> to 620596b893d84291b88b90dc9e317612 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "620596b893d84291b88b90dc9e317612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:39.543494 16548 raft_consensus.cc:482] T 4cdcdde158054cdaabc1856a90fe1302 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:39.543646 16548 raft_consensus.cc:504] T 4cdcdde158054cdaabc1856a90fe1302 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:39.544071 16548 leader_election.cc:288] T 4cdcdde158054cdaabc1856a90fe1302 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:39.544489 15088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcdde158054cdaabc1856a90fe1302" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b602f6099cbb4758847e7e9550685372" is_pre_election: true
I20240503 02:00:39.544490 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcdde158054cdaabc1856a90fe1302" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c05f7620bb884392931d3243a0b47a9e" is_pre_election: true
I20240503 02:00:39.544647 15088 raft_consensus.cc:2454] T 4cdcdde158054cdaabc1856a90fe1302 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 0.
I20240503 02:00:39.544642 15351 raft_consensus.cc:2454] T 4cdcdde158054cdaabc1856a90fe1302 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 0.
I20240503 02:00:39.544952 15154 leader_election.cc:302] T 4cdcdde158054cdaabc1856a90fe1302 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, b602f6099cbb4758847e7e9550685372; no voters:
I20240503 02:00:39.545145 16548 raft_consensus.cc:2781] T 4cdcdde158054cdaabc1856a90fe1302 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:39.545212 16548 raft_consensus.cc:482] T 4cdcdde158054cdaabc1856a90fe1302 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:39.545238 16548 raft_consensus.cc:3037] T 4cdcdde158054cdaabc1856a90fe1302 P 620596b893d84291b88b90dc9e317612 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:39.546083 16548 raft_consensus.cc:504] T 4cdcdde158054cdaabc1856a90fe1302 P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:39.546427 16548 leader_election.cc:288] T 4cdcdde158054cdaabc1856a90fe1302 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 1 election: Requested vote from peers b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:39.546821 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcdde158054cdaabc1856a90fe1302" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:39.546820 15088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcdde158054cdaabc1856a90fe1302" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:39.546957 15088 raft_consensus.cc:3037] T 4cdcdde158054cdaabc1856a90fe1302 P b602f6099cbb4758847e7e9550685372 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:39.546957 15351 raft_consensus.cc:3037] T 4cdcdde158054cdaabc1856a90fe1302 P c05f7620bb884392931d3243a0b47a9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:39.547940 15088 raft_consensus.cc:2454] T 4cdcdde158054cdaabc1856a90fe1302 P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 1.
I20240503 02:00:39.547923 15351 raft_consensus.cc:2454] T 4cdcdde158054cdaabc1856a90fe1302 P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 1.
I20240503 02:00:39.548207 15155 leader_election.cc:302] T 4cdcdde158054cdaabc1856a90fe1302 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, c05f7620bb884392931d3243a0b47a9e; no voters:
I20240503 02:00:39.548401 16548 raft_consensus.cc:2781] T 4cdcdde158054cdaabc1856a90fe1302 P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:39.548485 16548 raft_consensus.cc:686] T 4cdcdde158054cdaabc1856a90fe1302 P 620596b893d84291b88b90dc9e317612 [term 1 LEADER]: Becoming Leader. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:39.548628 16548 consensus_queue.cc:237] T 4cdcdde158054cdaabc1856a90fe1302 P 620596b893d84291b88b90dc9e317612 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:39.550648 14815 catalog_manager.cc:5484] T 4cdcdde158054cdaabc1856a90fe1302 P 620596b893d84291b88b90dc9e317612 reported cstate change: term changed from 0 to 1, leader changed from <none> to 620596b893d84291b88b90dc9e317612 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "620596b893d84291b88b90dc9e317612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } }
W20240503 02:00:39.688570 16552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:39.688941 16552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:39.692500 16552 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240503 02:00:39.787709 15351 raft_consensus.cc:1261] T 4cdcdde158054cdaabc1856a90fe1302 P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Refusing update from remote peer 620596b893d84291b88b90dc9e317612: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:39.787709 15088 raft_consensus.cc:1261] T 4cdcdde158054cdaabc1856a90fe1302 P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Refusing update from remote peer 620596b893d84291b88b90dc9e317612: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:39.788200 16458 consensus_queue.cc:1035] T 4cdcdde158054cdaabc1856a90fe1302 P 620596b893d84291b88b90dc9e317612 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:39.788357 16548 consensus_queue.cc:1035] T 4cdcdde158054cdaabc1856a90fe1302 P 620596b893d84291b88b90dc9e317612 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:39.790781 15089 raft_consensus.cc:1261] T de805ef7511d4ee9b9bbed67e0a4186c P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Refusing update from remote peer 620596b893d84291b88b90dc9e317612: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:39.791142 16458 consensus_queue.cc:1035] T de805ef7511d4ee9b9bbed67e0a4186c P 620596b893d84291b88b90dc9e317612 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:39.791203 15349 raft_consensus.cc:1261] T de805ef7511d4ee9b9bbed67e0a4186c P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Refusing update from remote peer 620596b893d84291b88b90dc9e317612: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:39.791551 16458 consensus_queue.cc:1035] T de805ef7511d4ee9b9bbed67e0a4186c P 620596b893d84291b88b90dc9e317612 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:39.793798 14957 raft_consensus.cc:1261] T 7b4f203f4048490ba599353a4cd92748 P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Refusing update from remote peer 620596b893d84291b88b90dc9e317612: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:39.794202 16458 consensus_queue.cc:1035] T 7b4f203f4048490ba599353a4cd92748 P 620596b893d84291b88b90dc9e317612 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:39.797473 15087 raft_consensus.cc:1261] T 7b4f203f4048490ba599353a4cd92748 P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Refusing update from remote peer 620596b893d84291b88b90dc9e317612: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:39.797963 16458 consensus_queue.cc:1035] T 7b4f203f4048490ba599353a4cd92748 P 620596b893d84291b88b90dc9e317612 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 123.438 ms
time per row: 0.0617189 ms
I20240503 02:00:39.863612 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
/tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:43821
--local_ip_for_outbound_sockets=127.0.99.133
--tserver_master_addrs=127.0.99.190:38675
--webserver_port=41511
--webserver_interface=127.0.99.133
--builtin_ntp_servers=127.0.99.148:43797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 02:00:39.979969 16607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:39.980233 16607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:39.980295 16607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:39.983573 16607 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:00:39.983675 16607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:39.983772 16607 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 02:00:39.987927 16607 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:43821
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=41511
--tserver_master_addrs=127.0.99.190:38675
--never_fsync=true
--heap_profile_path=/tmp/kudu.16607
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:39.988981 16607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:39.990190 16607 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:39.999222 16612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:39.999281 16615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:39.999466 16607 server_base.cc:1034] running on GCE node
W20240503 02:00:39.999222 16613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:39.999939 16607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:40.000676 16607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:40.001864 16607 hybrid_clock.cc:648] HybridClock initialized: now 1714701640001838 us; error 46 us; skew 500 ppm
I20240503 02:00:40.008677 16607 webserver.cc:460] Webserver started at http://127.0.99.133:41511/ using document root <none> and password file <none>
I20240503 02:00:40.009322 16607 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:40.009403 16607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:40.013546 16607 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 02:00:40.015705 16621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:40.016901 16607 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 02:00:40.017040 16607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "07a1b0ea41ae45b3945dc0d17d405d61"
format_stamp: "Formatted at 2024-05-03 02:00:29 on dist-test-slave-pz5w"
I20240503 02:00:40.017524 16607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:40.037621 16607 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:40.038295 16607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:40.038463 16607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:40.039029 16607 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:40.040230 16607 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:40.040285 16607 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:40.040335 16607 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:40.040364 16607 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:40.049757 16607 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:43821
I20240503 02:00:40.049791 16734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:43821 every 8 connection(s)
I20240503 02:00:40.051012 16607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 02:00:40.060046 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu as pid 16607
I20240503 02:00:40.060541 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
/tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:42067
--local_ip_for_outbound_sockets=127.0.99.134
--tserver_master_addrs=127.0.99.190:38675
--webserver_port=33791
--webserver_interface=127.0.99.134
--builtin_ntp_servers=127.0.99.148:43797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:00:40.061388 16735 heartbeater.cc:347] Connected to a master server at 127.0.99.190:38675
I20240503 02:00:40.061647 16735 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:40.062340 16735 heartbeater.cc:511] Master 127.0.99.190:38675 requested a full tablet report, sending...
I20240503 02:00:40.063737 14811 ts_manager.cc:194] Re-registered known tserver with Master: 07a1b0ea41ae45b3945dc0d17d405d61 (127.0.99.133:43821)
I20240503 02:00:40.064486 14811 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:39311
W20240503 02:00:40.183811 16738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:40.184082 16738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:40.184151 16738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:40.187623 16738 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:00:40.187717 16738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:40.187820 16738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 02:00:40.192461 16738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:42067
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=33791
--tserver_master_addrs=127.0.99.190:38675
--never_fsync=true
--heap_profile_path=/tmp/kudu.16738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:40.193449 16738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:40.194628 16738 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:40.203543 16743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:40.203543 16746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:40.203799 16744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:40.203748 16738 server_base.cc:1034] running on GCE node
I20240503 02:00:40.204267 16738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:40.204947 16738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:40.206115 16738 hybrid_clock.cc:648] HybridClock initialized: now 1714701640206089 us; error 49 us; skew 500 ppm
I20240503 02:00:40.208369 16738 webserver.cc:460] Webserver started at http://127.0.99.134:33791/ using document root <none> and password file <none>
I20240503 02:00:40.209106 16738 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:40.209184 16738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:40.212931 16738 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 02:00:40.215051 16752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:40.216151 16738 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:40.216277 16738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "7262cfc0983543f7b62e31d7a00ee5b2"
format_stamp: "Formatted at 2024-05-03 02:00:29 on dist-test-slave-pz5w"
I20240503 02:00:40.216781 16738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:40.262424 16738 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:40.263072 16738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:40.263240 16738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:40.263804 16738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:40.264784 16738 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:40.264833 16738 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:40.264891 16738 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:40.264915 16738 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:40.273598 16738 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:42067
I20240503 02:00:40.273631 16865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:42067 every 8 connection(s)
I20240503 02:00:40.274802 16738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 02:00:40.277702 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu as pid 16738
I20240503 02:00:40.278146 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
/tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:44193
--local_ip_for_outbound_sockets=127.0.99.135
--tserver_master_addrs=127.0.99.190:38675
--webserver_port=42245
--webserver_interface=127.0.99.135
--builtin_ntp_servers=127.0.99.148:43797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:00:40.292238 16866 heartbeater.cc:347] Connected to a master server at 127.0.99.190:38675
I20240503 02:00:40.292544 16866 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:40.293251 16866 heartbeater.cc:511] Master 127.0.99.190:38675 requested a full tablet report, sending...
I20240503 02:00:40.294368 14811 ts_manager.cc:194] Re-registered known tserver with Master: 7262cfc0983543f7b62e31d7a00ee5b2 (127.0.99.134:42067)
I20240503 02:00:40.294929 14811 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:58583
W20240503 02:00:40.412652 16869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:40.412910 16869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:40.412966 16869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:40.416365 16869 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:00:40.416513 16869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:40.416613 16869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 02:00:40.421109 16869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:44193
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=42245
--tserver_master_addrs=127.0.99.190:38675
--never_fsync=true
--heap_profile_path=/tmp/kudu.16869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:40.422319 16869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:40.423877 16869 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:40.432907 16877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:40.432907 16874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:40.432914 16875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:40.433166 16869 server_base.cc:1034] running on GCE node
I20240503 02:00:40.433619 16869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:40.434243 16869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:40.435415 16869 hybrid_clock.cc:648] HybridClock initialized: now 1714701640435395 us; error 31 us; skew 500 ppm
I20240503 02:00:40.437711 16869 webserver.cc:460] Webserver started at http://127.0.99.135:42245/ using document root <none> and password file <none>
I20240503 02:00:40.438316 16869 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:40.438395 16869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:40.442183 16869 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 02:00:40.444365 16883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:40.445691 16869 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:40.445830 16869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "ce266197c0f4472ead9d54d16e706204"
format_stamp: "Formatted at 2024-05-03 02:00:30 on dist-test-slave-pz5w"
I20240503 02:00:40.446310 16869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:40.459147 16869 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:40.459829 16869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:40.459993 16869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:40.460574 16869 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:40.461506 16869 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:40.461557 16869 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:40.461620 16869 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:40.461644 16869 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:40.471954 16869 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:44193
I20240503 02:00:40.471974 16997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:44193 every 8 connection(s)
I20240503 02:00:40.473179 16869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 02:00:40.476982 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu as pid 16869
I20240503 02:00:40.477391 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
/tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:41611
--local_ip_for_outbound_sockets=127.0.99.136
--tserver_master_addrs=127.0.99.190:38675
--webserver_port=45309
--webserver_interface=127.0.99.136
--builtin_ntp_servers=127.0.99.148:43797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:00:40.485129 16998 heartbeater.cc:347] Connected to a master server at 127.0.99.190:38675
I20240503 02:00:40.485414 16998 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:40.486200 16998 heartbeater.cc:511] Master 127.0.99.190:38675 requested a full tablet report, sending...
I20240503 02:00:40.487399 14811 ts_manager.cc:194] Re-registered known tserver with Master: ce266197c0f4472ead9d54d16e706204 (127.0.99.135:44193)
I20240503 02:00:40.487994 14811 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:37907
W20240503 02:00:40.598644 17001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:40.598910 17001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:40.598954 17001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:40.602856 17001 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:00:40.602944 17001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:40.603031 17001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240503 02:00:40.607410 17001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:41611
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=45309
--tserver_master_addrs=127.0.99.190:38675
--never_fsync=true
--heap_profile_path=/tmp/kudu.17001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:40.608675 17001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:40.610037 17001 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:40.618731 17006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:40.618861 17007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:40.619392 17009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:40.619607 17001 server_base.cc:1034] running on GCE node
I20240503 02:00:40.620051 17001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:40.620708 17001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:40.621922 17001 hybrid_clock.cc:648] HybridClock initialized: now 1714701640621884 us; error 53 us; skew 500 ppm
I20240503 02:00:40.624212 17001 webserver.cc:460] Webserver started at http://127.0.99.136:45309/ using document root <none> and password file <none>
I20240503 02:00:40.624838 17001 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:40.624914 17001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:40.628829 17001 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 02:00:40.631570 17016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:40.633373 17001 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:40.633517 17001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "223c02bbc6d9479595889809d7384dae"
format_stamp: "Formatted at 2024-05-03 02:00:30 on dist-test-slave-pz5w"
I20240503 02:00:40.633977 17001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:40.651383 17001 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:40.652221 17001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:40.652455 17001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:40.653144 17001 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:40.654189 17001 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:40.654242 17001 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:40.654289 17001 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:40.654335 17001 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:40.663559 17001 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:41611
I20240503 02:00:40.664433 17129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:41611 every 8 connection(s)
I20240503 02:00:40.664847 17001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQxiMKQ/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714701555658200-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 02:00:40.674083 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQxiMKQ/build/debug/bin/kudu as pid 17001
I20240503 02:00:40.674343 398 test_util.cc:268] Using random seed: 372255782
I20240503 02:00:40.675222 17130 heartbeater.cc:347] Connected to a master server at 127.0.99.190:38675
I20240503 02:00:40.675523 17130 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:40.676261 17130 heartbeater.cc:511] Master 127.0.99.190:38675 requested a full tablet report, sending...
I20240503 02:00:40.677563 14811 ts_manager.cc:194] Re-registered known tserver with Master: 223c02bbc6d9479595889809d7384dae (127.0.99.136:41611)
I20240503 02:00:40.678123 14811 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:40593
I20240503 02:00:40.686105 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_0 already exists
I20240503 02:00:40.686379 398 test_util.cc:268] Using random seed: 372267815
I20240503 02:00:40.695705 15349 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4cdcdde158054cdaabc1856a90fe1302"
dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
from {username='slave'} at 127.0.0.1:50864
I20240503 02:00:40.695924 15349 raft_consensus.cc:482] T 4cdcdde158054cdaabc1856a90fe1302 P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:40.696000 15349 raft_consensus.cc:3037] T 4cdcdde158054cdaabc1856a90fe1302 P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.697153 15349 raft_consensus.cc:504] T 4cdcdde158054cdaabc1856a90fe1302 P c05f7620bb884392931d3243a0b47a9e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:40.697530 15349 leader_election.cc:288] T 4cdcdde158054cdaabc1856a90fe1302 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 2 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
I20240503 02:00:40.698107 15219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcdde158054cdaabc1856a90fe1302" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:40.698222 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_1 already exists
I20240503 02:00:40.698247 15219 raft_consensus.cc:3032] T 4cdcdde158054cdaabc1856a90fe1302 P 620596b893d84291b88b90dc9e317612 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:00:40.698287 15219 raft_consensus.cc:729] T 4cdcdde158054cdaabc1856a90fe1302 P 620596b893d84291b88b90dc9e317612 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:40.698410 398 test_util.cc:268] Using random seed: 372279849
I20240503 02:00:40.698487 15219 consensus_queue.cc:260] T 4cdcdde158054cdaabc1856a90fe1302 P 620596b893d84291b88b90dc9e317612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:40.698776 15219 raft_consensus.cc:3037] T 4cdcdde158054cdaabc1856a90fe1302 P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.698879 15087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcdde158054cdaabc1856a90fe1302" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:40.698995 15087 raft_consensus.cc:3037] T 4cdcdde158054cdaabc1856a90fe1302 P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.699903 15219 raft_consensus.cc:2454] T 4cdcdde158054cdaabc1856a90fe1302 P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 2.
I20240503 02:00:40.699913 15087 raft_consensus.cc:2454] T 4cdcdde158054cdaabc1856a90fe1302 P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 2.
I20240503 02:00:40.700664 15285 leader_election.cc:302] T 4cdcdde158054cdaabc1856a90fe1302 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b602f6099cbb4758847e7e9550685372, c05f7620bb884392931d3243a0b47a9e; no voters:
I20240503 02:00:40.701246 16986 raft_consensus.cc:2781] T 4cdcdde158054cdaabc1856a90fe1302 P c05f7620bb884392931d3243a0b47a9e [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:40.701334 16986 raft_consensus.cc:686] T 4cdcdde158054cdaabc1856a90fe1302 P c05f7620bb884392931d3243a0b47a9e [term 2 LEADER]: Becoming Leader. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:40.701486 16986 consensus_queue.cc:237] T 4cdcdde158054cdaabc1856a90fe1302 P c05f7620bb884392931d3243a0b47a9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:40.703003 15349 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "de805ef7511d4ee9b9bbed67e0a4186c"
dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
from {username='slave'} at 127.0.0.1:50864
I20240503 02:00:40.703156 15349 raft_consensus.cc:482] T de805ef7511d4ee9b9bbed67e0a4186c P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:40.703222 15349 raft_consensus.cc:3037] T de805ef7511d4ee9b9bbed67e0a4186c P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.703644 14811 catalog_manager.cc:5484] T 4cdcdde158054cdaabc1856a90fe1302 P c05f7620bb884392931d3243a0b47a9e reported cstate change: term changed from 1 to 2, leader changed from 620596b893d84291b88b90dc9e317612 (127.0.99.131) to c05f7620bb884392931d3243a0b47a9e (127.0.99.132). New cstate: current_term: 2 leader_uuid: "c05f7620bb884392931d3243a0b47a9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:40.704268 15349 raft_consensus.cc:504] T de805ef7511d4ee9b9bbed67e0a4186c P c05f7620bb884392931d3243a0b47a9e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:40.704640 15349 leader_election.cc:288] T de805ef7511d4ee9b9bbed67e0a4186c P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 2 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
I20240503 02:00:40.705152 15089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de805ef7511d4ee9b9bbed67e0a4186c" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:40.705288 15089 raft_consensus.cc:3037] T de805ef7511d4ee9b9bbed67e0a4186c P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.706286 15219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de805ef7511d4ee9b9bbed67e0a4186c" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:40.706437 15219 raft_consensus.cc:3032] T de805ef7511d4ee9b9bbed67e0a4186c P 620596b893d84291b88b90dc9e317612 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:00:40.706491 15219 raft_consensus.cc:729] T de805ef7511d4ee9b9bbed67e0a4186c P 620596b893d84291b88b90dc9e317612 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:40.706507 15089 raft_consensus.cc:2454] T de805ef7511d4ee9b9bbed67e0a4186c P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 2.
I20240503 02:00:40.706657 15219 consensus_queue.cc:260] T de805ef7511d4ee9b9bbed67e0a4186c P 620596b893d84291b88b90dc9e317612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:40.706854 15219 raft_consensus.cc:3037] T de805ef7511d4ee9b9bbed67e0a4186c P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.706840 15285 leader_election.cc:302] T de805ef7511d4ee9b9bbed67e0a4186c P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b602f6099cbb4758847e7e9550685372, c05f7620bb884392931d3243a0b47a9e; no voters:
I20240503 02:00:40.707080 16986 raft_consensus.cc:2781] T de805ef7511d4ee9b9bbed67e0a4186c P c05f7620bb884392931d3243a0b47a9e [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:40.707197 16986 raft_consensus.cc:686] T de805ef7511d4ee9b9bbed67e0a4186c P c05f7620bb884392931d3243a0b47a9e [term 2 LEADER]: Becoming Leader. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:40.707335 16986 consensus_queue.cc:237] T de805ef7511d4ee9b9bbed67e0a4186c P c05f7620bb884392931d3243a0b47a9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:40.707896 15219 raft_consensus.cc:2454] T de805ef7511d4ee9b9bbed67e0a4186c P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 2.
I20240503 02:00:40.708425 15349 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "21d92d412a5247e8907c5c41d253ad2e"
dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
from {username='slave'} at 127.0.0.1:50864
I20240503 02:00:40.708544 15349 raft_consensus.cc:482] T 21d92d412a5247e8907c5c41d253ad2e P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:40.708592 15349 raft_consensus.cc:3037] T 21d92d412a5247e8907c5c41d253ad2e P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.709993 15349 raft_consensus.cc:504] T 21d92d412a5247e8907c5c41d253ad2e P c05f7620bb884392931d3243a0b47a9e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:40.710968 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21d92d412a5247e8907c5c41d253ad2e" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:40.711118 14957 raft_consensus.cc:3032] T 21d92d412a5247e8907c5c41d253ad2e P dcfdfa62edc34dab9525e2712d487582 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:00:40.711170 14957 raft_consensus.cc:729] T 21d92d412a5247e8907c5c41d253ad2e P dcfdfa62edc34dab9525e2712d487582 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:40.711354 14957 consensus_queue.cc:260] T 21d92d412a5247e8907c5c41d253ad2e P dcfdfa62edc34dab9525e2712d487582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:40.711656 14957 raft_consensus.cc:3037] T 21d92d412a5247e8907c5c41d253ad2e P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.712765 14957 raft_consensus.cc:2454] T 21d92d412a5247e8907c5c41d253ad2e P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 2.
I20240503 02:00:40.713110 15284 leader_election.cc:302] T 21d92d412a5247e8907c5c41d253ad2e P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c05f7620bb884392931d3243a0b47a9e, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:40.713295 16474 raft_consensus.cc:2781] T 21d92d412a5247e8907c5c41d253ad2e P c05f7620bb884392931d3243a0b47a9e [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:40.713373 16474 raft_consensus.cc:686] T 21d92d412a5247e8907c5c41d253ad2e P c05f7620bb884392931d3243a0b47a9e [term 2 LEADER]: Becoming Leader. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:40.713498 16474 consensus_queue.cc:237] T 21d92d412a5247e8907c5c41d253ad2e P c05f7620bb884392931d3243a0b47a9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:40.716960 15219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21d92d412a5247e8907c5c41d253ad2e" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:40.717125 15219 raft_consensus.cc:3037] T 21d92d412a5247e8907c5c41d253ad2e P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.718037 15219 raft_consensus.cc:2454] T 21d92d412a5247e8907c5c41d253ad2e P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 2.
I20240503 02:00:40.718467 15349 leader_election.cc:288] T 21d92d412a5247e8907c5c41d253ad2e P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 2 election: Requested vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437)
I20240503 02:00:40.719496 15349 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9e9899666b15498791f62dab6d9dda16"
dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
from {username='slave'} at 127.0.0.1:50864
I20240503 02:00:40.719633 15349 raft_consensus.cc:482] T 9e9899666b15498791f62dab6d9dda16 P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:40.719696 15349 raft_consensus.cc:3037] T 9e9899666b15498791f62dab6d9dda16 P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.720788 15349 raft_consensus.cc:504] T 9e9899666b15498791f62dab6d9dda16 P c05f7620bb884392931d3243a0b47a9e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:40.721187 15349 leader_election.cc:288] T 9e9899666b15498791f62dab6d9dda16 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 2 election: Requested vote from peers b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207), 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437)
I20240503 02:00:40.721611 15219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e9899666b15498791f62dab6d9dda16" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:40.721729 15219 raft_consensus.cc:3032] T 9e9899666b15498791f62dab6d9dda16 P 620596b893d84291b88b90dc9e317612 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:00:40.721767 15219 raft_consensus.cc:729] T 9e9899666b15498791f62dab6d9dda16 P 620596b893d84291b88b90dc9e317612 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:40.721933 15219 consensus_queue.cc:260] T 9e9899666b15498791f62dab6d9dda16 P 620596b893d84291b88b90dc9e317612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:40.722144 15219 raft_consensus.cc:3037] T 9e9899666b15498791f62dab6d9dda16 P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.723171 15089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e9899666b15498791f62dab6d9dda16" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:40.723259 15219 raft_consensus.cc:2454] T 9e9899666b15498791f62dab6d9dda16 P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 2.
I20240503 02:00:40.723304 15089 raft_consensus.cc:3037] T 9e9899666b15498791f62dab6d9dda16 P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.723464 14811 catalog_manager.cc:5484] T de805ef7511d4ee9b9bbed67e0a4186c P c05f7620bb884392931d3243a0b47a9e reported cstate change: term changed from 1 to 2, leader changed from 620596b893d84291b88b90dc9e317612 (127.0.99.131) to c05f7620bb884392931d3243a0b47a9e (127.0.99.132). New cstate: current_term: 2 leader_uuid: "c05f7620bb884392931d3243a0b47a9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:40.723742 14811 catalog_manager.cc:5484] T 21d92d412a5247e8907c5c41d253ad2e P c05f7620bb884392931d3243a0b47a9e reported cstate change: term changed from 1 to 2, leader changed from dcfdfa62edc34dab9525e2712d487582 (127.0.99.129) to c05f7620bb884392931d3243a0b47a9e (127.0.99.132). New cstate: current_term: 2 leader_uuid: "c05f7620bb884392931d3243a0b47a9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:40.724318 15089 raft_consensus.cc:2454] T 9e9899666b15498791f62dab6d9dda16 P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 2.
I20240503 02:00:40.724514 15286 leader_election.cc:302] T 9e9899666b15498791f62dab6d9dda16 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, c05f7620bb884392931d3243a0b47a9e; no voters:
I20240503 02:00:40.724735 17162 raft_consensus.cc:2781] T 9e9899666b15498791f62dab6d9dda16 P c05f7620bb884392931d3243a0b47a9e [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:40.724820 17162 raft_consensus.cc:686] T 9e9899666b15498791f62dab6d9dda16 P c05f7620bb884392931d3243a0b47a9e [term 2 LEADER]: Becoming Leader. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:40.725265 17162 consensus_queue.cc:237] T 9e9899666b15498791f62dab6d9dda16 P c05f7620bb884392931d3243a0b47a9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:40.728547 15349 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "50c5abd860ae4b09bb289e9c7550c84e"
dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
from {username='slave'} at 127.0.0.1:50864
I20240503 02:00:40.728693 15349 raft_consensus.cc:465] T 50c5abd860ae4b09bb289e9c7550c84e P c05f7620bb884392931d3243a0b47a9e [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240503 02:00:40.729429 15349 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7fc3330dfd0644f6a303659b960124f4"
dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
from {username='slave'} at 127.0.0.1:50864
I20240503 02:00:40.729566 15349 raft_consensus.cc:482] T 7fc3330dfd0644f6a303659b960124f4 P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:40.729652 15349 raft_consensus.cc:3037] T 7fc3330dfd0644f6a303659b960124f4 P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.730616 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_2 already exists
I20240503 02:00:40.730690 15349 raft_consensus.cc:504] T 7fc3330dfd0644f6a303659b960124f4 P c05f7620bb884392931d3243a0b47a9e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:40.730810 398 test_util.cc:268] Using random seed: 372312248
I20240503 02:00:40.731086 15349 leader_election.cc:288] T 7fc3330dfd0644f6a303659b960124f4 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 2 election: Requested vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437)
I20240503 02:00:40.731702 14956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fc3330dfd0644f6a303659b960124f4" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:40.731829 14956 raft_consensus.cc:3037] T 7fc3330dfd0644f6a303659b960124f4 P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.732443 15349 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c74522fbb5c64d27b5c95f05896b0716"
dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
from {username='slave'} at 127.0.0.1:50864
I20240503 02:00:40.732479 15219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fc3330dfd0644f6a303659b960124f4" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:40.732555 15349 raft_consensus.cc:482] T c74522fbb5c64d27b5c95f05896b0716 P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:40.732605 15349 raft_consensus.cc:3037] T c74522fbb5c64d27b5c95f05896b0716 P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.732606 15219 raft_consensus.cc:3032] T 7fc3330dfd0644f6a303659b960124f4 P 620596b893d84291b88b90dc9e317612 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:00:40.732645 15219 raft_consensus.cc:729] T 7fc3330dfd0644f6a303659b960124f4 P 620596b893d84291b88b90dc9e317612 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:40.732782 15219 consensus_queue.cc:260] T 7fc3330dfd0644f6a303659b960124f4 P 620596b893d84291b88b90dc9e317612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:40.732903 14956 raft_consensus.cc:2454] T 7fc3330dfd0644f6a303659b960124f4 P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 2.
I20240503 02:00:40.732946 15219 raft_consensus.cc:3037] T 7fc3330dfd0644f6a303659b960124f4 P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.733551 15349 raft_consensus.cc:504] T c74522fbb5c64d27b5c95f05896b0716 P c05f7620bb884392931d3243a0b47a9e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:40.733857 15219 raft_consensus.cc:2454] T 7fc3330dfd0644f6a303659b960124f4 P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 2.
I20240503 02:00:40.734544 15286 leader_election.cc:302] T 7fc3330dfd0644f6a303659b960124f4 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, c05f7620bb884392931d3243a0b47a9e; no voters:
I20240503 02:00:40.734704 15219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c74522fbb5c64d27b5c95f05896b0716" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:40.734809 15219 raft_consensus.cc:3037] T c74522fbb5c64d27b5c95f05896b0716 P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.734884 16474 raft_consensus.cc:2781] T 7fc3330dfd0644f6a303659b960124f4 P c05f7620bb884392931d3243a0b47a9e [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:40.734953 16474 raft_consensus.cc:686] T 7fc3330dfd0644f6a303659b960124f4 P c05f7620bb884392931d3243a0b47a9e [term 2 LEADER]: Becoming Leader. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:40.735074 16474 consensus_queue.cc:237] T 7fc3330dfd0644f6a303659b960124f4 P c05f7620bb884392931d3243a0b47a9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:40.735849 15219 raft_consensus.cc:2454] T c74522fbb5c64d27b5c95f05896b0716 P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 2.
I20240503 02:00:40.736083 15089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c74522fbb5c64d27b5c95f05896b0716" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:40.736198 15089 raft_consensus.cc:3032] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:00:40.736239 15089 raft_consensus.cc:729] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:40.736454 15089 consensus_queue.cc:260] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:40.736631 15349 leader_election.cc:288] T c74522fbb5c64d27b5c95f05896b0716 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 2 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
I20240503 02:00:40.736752 15089 raft_consensus.cc:3037] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.736903 15286 leader_election.cc:302] T c74522fbb5c64d27b5c95f05896b0716 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, c05f7620bb884392931d3243a0b47a9e; no voters:
I20240503 02:00:40.737604 17162 raft_consensus.cc:2781] T c74522fbb5c64d27b5c95f05896b0716 P c05f7620bb884392931d3243a0b47a9e [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:40.737679 15089 raft_consensus.cc:2454] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 2.
I20240503 02:00:40.737702 17162 raft_consensus.cc:686] T c74522fbb5c64d27b5c95f05896b0716 P c05f7620bb884392931d3243a0b47a9e [term 2 LEADER]: Becoming Leader. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:40.737825 17162 consensus_queue.cc:237] T c74522fbb5c64d27b5c95f05896b0716 P c05f7620bb884392931d3243a0b47a9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:40.738909 15349 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6d2a1296a54a4b92a03c12954de0cc34"
dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
from {username='slave'} at 127.0.0.1:50864
I20240503 02:00:40.739143 15349 raft_consensus.cc:482] T 6d2a1296a54a4b92a03c12954de0cc34 P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:40.740062 15349 raft_consensus.cc:3037] T 6d2a1296a54a4b92a03c12954de0cc34 P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.741067 15349 raft_consensus.cc:504] T 6d2a1296a54a4b92a03c12954de0cc34 P c05f7620bb884392931d3243a0b47a9e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:40.741459 15349 leader_election.cc:288] T 6d2a1296a54a4b92a03c12954de0cc34 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 2 election: Requested vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
I20240503 02:00:40.741405 14811 catalog_manager.cc:5484] T 9e9899666b15498791f62dab6d9dda16 P c05f7620bb884392931d3243a0b47a9e reported cstate change: term changed from 1 to 2, leader changed from 620596b893d84291b88b90dc9e317612 (127.0.99.131) to c05f7620bb884392931d3243a0b47a9e (127.0.99.132). New cstate: current_term: 2 leader_uuid: "c05f7620bb884392931d3243a0b47a9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:40.741708 14811 catalog_manager.cc:5484] T c74522fbb5c64d27b5c95f05896b0716 P c05f7620bb884392931d3243a0b47a9e reported cstate change: term changed from 1 to 2, leader changed from b602f6099cbb4758847e7e9550685372 (127.0.99.130) to c05f7620bb884392931d3243a0b47a9e (127.0.99.132). New cstate: current_term: 2 leader_uuid: "c05f7620bb884392931d3243a0b47a9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:40.741964 14811 catalog_manager.cc:5484] T 7fc3330dfd0644f6a303659b960124f4 P c05f7620bb884392931d3243a0b47a9e reported cstate change: term changed from 1 to 2, leader changed from 620596b893d84291b88b90dc9e317612 (127.0.99.131) to c05f7620bb884392931d3243a0b47a9e (127.0.99.132). New cstate: current_term: 2 leader_uuid: "c05f7620bb884392931d3243a0b47a9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:40.743897 15089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2a1296a54a4b92a03c12954de0cc34" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:40.744028 15089 raft_consensus.cc:3032] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:00:40.744068 15089 raft_consensus.cc:729] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:40.744578 15089 consensus_queue.cc:260] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:40.744769 15089 raft_consensus.cc:3037] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.745692 14956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2a1296a54a4b92a03c12954de0cc34" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:40.745815 14956 raft_consensus.cc:3037] T 6d2a1296a54a4b92a03c12954de0cc34 P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.746824 14956 raft_consensus.cc:2454] T 6d2a1296a54a4b92a03c12954de0cc34 P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 2.
I20240503 02:00:40.747583 15284 leader_election.cc:302] T 6d2a1296a54a4b92a03c12954de0cc34 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c05f7620bb884392931d3243a0b47a9e, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:40.747850 16986 raft_consensus.cc:2781] T 6d2a1296a54a4b92a03c12954de0cc34 P c05f7620bb884392931d3243a0b47a9e [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:40.747928 16986 raft_consensus.cc:686] T 6d2a1296a54a4b92a03c12954de0cc34 P c05f7620bb884392931d3243a0b47a9e [term 2 LEADER]: Becoming Leader. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
W20240503 02:00:40.748021 15048 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34006: Illegal state: replica b602f6099cbb4758847e7e9550685372 is not leader of this config: current role FOLLOWER
I20240503 02:00:40.748056 16986 consensus_queue.cc:237] T 6d2a1296a54a4b92a03c12954de0cc34 P c05f7620bb884392931d3243a0b47a9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:40.749588 15089 raft_consensus.cc:2454] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 2.
I20240503 02:00:40.749876 15349 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5736c284a85540a1bbab2be6df672fc1"
dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
from {username='slave'} at 127.0.0.1:50864
I20240503 02:00:40.750141 15349 raft_consensus.cc:465] T 5736c284a85540a1bbab2be6df672fc1 P c05f7620bb884392931d3243a0b47a9e [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240503 02:00:40.750941 15351 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eade118a36da42e3ac08b96a13893a5c"
dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
from {username='slave'} at 127.0.0.1:50864
I20240503 02:00:40.751045 15351 raft_consensus.cc:465] T eade118a36da42e3ac08b96a13893a5c P c05f7620bb884392931d3243a0b47a9e [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240503 02:00:40.754211 15349 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9a74cca2988a4f219f6fd9b07ade42d8"
dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
from {username='slave'} at 127.0.0.1:50864
I20240503 02:00:40.754360 15349 raft_consensus.cc:482] T 9a74cca2988a4f219f6fd9b07ade42d8 P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:40.754415 15349 raft_consensus.cc:3037] T 9a74cca2988a4f219f6fd9b07ade42d8 P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.755522 15349 raft_consensus.cc:504] T 9a74cca2988a4f219f6fd9b07ade42d8 P c05f7620bb884392931d3243a0b47a9e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:40.756183 14956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a74cca2988a4f219f6fd9b07ade42d8" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:40.756300 14956 raft_consensus.cc:3037] T 9a74cca2988a4f219f6fd9b07ade42d8 P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.757328 14956 raft_consensus.cc:2454] T 9a74cca2988a4f219f6fd9b07ade42d8 P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 2.
I20240503 02:00:40.757978 15284 leader_election.cc:302] T 9a74cca2988a4f219f6fd9b07ade42d8 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c05f7620bb884392931d3243a0b47a9e, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:40.758297 15349 leader_election.cc:288] T 9a74cca2988a4f219f6fd9b07ade42d8 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 2 election: Requested vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437)
W20240503 02:00:40.758630 14915 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47940: Illegal state: replica dcfdfa62edc34dab9525e2712d487582 is not leader of this config: current role FOLLOWER
I20240503 02:00:40.758819 17162 raft_consensus.cc:2781] T 9a74cca2988a4f219f6fd9b07ade42d8 P c05f7620bb884392931d3243a0b47a9e [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:40.758913 17162 raft_consensus.cc:686] T 9a74cca2988a4f219f6fd9b07ade42d8 P c05f7620bb884392931d3243a0b47a9e [term 2 LEADER]: Becoming Leader. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:40.759047 17162 consensus_queue.cc:237] T 9a74cca2988a4f219f6fd9b07ade42d8 P c05f7620bb884392931d3243a0b47a9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:40.759429 14816 catalog_manager.cc:5484] T 6d2a1296a54a4b92a03c12954de0cc34 P c05f7620bb884392931d3243a0b47a9e reported cstate change: term changed from 1 to 2, leader changed from b602f6099cbb4758847e7e9550685372 (127.0.99.130) to c05f7620bb884392931d3243a0b47a9e (127.0.99.132). New cstate: current_term: 2 leader_uuid: "c05f7620bb884392931d3243a0b47a9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:40.760601 15220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a74cca2988a4f219f6fd9b07ade42d8" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:40.760741 15220 raft_consensus.cc:3032] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:00:40.760780 15220 raft_consensus.cc:729] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:40.760929 15220 consensus_queue.cc:260] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:40.761128 15220 raft_consensus.cc:3037] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.762025 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_3 already exists
I20240503 02:00:40.762158 15220 raft_consensus.cc:2454] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 2.
I20240503 02:00:40.762194 398 test_util.cc:268] Using random seed: 372343633
I20240503 02:00:40.763965 15349 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d805ee04df39406ba30b3b3b9e349507"
dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
from {username='slave'} at 127.0.0.1:50864
I20240503 02:00:40.764245 15349 raft_consensus.cc:465] T d805ee04df39406ba30b3b3b9e349507 P c05f7620bb884392931d3243a0b47a9e [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240503 02:00:40.765385 15349 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6798a9ed0cf441aca7b8ae1b05c001f3"
dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
from {username='slave'} at 127.0.0.1:50864
I20240503 02:00:40.765503 15349 raft_consensus.cc:482] T 6798a9ed0cf441aca7b8ae1b05c001f3 P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:40.765565 15349 raft_consensus.cc:3037] T 6798a9ed0cf441aca7b8ae1b05c001f3 P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.766898 15349 raft_consensus.cc:504] T 6798a9ed0cf441aca7b8ae1b05c001f3 P c05f7620bb884392931d3243a0b47a9e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:40.767462 14816 catalog_manager.cc:5484] T 9a74cca2988a4f219f6fd9b07ade42d8 P c05f7620bb884392931d3243a0b47a9e reported cstate change: term changed from 1 to 2, leader changed from 620596b893d84291b88b90dc9e317612 (127.0.99.131) to c05f7620bb884392931d3243a0b47a9e (127.0.99.132). New cstate: current_term: 2 leader_uuid: "c05f7620bb884392931d3243a0b47a9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:40.767629 15219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6798a9ed0cf441aca7b8ae1b05c001f3" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:40.767755 15219 raft_consensus.cc:3037] T 6798a9ed0cf441aca7b8ae1b05c001f3 P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.768807 15219 raft_consensus.cc:2454] T 6798a9ed0cf441aca7b8ae1b05c001f3 P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 2.
I20240503 02:00:40.770151 15286 leader_election.cc:302] T 6798a9ed0cf441aca7b8ae1b05c001f3 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, c05f7620bb884392931d3243a0b47a9e; no voters:
I20240503 02:00:40.770866 17163 raft_consensus.cc:2781] T 6798a9ed0cf441aca7b8ae1b05c001f3 P c05f7620bb884392931d3243a0b47a9e [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:40.770962 17163 raft_consensus.cc:686] T 6798a9ed0cf441aca7b8ae1b05c001f3 P c05f7620bb884392931d3243a0b47a9e [term 2 LEADER]: Becoming Leader. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:40.771102 17163 consensus_queue.cc:237] T 6798a9ed0cf441aca7b8ae1b05c001f3 P c05f7620bb884392931d3243a0b47a9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:40.772183 15349 leader_election.cc:288] T 6798a9ed0cf441aca7b8ae1b05c001f3 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 2 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817)
I20240503 02:00:40.772760 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6798a9ed0cf441aca7b8ae1b05c001f3" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:40.772886 14957 raft_consensus.cc:3032] T 6798a9ed0cf441aca7b8ae1b05c001f3 P dcfdfa62edc34dab9525e2712d487582 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:00:40.772925 14957 raft_consensus.cc:729] T 6798a9ed0cf441aca7b8ae1b05c001f3 P dcfdfa62edc34dab9525e2712d487582 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:40.773056 14957 consensus_queue.cc:260] T 6798a9ed0cf441aca7b8ae1b05c001f3 P dcfdfa62edc34dab9525e2712d487582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:40.773173 14957 raft_consensus.cc:3037] T 6798a9ed0cf441aca7b8ae1b05c001f3 P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.773249 15219 raft_consensus.cc:1226] T 6798a9ed0cf441aca7b8ae1b05c001f3 P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Rejecting Update request from peer dcfdfa62edc34dab9525e2712d487582 for earlier term 1. Current term is 2. Ops: [1.4-1.4]
I20240503 02:00:40.773725 15349 raft_consensus.cc:1226] T 6798a9ed0cf441aca7b8ae1b05c001f3 P c05f7620bb884392931d3243a0b47a9e [term 2 LEADER]: Rejecting Update request from peer dcfdfa62edc34dab9525e2712d487582 for earlier term 1. Current term is 2. Ops: [1.4-1.4]
I20240503 02:00:40.774080 15089 raft_consensus.cc:1261] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Refusing update from remote peer c05f7620bb884392931d3243a0b47a9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:00:40.774292 14957 raft_consensus.cc:2396] T 6798a9ed0cf441aca7b8ae1b05c001f3 P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate c05f7620bb884392931d3243a0b47a9e for term 2 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 02:00:40.775437 17163 consensus_queue.cc:1035] T 6d2a1296a54a4b92a03c12954de0cc34 P c05f7620bb884392931d3243a0b47a9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:40.775789 15349 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ebd7cdd284a74741b338e9f4778502b5"
dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
from {username='slave'} at 127.0.0.1:50864
I20240503 02:00:40.775725 14816 catalog_manager.cc:5484] T 6798a9ed0cf441aca7b8ae1b05c001f3 P c05f7620bb884392931d3243a0b47a9e reported cstate change: term changed from 1 to 2, leader changed from dcfdfa62edc34dab9525e2712d487582 (127.0.99.129) to c05f7620bb884392931d3243a0b47a9e (127.0.99.132). New cstate: current_term: 2 leader_uuid: "c05f7620bb884392931d3243a0b47a9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:40.775902 15349 raft_consensus.cc:465] T ebd7cdd284a74741b338e9f4778502b5 P c05f7620bb884392931d3243a0b47a9e [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240503 02:00:40.776324 14957 raft_consensus.cc:1261] T 6d2a1296a54a4b92a03c12954de0cc34 P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Refusing update from remote peer c05f7620bb884392931d3243a0b47a9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:00:40.776914 17163 consensus_queue.cc:1035] T 6d2a1296a54a4b92a03c12954de0cc34 P c05f7620bb884392931d3243a0b47a9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:40.777488 15349 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5d448658ed3b460fb1725a26a7c743e8"
dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
from {username='slave'} at 127.0.0.1:50864
I20240503 02:00:40.777601 15349 raft_consensus.cc:482] T 5d448658ed3b460fb1725a26a7c743e8 P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:40.777659 15349 raft_consensus.cc:3037] T 5d448658ed3b460fb1725a26a7c743e8 P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.778756 15349 raft_consensus.cc:504] T 5d448658ed3b460fb1725a26a7c743e8 P c05f7620bb884392931d3243a0b47a9e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:40.779170 15349 leader_election.cc:288] T 5d448658ed3b460fb1725a26a7c743e8 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 2 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817)
I20240503 02:00:40.779486 15219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d448658ed3b460fb1725a26a7c743e8" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:40.779621 15219 raft_consensus.cc:3037] T 5d448658ed3b460fb1725a26a7c743e8 P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.780758 15219 raft_consensus.cc:2454] T 5d448658ed3b460fb1725a26a7c743e8 P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 2.
I20240503 02:00:40.780977 14956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d448658ed3b460fb1725a26a7c743e8" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:40.781096 14956 raft_consensus.cc:3032] T 5d448658ed3b460fb1725a26a7c743e8 P dcfdfa62edc34dab9525e2712d487582 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:00:40.781139 14956 raft_consensus.cc:729] T 5d448658ed3b460fb1725a26a7c743e8 P dcfdfa62edc34dab9525e2712d487582 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:40.781299 14956 consensus_queue.cc:260] T 5d448658ed3b460fb1725a26a7c743e8 P dcfdfa62edc34dab9525e2712d487582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:40.781464 14956 raft_consensus.cc:3037] T 5d448658ed3b460fb1725a26a7c743e8 P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.782452 14956 raft_consensus.cc:2454] T 5d448658ed3b460fb1725a26a7c743e8 P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 2.
I20240503 02:00:40.783421 15349 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "454a24fd6d964d73ae3d5057f90fe36d"
dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
from {username='slave'} at 127.0.0.1:50864
I20240503 02:00:40.783550 15349 raft_consensus.cc:465] T 454a24fd6d964d73ae3d5057f90fe36d P c05f7620bb884392931d3243a0b47a9e [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240503 02:00:40.784158 15349 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0e3bb299e61148fcb355641e0134b3df"
dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
from {username='slave'} at 127.0.0.1:50864
I20240503 02:00:40.784261 15349 raft_consensus.cc:465] T 0e3bb299e61148fcb355641e0134b3df P c05f7620bb884392931d3243a0b47a9e [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240503 02:00:40.784438 15286 leader_election.cc:302] T 5d448658ed3b460fb1725a26a7c743e8 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, c05f7620bb884392931d3243a0b47a9e; no voters:
I20240503 02:00:40.784670 17163 raft_consensus.cc:2781] T 5d448658ed3b460fb1725a26a7c743e8 P c05f7620bb884392931d3243a0b47a9e [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:40.784754 17163 raft_consensus.cc:686] T 5d448658ed3b460fb1725a26a7c743e8 P c05f7620bb884392931d3243a0b47a9e [term 2 LEADER]: Becoming Leader. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:40.784780 15349 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "590736f5f1ca4703821c409d882c2f77"
dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
from {username='slave'} at 127.0.0.1:50864
I20240503 02:00:40.784871 15349 raft_consensus.cc:482] T 590736f5f1ca4703821c409d882c2f77 P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:40.784935 15349 raft_consensus.cc:3037] T 590736f5f1ca4703821c409d882c2f77 P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.785960 15349 raft_consensus.cc:504] T 590736f5f1ca4703821c409d882c2f77 P c05f7620bb884392931d3243a0b47a9e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:40.786517 15349 leader_election.cc:288] T 590736f5f1ca4703821c409d882c2f77 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 2 election: Requested vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
I20240503 02:00:40.786973 15086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "590736f5f1ca4703821c409d882c2f77" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:40.787106 15086 raft_consensus.cc:3037] T 590736f5f1ca4703821c409d882c2f77 P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.788116 15086 raft_consensus.cc:2454] T 590736f5f1ca4703821c409d882c2f77 P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 2.
I20240503 02:00:40.788807 15351 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "11275e009edd4354b0807b7f015023be"
dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
from {username='slave'} at 127.0.0.1:50864
I20240503 02:00:40.788856 15285 leader_election.cc:302] T 590736f5f1ca4703821c409d882c2f77 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b602f6099cbb4758847e7e9550685372, c05f7620bb884392931d3243a0b47a9e; no voters:
I20240503 02:00:40.788944 15351 raft_consensus.cc:482] T 11275e009edd4354b0807b7f015023be P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:40.789000 15351 raft_consensus.cc:3037] T 11275e009edd4354b0807b7f015023be P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.789050 17162 raft_consensus.cc:2781] T 590736f5f1ca4703821c409d882c2f77 P c05f7620bb884392931d3243a0b47a9e [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:40.789119 17162 raft_consensus.cc:686] T 590736f5f1ca4703821c409d882c2f77 P c05f7620bb884392931d3243a0b47a9e [term 2 LEADER]: Becoming Leader. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:40.789240 17162 consensus_queue.cc:237] T 590736f5f1ca4703821c409d882c2f77 P c05f7620bb884392931d3243a0b47a9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:40.784878 17163 consensus_queue.cc:237] T 5d448658ed3b460fb1725a26a7c743e8 P c05f7620bb884392931d3243a0b47a9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:40.789873 14956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "590736f5f1ca4703821c409d882c2f77" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:40.789990 14956 raft_consensus.cc:3032] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:00:40.790032 14956 raft_consensus.cc:729] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:40.789994 15351 raft_consensus.cc:504] T 11275e009edd4354b0807b7f015023be P c05f7620bb884392931d3243a0b47a9e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:40.790163 14956 consensus_queue.cc:260] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:40.790289 15351 leader_election.cc:288] T 11275e009edd4354b0807b7f015023be P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 2 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817)
I20240503 02:00:40.790336 14956 raft_consensus.cc:3037] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.791319 14956 raft_consensus.cc:2454] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 2.
I20240503 02:00:40.793530 15351 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ca0d7d3e4d59496c9a9a59cde24b9247"
dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
from {username='slave'} at 127.0.0.1:50864
I20240503 02:00:40.793673 15351 raft_consensus.cc:482] T ca0d7d3e4d59496c9a9a59cde24b9247 P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:40.793731 15351 raft_consensus.cc:3037] T ca0d7d3e4d59496c9a9a59cde24b9247 P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.794720 15351 raft_consensus.cc:504] T ca0d7d3e4d59496c9a9a59cde24b9247 P c05f7620bb884392931d3243a0b47a9e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:40.794821 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11275e009edd4354b0807b7f015023be" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:40.794941 14957 raft_consensus.cc:3037] T 11275e009edd4354b0807b7f015023be P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.795099 15351 leader_election.cc:288] T ca0d7d3e4d59496c9a9a59cde24b9247 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 2 election: Requested vote from peers b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207), dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817)
I20240503 02:00:40.795049 14816 catalog_manager.cc:5484] T 5d448658ed3b460fb1725a26a7c743e8 P c05f7620bb884392931d3243a0b47a9e reported cstate change: term changed from 1 to 2, leader changed from dcfdfa62edc34dab9525e2712d487582 (127.0.99.129) to c05f7620bb884392931d3243a0b47a9e (127.0.99.132). New cstate: current_term: 2 leader_uuid: "c05f7620bb884392931d3243a0b47a9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:40.795956 14957 raft_consensus.cc:2454] T 11275e009edd4354b0807b7f015023be P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 2.
I20240503 02:00:40.798492 15351 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5da91b7af68e4cf597dc89a017ee0197"
dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
from {username='slave'} at 127.0.0.1:50864
I20240503 02:00:40.798683 15351 raft_consensus.cc:465] T 5da91b7af68e4cf597dc89a017ee0197 P c05f7620bb884392931d3243a0b47a9e [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240503 02:00:40.798945 15349 raft_consensus.cc:1226] T 11275e009edd4354b0807b7f015023be P c05f7620bb884392931d3243a0b47a9e [term 2 FOLLOWER]: Rejecting Update request from peer 620596b893d84291b88b90dc9e317612 for earlier term 1. Current term is 2. Ops: []
I20240503 02:00:40.799332 16264 consensus_queue.cc:1046] T 11275e009edd4354b0807b7f015023be P 620596b893d84291b88b90dc9e317612 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:40.799558 16264 raft_consensus.cc:3032] T 11275e009edd4354b0807b7f015023be P 620596b893d84291b88b90dc9e317612 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:00:40.799608 16264 raft_consensus.cc:729] T 11275e009edd4354b0807b7f015023be P 620596b893d84291b88b90dc9e317612 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:40.799744 16264 consensus_queue.cc:260] T 11275e009edd4354b0807b7f015023be P 620596b893d84291b88b90dc9e317612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:40.799912 16264 raft_consensus.cc:3037] T 11275e009edd4354b0807b7f015023be P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.799978 15219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11275e009edd4354b0807b7f015023be" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:40.800174 15349 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0989d98050744df6a1c699cb86ada505"
dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
from {username='slave'} at 127.0.0.1:50864
I20240503 02:00:40.800284 15349 raft_consensus.cc:482] T 0989d98050744df6a1c699cb86ada505 P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:40.800339 15349 raft_consensus.cc:3037] T 0989d98050744df6a1c699cb86ada505 P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.801246 15349 raft_consensus.cc:504] T 0989d98050744df6a1c699cb86ada505 P c05f7620bb884392931d3243a0b47a9e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:40.802012 15086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0989d98050744df6a1c699cb86ada505" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:40.802136 15086 raft_consensus.cc:3032] T 0989d98050744df6a1c699cb86ada505 P b602f6099cbb4758847e7e9550685372 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:00:40.802188 15086 raft_consensus.cc:729] T 0989d98050744df6a1c699cb86ada505 P b602f6099cbb4758847e7e9550685372 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:40.802191 15219 raft_consensus.cc:2454] T 11275e009edd4354b0807b7f015023be P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 2.
I20240503 02:00:40.802347 15086 consensus_queue.cc:260] T 0989d98050744df6a1c699cb86ada505 P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:40.802531 15086 raft_consensus.cc:3037] T 0989d98050744df6a1c699cb86ada505 P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.802961 15286 leader_election.cc:302] T 11275e009edd4354b0807b7f015023be P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, c05f7620bb884392931d3243a0b47a9e; no voters:
I20240503 02:00:40.803257 17162 raft_consensus.cc:2781] T 11275e009edd4354b0807b7f015023be P c05f7620bb884392931d3243a0b47a9e [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:40.803341 17162 raft_consensus.cc:686] T 11275e009edd4354b0807b7f015023be P c05f7620bb884392931d3243a0b47a9e [term 2 LEADER]: Becoming Leader. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:40.803484 15086 raft_consensus.cc:2454] T 0989d98050744df6a1c699cb86ada505 P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 2.
I20240503 02:00:40.803469 17162 consensus_queue.cc:237] T 11275e009edd4354b0807b7f015023be P c05f7620bb884392931d3243a0b47a9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:40.803886 15285 leader_election.cc:302] T 0989d98050744df6a1c699cb86ada505 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b602f6099cbb4758847e7e9550685372, c05f7620bb884392931d3243a0b47a9e; no voters:
I20240503 02:00:40.804060 17163 raft_consensus.cc:2781] T 0989d98050744df6a1c699cb86ada505 P c05f7620bb884392931d3243a0b47a9e [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:40.804141 17163 raft_consensus.cc:686] T 0989d98050744df6a1c699cb86ada505 P c05f7620bb884392931d3243a0b47a9e [term 2 LEADER]: Becoming Leader. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:40.804265 17163 consensus_queue.cc:237] T 0989d98050744df6a1c699cb86ada505 P c05f7620bb884392931d3243a0b47a9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:40.805030 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca0d7d3e4d59496c9a9a59cde24b9247" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:40.805104 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0989d98050744df6a1c699cb86ada505" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:40.805155 14957 raft_consensus.cc:3032] T ca0d7d3e4d59496c9a9a59cde24b9247 P dcfdfa62edc34dab9525e2712d487582 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:00:40.805195 14957 raft_consensus.cc:729] T ca0d7d3e4d59496c9a9a59cde24b9247 P dcfdfa62edc34dab9525e2712d487582 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:40.805208 14958 raft_consensus.cc:3037] T 0989d98050744df6a1c699cb86ada505 P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.805320 14957 consensus_queue.cc:260] T ca0d7d3e4d59496c9a9a59cde24b9247 P dcfdfa62edc34dab9525e2712d487582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:40.805492 14957 raft_consensus.cc:3037] T ca0d7d3e4d59496c9a9a59cde24b9247 P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.805619 15087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca0d7d3e4d59496c9a9a59cde24b9247" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:40.805720 15087 raft_consensus.cc:3037] T ca0d7d3e4d59496c9a9a59cde24b9247 P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.806344 14958 raft_consensus.cc:2454] T 0989d98050744df6a1c699cb86ada505 P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 2.
I20240503 02:00:40.806628 15087 raft_consensus.cc:2454] T ca0d7d3e4d59496c9a9a59cde24b9247 P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 2.
I20240503 02:00:40.807341 15285 leader_election.cc:302] T ca0d7d3e4d59496c9a9a59cde24b9247 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b602f6099cbb4758847e7e9550685372, c05f7620bb884392931d3243a0b47a9e; no voters:
I20240503 02:00:40.807726 16986 raft_consensus.cc:2781] T ca0d7d3e4d59496c9a9a59cde24b9247 P c05f7620bb884392931d3243a0b47a9e [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:40.807806 16986 raft_consensus.cc:686] T ca0d7d3e4d59496c9a9a59cde24b9247 P c05f7620bb884392931d3243a0b47a9e [term 2 LEADER]: Becoming Leader. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:40.807940 14957 raft_consensus.cc:2454] T ca0d7d3e4d59496c9a9a59cde24b9247 P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 2.
I20240503 02:00:40.807916 16986 consensus_queue.cc:237] T ca0d7d3e4d59496c9a9a59cde24b9247 P c05f7620bb884392931d3243a0b47a9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
W20240503 02:00:40.808040 15048 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33982: Illegal state: replica b602f6099cbb4758847e7e9550685372 is not leader of this config: current role FOLLOWER
I20240503 02:00:40.808760 14816 catalog_manager.cc:5484] T 590736f5f1ca4703821c409d882c2f77 P c05f7620bb884392931d3243a0b47a9e reported cstate change: term changed from 1 to 2, leader changed from dcfdfa62edc34dab9525e2712d487582 (127.0.99.129) to c05f7620bb884392931d3243a0b47a9e (127.0.99.132). New cstate: current_term: 2 leader_uuid: "c05f7620bb884392931d3243a0b47a9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:40.809067 14816 catalog_manager.cc:5484] T 0989d98050744df6a1c699cb86ada505 P c05f7620bb884392931d3243a0b47a9e reported cstate change: term changed from 1 to 2, leader changed from b602f6099cbb4758847e7e9550685372 (127.0.99.130) to c05f7620bb884392931d3243a0b47a9e (127.0.99.132). New cstate: current_term: 2 leader_uuid: "c05f7620bb884392931d3243a0b47a9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:40.809295 14816 catalog_manager.cc:5484] T 11275e009edd4354b0807b7f015023be P c05f7620bb884392931d3243a0b47a9e reported cstate change: term changed from 1 to 2, leader changed from 620596b893d84291b88b90dc9e317612 (127.0.99.131) to c05f7620bb884392931d3243a0b47a9e (127.0.99.132). New cstate: current_term: 2 leader_uuid: "c05f7620bb884392931d3243a0b47a9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:40.811096 15349 leader_election.cc:288] T 0989d98050744df6a1c699cb86ada505 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 2 election: Requested vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
W20240503 02:00:40.812312 14915 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47940: Illegal state: replica dcfdfa62edc34dab9525e2712d487582 is not leader of this config: current role FOLLOWER
W20240503 02:00:40.817907 15176 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57618: Illegal state: replica 620596b893d84291b88b90dc9e317612 is not leader of this config: current role FOLLOWER
I20240503 02:00:40.818367 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_4 already exists
I20240503 02:00:40.818545 398 test_util.cc:268] Using random seed: 372399983
I20240503 02:00:40.822105 14956 raft_consensus.cc:1261] T ca0d7d3e4d59496c9a9a59cde24b9247 P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Refusing update from remote peer c05f7620bb884392931d3243a0b47a9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:00:40.822719 17162 consensus_queue.cc:1035] T ca0d7d3e4d59496c9a9a59cde24b9247 P c05f7620bb884392931d3243a0b47a9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:40.822839 15219 raft_consensus.cc:1261] T c74522fbb5c64d27b5c95f05896b0716 P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Refusing update from remote peer c05f7620bb884392931d3243a0b47a9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:00:40.823275 17162 consensus_queue.cc:1035] T c74522fbb5c64d27b5c95f05896b0716 P c05f7620bb884392931d3243a0b47a9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:40.824461 15086 raft_consensus.cc:1261] T ca0d7d3e4d59496c9a9a59cde24b9247 P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Refusing update from remote peer c05f7620bb884392931d3243a0b47a9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:00:40.824461 15087 raft_consensus.cc:1261] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Refusing update from remote peer c05f7620bb884392931d3243a0b47a9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:00:40.825151 17162 consensus_queue.cc:1035] T ca0d7d3e4d59496c9a9a59cde24b9247 P c05f7620bb884392931d3243a0b47a9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:40.825264 17163 consensus_queue.cc:1035] T c74522fbb5c64d27b5c95f05896b0716 P c05f7620bb884392931d3243a0b47a9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:40.827240 14816 catalog_manager.cc:5484] T ca0d7d3e4d59496c9a9a59cde24b9247 P dcfdfa62edc34dab9525e2712d487582 reported cstate change: term changed from 1 to 2, leader changed from dcfdfa62edc34dab9525e2712d487582 (127.0.99.129) to c05f7620bb884392931d3243a0b47a9e (127.0.99.132). New cstate: current_term: 2 leader_uuid: "c05f7620bb884392931d3243a0b47a9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } }
W20240503 02:00:40.833117 14915 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47996: Illegal state: replica dcfdfa62edc34dab9525e2712d487582 is not leader of this config: current role FOLLOWER
I20240503 02:00:40.837872 14957 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7b4f203f4048490ba599353a4cd92748"
dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
from {username='slave'} at 127.0.0.1:47980
I20240503 02:00:40.838073 14957 raft_consensus.cc:482] T 7b4f203f4048490ba599353a4cd92748 P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:40.838135 14957 raft_consensus.cc:3037] T 7b4f203f4048490ba599353a4cd92748 P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.839205 14957 raft_consensus.cc:504] T 7b4f203f4048490ba599353a4cd92748 P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:40.839614 14957 leader_election.cc:288] T 7b4f203f4048490ba599353a4cd92748 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 2 election: Requested vote from peers b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207), 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437)
I20240503 02:00:40.842092 15220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4f203f4048490ba599353a4cd92748" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:40.842244 15220 raft_consensus.cc:3032] T 7b4f203f4048490ba599353a4cd92748 P 620596b893d84291b88b90dc9e317612 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:00:40.842286 15220 raft_consensus.cc:729] T 7b4f203f4048490ba599353a4cd92748 P 620596b893d84291b88b90dc9e317612 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:40.842428 15220 consensus_queue.cc:260] T 7b4f203f4048490ba599353a4cd92748 P 620596b893d84291b88b90dc9e317612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:40.842628 15220 raft_consensus.cc:3037] T 7b4f203f4048490ba599353a4cd92748 P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.843744 15220 raft_consensus.cc:2454] T 7b4f203f4048490ba599353a4cd92748 P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 2.
I20240503 02:00:40.844146 14893 leader_election.cc:302] T 7b4f203f4048490ba599353a4cd92748 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:40.844347 17225 raft_consensus.cc:2781] T 7b4f203f4048490ba599353a4cd92748 P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:40.844453 17225 raft_consensus.cc:686] T 7b4f203f4048490ba599353a4cd92748 P dcfdfa62edc34dab9525e2712d487582 [term 2 LEADER]: Becoming Leader. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:40.844610 17225 consensus_queue.cc:237] T 7b4f203f4048490ba599353a4cd92748 P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:40.846477 15087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4f203f4048490ba599353a4cd92748" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:40.846617 15087 raft_consensus.cc:3037] T 7b4f203f4048490ba599353a4cd92748 P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.847702 15087 raft_consensus.cc:2454] T 7b4f203f4048490ba599353a4cd92748 P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 2.
I20240503 02:00:40.849014 14956 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "21d92d412a5247e8907c5c41d253ad2e"
dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
from {username='slave'} at 127.0.0.1:47980
I20240503 02:00:40.849161 14956 raft_consensus.cc:482] T 21d92d412a5247e8907c5c41d253ad2e P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:40.849215 14956 raft_consensus.cc:3037] T 21d92d412a5247e8907c5c41d253ad2e P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:40.850335 14956 raft_consensus.cc:504] T 21d92d412a5247e8907c5c41d253ad2e P dcfdfa62edc34dab9525e2712d487582 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:40.850726 14956 leader_election.cc:288] T 21d92d412a5247e8907c5c41d253ad2e P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 3 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:40.851101 15219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21d92d412a5247e8907c5c41d253ad2e" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:40.851228 15219 raft_consensus.cc:3037] T 21d92d412a5247e8907c5c41d253ad2e P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:40.851234 15349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21d92d412a5247e8907c5c41d253ad2e" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:40.851349 15349 raft_consensus.cc:3032] T 21d92d412a5247e8907c5c41d253ad2e P c05f7620bb884392931d3243a0b47a9e [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:00:40.851392 15349 raft_consensus.cc:729] T 21d92d412a5247e8907c5c41d253ad2e P c05f7620bb884392931d3243a0b47a9e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:40.851572 15349 consensus_queue.cc:260] T 21d92d412a5247e8907c5c41d253ad2e P c05f7620bb884392931d3243a0b47a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:40.851827 15349 raft_consensus.cc:3037] T 21d92d412a5247e8907c5c41d253ad2e P c05f7620bb884392931d3243a0b47a9e [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:40.852223 15219 raft_consensus.cc:2454] T 21d92d412a5247e8907c5c41d253ad2e P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 3.
I20240503 02:00:40.852711 14893 leader_election.cc:302] T 21d92d412a5247e8907c5c41d253ad2e P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:40.852881 15349 raft_consensus.cc:2396] T 21d92d412a5247e8907c5c41d253ad2e P c05f7620bb884392931d3243a0b47a9e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate dcfdfa62edc34dab9525e2712d487582 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 02:00:40.853179 14815 catalog_manager.cc:5484] T 7b4f203f4048490ba599353a4cd92748 P dcfdfa62edc34dab9525e2712d487582 reported cstate change: term changed from 1 to 2, leader changed from 620596b893d84291b88b90dc9e317612 (127.0.99.131) to dcfdfa62edc34dab9525e2712d487582 (127.0.99.129). New cstate: current_term: 2 leader_uuid: "dcfdfa62edc34dab9525e2712d487582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:40.853793 17225 raft_consensus.cc:2781] T 21d92d412a5247e8907c5c41d253ad2e P dcfdfa62edc34dab9525e2712d487582 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:00:40.853884 17225 raft_consensus.cc:686] T 21d92d412a5247e8907c5c41d253ad2e P dcfdfa62edc34dab9525e2712d487582 [term 3 LEADER]: Becoming Leader. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:40.854045 17225 consensus_queue.cc:237] T 21d92d412a5247e8907c5c41d253ad2e P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:40.854238 14958 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "830da5cb4972453cb9bf2145a0839ced"
dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
from {username='slave'} at 127.0.0.1:47980
I20240503 02:00:40.854342 14958 raft_consensus.cc:482] T 830da5cb4972453cb9bf2145a0839ced P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:40.854391 14958 raft_consensus.cc:3037] T 830da5cb4972453cb9bf2145a0839ced P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.855413 14958 raft_consensus.cc:504] T 830da5cb4972453cb9bf2145a0839ced P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:40.855886 14958 leader_election.cc:288] T 830da5cb4972453cb9bf2145a0839ced P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 2 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
I20240503 02:00:40.856320 14956 raft_consensus.cc:1261] T 11275e009edd4354b0807b7f015023be P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Refusing update from remote peer c05f7620bb884392931d3243a0b47a9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:00:40.856328 15220 raft_consensus.cc:1261] T 11275e009edd4354b0807b7f015023be P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Refusing update from remote peer c05f7620bb884392931d3243a0b47a9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:00:40.856823 17162 consensus_queue.cc:1035] T 11275e009edd4354b0807b7f015023be P c05f7620bb884392931d3243a0b47a9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:40.857599 17162 consensus_queue.cc:1035] T 11275e009edd4354b0807b7f015023be P c05f7620bb884392931d3243a0b47a9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:40.858816 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_5 already exists
I20240503 02:00:40.858986 398 test_util.cc:268] Using random seed: 372440424
I20240503 02:00:40.859724 14958 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "50c5abd860ae4b09bb289e9c7550c84e"
dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
from {username='slave'} at 127.0.0.1:47980
I20240503 02:00:40.859855 14958 raft_consensus.cc:482] T 50c5abd860ae4b09bb289e9c7550c84e P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:40.859916 14958 raft_consensus.cc:3037] T 50c5abd860ae4b09bb289e9c7550c84e P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.860531 15089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "830da5cb4972453cb9bf2145a0839ced" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:40.860666 15089 raft_consensus.cc:3032] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:00:40.860706 15089 raft_consensus.cc:729] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:40.860836 15089 consensus_queue.cc:260] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:40.860998 15089 raft_consensus.cc:3037] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.862082 15089 raft_consensus.cc:2454] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 2.
I20240503 02:00:40.862804 14892 leader_election.cc:302] T 830da5cb4972453cb9bf2145a0839ced P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b602f6099cbb4758847e7e9550685372, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:40.863099 17225 raft_consensus.cc:2781] T 830da5cb4972453cb9bf2145a0839ced P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:40.863179 17225 raft_consensus.cc:686] T 830da5cb4972453cb9bf2145a0839ced P dcfdfa62edc34dab9525e2712d487582 [term 2 LEADER]: Becoming Leader. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:40.863330 17225 consensus_queue.cc:237] T 830da5cb4972453cb9bf2145a0839ced P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:40.863629 14958 raft_consensus.cc:504] T 50c5abd860ae4b09bb289e9c7550c84e P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:40.863946 14958 leader_election.cc:288] T 50c5abd860ae4b09bb289e9c7550c84e P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 2 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:40.864629 15349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50c5abd860ae4b09bb289e9c7550c84e" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:40.864743 15349 raft_consensus.cc:3032] T 50c5abd860ae4b09bb289e9c7550c84e P c05f7620bb884392931d3243a0b47a9e [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:00:40.864784 15349 raft_consensus.cc:729] T 50c5abd860ae4b09bb289e9c7550c84e P c05f7620bb884392931d3243a0b47a9e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:40.864907 15349 consensus_queue.cc:260] T 50c5abd860ae4b09bb289e9c7550c84e P c05f7620bb884392931d3243a0b47a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:40.865064 15349 raft_consensus.cc:3037] T 50c5abd860ae4b09bb289e9c7550c84e P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.865641 14815 catalog_manager.cc:5484] T 21d92d412a5247e8907c5c41d253ad2e P dcfdfa62edc34dab9525e2712d487582 reported cstate change: term changed from 2 to 3, leader changed from c05f7620bb884392931d3243a0b47a9e (127.0.99.132) to dcfdfa62edc34dab9525e2712d487582 (127.0.99.129). New cstate: current_term: 3 leader_uuid: "dcfdfa62edc34dab9525e2712d487582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:40.866068 15349 raft_consensus.cc:2454] T 50c5abd860ae4b09bb289e9c7550c84e P c05f7620bb884392931d3243a0b47a9e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 2.
I20240503 02:00:40.866461 14893 leader_election.cc:302] T 50c5abd860ae4b09bb289e9c7550c84e P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c05f7620bb884392931d3243a0b47a9e, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:40.866945 17210 raft_consensus.cc:2781] T 50c5abd860ae4b09bb289e9c7550c84e P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:40.867023 17210 raft_consensus.cc:686] T 50c5abd860ae4b09bb289e9c7550c84e P dcfdfa62edc34dab9525e2712d487582 [term 2 LEADER]: Becoming Leader. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:40.867054 15218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50c5abd860ae4b09bb289e9c7550c84e" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:40.867201 15218 raft_consensus.cc:3037] T 50c5abd860ae4b09bb289e9c7550c84e P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.867161 17210 consensus_queue.cc:237] T 50c5abd860ae4b09bb289e9c7550c84e P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:40.868331 15218 raft_consensus.cc:2454] T 50c5abd860ae4b09bb289e9c7550c84e P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 2.
I20240503 02:00:40.868438 14955 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6d2a1296a54a4b92a03c12954de0cc34"
dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
from {username='slave'} at 127.0.0.1:47980
I20240503 02:00:40.868567 14955 raft_consensus.cc:482] T 6d2a1296a54a4b92a03c12954de0cc34 P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:40.868625 14955 raft_consensus.cc:3037] T 6d2a1296a54a4b92a03c12954de0cc34 P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:40.869197 15219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "830da5cb4972453cb9bf2145a0839ced" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:40.869308 15219 raft_consensus.cc:3037] T 830da5cb4972453cb9bf2145a0839ced P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.869665 14955 raft_consensus.cc:504] T 6d2a1296a54a4b92a03c12954de0cc34 P dcfdfa62edc34dab9525e2712d487582 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:40.869999 14955 leader_election.cc:288] T 6d2a1296a54a4b92a03c12954de0cc34 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 3 election: Requested vote from peers c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
I20240503 02:00:40.870270 15219 raft_consensus.cc:2454] T 830da5cb4972453cb9bf2145a0839ced P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 2.
I20240503 02:00:40.871075 15089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2a1296a54a4b92a03c12954de0cc34" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:40.871204 15089 raft_consensus.cc:3037] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:40.871841 14957 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7fc3330dfd0644f6a303659b960124f4"
dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
from {username='slave'} at 127.0.0.1:47980
I20240503 02:00:40.871949 14957 raft_consensus.cc:482] T 7fc3330dfd0644f6a303659b960124f4 P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:40.872001 14957 raft_consensus.cc:3037] T 7fc3330dfd0644f6a303659b960124f4 P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:40.872215 15089 raft_consensus.cc:2454] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 3.
I20240503 02:00:40.872606 15088 raft_consensus.cc:1226] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Rejecting Update request from peer c05f7620bb884392931d3243a0b47a9e for earlier term 2. Current term is 3. Ops: [2.7-2.7]
I20240503 02:00:40.872954 14957 raft_consensus.cc:504] T 7fc3330dfd0644f6a303659b960124f4 P dcfdfa62edc34dab9525e2712d487582 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:40.873004 16474 consensus_queue.cc:1046] T 6d2a1296a54a4b92a03c12954de0cc34 P c05f7620bb884392931d3243a0b47a9e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 }, Status: INVALID_TERM, Last received: 2.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:40.873204 17163 raft_consensus.cc:3032] T 6d2a1296a54a4b92a03c12954de0cc34 P c05f7620bb884392931d3243a0b47a9e [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:00:40.873270 17163 raft_consensus.cc:729] T 6d2a1296a54a4b92a03c12954de0cc34 P c05f7620bb884392931d3243a0b47a9e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:40.873314 14957 leader_election.cc:288] T 7fc3330dfd0644f6a303659b960124f4 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 3 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:40.873412 17163 consensus_queue.cc:260] T 6d2a1296a54a4b92a03c12954de0cc34 P c05f7620bb884392931d3243a0b47a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:40.873616 17163 raft_consensus.cc:3037] T 6d2a1296a54a4b92a03c12954de0cc34 P c05f7620bb884392931d3243a0b47a9e [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:40.874018 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fc3330dfd0644f6a303659b960124f4" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:40.874123 15351 raft_consensus.cc:3032] T 7fc3330dfd0644f6a303659b960124f4 P c05f7620bb884392931d3243a0b47a9e [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:00:40.874159 15351 raft_consensus.cc:729] T 7fc3330dfd0644f6a303659b960124f4 P c05f7620bb884392931d3243a0b47a9e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:40.874279 15351 consensus_queue.cc:260] T 7fc3330dfd0644f6a303659b960124f4 P c05f7620bb884392931d3243a0b47a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:40.874408 15351 raft_consensus.cc:3037] T 7fc3330dfd0644f6a303659b960124f4 P c05f7620bb884392931d3243a0b47a9e [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:40.874722 14892 leader_election.cc:302] T 6d2a1296a54a4b92a03c12954de0cc34 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b602f6099cbb4758847e7e9550685372, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:40.875039 17210 raft_consensus.cc:2781] T 6d2a1296a54a4b92a03c12954de0cc34 P dcfdfa62edc34dab9525e2712d487582 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:00:40.875121 15349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2a1296a54a4b92a03c12954de0cc34" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:40.875133 17210 raft_consensus.cc:686] T 6d2a1296a54a4b92a03c12954de0cc34 P dcfdfa62edc34dab9525e2712d487582 [term 3 LEADER]: Becoming Leader. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:40.875337 15351 raft_consensus.cc:2396] T 7fc3330dfd0644f6a303659b960124f4 P c05f7620bb884392931d3243a0b47a9e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate dcfdfa62edc34dab9525e2712d487582 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 02:00:40.875285 17210 consensus_queue.cc:237] T 6d2a1296a54a4b92a03c12954de0cc34 P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:40.876155 15349 raft_consensus.cc:2454] T 6d2a1296a54a4b92a03c12954de0cc34 P c05f7620bb884392931d3243a0b47a9e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 3.
I20240503 02:00:40.876958 14957 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6798a9ed0cf441aca7b8ae1b05c001f3"
dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
from {username='slave'} at 127.0.0.1:47980
I20240503 02:00:40.877081 14957 raft_consensus.cc:482] T 6798a9ed0cf441aca7b8ae1b05c001f3 P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
W20240503 02:00:40.877074 15311 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50888: Illegal state: replica c05f7620bb884392931d3243a0b47a9e is not leader of this config: current role FOLLOWER
I20240503 02:00:40.877135 14957 raft_consensus.cc:3037] T 6798a9ed0cf441aca7b8ae1b05c001f3 P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:40.877739 15218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fc3330dfd0644f6a303659b960124f4" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:40.877852 15218 raft_consensus.cc:3037] T 7fc3330dfd0644f6a303659b960124f4 P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:40.878086 14957 raft_consensus.cc:504] T 6798a9ed0cf441aca7b8ae1b05c001f3 P dcfdfa62edc34dab9525e2712d487582 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:40.878618 14957 leader_election.cc:288] T 6798a9ed0cf441aca7b8ae1b05c001f3 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 3 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:40.879925 14957 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0989d98050744df6a1c699cb86ada505"
dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
from {username='slave'} at 127.0.0.1:47980
I20240503 02:00:40.880048 14957 raft_consensus.cc:482] T 0989d98050744df6a1c699cb86ada505 P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:40.880105 14957 raft_consensus.cc:3037] T 0989d98050744df6a1c699cb86ada505 P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:40.880079 15349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6798a9ed0cf441aca7b8ae1b05c001f3" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:40.880185 15349 raft_consensus.cc:3032] T 6798a9ed0cf441aca7b8ae1b05c001f3 P c05f7620bb884392931d3243a0b47a9e [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:00:40.880219 15349 raft_consensus.cc:729] T 6798a9ed0cf441aca7b8ae1b05c001f3 P c05f7620bb884392931d3243a0b47a9e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:40.880299 15217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6798a9ed0cf441aca7b8ae1b05c001f3" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:40.880358 15349 consensus_queue.cc:260] T 6798a9ed0cf441aca7b8ae1b05c001f3 P c05f7620bb884392931d3243a0b47a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:40.880445 15217 raft_consensus.cc:3037] T 6798a9ed0cf441aca7b8ae1b05c001f3 P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:40.880527 15349 raft_consensus.cc:3037] T 6798a9ed0cf441aca7b8ae1b05c001f3 P c05f7620bb884392931d3243a0b47a9e [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:40.881094 14957 raft_consensus.cc:504] T 0989d98050744df6a1c699cb86ada505 P dcfdfa62edc34dab9525e2712d487582 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:40.881398 15217 raft_consensus.cc:2454] T 6798a9ed0cf441aca7b8ae1b05c001f3 P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 3.
I20240503 02:00:40.881491 15349 raft_consensus.cc:2396] T 6798a9ed0cf441aca7b8ae1b05c001f3 P c05f7620bb884392931d3243a0b47a9e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate dcfdfa62edc34dab9525e2712d487582 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20240503 02:00:40.881887 15089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0989d98050744df6a1c699cb86ada505" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:40.882014 15089 raft_consensus.cc:3037] T 0989d98050744df6a1c699cb86ada505 P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:40.882823 15089 raft_consensus.cc:2454] T 0989d98050744df6a1c699cb86ada505 P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 3.
I20240503 02:00:40.883054 15218 raft_consensus.cc:2454] T 7fc3330dfd0644f6a303659b960124f4 P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 3.
I20240503 02:00:40.883517 14892 leader_election.cc:302] T 0989d98050744df6a1c699cb86ada505 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b602f6099cbb4758847e7e9550685372, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:40.884153 15220 raft_consensus.cc:1261] T 5d448658ed3b460fb1725a26a7c743e8 P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Refusing update from remote peer c05f7620bb884392931d3243a0b47a9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:00:40.884703 17162 consensus_queue.cc:1035] T 5d448658ed3b460fb1725a26a7c743e8 P c05f7620bb884392931d3243a0b47a9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:40.885313 17210 raft_consensus.cc:2781] T 0989d98050744df6a1c699cb86ada505 P dcfdfa62edc34dab9525e2712d487582 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:00:40.885409 17210 raft_consensus.cc:686] T 0989d98050744df6a1c699cb86ada505 P dcfdfa62edc34dab9525e2712d487582 [term 3 LEADER]: Becoming Leader. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:40.885557 17210 consensus_queue.cc:237] T 0989d98050744df6a1c699cb86ada505 P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
W20240503 02:00:40.886724 14915 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47940: Illegal state: replica dcfdfa62edc34dab9525e2712d487582 is not leader of this config: current role FOLLOWER
I20240503 02:00:40.887037 14957 leader_election.cc:288] T 0989d98050744df6a1c699cb86ada505 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 3 election: Requested vote from peers c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
I20240503 02:00:40.887478 14893 leader_election.cc:302] T 6798a9ed0cf441aca7b8ae1b05c001f3 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 620596b893d84291b88b90dc9e317612, dcfdfa62edc34dab9525e2712d487582; no voters: c05f7620bb884392931d3243a0b47a9e
I20240503 02:00:40.887725 14893 leader_election.cc:302] T 7fc3330dfd0644f6a303659b960124f4 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 620596b893d84291b88b90dc9e317612, dcfdfa62edc34dab9525e2712d487582; no voters: c05f7620bb884392931d3243a0b47a9e
I20240503 02:00:40.888051 17225 raft_consensus.cc:2781] T 6798a9ed0cf441aca7b8ae1b05c001f3 P dcfdfa62edc34dab9525e2712d487582 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:00:40.888147 17225 raft_consensus.cc:686] T 6798a9ed0cf441aca7b8ae1b05c001f3 P dcfdfa62edc34dab9525e2712d487582 [term 3 LEADER]: Becoming Leader. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:40.888273 17225 consensus_queue.cc:237] T 6798a9ed0cf441aca7b8ae1b05c001f3 P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:40.888748 14816 catalog_manager.cc:5484] T 6d2a1296a54a4b92a03c12954de0cc34 P dcfdfa62edc34dab9525e2712d487582 reported cstate change: term changed from 2 to 3, leader changed from c05f7620bb884392931d3243a0b47a9e (127.0.99.132) to dcfdfa62edc34dab9525e2712d487582 (127.0.99.129). New cstate: current_term: 3 leader_uuid: "dcfdfa62edc34dab9525e2712d487582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:40.888880 17225 raft_consensus.cc:2781] T 7fc3330dfd0644f6a303659b960124f4 P dcfdfa62edc34dab9525e2712d487582 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:00:40.888938 17225 raft_consensus.cc:686] T 7fc3330dfd0644f6a303659b960124f4 P dcfdfa62edc34dab9525e2712d487582 [term 3 LEADER]: Becoming Leader. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:40.889020 14816 catalog_manager.cc:5484] T 50c5abd860ae4b09bb289e9c7550c84e P dcfdfa62edc34dab9525e2712d487582 reported cstate change: term changed from 1 to 2, leader changed from c05f7620bb884392931d3243a0b47a9e (127.0.99.132) to dcfdfa62edc34dab9525e2712d487582 (127.0.99.129). New cstate: current_term: 2 leader_uuid: "dcfdfa62edc34dab9525e2712d487582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:40.889051 17225 consensus_queue.cc:237] T 7fc3330dfd0644f6a303659b960124f4 P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:40.889214 14816 catalog_manager.cc:5484] T 830da5cb4972453cb9bf2145a0839ced P dcfdfa62edc34dab9525e2712d487582 reported cstate change: term changed from 1 to 2, leader changed from b602f6099cbb4758847e7e9550685372 (127.0.99.130) to dcfdfa62edc34dab9525e2712d487582 (127.0.99.129). New cstate: current_term: 2 leader_uuid: "dcfdfa62edc34dab9525e2712d487582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:40.898485 14816 catalog_manager.cc:5484] T 7fc3330dfd0644f6a303659b960124f4 P dcfdfa62edc34dab9525e2712d487582 reported cstate change: term changed from 2 to 3, leader changed from c05f7620bb884392931d3243a0b47a9e (127.0.99.132) to dcfdfa62edc34dab9525e2712d487582 (127.0.99.129). New cstate: current_term: 3 leader_uuid: "dcfdfa62edc34dab9525e2712d487582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:40.898816 14816 catalog_manager.cc:5484] T 6798a9ed0cf441aca7b8ae1b05c001f3 P dcfdfa62edc34dab9525e2712d487582 reported cstate change: term changed from 2 to 3, leader changed from c05f7620bb884392931d3243a0b47a9e (127.0.99.132) to dcfdfa62edc34dab9525e2712d487582 (127.0.99.129). New cstate: current_term: 3 leader_uuid: "dcfdfa62edc34dab9525e2712d487582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:40.899017 14816 catalog_manager.cc:5484] T 0989d98050744df6a1c699cb86ada505 P dcfdfa62edc34dab9525e2712d487582 reported cstate change: term changed from 2 to 3, leader changed from c05f7620bb884392931d3243a0b47a9e (127.0.99.132) to dcfdfa62edc34dab9525e2712d487582 (127.0.99.129). New cstate: current_term: 3 leader_uuid: "dcfdfa62edc34dab9525e2712d487582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:40.904441 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_6 already exists
I20240503 02:00:40.904646 398 test_util.cc:268] Using random seed: 372486083
I20240503 02:00:40.905136 14956 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d805ee04df39406ba30b3b3b9e349507"
dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
from {username='slave'} at 127.0.0.1:47980
I20240503 02:00:40.905267 14956 raft_consensus.cc:482] T d805ee04df39406ba30b3b3b9e349507 P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:40.905320 14956 raft_consensus.cc:3037] T d805ee04df39406ba30b3b3b9e349507 P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.906364 14956 raft_consensus.cc:504] T d805ee04df39406ba30b3b3b9e349507 P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:40.906762 14956 leader_election.cc:288] T d805ee04df39406ba30b3b3b9e349507 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 2 election: Requested vote from peers c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
I20240503 02:00:40.907430 15086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d805ee04df39406ba30b3b3b9e349507" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:40.907572 15086 raft_consensus.cc:3037] T d805ee04df39406ba30b3b3b9e349507 P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.908840 15086 raft_consensus.cc:2454] T d805ee04df39406ba30b3b3b9e349507 P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 2.
I20240503 02:00:40.908876 15349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0989d98050744df6a1c699cb86ada505" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:40.908988 15349 raft_consensus.cc:3032] T 0989d98050744df6a1c699cb86ada505 P c05f7620bb884392931d3243a0b47a9e [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:00:40.909025 15349 raft_consensus.cc:729] T 0989d98050744df6a1c699cb86ada505 P c05f7620bb884392931d3243a0b47a9e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:40.909052 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d805ee04df39406ba30b3b3b9e349507" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:40.909123 15351 raft_consensus.cc:3032] T d805ee04df39406ba30b3b3b9e349507 P c05f7620bb884392931d3243a0b47a9e [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:00:40.909152 15351 raft_consensus.cc:729] T d805ee04df39406ba30b3b3b9e349507 P c05f7620bb884392931d3243a0b47a9e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:40.909183 15349 consensus_queue.cc:260] T 0989d98050744df6a1c699cb86ada505 P c05f7620bb884392931d3243a0b47a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:40.909245 15351 consensus_queue.cc:260] T d805ee04df39406ba30b3b3b9e349507 P c05f7620bb884392931d3243a0b47a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:40.909309 15349 raft_consensus.cc:3037] T 0989d98050744df6a1c699cb86ada505 P c05f7620bb884392931d3243a0b47a9e [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:40.909361 15351 raft_consensus.cc:3037] T d805ee04df39406ba30b3b3b9e349507 P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.910313 15351 raft_consensus.cc:2454] T d805ee04df39406ba30b3b3b9e349507 P c05f7620bb884392931d3243a0b47a9e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 2.
I20240503 02:00:40.910655 14955 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2b757c430aaa42f382d174ca57f5fcfc"
dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
from {username='slave'} at 127.0.0.1:47980
I20240503 02:00:40.910780 14955 raft_consensus.cc:482] T 2b757c430aaa42f382d174ca57f5fcfc P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:40.910835 14955 raft_consensus.cc:3037] T 2b757c430aaa42f382d174ca57f5fcfc P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.911774 14955 raft_consensus.cc:504] T 2b757c430aaa42f382d174ca57f5fcfc P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:40.912189 14955 leader_election.cc:288] T 2b757c430aaa42f382d174ca57f5fcfc P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 2 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
I20240503 02:00:40.912627 15088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b757c430aaa42f382d174ca57f5fcfc" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:40.912619 15219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b757c430aaa42f382d174ca57f5fcfc" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:40.912736 15219 raft_consensus.cc:3037] T 2b757c430aaa42f382d174ca57f5fcfc P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.912744 15088 raft_consensus.cc:3032] T 2b757c430aaa42f382d174ca57f5fcfc P b602f6099cbb4758847e7e9550685372 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:00:40.912781 15088 raft_consensus.cc:729] T 2b757c430aaa42f382d174ca57f5fcfc P b602f6099cbb4758847e7e9550685372 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:40.912916 15088 consensus_queue.cc:260] T 2b757c430aaa42f382d174ca57f5fcfc P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:40.913074 15088 raft_consensus.cc:3037] T 2b757c430aaa42f382d174ca57f5fcfc P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.913734 15219 raft_consensus.cc:2454] T 2b757c430aaa42f382d174ca57f5fcfc P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 2.
I20240503 02:00:40.913992 15088 raft_consensus.cc:2454] T 2b757c430aaa42f382d174ca57f5fcfc P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 2.
I20240503 02:00:40.914491 14892 leader_election.cc:302] T d805ee04df39406ba30b3b3b9e349507 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b602f6099cbb4758847e7e9550685372, dcfdfa62edc34dab9525e2712d487582; no voters:
W20240503 02:00:40.914599 15178 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57646: Illegal state: replica 620596b893d84291b88b90dc9e317612 is not leader of this config: current role FOLLOWER
I20240503 02:00:40.914695 14892 leader_election.cc:302] T 2b757c430aaa42f382d174ca57f5fcfc P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b602f6099cbb4758847e7e9550685372, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:40.910310 15349 raft_consensus.cc:2396] T 0989d98050744df6a1c699cb86ada505 P c05f7620bb884392931d3243a0b47a9e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate dcfdfa62edc34dab9525e2712d487582 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 02:00:40.915642 16590 raft_consensus.cc:2781] T d805ee04df39406ba30b3b3b9e349507 P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:40.915721 16590 raft_consensus.cc:686] T d805ee04df39406ba30b3b3b9e349507 P dcfdfa62edc34dab9525e2712d487582 [term 2 LEADER]: Becoming Leader. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:40.915858 16590 consensus_queue.cc:237] T d805ee04df39406ba30b3b3b9e349507 P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:40.916442 16590 raft_consensus.cc:2781] T 2b757c430aaa42f382d174ca57f5fcfc P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:40.916509 16590 raft_consensus.cc:686] T 2b757c430aaa42f382d174ca57f5fcfc P dcfdfa62edc34dab9525e2712d487582 [term 2 LEADER]: Becoming Leader. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:40.916616 16590 consensus_queue.cc:237] T 2b757c430aaa42f382d174ca57f5fcfc P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:40.917769 14955 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5d448658ed3b460fb1725a26a7c743e8"
dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
from {username='slave'} at 127.0.0.1:47980
I20240503 02:00:40.917884 14955 raft_consensus.cc:482] T 5d448658ed3b460fb1725a26a7c743e8 P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:40.917932 14955 raft_consensus.cc:3037] T 5d448658ed3b460fb1725a26a7c743e8 P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:40.918960 14955 raft_consensus.cc:504] T 5d448658ed3b460fb1725a26a7c743e8 P dcfdfa62edc34dab9525e2712d487582 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:40.919328 14955 leader_election.cc:288] T 5d448658ed3b460fb1725a26a7c743e8 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 3 election: Requested vote from peers c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009), 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437)
I20240503 02:00:40.920254 15219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d448658ed3b460fb1725a26a7c743e8" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:40.920536 15219 raft_consensus.cc:3037] T 5d448658ed3b460fb1725a26a7c743e8 P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:40.920717 15349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d448658ed3b460fb1725a26a7c743e8" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:40.920828 15349 raft_consensus.cc:3032] T 5d448658ed3b460fb1725a26a7c743e8 P c05f7620bb884392931d3243a0b47a9e [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:00:40.920861 15349 raft_consensus.cc:729] T 5d448658ed3b460fb1725a26a7c743e8 P c05f7620bb884392931d3243a0b47a9e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:40.921000 15349 consensus_queue.cc:260] T 5d448658ed3b460fb1725a26a7c743e8 P c05f7620bb884392931d3243a0b47a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:40.921136 15349 raft_consensus.cc:3037] T 5d448658ed3b460fb1725a26a7c743e8 P c05f7620bb884392931d3243a0b47a9e [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:40.921615 15219 raft_consensus.cc:2396] T 5d448658ed3b460fb1725a26a7c743e8 P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate dcfdfa62edc34dab9525e2712d487582 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 02:00:40.922178 15349 raft_consensus.cc:2396] T 5d448658ed3b460fb1725a26a7c743e8 P c05f7620bb884392931d3243a0b47a9e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate dcfdfa62edc34dab9525e2712d487582 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 02:00:40.922341 14956 raft_consensus.cc:1226] T 5d448658ed3b460fb1725a26a7c743e8 P dcfdfa62edc34dab9525e2712d487582 [term 3 FOLLOWER]: Rejecting Update request from peer c05f7620bb884392931d3243a0b47a9e for earlier term 2. Current term is 3. Ops: []
I20240503 02:00:40.922547 14893 leader_election.cc:302] T 5d448658ed3b460fb1725a26a7c743e8 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dcfdfa62edc34dab9525e2712d487582; no voters: 620596b893d84291b88b90dc9e317612, c05f7620bb884392931d3243a0b47a9e
I20240503 02:00:40.922766 17225 raft_consensus.cc:2726] T 5d448658ed3b460fb1725a26a7c743e8 P dcfdfa62edc34dab9525e2712d487582 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240503 02:00:40.926574 15219 raft_consensus.cc:1261] T 7fc3330dfd0644f6a303659b960124f4 P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Refusing update from remote peer dcfdfa62edc34dab9525e2712d487582: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 02:00:40.926707 14816 catalog_manager.cc:5484] T d805ee04df39406ba30b3b3b9e349507 P dcfdfa62edc34dab9525e2712d487582 reported cstate change: term changed from 1 to 2, leader changed from c05f7620bb884392931d3243a0b47a9e (127.0.99.132) to dcfdfa62edc34dab9525e2712d487582 (127.0.99.129). New cstate: current_term: 2 leader_uuid: "dcfdfa62edc34dab9525e2712d487582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:40.926987 14816 catalog_manager.cc:5484] T 2b757c430aaa42f382d174ca57f5fcfc P dcfdfa62edc34dab9525e2712d487582 reported cstate change: term changed from 1 to 2, leader changed from b602f6099cbb4758847e7e9550685372 (127.0.99.130) to dcfdfa62edc34dab9525e2712d487582 (127.0.99.129). New cstate: current_term: 2 leader_uuid: "dcfdfa62edc34dab9525e2712d487582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:40.927139 17225 consensus_queue.cc:1035] T 7fc3330dfd0644f6a303659b960124f4 P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Connected to new peer: Peer: permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:40.929083 15349 raft_consensus.cc:1261] T 7fc3330dfd0644f6a303659b960124f4 P c05f7620bb884392931d3243a0b47a9e [term 3 FOLLOWER]: Refusing update from remote peer dcfdfa62edc34dab9525e2712d487582: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 02:00:40.927165 14958 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ca0d7d3e4d59496c9a9a59cde24b9247"
dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
from {username='slave'} at 127.0.0.1:47980
I20240503 02:00:40.931321 14958 raft_consensus.cc:482] T ca0d7d3e4d59496c9a9a59cde24b9247 P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:40.931402 14958 raft_consensus.cc:3037] T ca0d7d3e4d59496c9a9a59cde24b9247 P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:40.932458 14958 raft_consensus.cc:504] T ca0d7d3e4d59496c9a9a59cde24b9247 P dcfdfa62edc34dab9525e2712d487582 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
W20240503 02:00:40.932560 15311 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50904: Illegal state: replica c05f7620bb884392931d3243a0b47a9e is not leader of this config: current role FOLLOWER
I20240503 02:00:40.932890 14958 leader_election.cc:288] T ca0d7d3e4d59496c9a9a59cde24b9247 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 3 election: Requested vote from peers c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
I20240503 02:00:40.933646 17225 consensus_queue.cc:1035] T 7fc3330dfd0644f6a303659b960124f4 P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:40.933763 17225 consensus_queue.cc:1225] T 7fc3330dfd0644f6a303659b960124f4 P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Peer c05f7620bb884392931d3243a0b47a9e log is divergent from this leader: its last log entry 2.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240503 02:00:40.934015 15089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca0d7d3e4d59496c9a9a59cde24b9247" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:40.934132 15089 raft_consensus.cc:3037] T ca0d7d3e4d59496c9a9a59cde24b9247 P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:40.934228 15349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca0d7d3e4d59496c9a9a59cde24b9247" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:40.934345 15349 raft_consensus.cc:3032] T ca0d7d3e4d59496c9a9a59cde24b9247 P c05f7620bb884392931d3243a0b47a9e [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:00:40.934314 14956 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6d98ba8805024eb79bf5df7a8225d784"
dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
from {username='slave'} at 127.0.0.1:47980
I20240503 02:00:40.934386 15349 raft_consensus.cc:729] T ca0d7d3e4d59496c9a9a59cde24b9247 P c05f7620bb884392931d3243a0b47a9e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:40.934515 14956 raft_consensus.cc:482] T 6d98ba8805024eb79bf5df7a8225d784 P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:40.934581 14956 raft_consensus.cc:3037] T 6d98ba8805024eb79bf5df7a8225d784 P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:00:40.935561 15176 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57610: Illegal state: replica 620596b893d84291b88b90dc9e317612 is not leader of this config: current role FOLLOWER
I20240503 02:00:40.936914 15089 raft_consensus.cc:2454] T ca0d7d3e4d59496c9a9a59cde24b9247 P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 3.
I20240503 02:00:40.937350 14892 leader_election.cc:302] T ca0d7d3e4d59496c9a9a59cde24b9247 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b602f6099cbb4758847e7e9550685372, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:40.937608 16590 raft_consensus.cc:2781] T ca0d7d3e4d59496c9a9a59cde24b9247 P dcfdfa62edc34dab9525e2712d487582 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:00:40.937716 16590 raft_consensus.cc:686] T ca0d7d3e4d59496c9a9a59cde24b9247 P dcfdfa62edc34dab9525e2712d487582 [term 3 LEADER]: Becoming Leader. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:40.937994 15351 pending_rounds.cc:76] T 7fc3330dfd0644f6a303659b960124f4 P c05f7620bb884392931d3243a0b47a9e: Aborting all ops after (but not including) 3
I20240503 02:00:40.938050 15351 pending_rounds.cc:97] T 7fc3330dfd0644f6a303659b960124f4 P c05f7620bb884392931d3243a0b47a9e: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240503 02:00:40.938107 15351 raft_consensus.cc:2866] T 7fc3330dfd0644f6a303659b960124f4 P c05f7620bb884392931d3243a0b47a9e [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 02:00:40.938066 16590 consensus_queue.cc:237] T ca0d7d3e4d59496c9a9a59cde24b9247 P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:40.940696 15349 consensus_queue.cc:260] T ca0d7d3e4d59496c9a9a59cde24b9247 P c05f7620bb884392931d3243a0b47a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:40.940941 15349 raft_consensus.cc:3037] T ca0d7d3e4d59496c9a9a59cde24b9247 P c05f7620bb884392931d3243a0b47a9e [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:40.942157 15349 raft_consensus.cc:2454] T ca0d7d3e4d59496c9a9a59cde24b9247 P c05f7620bb884392931d3243a0b47a9e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 3.
I20240503 02:00:40.942256 14956 raft_consensus.cc:504] T 6d98ba8805024eb79bf5df7a8225d784 P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:40.942364 14815 catalog_manager.cc:5484] T ca0d7d3e4d59496c9a9a59cde24b9247 P dcfdfa62edc34dab9525e2712d487582 reported cstate change: term changed from 2 to 3, leader changed from c05f7620bb884392931d3243a0b47a9e (127.0.99.132) to dcfdfa62edc34dab9525e2712d487582 (127.0.99.129). New cstate: current_term: 3 leader_uuid: "dcfdfa62edc34dab9525e2712d487582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:40.942737 14956 leader_election.cc:288] T 6d98ba8805024eb79bf5df7a8225d784 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 2 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
I20240503 02:00:40.943552 15220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d98ba8805024eb79bf5df7a8225d784" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:40.943683 15220 raft_consensus.cc:3032] T 6d98ba8805024eb79bf5df7a8225d784 P 620596b893d84291b88b90dc9e317612 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:00:40.943728 15220 raft_consensus.cc:729] T 6d98ba8805024eb79bf5df7a8225d784 P 620596b893d84291b88b90dc9e317612 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:40.943863 15220 consensus_queue.cc:260] T 6d98ba8805024eb79bf5df7a8225d784 P 620596b893d84291b88b90dc9e317612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:40.944029 15220 raft_consensus.cc:3037] T 6d98ba8805024eb79bf5df7a8225d784 P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:00:40.944082 14915 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47996: Illegal state: replica dcfdfa62edc34dab9525e2712d487582 is not leader of this config: current role FOLLOWER
I20240503 02:00:40.944331 14956 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "42ceeb7b3c4146cd8a2e18b3e9ae987c"
dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
from {username='slave'} at 127.0.0.1:47980
I20240503 02:00:40.944475 14956 raft_consensus.cc:482] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:40.944530 14956 raft_consensus.cc:3037] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.946012 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_7 already exists
I20240503 02:00:40.946197 398 test_util.cc:268] Using random seed: 372527634
I20240503 02:00:40.945632 14956 raft_consensus.cc:504] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:40.947083 15217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ceeb7b3c4146cd8a2e18b3e9ae987c" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:40.947198 15217 raft_consensus.cc:3032] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P 620596b893d84291b88b90dc9e317612 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:00:40.947238 15217 raft_consensus.cc:729] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P 620596b893d84291b88b90dc9e317612 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:40.947288 15220 raft_consensus.cc:2454] T 6d98ba8805024eb79bf5df7a8225d784 P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 2.
I20240503 02:00:40.947362 15217 consensus_queue.cc:260] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P 620596b893d84291b88b90dc9e317612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:40.947533 15217 raft_consensus.cc:3037] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.947744 14893 leader_election.cc:302] T 6d98ba8805024eb79bf5df7a8225d784 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:40.948171 15085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ceeb7b3c4146cd8a2e18b3e9ae987c" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:40.948314 15085 raft_consensus.cc:3037] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.948592 15217 raft_consensus.cc:2454] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 2.
I20240503 02:00:40.948171 15087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d98ba8805024eb79bf5df7a8225d784" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:40.949440 15087 raft_consensus.cc:3037] T 6d98ba8805024eb79bf5df7a8225d784 P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.950317 14956 leader_election.cc:288] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 2 election: Requested vote from peers b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207), 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437)
I20240503 02:00:40.950332 15087 raft_consensus.cc:2454] T 6d98ba8805024eb79bf5df7a8225d784 P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 2.
I20240503 02:00:40.949324 15085 raft_consensus.cc:2454] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 2.
I20240503 02:00:40.950703 14892 leader_election.cc:302] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b602f6099cbb4758847e7e9550685372, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:40.954010 14957 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6b0c8b72e1884529a8a9b83d2a734d71"
dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
from {username='slave'} at 127.0.0.1:47980
I20240503 02:00:40.954169 14957 raft_consensus.cc:482] T 6b0c8b72e1884529a8a9b83d2a734d71 P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:40.954224 14957 raft_consensus.cc:3037] T 6b0c8b72e1884529a8a9b83d2a734d71 P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.954506 17277 raft_consensus.cc:2781] T 6d98ba8805024eb79bf5df7a8225d784 P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:40.954679 17277 raft_consensus.cc:686] T 6d98ba8805024eb79bf5df7a8225d784 P dcfdfa62edc34dab9525e2712d487582 [term 2 LEADER]: Becoming Leader. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:40.954867 17277 consensus_queue.cc:237] T 6d98ba8805024eb79bf5df7a8225d784 P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:40.955296 14957 raft_consensus.cc:504] T 6b0c8b72e1884529a8a9b83d2a734d71 P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:40.955631 14957 leader_election.cc:288] T 6b0c8b72e1884529a8a9b83d2a734d71 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 2 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
I20240503 02:00:40.954015 17210 raft_consensus.cc:2781] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:40.957271 17210 raft_consensus.cc:686] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P dcfdfa62edc34dab9525e2712d487582 [term 2 LEADER]: Becoming Leader. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:40.957392 17210 consensus_queue.cc:237] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
W20240503 02:00:40.959621 15311 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50904: Illegal state: replica c05f7620bb884392931d3243a0b47a9e is not leader of this config: current role FOLLOWER
I20240503 02:00:40.960333 15085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b0c8b72e1884529a8a9b83d2a734d71" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:40.960498 15085 raft_consensus.cc:3037] T 6b0c8b72e1884529a8a9b83d2a734d71 P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.961431 15085 raft_consensus.cc:2396] T 6b0c8b72e1884529a8a9b83d2a734d71 P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate dcfdfa62edc34dab9525e2712d487582 for term 2 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20240503 02:00:40.962533 14957 raft_consensus.cc:1226] T 6b0c8b72e1884529a8a9b83d2a734d71 P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Rejecting Update request from peer 620596b893d84291b88b90dc9e317612 for earlier term 1. Current term is 2. Ops: [1.5-1.5]
I20240503 02:00:40.963055 16264 consensus_queue.cc:1046] T 6b0c8b72e1884529a8a9b83d2a734d71 P 620596b893d84291b88b90dc9e317612 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:40.963253 16264 raft_consensus.cc:3032] T 6b0c8b72e1884529a8a9b83d2a734d71 P 620596b893d84291b88b90dc9e317612 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:00:40.963299 16264 raft_consensus.cc:729] T 6b0c8b72e1884529a8a9b83d2a734d71 P 620596b893d84291b88b90dc9e317612 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:40.963444 16264 consensus_queue.cc:260] T 6b0c8b72e1884529a8a9b83d2a734d71 P 620596b893d84291b88b90dc9e317612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:40.963624 16264 raft_consensus.cc:3037] T 6b0c8b72e1884529a8a9b83d2a734d71 P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:00:40.966537 15176 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57610: Illegal state: replica 620596b893d84291b88b90dc9e317612 is not leader of this config: current role FOLLOWER
I20240503 02:00:40.966760 15218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b0c8b72e1884529a8a9b83d2a734d71" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:40.966910 15218 raft_consensus.cc:2396] T 6b0c8b72e1884529a8a9b83d2a734d71 P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate dcfdfa62edc34dab9525e2712d487582 for term 2 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20240503 02:00:40.967758 14893 leader_election.cc:302] T 6b0c8b72e1884529a8a9b83d2a734d71 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dcfdfa62edc34dab9525e2712d487582; no voters: 620596b893d84291b88b90dc9e317612, b602f6099cbb4758847e7e9550685372
I20240503 02:00:40.967993 17277 raft_consensus.cc:2726] T 6b0c8b72e1884529a8a9b83d2a734d71 P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240503 02:00:40.968238 14957 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9a74cca2988a4f219f6fd9b07ade42d8"
dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
from {username='slave'} at 127.0.0.1:47980
I20240503 02:00:40.968354 14957 raft_consensus.cc:482] T 9a74cca2988a4f219f6fd9b07ade42d8 P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:40.968428 14957 raft_consensus.cc:3037] T 9a74cca2988a4f219f6fd9b07ade42d8 P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:40.969285 14816 catalog_manager.cc:5484] T 6d98ba8805024eb79bf5df7a8225d784 P dcfdfa62edc34dab9525e2712d487582 reported cstate change: term changed from 1 to 2, leader changed from 620596b893d84291b88b90dc9e317612 (127.0.99.131) to dcfdfa62edc34dab9525e2712d487582 (127.0.99.129). New cstate: current_term: 2 leader_uuid: "dcfdfa62edc34dab9525e2712d487582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:40.969465 14957 raft_consensus.cc:504] T 9a74cca2988a4f219f6fd9b07ade42d8 P dcfdfa62edc34dab9525e2712d487582 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:40.969843 14957 leader_election.cc:288] T 9a74cca2988a4f219f6fd9b07ade42d8 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 3 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:40.970724 15219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a74cca2988a4f219f6fd9b07ade42d8" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:40.970837 15219 raft_consensus.cc:3037] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:40.971453 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a74cca2988a4f219f6fd9b07ade42d8" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:40.971580 15351 raft_consensus.cc:3032] T 9a74cca2988a4f219f6fd9b07ade42d8 P c05f7620bb884392931d3243a0b47a9e [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:00:40.971626 15351 raft_consensus.cc:729] T 9a74cca2988a4f219f6fd9b07ade42d8 P c05f7620bb884392931d3243a0b47a9e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:40.971755 15351 consensus_queue.cc:260] T 9a74cca2988a4f219f6fd9b07ade42d8 P c05f7620bb884392931d3243a0b47a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:40.971848 15219 raft_consensus.cc:2454] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 3.
I20240503 02:00:40.971901 15351 raft_consensus.cc:3037] T 9a74cca2988a4f219f6fd9b07ade42d8 P c05f7620bb884392931d3243a0b47a9e [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:40.972342 14893 leader_election.cc:302] T 9a74cca2988a4f219f6fd9b07ade42d8 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:40.972555 17277 raft_consensus.cc:2781] T 9a74cca2988a4f219f6fd9b07ade42d8 P dcfdfa62edc34dab9525e2712d487582 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:00:40.972628 17277 raft_consensus.cc:686] T 9a74cca2988a4f219f6fd9b07ade42d8 P dcfdfa62edc34dab9525e2712d487582 [term 3 LEADER]: Becoming Leader. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:40.972754 17277 consensus_queue.cc:237] T 9a74cca2988a4f219f6fd9b07ade42d8 P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:40.972918 15351 raft_consensus.cc:2396] T 9a74cca2988a4f219f6fd9b07ade42d8 P c05f7620bb884392931d3243a0b47a9e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate dcfdfa62edc34dab9525e2712d487582 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 02:00:40.973906 14957 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6b225db529f14e7386f21f42bf31493b"
dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
from {username='slave'} at 127.0.0.1:47980
I20240503 02:00:40.974040 14957 raft_consensus.cc:482] T 6b225db529f14e7386f21f42bf31493b P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:40.974104 14957 raft_consensus.cc:3037] T 6b225db529f14e7386f21f42bf31493b P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:00:40.974228 14915 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47996: Illegal state: replica dcfdfa62edc34dab9525e2712d487582 is not leader of this config: current role FOLLOWER
I20240503 02:00:40.974659 14815 catalog_manager.cc:5484] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P dcfdfa62edc34dab9525e2712d487582 reported cstate change: term changed from 1 to 2, leader changed from 620596b893d84291b88b90dc9e317612 (127.0.99.131) to dcfdfa62edc34dab9525e2712d487582 (127.0.99.129). New cstate: current_term: 2 leader_uuid: "dcfdfa62edc34dab9525e2712d487582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:40.975029 14957 raft_consensus.cc:504] T 6b225db529f14e7386f21f42bf31493b P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:40.975692 14957 leader_election.cc:288] T 6b225db529f14e7386f21f42bf31493b P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 2 election: Requested vote from peers b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207), 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437)
I20240503 02:00:40.978943 15085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b225db529f14e7386f21f42bf31493b" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:40.979110 15085 raft_consensus.cc:3037] T 6b225db529f14e7386f21f42bf31493b P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.979313 14957 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f85a361ce0254472b37526362d4edf0e"
dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
from {username='slave'} at 127.0.0.1:47980
I20240503 02:00:40.979429 14957 raft_consensus.cc:465] T f85a361ce0254472b37526362d4edf0e P dcfdfa62edc34dab9525e2712d487582 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240503 02:00:40.980232 15220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b225db529f14e7386f21f42bf31493b" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:40.980479 15220 raft_consensus.cc:3032] T 6b225db529f14e7386f21f42bf31493b P 620596b893d84291b88b90dc9e317612 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:00:40.980566 15220 raft_consensus.cc:729] T 6b225db529f14e7386f21f42bf31493b P 620596b893d84291b88b90dc9e317612 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:40.980767 15220 consensus_queue.cc:260] T 6b225db529f14e7386f21f42bf31493b P 620596b893d84291b88b90dc9e317612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:40.980926 15085 raft_consensus.cc:2454] T 6b225db529f14e7386f21f42bf31493b P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 2.
I20240503 02:00:40.981074 15220 raft_consensus.cc:3037] T 6b225db529f14e7386f21f42bf31493b P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.981511 14957 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "454a24fd6d964d73ae3d5057f90fe36d"
dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
from {username='slave'} at 127.0.0.1:47980
I20240503 02:00:40.981665 14957 raft_consensus.cc:482] T 454a24fd6d964d73ae3d5057f90fe36d P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:40.981734 14957 raft_consensus.cc:3037] T 454a24fd6d964d73ae3d5057f90fe36d P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.982143 14815 catalog_manager.cc:5484] T 9a74cca2988a4f219f6fd9b07ade42d8 P dcfdfa62edc34dab9525e2712d487582 reported cstate change: term changed from 2 to 3, leader changed from c05f7620bb884392931d3243a0b47a9e (127.0.99.132) to dcfdfa62edc34dab9525e2712d487582 (127.0.99.129). New cstate: current_term: 3 leader_uuid: "dcfdfa62edc34dab9525e2712d487582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:40.982398 15220 raft_consensus.cc:2454] T 6b225db529f14e7386f21f42bf31493b P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 2.
I20240503 02:00:40.982708 14957 raft_consensus.cc:504] T 454a24fd6d964d73ae3d5057f90fe36d P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:40.981222 14892 leader_election.cc:302] T 6b225db529f14e7386f21f42bf31493b P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b602f6099cbb4758847e7e9550685372, dcfdfa62edc34dab9525e2712d487582; no voters:
W20240503 02:00:40.984545 15311 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50926: Illegal state: replica c05f7620bb884392931d3243a0b47a9e is not leader of this config: current role FOLLOWER
I20240503 02:00:40.984623 16590 raft_consensus.cc:2781] T 6b225db529f14e7386f21f42bf31493b P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:40.984781 16590 raft_consensus.cc:686] T 6b225db529f14e7386f21f42bf31493b P dcfdfa62edc34dab9525e2712d487582 [term 2 LEADER]: Becoming Leader. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:40.984958 15085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "454a24fd6d964d73ae3d5057f90fe36d" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:40.985092 15085 raft_consensus.cc:3037] T 454a24fd6d964d73ae3d5057f90fe36d P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:40.985263 16590 consensus_queue.cc:237] T 6b225db529f14e7386f21f42bf31493b P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:40.986778 15085 raft_consensus.cc:2454] T 454a24fd6d964d73ae3d5057f90fe36d P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 2.
I20240503 02:00:40.987192 14892 leader_election.cc:302] T 454a24fd6d964d73ae3d5057f90fe36d P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b602f6099cbb4758847e7e9550685372, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:40.987356 17277 raft_consensus.cc:2781] T 454a24fd6d964d73ae3d5057f90fe36d P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:40.987424 17277 raft_consensus.cc:686] T 454a24fd6d964d73ae3d5057f90fe36d P dcfdfa62edc34dab9525e2712d487582 [term 2 LEADER]: Becoming Leader. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:40.987535 17277 consensus_queue.cc:237] T 454a24fd6d964d73ae3d5057f90fe36d P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:40.992039 14957 leader_election.cc:288] T 454a24fd6d964d73ae3d5057f90fe36d P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 2 election: Requested vote from peers b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:40.992689 14957 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0e3bb299e61148fcb355641e0134b3df"
dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
from {username='slave'} at 127.0.0.1:47980
I20240503 02:00:40.992817 14957 raft_consensus.cc:482] T 0e3bb299e61148fcb355641e0134b3df P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:40.992872 14957 raft_consensus.cc:3037] T 0e3bb299e61148fcb355641e0134b3df P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:00:40.992900 15311 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50904: Illegal state: replica c05f7620bb884392931d3243a0b47a9e is not leader of this config: current role FOLLOWER
I20240503 02:00:40.996170 14816 catalog_manager.cc:5484] T 6b225db529f14e7386f21f42bf31493b P dcfdfa62edc34dab9525e2712d487582 reported cstate change: term changed from 1 to 2, leader changed from 620596b893d84291b88b90dc9e317612 (127.0.99.131) to dcfdfa62edc34dab9525e2712d487582 (127.0.99.129). New cstate: current_term: 2 leader_uuid: "dcfdfa62edc34dab9525e2712d487582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:40.999255 15089 raft_consensus.cc:1226] T 454a24fd6d964d73ae3d5057f90fe36d P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Rejecting Update request from peer c05f7620bb884392931d3243a0b47a9e for earlier term 1. Current term is 2. Ops: []
I20240503 02:00:40.999671 17163 consensus_queue.cc:1046] T 454a24fd6d964d73ae3d5057f90fe36d P c05f7620bb884392931d3243a0b47a9e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:40.999893 17163 raft_consensus.cc:3032] T 454a24fd6d964d73ae3d5057f90fe36d P c05f7620bb884392931d3243a0b47a9e [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:00:40.999941 17163 raft_consensus.cc:729] T 454a24fd6d964d73ae3d5057f90fe36d P c05f7620bb884392931d3243a0b47a9e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:41.000077 17163 consensus_queue.cc:260] T 454a24fd6d964d73ae3d5057f90fe36d P c05f7620bb884392931d3243a0b47a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.000263 17163 raft_consensus.cc:3037] T 454a24fd6d964d73ae3d5057f90fe36d P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:41.001021 14957 raft_consensus.cc:504] T 0e3bb299e61148fcb355641e0134b3df P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
W20240503 02:00:41.001276 15176 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57610: Illegal state: replica 620596b893d84291b88b90dc9e317612 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.002697 15349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "454a24fd6d964d73ae3d5057f90fe36d" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.003355 15089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e3bb299e61148fcb355641e0134b3df" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:41.003471 15089 raft_consensus.cc:3037] T 0e3bb299e61148fcb355641e0134b3df P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:41.003688 15349 raft_consensus.cc:2454] T 454a24fd6d964d73ae3d5057f90fe36d P c05f7620bb884392931d3243a0b47a9e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 2.
I20240503 02:00:41.009441 15349 raft_consensus.cc:1261] T 9a74cca2988a4f219f6fd9b07ade42d8 P c05f7620bb884392931d3243a0b47a9e [term 3 FOLLOWER]: Refusing update from remote peer dcfdfa62edc34dab9525e2712d487582: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 02:00:41.009928 15089 raft_consensus.cc:2454] T 0e3bb299e61148fcb355641e0134b3df P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 2.
I20240503 02:00:41.010226 14892 leader_election.cc:302] T 0e3bb299e61148fcb355641e0134b3df P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b602f6099cbb4758847e7e9550685372, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:41.010406 16590 raft_consensus.cc:2781] T 0e3bb299e61148fcb355641e0134b3df P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:41.010479 16590 raft_consensus.cc:686] T 0e3bb299e61148fcb355641e0134b3df P dcfdfa62edc34dab9525e2712d487582 [term 2 LEADER]: Becoming Leader. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:41.010617 16590 consensus_queue.cc:237] T 0e3bb299e61148fcb355641e0134b3df P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.011626 15218 raft_consensus.cc:1261] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Refusing update from remote peer dcfdfa62edc34dab9525e2712d487582: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 02:00:41.012102 14957 leader_election.cc:288] T 0e3bb299e61148fcb355641e0134b3df P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 2 election: Requested vote from peers c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
W20240503 02:00:41.013432 14917 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47996: Illegal state: replica dcfdfa62edc34dab9525e2712d487582 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.013423 15349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e3bb299e61148fcb355641e0134b3df" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.013542 15349 raft_consensus.cc:3032] T 0e3bb299e61148fcb355641e0134b3df P c05f7620bb884392931d3243a0b47a9e [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:00:41.013595 15349 raft_consensus.cc:729] T 0e3bb299e61148fcb355641e0134b3df P c05f7620bb884392931d3243a0b47a9e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:41.014040 14956 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "590736f5f1ca4703821c409d882c2f77"
dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
from {username='slave'} at 127.0.0.1:47980
I20240503 02:00:41.014160 14956 raft_consensus.cc:482] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.014216 14956 raft_consensus.cc:3037] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:41.014283 14954 raft_consensus.cc:1226] T 0e3bb299e61148fcb355641e0134b3df P dcfdfa62edc34dab9525e2712d487582 [term 2 LEADER]: Rejecting Update request from peer c05f7620bb884392931d3243a0b47a9e for earlier term 1. Current term is 2. Ops: []
I20240503 02:00:41.015017 16474 consensus_queue.cc:1046] T 0e3bb299e61148fcb355641e0134b3df P c05f7620bb884392931d3243a0b47a9e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:41.015159 16474 consensus_queue.cc:1225] T 0e3bb299e61148fcb355641e0134b3df P c05f7620bb884392931d3243a0b47a9e [LEADER]: Peer dcfdfa62edc34dab9525e2712d487582 log is divergent from this leader: its last log entry 2.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240503 02:00:41.015252 14956 raft_consensus.cc:504] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.015609 14956 leader_election.cc:288] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 3 election: Requested vote from peers c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
I20240503 02:00:41.016467 15349 consensus_queue.cc:260] T 0e3bb299e61148fcb355641e0134b3df P c05f7620bb884392931d3243a0b47a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.016666 15349 raft_consensus.cc:3037] T 0e3bb299e61148fcb355641e0134b3df P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:41.016868 15087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "590736f5f1ca4703821c409d882c2f77" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:41.016988 15087 raft_consensus.cc:3037] T 590736f5f1ca4703821c409d882c2f77 P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:41.017798 15349 raft_consensus.cc:2454] T 0e3bb299e61148fcb355641e0134b3df P c05f7620bb884392931d3243a0b47a9e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 2.
W20240503 02:00:41.017940 16474 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20240503 02:00:41.017942 15087 raft_consensus.cc:2454] T 590736f5f1ca4703821c409d882c2f77 P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 3.
I20240503 02:00:41.018252 14892 leader_election.cc:302] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b602f6099cbb4758847e7e9550685372, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:41.018572 14956 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "11275e009edd4354b0807b7f015023be"
dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
from {username='slave'} at 127.0.0.1:47980
I20240503 02:00:41.018694 14956 raft_consensus.cc:482] T 11275e009edd4354b0807b7f015023be P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.018777 14956 raft_consensus.cc:3037] T 11275e009edd4354b0807b7f015023be P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:41.019117 16590 raft_consensus.cc:2781] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:00:41.019205 16590 raft_consensus.cc:686] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582 [term 3 LEADER]: Becoming Leader. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:41.019330 16590 consensus_queue.cc:237] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.019745 14956 raft_consensus.cc:504] T 11275e009edd4354b0807b7f015023be P dcfdfa62edc34dab9525e2712d487582 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.020044 14956 leader_election.cc:288] T 11275e009edd4354b0807b7f015023be P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 3 election: Requested vote from peers c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009), 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437)
I20240503 02:00:41.021227 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_8 already exists
I20240503 02:00:41.021411 398 test_util.cc:268] Using random seed: 372602849
I20240503 02:00:41.021718 15351 raft_consensus.cc:1261] T 50c5abd860ae4b09bb289e9c7550c84e P c05f7620bb884392931d3243a0b47a9e [term 2 FOLLOWER]: Refusing update from remote peer dcfdfa62edc34dab9525e2712d487582: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:00:41.022392 15349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "590736f5f1ca4703821c409d882c2f77" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.022495 15218 raft_consensus.cc:1261] T 50c5abd860ae4b09bb289e9c7550c84e P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Refusing update from remote peer dcfdfa62edc34dab9525e2712d487582: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:00:41.022544 15349 raft_consensus.cc:3032] T 590736f5f1ca4703821c409d882c2f77 P c05f7620bb884392931d3243a0b47a9e [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:00:41.022785 15349 raft_consensus.cc:729] T 590736f5f1ca4703821c409d882c2f77 P c05f7620bb884392931d3243a0b47a9e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:41.022976 17225 consensus_queue.cc:1035] T 9a74cca2988a4f219f6fd9b07ade42d8 P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:41.023077 17225 consensus_queue.cc:1225] T 9a74cca2988a4f219f6fd9b07ade42d8 P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Peer c05f7620bb884392931d3243a0b47a9e log is divergent from this leader: its last log entry 2.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240503 02:00:41.023600 15349 consensus_queue.cc:260] T 590736f5f1ca4703821c409d882c2f77 P c05f7620bb884392931d3243a0b47a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.024718 15349 raft_consensus.cc:3037] T 590736f5f1ca4703821c409d882c2f77 P c05f7620bb884392931d3243a0b47a9e [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:41.025409 17225 consensus_queue.cc:1035] T 50c5abd860ae4b09bb289e9c7550c84e P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:41.026942 15349 raft_consensus.cc:2396] T 590736f5f1ca4703821c409d882c2f77 P c05f7620bb884392931d3243a0b47a9e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate dcfdfa62edc34dab9525e2712d487582 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 02:00:41.027519 17225 consensus_queue.cc:1035] T 9a74cca2988a4f219f6fd9b07ade42d8 P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Connected to new peer: Peer: permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:41.027913 17225 consensus_queue.cc:1035] T 50c5abd860ae4b09bb289e9c7550c84e P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Connected to new peer: Peer: permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:41.028353 15218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11275e009edd4354b0807b7f015023be" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:41.028528 15218 raft_consensus.cc:3037] T 11275e009edd4354b0807b7f015023be P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:41.029548 15218 raft_consensus.cc:2454] T 11275e009edd4354b0807b7f015023be P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 3.
I20240503 02:00:41.030776 14816 catalog_manager.cc:5484] T 454a24fd6d964d73ae3d5057f90fe36d P dcfdfa62edc34dab9525e2712d487582 reported cstate change: term changed from 1 to 2, leader changed from c05f7620bb884392931d3243a0b47a9e (127.0.99.132) to dcfdfa62edc34dab9525e2712d487582 (127.0.99.129). New cstate: current_term: 2 leader_uuid: "dcfdfa62edc34dab9525e2712d487582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:41.031072 14816 catalog_manager.cc:5484] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582 reported cstate change: term changed from 2 to 3, leader changed from c05f7620bb884392931d3243a0b47a9e (127.0.99.132) to dcfdfa62edc34dab9525e2712d487582 (127.0.99.129). New cstate: current_term: 3 leader_uuid: "dcfdfa62edc34dab9525e2712d487582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:41.031260 14816 catalog_manager.cc:5484] T 0e3bb299e61148fcb355641e0134b3df P dcfdfa62edc34dab9525e2712d487582 reported cstate change: term changed from 1 to 2, leader changed from c05f7620bb884392931d3243a0b47a9e (127.0.99.132) to dcfdfa62edc34dab9525e2712d487582 (127.0.99.129). New cstate: current_term: 2 leader_uuid: "dcfdfa62edc34dab9525e2712d487582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:41.032696 15089 raft_consensus.cc:1261] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Refusing update from remote peer dcfdfa62edc34dab9525e2712d487582: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240503 02:00:41.033764 17277 consensus_queue.cc:1035] T 6d2a1296a54a4b92a03c12954de0cc34 P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:41.044498 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_9 already exists
W20240503 02:00:41.027295 15311 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50904: Illegal state: replica c05f7620bb884392931d3243a0b47a9e is not leader of this config: current role FOLLOWER
I20240503 02:00:41.027825 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11275e009edd4354b0807b7f015023be" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.051745 15351 raft_consensus.cc:3032] T 11275e009edd4354b0807b7f015023be P c05f7620bb884392931d3243a0b47a9e [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:00:41.051820 15351 raft_consensus.cc:729] T 11275e009edd4354b0807b7f015023be P c05f7620bb884392931d3243a0b47a9e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:41.051831 15347 raft_consensus.cc:1261] T 6d2a1296a54a4b92a03c12954de0cc34 P c05f7620bb884392931d3243a0b47a9e [term 3 FOLLOWER]: Refusing update from remote peer dcfdfa62edc34dab9525e2712d487582: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240503 02:00:41.034824 15350 pending_rounds.cc:76] T 9a74cca2988a4f219f6fd9b07ade42d8 P c05f7620bb884392931d3243a0b47a9e: Aborting all ops after (but not including) 3
I20240503 02:00:41.051954 15351 consensus_queue.cc:260] T 11275e009edd4354b0807b7f015023be P c05f7620bb884392931d3243a0b47a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.052011 15350 pending_rounds.cc:97] T 9a74cca2988a4f219f6fd9b07ade42d8 P c05f7620bb884392931d3243a0b47a9e: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240503 02:00:41.052053 15350 raft_consensus.cc:2866] T 9a74cca2988a4f219f6fd9b07ade42d8 P c05f7620bb884392931d3243a0b47a9e [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 02:00:41.052122 15351 raft_consensus.cc:3037] T 11275e009edd4354b0807b7f015023be P c05f7620bb884392931d3243a0b47a9e [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:41.053149 15351 raft_consensus.cc:2454] T 11275e009edd4354b0807b7f015023be P c05f7620bb884392931d3243a0b47a9e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 3.
I20240503 02:00:41.054086 16590 consensus_queue.cc:1035] T 6d2a1296a54a4b92a03c12954de0cc34 P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:41.054621 14893 leader_election.cc:302] T 11275e009edd4354b0807b7f015023be P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c05f7620bb884392931d3243a0b47a9e, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:41.054847 16590 raft_consensus.cc:2781] T 11275e009edd4354b0807b7f015023be P dcfdfa62edc34dab9525e2712d487582 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:00:41.054952 16590 raft_consensus.cc:686] T 11275e009edd4354b0807b7f015023be P dcfdfa62edc34dab9525e2712d487582 [term 3 LEADER]: Becoming Leader. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:41.055137 16590 consensus_queue.cc:237] T 11275e009edd4354b0807b7f015023be P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.056170 15350 raft_consensus.cc:1261] T 21d92d412a5247e8907c5c41d253ad2e P c05f7620bb884392931d3243a0b47a9e [term 3 FOLLOWER]: Refusing update from remote peer dcfdfa62edc34dab9525e2712d487582: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 02:00:41.056989 15219 raft_consensus.cc:1261] T 21d92d412a5247e8907c5c41d253ad2e P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Refusing update from remote peer dcfdfa62edc34dab9525e2712d487582: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 02:00:41.057143 17160 mvcc.cc:205] Tried to move back new op lower bound from 7023417921535504384 to 7023417921425940480. Current Snapshot: MvccSnapshot[applied={T|T < 7023417921535504384}]
I20240503 02:00:41.067214 16735 heartbeater.cc:503] Master 127.0.99.190:38675 was elected leader, sending a full tablet report...
I20240503 02:00:41.069608 15087 raft_consensus.cc:1261] T de805ef7511d4ee9b9bbed67e0a4186c P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Refusing update from remote peer c05f7620bb884392931d3243a0b47a9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:00:41.070809 15219 raft_consensus.cc:1261] T de805ef7511d4ee9b9bbed67e0a4186c P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Refusing update from remote peer c05f7620bb884392931d3243a0b47a9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:00:41.071295 17162 consensus_queue.cc:1035] T de805ef7511d4ee9b9bbed67e0a4186c P c05f7620bb884392931d3243a0b47a9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:41.071823 17162 consensus_queue.cc:1035] T de805ef7511d4ee9b9bbed67e0a4186c P c05f7620bb884392931d3243a0b47a9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:00:41.072006 15176 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57610: Illegal state: replica 620596b893d84291b88b90dc9e317612 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.081975 14815 catalog_manager.cc:5484] T 11275e009edd4354b0807b7f015023be P dcfdfa62edc34dab9525e2712d487582 reported cstate change: term changed from 2 to 3, leader changed from c05f7620bb884392931d3243a0b47a9e (127.0.99.132) to dcfdfa62edc34dab9525e2712d487582 (127.0.99.129). New cstate: current_term: 3 leader_uuid: "dcfdfa62edc34dab9525e2712d487582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:41.082226 17151 mvcc.cc:205] Tried to move back new op lower bound from 7023417921810784256 to 7023417920338837504. Current Snapshot: MvccSnapshot[applied={T|T < 7023417921810784256 or (T in {7023417921810784256})}]
W20240503 02:00:41.085669 14917 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47996: Illegal state: replica dcfdfa62edc34dab9525e2712d487582 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.092033 17210 consensus_queue.cc:1035] T 21d92d412a5247e8907c5c41d253ad2e P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Connected to new peer: Peer: permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:41.097656 17210 consensus_queue.cc:1035] T 21d92d412a5247e8907c5c41d253ad2e P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:41.097806 17210 consensus_queue.cc:1225] T 21d92d412a5247e8907c5c41d253ad2e P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Peer c05f7620bb884392931d3243a0b47a9e log is divergent from this leader: its last log entry 2.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
W20240503 02:00:41.103852 15311 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50904: Illegal state: replica c05f7620bb884392931d3243a0b47a9e is not leader of this config: current role FOLLOWER
I20240503 02:00:41.103915 15220 raft_consensus.cc:1261] T 9e9899666b15498791f62dab6d9dda16 P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Refusing update from remote peer c05f7620bb884392931d3243a0b47a9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:00:41.104452 17162 consensus_queue.cc:1035] T 9e9899666b15498791f62dab6d9dda16 P c05f7620bb884392931d3243a0b47a9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:41.107748 15350 pending_rounds.cc:76] T 21d92d412a5247e8907c5c41d253ad2e P c05f7620bb884392931d3243a0b47a9e: Aborting all ops after (but not including) 3
I20240503 02:00:41.107823 15350 pending_rounds.cc:97] T 21d92d412a5247e8907c5c41d253ad2e P c05f7620bb884392931d3243a0b47a9e: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240503 02:00:41.107853 15350 raft_consensus.cc:2866] T 21d92d412a5247e8907c5c41d253ad2e P c05f7620bb884392931d3243a0b47a9e [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 02:00:41.107843 15087 raft_consensus.cc:1261] T 9e9899666b15498791f62dab6d9dda16 P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Refusing update from remote peer c05f7620bb884392931d3243a0b47a9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:00:41.108314 17162 consensus_queue.cc:1035] T 9e9899666b15498791f62dab6d9dda16 P c05f7620bb884392931d3243a0b47a9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:41.115434 17319 mvcc.cc:205] Tried to move back new op lower bound from 7023417921572388864 to 7023417920993497088. Current Snapshot: MvccSnapshot[applied={T|T < 7023417921572388864 or (T in {7023417921572388864})}]
W20240503 02:00:41.123633 15176 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57610: Illegal state: replica 620596b893d84291b88b90dc9e317612 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.125893 15085 raft_consensus.cc:1261] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Refusing update from remote peer dcfdfa62edc34dab9525e2712d487582: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:00:41.126343 17225 consensus_queue.cc:1035] T 830da5cb4972453cb9bf2145a0839ced P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:00:41.127230 15311 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50926: Illegal state: replica c05f7620bb884392931d3243a0b47a9e is not leader of this config: current role FOLLOWER
I20240503 02:00:41.133637 15089 raft_consensus.cc:1261] T 6d98ba8805024eb79bf5df7a8225d784 P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Refusing update from remote peer dcfdfa62edc34dab9525e2712d487582: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:00:41.134189 17210 consensus_queue.cc:1035] T 6d98ba8805024eb79bf5df7a8225d784 P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:41.135468 15214 raft_consensus.cc:1261] T 830da5cb4972453cb9bf2145a0839ced P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Refusing update from remote peer dcfdfa62edc34dab9525e2712d487582: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20240503 02:00:41.136113 15048 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34010: Illegal state: replica b602f6099cbb4758847e7e9550685372 is not leader of this config: current role FOLLOWER
W20240503 02:00:41.139469 14917 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47996: Illegal state: replica dcfdfa62edc34dab9525e2712d487582 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.142729 15086 raft_consensus.cc:1261] T 0e3bb299e61148fcb355641e0134b3df P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Refusing update from remote peer dcfdfa62edc34dab9525e2712d487582: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:00:41.142746 15219 raft_consensus.cc:1261] T 6d98ba8805024eb79bf5df7a8225d784 P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Refusing update from remote peer dcfdfa62edc34dab9525e2712d487582: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:00:41.143342 17309 consensus_queue.cc:1035] T 0e3bb299e61148fcb355641e0134b3df P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:41.151022 17174 mvcc.cc:205] Tried to move back new op lower bound from 7023417922078162944 to 7023417921353453568. Current Snapshot: MvccSnapshot[applied={T|T < 7023417922078162944}]
I20240503 02:00:41.153993 15351 raft_consensus.cc:1261] T 0e3bb299e61148fcb355641e0134b3df P c05f7620bb884392931d3243a0b47a9e [term 2 FOLLOWER]: Refusing update from remote peer dcfdfa62edc34dab9525e2712d487582: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:00:41.156513 15086 raft_consensus.cc:1261] T 7b4f203f4048490ba599353a4cd92748 P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Refusing update from remote peer dcfdfa62edc34dab9525e2712d487582: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:00:41.157009 17225 consensus_queue.cc:1035] T 7b4f203f4048490ba599353a4cd92748 P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:00:41.158591 15310 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50904: Illegal state: replica c05f7620bb884392931d3243a0b47a9e is not leader of this config: current role FOLLOWER
I20240503 02:00:41.159170 16590 consensus_queue.cc:1035] T 0e3bb299e61148fcb355641e0134b3df P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:41.165943 17225 consensus_queue.cc:1035] T 830da5cb4972453cb9bf2145a0839ced P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Connected to new peer: Peer: permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:41.168723 17225 consensus_queue.cc:1035] T 6d98ba8805024eb79bf5df7a8225d784 P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Connected to new peer: Peer: permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:41.177943 15217 raft_consensus.cc:1261] T 7b4f203f4048490ba599353a4cd92748 P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Refusing update from remote peer dcfdfa62edc34dab9525e2712d487582: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20240503 02:00:41.179258 15176 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57610: Illegal state: replica 620596b893d84291b88b90dc9e317612 is not leader of this config: current role FOLLOWER
W20240503 02:00:41.202720 14915 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47996: Illegal state: replica dcfdfa62edc34dab9525e2712d487582 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.213672 15086 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "de805ef7511d4ee9b9bbed67e0a4186c"
dest_uuid: "b602f6099cbb4758847e7e9550685372"
from {username='slave'} at 127.0.0.1:34018
I20240503 02:00:41.213958 15086 raft_consensus.cc:482] T de805ef7511d4ee9b9bbed67e0a4186c P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.214092 15086 raft_consensus.cc:3037] T de805ef7511d4ee9b9bbed67e0a4186c P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:41.215389 15086 raft_consensus.cc:504] T de805ef7511d4ee9b9bbed67e0a4186c P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.215898 15086 leader_election.cc:288] T de805ef7511d4ee9b9bbed67e0a4186c P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 3 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:41.216614 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de805ef7511d4ee9b9bbed67e0a4186c" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.216864 15351 raft_consensus.cc:3032] T de805ef7511d4ee9b9bbed67e0a4186c P c05f7620bb884392931d3243a0b47a9e [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:00:41.216862 15217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de805ef7511d4ee9b9bbed67e0a4186c" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:41.216928 15351 raft_consensus.cc:729] T de805ef7511d4ee9b9bbed67e0a4186c P c05f7620bb884392931d3243a0b47a9e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:41.216989 15217 raft_consensus.cc:3037] T de805ef7511d4ee9b9bbed67e0a4186c P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:41.217073 15351 consensus_queue.cc:260] T de805ef7511d4ee9b9bbed67e0a4186c P c05f7620bb884392931d3243a0b47a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.217257 15351 raft_consensus.cc:3037] T de805ef7511d4ee9b9bbed67e0a4186c P c05f7620bb884392931d3243a0b47a9e [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:41.218048 15217 raft_consensus.cc:2454] T de805ef7511d4ee9b9bbed67e0a4186c P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 3.
I20240503 02:00:41.218254 15351 raft_consensus.cc:2454] T de805ef7511d4ee9b9bbed67e0a4186c P c05f7620bb884392931d3243a0b47a9e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 3.
I20240503 02:00:41.218730 15089 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7b4f203f4048490ba599353a4cd92748"
dest_uuid: "b602f6099cbb4758847e7e9550685372"
from {username='slave'} at 127.0.0.1:34018
I20240503 02:00:41.218863 15089 raft_consensus.cc:482] T 7b4f203f4048490ba599353a4cd92748 P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.218922 15089 raft_consensus.cc:3037] T 7b4f203f4048490ba599353a4cd92748 P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:41.219772 15089 raft_consensus.cc:504] T 7b4f203f4048490ba599353a4cd92748 P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.220180 15089 leader_election.cc:288] T 7b4f203f4048490ba599353a4cd92748 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 3 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817)
I20240503 02:00:41.220649 15024 leader_election.cc:302] T de805ef7511d4ee9b9bbed67e0a4186c P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b602f6099cbb4758847e7e9550685372, c05f7620bb884392931d3243a0b47a9e; no voters:
I20240503 02:00:41.220830 17176 raft_consensus.cc:2781] T de805ef7511d4ee9b9bbed67e0a4186c P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:00:41.220906 17176 raft_consensus.cc:686] T de805ef7511d4ee9b9bbed67e0a4186c P b602f6099cbb4758847e7e9550685372 [term 3 LEADER]: Becoming Leader. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.221060 17176 consensus_queue.cc:237] T de805ef7511d4ee9b9bbed67e0a4186c P b602f6099cbb4758847e7e9550685372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.221863 15088 raft_consensus.cc:1261] T 4cdcdde158054cdaabc1856a90fe1302 P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Refusing update from remote peer c05f7620bb884392931d3243a0b47a9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:00:41.222290 16474 consensus_queue.cc:1035] T 4cdcdde158054cdaabc1856a90fe1302 P c05f7620bb884392931d3243a0b47a9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:41.223254 14816 catalog_manager.cc:5484] T de805ef7511d4ee9b9bbed67e0a4186c P b602f6099cbb4758847e7e9550685372 reported cstate change: term changed from 2 to 3, leader changed from c05f7620bb884392931d3243a0b47a9e (127.0.99.132) to b602f6099cbb4758847e7e9550685372 (127.0.99.130). New cstate: current_term: 3 leader_uuid: "b602f6099cbb4758847e7e9550685372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:41.228576 15089 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "590736f5f1ca4703821c409d882c2f77"
dest_uuid: "b602f6099cbb4758847e7e9550685372"
from {username='slave'} at 127.0.0.1:34018
I20240503 02:00:41.228735 15089 raft_consensus.cc:482] T 590736f5f1ca4703821c409d882c2f77 P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.228806 15089 raft_consensus.cc:3037] T 590736f5f1ca4703821c409d882c2f77 P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Advancing to term 4
W20240503 02:00:41.228982 15311 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50904: Illegal state: replica c05f7620bb884392931d3243a0b47a9e is not leader of this config: current role FOLLOWER
I20240503 02:00:41.229740 15089 raft_consensus.cc:504] T 590736f5f1ca4703821c409d882c2f77 P b602f6099cbb4758847e7e9550685372 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.230087 15089 leader_election.cc:288] T 590736f5f1ca4703821c409d882c2f77 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 4 election: Requested vote from peers c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009), dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817)
I20240503 02:00:41.231408 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "590736f5f1ca4703821c409d882c2f77" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.231544 15351 raft_consensus.cc:3037] T 590736f5f1ca4703821c409d882c2f77 P c05f7620bb884392931d3243a0b47a9e [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.232497 15351 raft_consensus.cc:2396] T 590736f5f1ca4703821c409d882c2f77 P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b602f6099cbb4758847e7e9550685372 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 02:00:41.233644 15217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4f203f4048490ba599353a4cd92748" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:41.233779 15217 raft_consensus.cc:3037] T 7b4f203f4048490ba599353a4cd92748 P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:41.234095 15215 raft_consensus.cc:1261] T 4cdcdde158054cdaabc1856a90fe1302 P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Refusing update from remote peer c05f7620bb884392931d3243a0b47a9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:00:41.234520 15089 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "830da5cb4972453cb9bf2145a0839ced"
dest_uuid: "b602f6099cbb4758847e7e9550685372"
from {username='slave'} at 127.0.0.1:34018
I20240503 02:00:41.234807 15217 raft_consensus.cc:2454] T 7b4f203f4048490ba599353a4cd92748 P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 3.
I20240503 02:00:41.234932 15089 raft_consensus.cc:482] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.235062 15089 raft_consensus.cc:3037] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:41.235575 17163 consensus_queue.cc:1035] T 4cdcdde158054cdaabc1856a90fe1302 P c05f7620bb884392931d3243a0b47a9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:41.236325 15024 leader_election.cc:302] T 7b4f203f4048490ba599353a4cd92748 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, b602f6099cbb4758847e7e9550685372; no voters:
I20240503 02:00:41.236640 16496 raft_consensus.cc:2781] T 7b4f203f4048490ba599353a4cd92748 P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:00:41.236716 16496 raft_consensus.cc:686] T 7b4f203f4048490ba599353a4cd92748 P b602f6099cbb4758847e7e9550685372 [term 3 LEADER]: Becoming Leader. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.236858 16496 consensus_queue.cc:237] T 7b4f203f4048490ba599353a4cd92748 P b602f6099cbb4758847e7e9550685372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.236625 15089 raft_consensus.cc:504] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.240720 14811 catalog_manager.cc:5484] T 7b4f203f4048490ba599353a4cd92748 P b602f6099cbb4758847e7e9550685372 reported cstate change: term changed from 2 to 3, leader changed from dcfdfa62edc34dab9525e2712d487582 (127.0.99.129) to b602f6099cbb4758847e7e9550685372 (127.0.99.130). New cstate: current_term: 3 leader_uuid: "b602f6099cbb4758847e7e9550685372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:41.241274 15217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "830da5cb4972453cb9bf2145a0839ced" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:41.241410 15217 raft_consensus.cc:3037] T 830da5cb4972453cb9bf2145a0839ced P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:41.242527 15217 raft_consensus.cc:2454] T 830da5cb4972453cb9bf2145a0839ced P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 3.
I20240503 02:00:41.243441 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4f203f4048490ba599353a4cd92748" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:41.243588 14957 raft_consensus.cc:3032] T 7b4f203f4048490ba599353a4cd92748 P dcfdfa62edc34dab9525e2712d487582 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:00:41.243636 14957 raft_consensus.cc:729] T 7b4f203f4048490ba599353a4cd92748 P dcfdfa62edc34dab9525e2712d487582 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:41.243775 14957 consensus_queue.cc:260] T 7b4f203f4048490ba599353a4cd92748 P dcfdfa62edc34dab9525e2712d487582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.243932 14957 raft_consensus.cc:3037] T 7b4f203f4048490ba599353a4cd92748 P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:41.243984 14956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "590736f5f1ca4703821c409d882c2f77" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
W20240503 02:00:41.244021 14915 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48024: Illegal state: replica dcfdfa62edc34dab9525e2712d487582 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.244065 14956 raft_consensus.cc:3032] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 02:00:41.244097 14956 raft_consensus.cc:729] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:41.244197 14956 consensus_queue.cc:260] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.244304 14956 raft_consensus.cc:3037] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.245393 14956 raft_consensus.cc:2396] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b602f6099cbb4758847e7e9550685372 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 02:00:41.246599 14957 raft_consensus.cc:2454] T 7b4f203f4048490ba599353a4cd92748 P dcfdfa62edc34dab9525e2712d487582 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 3.
I20240503 02:00:41.246856 15024 leader_election.cc:302] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, b602f6099cbb4758847e7e9550685372; no voters:
I20240503 02:00:41.247058 17176 raft_consensus.cc:2781] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:00:41.247133 17176 raft_consensus.cc:686] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372 [term 3 LEADER]: Becoming Leader. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.247273 17176 consensus_queue.cc:237] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.249297 14816 catalog_manager.cc:5484] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372 reported cstate change: term changed from 2 to 3, leader changed from dcfdfa62edc34dab9525e2712d487582 (127.0.99.129) to b602f6099cbb4758847e7e9550685372 (127.0.99.130). New cstate: current_term: 3 leader_uuid: "b602f6099cbb4758847e7e9550685372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:41.255090 15089 leader_election.cc:288] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 3 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817)
I20240503 02:00:41.255920 15089 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9e9899666b15498791f62dab6d9dda16"
dest_uuid: "b602f6099cbb4758847e7e9550685372"
from {username='slave'} at 127.0.0.1:34018
I20240503 02:00:41.256079 15089 raft_consensus.cc:482] T 9e9899666b15498791f62dab6d9dda16 P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.256142 15089 raft_consensus.cc:3037] T 9e9899666b15498791f62dab6d9dda16 P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Advancing to term 3
W20240503 02:00:41.256172 15176 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57610: Illegal state: replica 620596b893d84291b88b90dc9e317612 is not leader of this config: current role FOLLOWER
W20240503 02:00:41.256717 14915 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48020: Illegal state: replica dcfdfa62edc34dab9525e2712d487582 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.257953 15089 raft_consensus.cc:504] T 9e9899666b15498791f62dab6d9dda16 P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:41.258360 15089 leader_election.cc:288] T 9e9899666b15498791f62dab6d9dda16 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 3 election: Requested vote from peers c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009), 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437)
I20240503 02:00:41.258816 15217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e9899666b15498791f62dab6d9dda16" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:41.258941 15217 raft_consensus.cc:3037] T 9e9899666b15498791f62dab6d9dda16 P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:41.258828 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e9899666b15498791f62dab6d9dda16" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.259075 15351 raft_consensus.cc:3032] T 9e9899666b15498791f62dab6d9dda16 P c05f7620bb884392931d3243a0b47a9e [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:00:41.259119 15351 raft_consensus.cc:729] T 9e9899666b15498791f62dab6d9dda16 P c05f7620bb884392931d3243a0b47a9e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:41.259320 15351 consensus_queue.cc:260] T 9e9899666b15498791f62dab6d9dda16 P c05f7620bb884392931d3243a0b47a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:41.259418 15088 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6d2a1296a54a4b92a03c12954de0cc34"
dest_uuid: "b602f6099cbb4758847e7e9550685372"
from {username='slave'} at 127.0.0.1:34018
I20240503 02:00:41.259517 15088 raft_consensus.cc:482] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.259531 15351 raft_consensus.cc:3037] T 9e9899666b15498791f62dab6d9dda16 P c05f7620bb884392931d3243a0b47a9e [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:41.259569 15088 raft_consensus.cc:3037] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.260551 15351 raft_consensus.cc:2454] T 9e9899666b15498791f62dab6d9dda16 P c05f7620bb884392931d3243a0b47a9e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 3.
I20240503 02:00:41.260555 15088 raft_consensus.cc:504] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.260834 15024 leader_election.cc:302] T 9e9899666b15498791f62dab6d9dda16 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b602f6099cbb4758847e7e9550685372, c05f7620bb884392931d3243a0b47a9e; no voters:
I20240503 02:00:41.260921 15088 leader_election.cc:288] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 4 election: Requested vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:41.261113 16496 raft_consensus.cc:2781] T 9e9899666b15498791f62dab6d9dda16 P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:00:41.261195 16496 raft_consensus.cc:686] T 9e9899666b15498791f62dab6d9dda16 P b602f6099cbb4758847e7e9550685372 [term 3 LEADER]: Becoming Leader. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.261323 16496 consensus_queue.cc:237] T 9e9899666b15498791f62dab6d9dda16 P b602f6099cbb4758847e7e9550685372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:41.262120 15349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2a1296a54a4b92a03c12954de0cc34" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 4 candidate_status { last_received { term: 3 index: 10 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.262238 15349 raft_consensus.cc:3037] T 6d2a1296a54a4b92a03c12954de0cc34 P c05f7620bb884392931d3243a0b47a9e [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.262387 15088 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5736c284a85540a1bbab2be6df672fc1"
dest_uuid: "b602f6099cbb4758847e7e9550685372"
from {username='slave'} at 127.0.0.1:34018
I20240503 02:00:41.262485 15088 raft_consensus.cc:482] T 5736c284a85540a1bbab2be6df672fc1 P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.262531 15088 raft_consensus.cc:3037] T 5736c284a85540a1bbab2be6df672fc1 P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:41.263221 15349 raft_consensus.cc:2454] T 6d2a1296a54a4b92a03c12954de0cc34 P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 4.
I20240503 02:00:41.263605 15088 raft_consensus.cc:504] T 5736c284a85540a1bbab2be6df672fc1 P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.263731 15024 leader_election.cc:302] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b602f6099cbb4758847e7e9550685372, c05f7620bb884392931d3243a0b47a9e; no voters:
I20240503 02:00:41.263983 15088 leader_election.cc:288] T 5736c284a85540a1bbab2be6df672fc1 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 2 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:41.263995 16496 raft_consensus.cc:2781] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 02:00:41.264287 16496 raft_consensus.cc:686] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372 [term 4 LEADER]: Becoming Leader. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.264439 16496 consensus_queue.cc:237] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 3.10, Last appended by leader: 10, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.264561 15088 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eade118a36da42e3ac08b96a13893a5c"
dest_uuid: "b602f6099cbb4758847e7e9550685372"
from {username='slave'} at 127.0.0.1:34018
I20240503 02:00:41.264667 15088 raft_consensus.cc:482] T eade118a36da42e3ac08b96a13893a5c P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.264712 15088 raft_consensus.cc:3037] T eade118a36da42e3ac08b96a13893a5c P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:00:41.264905 15311 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50950: Illegal state: replica c05f7620bb884392931d3243a0b47a9e is not leader of this config: current role FOLLOWER
I20240503 02:00:41.265638 15349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5736c284a85540a1bbab2be6df672fc1" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.265745 15349 raft_consensus.cc:3032] T 5736c284a85540a1bbab2be6df672fc1 P c05f7620bb884392931d3243a0b47a9e [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:00:41.265781 15349 raft_consensus.cc:729] T 5736c284a85540a1bbab2be6df672fc1 P c05f7620bb884392931d3243a0b47a9e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:41.265764 15088 raft_consensus.cc:504] T eade118a36da42e3ac08b96a13893a5c P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:41.265901 15349 consensus_queue.cc:260] T 5736c284a85540a1bbab2be6df672fc1 P c05f7620bb884392931d3243a0b47a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.266063 15349 raft_consensus.cc:3037] T 5736c284a85540a1bbab2be6df672fc1 P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:41.266070 15088 leader_election.cc:288] T eade118a36da42e3ac08b96a13893a5c P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 2 election: Requested vote from peers c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009), 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437)
I20240503 02:00:41.266992 15349 raft_consensus.cc:2454] T 5736c284a85540a1bbab2be6df672fc1 P c05f7620bb884392931d3243a0b47a9e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 2.
I20240503 02:00:41.266994 15217 raft_consensus.cc:2454] T 9e9899666b15498791f62dab6d9dda16 P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 3.
I20240503 02:00:41.267104 14816 catalog_manager.cc:5484] T 9e9899666b15498791f62dab6d9dda16 P b602f6099cbb4758847e7e9550685372 reported cstate change: term changed from 2 to 3, leader changed from c05f7620bb884392931d3243a0b47a9e (127.0.99.132) to b602f6099cbb4758847e7e9550685372 (127.0.99.130). New cstate: current_term: 3 leader_uuid: "b602f6099cbb4758847e7e9550685372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:41.267386 15086 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4cdcdde158054cdaabc1856a90fe1302"
dest_uuid: "b602f6099cbb4758847e7e9550685372"
from {username='slave'} at 127.0.0.1:34018
I20240503 02:00:41.267511 15086 raft_consensus.cc:482] T 4cdcdde158054cdaabc1856a90fe1302 P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.267581 15086 raft_consensus.cc:3037] T 4cdcdde158054cdaabc1856a90fe1302 P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:41.268561 15086 raft_consensus.cc:504] T 4cdcdde158054cdaabc1856a90fe1302 P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.268922 15086 leader_election.cc:288] T 4cdcdde158054cdaabc1856a90fe1302 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 3 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:41.270366 15086 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6d98ba8805024eb79bf5df7a8225d784"
dest_uuid: "b602f6099cbb4758847e7e9550685372"
from {username='slave'} at 127.0.0.1:34018
I20240503 02:00:41.270521 15086 raft_consensus.cc:482] T 6d98ba8805024eb79bf5df7a8225d784 P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.270586 15086 raft_consensus.cc:3037] T 6d98ba8805024eb79bf5df7a8225d784 P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:41.272406 15024 leader_election.cc:302] T 5736c284a85540a1bbab2be6df672fc1 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b602f6099cbb4758847e7e9550685372, c05f7620bb884392931d3243a0b47a9e; no voters:
I20240503 02:00:41.270048 15217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5736c284a85540a1bbab2be6df672fc1" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:41.273108 17176 raft_consensus.cc:2781] T 5736c284a85540a1bbab2be6df672fc1 P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:41.273192 17176 raft_consensus.cc:686] T 5736c284a85540a1bbab2be6df672fc1 P b602f6099cbb4758847e7e9550685372 [term 2 LEADER]: Becoming Leader. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.273211 15217 raft_consensus.cc:3037] T 5736c284a85540a1bbab2be6df672fc1 P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:41.273327 17176 consensus_queue.cc:237] T 5736c284a85540a1bbab2be6df672fc1 P b602f6099cbb4758847e7e9550685372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.274704 15349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eade118a36da42e3ac08b96a13893a5c" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.274832 15349 raft_consensus.cc:3032] T eade118a36da42e3ac08b96a13893a5c P c05f7620bb884392931d3243a0b47a9e [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:00:41.274869 15349 raft_consensus.cc:729] T eade118a36da42e3ac08b96a13893a5c P c05f7620bb884392931d3243a0b47a9e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:41.275002 15349 consensus_queue.cc:260] T eade118a36da42e3ac08b96a13893a5c P c05f7620bb884392931d3243a0b47a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 1.16, Last appended by leader: 16, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:41.275202 15349 raft_consensus.cc:3037] T eade118a36da42e3ac08b96a13893a5c P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:41.276192 15349 raft_consensus.cc:2454] T eade118a36da42e3ac08b96a13893a5c P c05f7620bb884392931d3243a0b47a9e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 2.
I20240503 02:00:41.276199 15086 raft_consensus.cc:504] T 6d98ba8805024eb79bf5df7a8225d784 P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.276667 15215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eade118a36da42e3ac08b96a13893a5c" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:41.276666 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcdde158054cdaabc1856a90fe1302" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.276787 15215 raft_consensus.cc:3037] T eade118a36da42e3ac08b96a13893a5c P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:41.276787 15351 raft_consensus.cc:3032] T 4cdcdde158054cdaabc1856a90fe1302 P c05f7620bb884392931d3243a0b47a9e [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:00:41.276825 15351 raft_consensus.cc:729] T 4cdcdde158054cdaabc1856a90fe1302 P c05f7620bb884392931d3243a0b47a9e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:41.276960 15351 consensus_queue.cc:260] T 4cdcdde158054cdaabc1856a90fe1302 P c05f7620bb884392931d3243a0b47a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.277137 15351 raft_consensus.cc:3037] T 4cdcdde158054cdaabc1856a90fe1302 P c05f7620bb884392931d3243a0b47a9e [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:41.277812 15215 raft_consensus.cc:2454] T eade118a36da42e3ac08b96a13893a5c P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 2.
I20240503 02:00:41.278028 15351 raft_consensus.cc:2454] T 4cdcdde158054cdaabc1856a90fe1302 P c05f7620bb884392931d3243a0b47a9e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 3.
I20240503 02:00:41.278672 15024 leader_election.cc:302] T eade118a36da42e3ac08b96a13893a5c P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b602f6099cbb4758847e7e9550685372, c05f7620bb884392931d3243a0b47a9e; no voters:
I20240503 02:00:41.278935 15024 leader_election.cc:302] T 4cdcdde158054cdaabc1856a90fe1302 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b602f6099cbb4758847e7e9550685372, c05f7620bb884392931d3243a0b47a9e; no voters:
I20240503 02:00:41.279063 16496 raft_consensus.cc:2781] T eade118a36da42e3ac08b96a13893a5c P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:41.279282 16496 raft_consensus.cc:686] T eade118a36da42e3ac08b96a13893a5c P b602f6099cbb4758847e7e9550685372 [term 2 LEADER]: Becoming Leader. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.279426 16496 consensus_queue.cc:237] T eade118a36da42e3ac08b96a13893a5c P b602f6099cbb4758847e7e9550685372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 1.16, Last appended by leader: 16, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:41.280011 16496 raft_consensus.cc:2781] T 4cdcdde158054cdaabc1856a90fe1302 P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:00:41.280073 16496 raft_consensus.cc:686] T 4cdcdde158054cdaabc1856a90fe1302 P b602f6099cbb4758847e7e9550685372 [term 3 LEADER]: Becoming Leader. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.280167 16496 consensus_queue.cc:237] T 4cdcdde158054cdaabc1856a90fe1302 P b602f6099cbb4758847e7e9550685372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
W20240503 02:00:41.281683 14915 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47996: Illegal state: replica dcfdfa62edc34dab9525e2712d487582 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.284607 14816 catalog_manager.cc:5484] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372 reported cstate change: term changed from 3 to 4, leader changed from dcfdfa62edc34dab9525e2712d487582 (127.0.99.129) to b602f6099cbb4758847e7e9550685372 (127.0.99.130). New cstate: current_term: 4 leader_uuid: "b602f6099cbb4758847e7e9550685372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:41.285030 14816 catalog_manager.cc:5484] T 5736c284a85540a1bbab2be6df672fc1 P b602f6099cbb4758847e7e9550685372 reported cstate change: term changed from 1 to 2, leader changed from c05f7620bb884392931d3243a0b47a9e (127.0.99.132) to b602f6099cbb4758847e7e9550685372 (127.0.99.130). New cstate: current_term: 2 leader_uuid: "b602f6099cbb4758847e7e9550685372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } }
W20240503 02:00:41.285710 15048 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34032: Illegal state: replica b602f6099cbb4758847e7e9550685372 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.286073 14816 catalog_manager.cc:5484] T 4cdcdde158054cdaabc1856a90fe1302 P b602f6099cbb4758847e7e9550685372 reported cstate change: term changed from 2 to 3, leader changed from c05f7620bb884392931d3243a0b47a9e (127.0.99.132) to b602f6099cbb4758847e7e9550685372 (127.0.99.130). New cstate: current_term: 3 leader_uuid: "b602f6099cbb4758847e7e9550685372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:41.286419 14816 catalog_manager.cc:5484] T eade118a36da42e3ac08b96a13893a5c P b602f6099cbb4758847e7e9550685372 reported cstate change: term changed from 1 to 2, leader changed from c05f7620bb884392931d3243a0b47a9e (127.0.99.132) to b602f6099cbb4758847e7e9550685372 (127.0.99.130). New cstate: current_term: 2 leader_uuid: "b602f6099cbb4758847e7e9550685372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } }
W20240503 02:00:41.288048 15311 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50880: Illegal state: replica c05f7620bb884392931d3243a0b47a9e is not leader of this config: current role FOLLOWER
I20240503 02:00:41.289554 15214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcdde158054cdaabc1856a90fe1302" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:41.289750 15214 raft_consensus.cc:3037] T 4cdcdde158054cdaabc1856a90fe1302 P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:41.290788 15214 raft_consensus.cc:2454] T 4cdcdde158054cdaabc1856a90fe1302 P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 3.
I20240503 02:00:41.292841 15217 raft_consensus.cc:2454] T 5736c284a85540a1bbab2be6df672fc1 P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 2.
I20240503 02:00:41.293397 15086 leader_election.cc:288] T 6d98ba8805024eb79bf5df7a8225d784 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 3 election: Requested vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437)
W20240503 02:00:41.295188 14915 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48020: Illegal state: replica dcfdfa62edc34dab9525e2712d487582 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.295789 15217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d98ba8805024eb79bf5df7a8225d784" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
W20240503 02:00:41.296173 15311 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50940: Illegal state: replica c05f7620bb884392931d3243a0b47a9e is not leader of this config: current role FOLLOWER
I20240503 02:00:41.296476 15086 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "42ceeb7b3c4146cd8a2e18b3e9ae987c"
dest_uuid: "b602f6099cbb4758847e7e9550685372"
from {username='slave'} at 127.0.0.1:34018
I20240503 02:00:41.296599 15086 raft_consensus.cc:482] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.296681 15086 raft_consensus.cc:3037] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:41.297410 16866 heartbeater.cc:503] Master 127.0.99.190:38675 was elected leader, sending a full tablet report...
I20240503 02:00:41.297432 15351 raft_consensus.cc:1261] T eade118a36da42e3ac08b96a13893a5c P c05f7620bb884392931d3243a0b47a9e [term 2 FOLLOWER]: Refusing update from remote peer b602f6099cbb4758847e7e9550685372: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 2 index: 18. (index mismatch)
I20240503 02:00:41.297753 15086 raft_consensus.cc:504] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:41.297835 16496 consensus_queue.cc:1035] T eade118a36da42e3ac08b96a13893a5c P b602f6099cbb4758847e7e9550685372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20240503 02:00:41.298132 15086 leader_election.cc:288] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 3 election: Requested vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437)
I20240503 02:00:41.298461 15217 raft_consensus.cc:1261] T eade118a36da42e3ac08b96a13893a5c P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Refusing update from remote peer b602f6099cbb4758847e7e9550685372: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 2 index: 18. (index mismatch)
I20240503 02:00:41.302167 15217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ceeb7b3c4146cd8a2e18b3e9ae987c" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:41.302318 15217 raft_consensus.cc:3037] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:41.303390 15217 raft_consensus.cc:2454] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 3.
I20240503 02:00:41.303697 15086 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6b0c8b72e1884529a8a9b83d2a734d71"
dest_uuid: "b602f6099cbb4758847e7e9550685372"
from {username='slave'} at 127.0.0.1:34018
I20240503 02:00:41.303822 15086 raft_consensus.cc:482] T 6b0c8b72e1884529a8a9b83d2a734d71 P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
W20240503 02:00:41.304508 15311 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50950: Illegal state: replica c05f7620bb884392931d3243a0b47a9e is not leader of this config: current role FOLLOWER
I20240503 02:00:41.303871 15086 raft_consensus.cc:3037] T 6b0c8b72e1884529a8a9b83d2a734d71 P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:41.305905 15086 raft_consensus.cc:504] T 6b0c8b72e1884529a8a9b83d2a734d71 P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.306341 15086 leader_election.cc:288] T 6b0c8b72e1884529a8a9b83d2a734d71 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 3 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817)
I20240503 02:00:41.312496 15089 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0989d98050744df6a1c699cb86ada505"
dest_uuid: "b602f6099cbb4758847e7e9550685372"
from {username='slave'} at 127.0.0.1:34018
I20240503 02:00:41.312672 15089 raft_consensus.cc:482] T 0989d98050744df6a1c699cb86ada505 P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.312738 15089 raft_consensus.cc:3037] T 0989d98050744df6a1c699cb86ada505 P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Advancing to term 4
W20240503 02:00:41.312826 15311 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50904: Illegal state: replica c05f7620bb884392931d3243a0b47a9e is not leader of this config: current role FOLLOWER
I20240503 02:00:41.313793 15089 raft_consensus.cc:504] T 0989d98050744df6a1c699cb86ada505 P b602f6099cbb4758847e7e9550685372 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.313908 15024 leader_election.cc:302] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, b602f6099cbb4758847e7e9550685372; no voters:
W20240503 02:00:41.313973 15048 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34032: Illegal state: replica b602f6099cbb4758847e7e9550685372 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.314205 15089 leader_election.cc:288] T 0989d98050744df6a1c699cb86ada505 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 4 election: Requested vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:41.314471 17176 consensus_queue.cc:1035] T eade118a36da42e3ac08b96a13893a5c P b602f6099cbb4758847e7e9550685372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20240503 02:00:41.314765 17176 raft_consensus.cc:2781] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:00:41.314828 17176 raft_consensus.cc:686] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P b602f6099cbb4758847e7e9550685372 [term 3 LEADER]: Becoming Leader. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.314958 17176 consensus_queue.cc:237] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P b602f6099cbb4758847e7e9550685372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:41.315524 15089 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d805ee04df39406ba30b3b3b9e349507"
dest_uuid: "b602f6099cbb4758847e7e9550685372"
from {username='slave'} at 127.0.0.1:34018
I20240503 02:00:41.315629 15089 raft_consensus.cc:482] T d805ee04df39406ba30b3b3b9e349507 P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.315676 15089 raft_consensus.cc:3037] T d805ee04df39406ba30b3b3b9e349507 P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:41.316138 15219 raft_consensus.cc:1261] T 9e9899666b15498791f62dab6d9dda16 P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Refusing update from remote peer b602f6099cbb4758847e7e9550685372: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 3 index: 8. (index mismatch)
I20240503 02:00:41.316684 15351 raft_consensus.cc:1261] T 9e9899666b15498791f62dab6d9dda16 P c05f7620bb884392931d3243a0b47a9e [term 3 FOLLOWER]: Refusing update from remote peer b602f6099cbb4758847e7e9550685372: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 3 index: 8. (index mismatch)
I20240503 02:00:41.318928 14811 catalog_manager.cc:5484] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P b602f6099cbb4758847e7e9550685372 reported cstate change: term changed from 2 to 3, leader changed from dcfdfa62edc34dab9525e2712d487582 (127.0.99.129) to b602f6099cbb4758847e7e9550685372 (127.0.99.130). New cstate: current_term: 3 leader_uuid: "b602f6099cbb4758847e7e9550685372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:41.320338 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0989d98050744df6a1c699cb86ada505" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.320498 15351 raft_consensus.cc:3037] T 0989d98050744df6a1c699cb86ada505 P c05f7620bb884392931d3243a0b47a9e [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.321614 15351 raft_consensus.cc:2396] T 0989d98050744df6a1c699cb86ada505 P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b602f6099cbb4758847e7e9550685372 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 02:00:41.325716 15089 raft_consensus.cc:504] T d805ee04df39406ba30b3b3b9e349507 P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.325748 17332 consensus_queue.cc:1035] T 9e9899666b15498791f62dab6d9dda16 P b602f6099cbb4758847e7e9550685372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:41.326071 16496 consensus_queue.cc:1035] T 9e9899666b15498791f62dab6d9dda16 P b602f6099cbb4758847e7e9550685372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:41.326454 15089 leader_election.cc:288] T d805ee04df39406ba30b3b3b9e349507 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 3 election: Requested vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:41.326504 15217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b0c8b72e1884529a8a9b83d2a734d71" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 3 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:41.326623 15217 raft_consensus.cc:3037] T 6b0c8b72e1884529a8a9b83d2a734d71 P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:41.327065 15089 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c74522fbb5c64d27b5c95f05896b0716"
dest_uuid: "b602f6099cbb4758847e7e9550685372"
from {username='slave'} at 127.0.0.1:34018
I20240503 02:00:41.327183 15089 raft_consensus.cc:482] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.327243 15089 raft_consensus.cc:3037] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:41.327926 15086 raft_consensus.cc:1226] T 0989d98050744df6a1c699cb86ada505 P b602f6099cbb4758847e7e9550685372 [term 4 FOLLOWER]: Rejecting Update request from peer dcfdfa62edc34dab9525e2712d487582 for earlier term 3. Current term is 4. Ops: []
I20240503 02:00:41.328271 17225 consensus_queue.cc:1046] T 0989d98050744df6a1c699cb86ada505 P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:41.328496 17225 raft_consensus.cc:3032] T 0989d98050744df6a1c699cb86ada505 P dcfdfa62edc34dab9525e2712d487582 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 02:00:41.328559 17225 raft_consensus.cc:729] T 0989d98050744df6a1c699cb86ada505 P dcfdfa62edc34dab9525e2712d487582 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:41.328711 15217 raft_consensus.cc:2454] T 6b0c8b72e1884529a8a9b83d2a734d71 P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 3.
I20240503 02:00:41.328702 17225 consensus_queue.cc:260] T 0989d98050744df6a1c699cb86ada505 P dcfdfa62edc34dab9525e2712d487582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.328851 17225 raft_consensus.cc:3037] T 0989d98050744df6a1c699cb86ada505 P dcfdfa62edc34dab9525e2712d487582 [term 3 FOLLOWER]: Advancing to term 4
W20240503 02:00:41.329066 14915 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48020: Illegal state: replica dcfdfa62edc34dab9525e2712d487582 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.331144 15089 raft_consensus.cc:504] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.331578 15089 leader_election.cc:288] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 3 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:41.332489 15089 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2b757c430aaa42f382d174ca57f5fcfc"
dest_uuid: "b602f6099cbb4758847e7e9550685372"
from {username='slave'} at 127.0.0.1:34018
I20240503 02:00:41.332684 15089 raft_consensus.cc:482] T 2b757c430aaa42f382d174ca57f5fcfc P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.332794 15089 raft_consensus.cc:3037] T 2b757c430aaa42f382d174ca57f5fcfc P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:41.333951 15089 raft_consensus.cc:504] T 2b757c430aaa42f382d174ca57f5fcfc P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.334439 15089 leader_election.cc:288] T 2b757c430aaa42f382d174ca57f5fcfc P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 3 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817)
I20240503 02:00:41.335188 15089 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6b225db529f14e7386f21f42bf31493b"
dest_uuid: "b602f6099cbb4758847e7e9550685372"
from {username='slave'} at 127.0.0.1:34018
I20240503 02:00:41.335357 15089 raft_consensus.cc:482] T 6b225db529f14e7386f21f42bf31493b P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.335465 15089 raft_consensus.cc:3037] T 6b225db529f14e7386f21f42bf31493b P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:41.336483 15089 raft_consensus.cc:504] T 6b225db529f14e7386f21f42bf31493b P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.336959 15089 leader_election.cc:288] T 6b225db529f14e7386f21f42bf31493b P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 3 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817)
W20240503 02:00:41.338559 15311 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50950: Illegal state: replica c05f7620bb884392931d3243a0b47a9e is not leader of this config: current role FOLLOWER
W20240503 02:00:41.339327 15176 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57610: Illegal state: replica 620596b893d84291b88b90dc9e317612 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.337682 15089 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ebd7cdd284a74741b338e9f4778502b5"
dest_uuid: "b602f6099cbb4758847e7e9550685372"
from {username='slave'} at 127.0.0.1:34018
I20240503 02:00:41.340031 15089 raft_consensus.cc:482] T ebd7cdd284a74741b338e9f4778502b5 P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.340102 15349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d805ee04df39406ba30b3b3b9e349507" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 3 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.340173 15089 raft_consensus.cc:3037] T ebd7cdd284a74741b338e9f4778502b5 P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:41.340245 15349 raft_consensus.cc:3037] T d805ee04df39406ba30b3b3b9e349507 P c05f7620bb884392931d3243a0b47a9e [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:41.341403 15349 raft_consensus.cc:2454] T d805ee04df39406ba30b3b3b9e349507 P c05f7620bb884392931d3243a0b47a9e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 3.
I20240503 02:00:41.341416 15089 raft_consensus.cc:504] T ebd7cdd284a74741b338e9f4778502b5 P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.341922 15024 leader_election.cc:302] T 6b0c8b72e1884529a8a9b83d2a734d71 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, b602f6099cbb4758847e7e9550685372; no voters:
I20240503 02:00:41.342103 15089 leader_election.cc:288] T ebd7cdd284a74741b338e9f4778502b5 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 2 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:41.342126 17332 raft_consensus.cc:2781] T 6b0c8b72e1884529a8a9b83d2a734d71 P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:00:41.342276 17332 raft_consensus.cc:686] T 6b0c8b72e1884529a8a9b83d2a734d71 P b602f6099cbb4758847e7e9550685372 [term 3 LEADER]: Becoming Leader. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.342439 17332 consensus_queue.cc:237] T 6b0c8b72e1884529a8a9b83d2a734d71 P b602f6099cbb4758847e7e9550685372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.342499 15219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c74522fbb5c64d27b5c95f05896b0716" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 3 candidate_status { last_received { term: 2 index: 25 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:41.342618 15219 raft_consensus.cc:3037] T c74522fbb5c64d27b5c95f05896b0716 P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:41.343008 15089 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f85a361ce0254472b37526362d4edf0e"
dest_uuid: "b602f6099cbb4758847e7e9550685372"
from {username='slave'} at 127.0.0.1:34018
I20240503 02:00:41.343185 15089 raft_consensus.cc:482] T f85a361ce0254472b37526362d4edf0e P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.343291 15089 raft_consensus.cc:3037] T f85a361ce0254472b37526362d4edf0e P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:41.343644 15219 raft_consensus.cc:2454] T c74522fbb5c64d27b5c95f05896b0716 P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 3.
I20240503 02:00:41.344358 15089 raft_consensus.cc:504] T f85a361ce0254472b37526362d4edf0e P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:41.344805 15089 leader_election.cc:288] T f85a361ce0254472b37526362d4edf0e P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 2 election: Requested vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437)
I20240503 02:00:41.345023 15219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b757c430aaa42f382d174ca57f5fcfc" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 3 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:41.345156 15219 raft_consensus.cc:3037] T 2b757c430aaa42f382d174ca57f5fcfc P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:41.346141 15219 raft_consensus.cc:2454] T 2b757c430aaa42f382d174ca57f5fcfc P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 3.
I20240503 02:00:41.346503 15089 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "454a24fd6d964d73ae3d5057f90fe36d"
dest_uuid: "b602f6099cbb4758847e7e9550685372"
from {username='slave'} at 127.0.0.1:34018
I20240503 02:00:41.346628 15089 raft_consensus.cc:482] T 454a24fd6d964d73ae3d5057f90fe36d P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.346601 15349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c74522fbb5c64d27b5c95f05896b0716" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 3 candidate_status { last_received { term: 2 index: 25 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.346683 15089 raft_consensus.cc:3037] T 454a24fd6d964d73ae3d5057f90fe36d P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:41.346714 15349 raft_consensus.cc:3032] T c74522fbb5c64d27b5c95f05896b0716 P c05f7620bb884392931d3243a0b47a9e [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:00:41.346760 15349 raft_consensus.cc:729] T c74522fbb5c64d27b5c95f05896b0716 P c05f7620bb884392931d3243a0b47a9e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:41.346935 15349 consensus_queue.cc:260] T c74522fbb5c64d27b5c95f05896b0716 P c05f7620bb884392931d3243a0b47a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 2.25, Last appended by leader: 25, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.347148 15349 raft_consensus.cc:3037] T c74522fbb5c64d27b5c95f05896b0716 P c05f7620bb884392931d3243a0b47a9e [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:41.348341 15349 raft_consensus.cc:2454] T c74522fbb5c64d27b5c95f05896b0716 P c05f7620bb884392931d3243a0b47a9e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 3.
I20240503 02:00:41.348939 15219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b225db529f14e7386f21f42bf31493b" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 3 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:41.349090 15219 raft_consensus.cc:3037] T 6b225db529f14e7386f21f42bf31493b P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:41.349150 15089 raft_consensus.cc:504] T 454a24fd6d964d73ae3d5057f90fe36d P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.350159 15215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebd7cdd284a74741b338e9f4778502b5" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:41.350270 15215 raft_consensus.cc:3037] T ebd7cdd284a74741b338e9f4778502b5 P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:41.350448 15089 leader_election.cc:288] T 454a24fd6d964d73ae3d5057f90fe36d P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 3 election: Requested vote from peers c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009), dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817)
W20240503 02:00:41.350468 15048 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34032: Illegal state: replica b602f6099cbb4758847e7e9550685372 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.351212 15215 raft_consensus.cc:2454] T ebd7cdd284a74741b338e9f4778502b5 P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 2.
I20240503 02:00:41.351413 15089 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0e3bb299e61148fcb355641e0134b3df"
dest_uuid: "b602f6099cbb4758847e7e9550685372"
from {username='slave'} at 127.0.0.1:34018
I20240503 02:00:41.351528 15219 raft_consensus.cc:2454] T 6b225db529f14e7386f21f42bf31493b P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 3.
I20240503 02:00:41.351758 15089 raft_consensus.cc:482] T 0e3bb299e61148fcb355641e0134b3df P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.351833 15089 raft_consensus.cc:3037] T 0e3bb299e61148fcb355641e0134b3df P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:41.352193 15349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebd7cdd284a74741b338e9f4778502b5" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.352320 15349 raft_consensus.cc:3032] T ebd7cdd284a74741b338e9f4778502b5 P c05f7620bb884392931d3243a0b47a9e [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:00:41.352367 15349 raft_consensus.cc:729] T ebd7cdd284a74741b338e9f4778502b5 P c05f7620bb884392931d3243a0b47a9e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:41.352571 15349 consensus_queue.cc:260] T ebd7cdd284a74741b338e9f4778502b5 P c05f7620bb884392931d3243a0b47a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.352756 15349 raft_consensus.cc:3037] T ebd7cdd284a74741b338e9f4778502b5 P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:41.353132 15024 leader_election.cc:302] T d805ee04df39406ba30b3b3b9e349507 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b602f6099cbb4758847e7e9550685372, c05f7620bb884392931d3243a0b47a9e; no voters:
I20240503 02:00:41.353365 17176 raft_consensus.cc:2781] T d805ee04df39406ba30b3b3b9e349507 P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:00:41.353544 17176 raft_consensus.cc:686] T d805ee04df39406ba30b3b3b9e349507 P b602f6099cbb4758847e7e9550685372 [term 3 LEADER]: Becoming Leader. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.353760 17176 consensus_queue.cc:237] T d805ee04df39406ba30b3b3b9e349507 P b602f6099cbb4758847e7e9550685372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.354092 15349 raft_consensus.cc:2454] T ebd7cdd284a74741b338e9f4778502b5 P c05f7620bb884392931d3243a0b47a9e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 2.
I20240503 02:00:41.354522 15089 raft_consensus.cc:504] T 0e3bb299e61148fcb355641e0134b3df P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.354722 15024 leader_election.cc:302] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b602f6099cbb4758847e7e9550685372, c05f7620bb884392931d3243a0b47a9e; no voters:
I20240503 02:00:41.354884 17176 raft_consensus.cc:2781] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:00:41.354988 17176 raft_consensus.cc:686] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372 [term 3 LEADER]: Becoming Leader. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.355185 17176 consensus_queue.cc:237] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 2.25, Last appended by leader: 25, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.355880 15024 leader_election.cc:302] T ebd7cdd284a74741b338e9f4778502b5 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b602f6099cbb4758847e7e9550685372, c05f7620bb884392931d3243a0b47a9e; no voters:
I20240503 02:00:41.356501 17176 raft_consensus.cc:2781] T ebd7cdd284a74741b338e9f4778502b5 P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:41.356591 17176 raft_consensus.cc:686] T ebd7cdd284a74741b338e9f4778502b5 P b602f6099cbb4758847e7e9550685372 [term 2 LEADER]: Becoming Leader. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.356705 17176 consensus_queue.cc:237] T ebd7cdd284a74741b338e9f4778502b5 P b602f6099cbb4758847e7e9550685372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
W20240503 02:00:41.358332 14893 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.131:42437, user_credentials={real_user=slave}} blocked reactor thread for 57049.2us
I20240503 02:00:41.358587 15089 leader_election.cc:288] T 0e3bb299e61148fcb355641e0134b3df P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 3 election: Requested vote from peers c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009), dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817)
I20240503 02:00:41.358556 15217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85a361ce0254472b37526362d4edf0e" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 2 candidate_status { last_received { term: 1 index: 19 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:41.358815 15217 raft_consensus.cc:3037] T f85a361ce0254472b37526362d4edf0e P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:41.359369 15089 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ca0d7d3e4d59496c9a9a59cde24b9247"
dest_uuid: "b602f6099cbb4758847e7e9550685372"
from {username='slave'} at 127.0.0.1:34018
I20240503 02:00:41.359504 15089 raft_consensus.cc:482] T ca0d7d3e4d59496c9a9a59cde24b9247 P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.359567 15089 raft_consensus.cc:3037] T ca0d7d3e4d59496c9a9a59cde24b9247 P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.360080 15217 raft_consensus.cc:2454] T f85a361ce0254472b37526362d4edf0e P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 2.
I20240503 02:00:41.360283 15349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "454a24fd6d964d73ae3d5057f90fe36d" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.361446 15349 raft_consensus.cc:3037] T 454a24fd6d964d73ae3d5057f90fe36d P c05f7620bb884392931d3243a0b47a9e [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:41.362666 15349 raft_consensus.cc:2454] T 454a24fd6d964d73ae3d5057f90fe36d P c05f7620bb884392931d3243a0b47a9e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 3.
I20240503 02:00:41.362671 15089 raft_consensus.cc:504] T ca0d7d3e4d59496c9a9a59cde24b9247 P b602f6099cbb4758847e7e9550685372 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.363067 15089 leader_election.cc:288] T ca0d7d3e4d59496c9a9a59cde24b9247 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 4 election: Requested vote from peers c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009), dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817)
I20240503 02:00:41.364043 15089 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5da91b7af68e4cf597dc89a017ee0197"
dest_uuid: "b602f6099cbb4758847e7e9550685372"
from {username='slave'} at 127.0.0.1:34018
W20240503 02:00:41.364189 14915 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48020: Illegal state: replica dcfdfa62edc34dab9525e2712d487582 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.364216 15089 raft_consensus.cc:482] T 5da91b7af68e4cf597dc89a017ee0197 P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.364332 15089 raft_consensus.cc:3037] T 5da91b7af68e4cf597dc89a017ee0197 P b602f6099cbb4758847e7e9550685372 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:41.365341 15089 raft_consensus.cc:504] T 5da91b7af68e4cf597dc89a017ee0197 P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.365762 15089 leader_election.cc:288] T 5da91b7af68e4cf597dc89a017ee0197 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 2 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:41.366732 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e3bb299e61148fcb355641e0134b3df" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.366858 15351 raft_consensus.cc:3037] T 0e3bb299e61148fcb355641e0134b3df P c05f7620bb884392931d3243a0b47a9e [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:41.368075 15351 raft_consensus.cc:2454] T 0e3bb299e61148fcb355641e0134b3df P c05f7620bb884392931d3243a0b47a9e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 3.
W20240503 02:00:41.369040 14915 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47996: Illegal state: replica dcfdfa62edc34dab9525e2712d487582 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.369068 15024 leader_election.cc:302] T 2b757c430aaa42f382d174ca57f5fcfc P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, b602f6099cbb4758847e7e9550685372; no voters:
I20240503 02:00:41.369290 15024 leader_election.cc:302] T 6b225db529f14e7386f21f42bf31493b P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, b602f6099cbb4758847e7e9550685372; no voters:
I20240503 02:00:41.369457 17331 raft_consensus.cc:2781] T 2b757c430aaa42f382d174ca57f5fcfc P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:00:41.369596 17331 raft_consensus.cc:686] T 2b757c430aaa42f382d174ca57f5fcfc P b602f6099cbb4758847e7e9550685372 [term 3 LEADER]: Becoming Leader. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.369748 17331 consensus_queue.cc:237] T 2b757c430aaa42f382d174ca57f5fcfc P b602f6099cbb4758847e7e9550685372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
W20240503 02:00:41.371045 15311 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50888: Illegal state: replica c05f7620bb884392931d3243a0b47a9e is not leader of this config: current role FOLLOWER
I20240503 02:00:41.371934 14815 catalog_manager.cc:5484] T 6b0c8b72e1884529a8a9b83d2a734d71 P b602f6099cbb4758847e7e9550685372 reported cstate change: term changed from 1 to 3, leader changed from 620596b893d84291b88b90dc9e317612 (127.0.99.131) to b602f6099cbb4758847e7e9550685372 (127.0.99.130). New cstate: current_term: 3 leader_uuid: "b602f6099cbb4758847e7e9550685372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:41.369473 16496 raft_consensus.cc:2781] T 6b225db529f14e7386f21f42bf31493b P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:00:41.375108 16496 raft_consensus.cc:686] T 6b225db529f14e7386f21f42bf31493b P b602f6099cbb4758847e7e9550685372 [term 3 LEADER]: Becoming Leader. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.375316 16496 consensus_queue.cc:237] T 6b225db529f14e7386f21f42bf31493b P b602f6099cbb4758847e7e9550685372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
W20240503 02:00:41.377883 15311 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50950: Illegal state: replica c05f7620bb884392931d3243a0b47a9e is not leader of this config: current role FOLLOWER
W20240503 02:00:41.378826 15048 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34006: Illegal state: replica b602f6099cbb4758847e7e9550685372 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.383360 15217 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "de805ef7511d4ee9b9bbed67e0a4186c"
dest_uuid: "620596b893d84291b88b90dc9e317612"
from {username='slave'} at 127.0.0.1:57680
I20240503 02:00:41.383638 15217 raft_consensus.cc:482] T de805ef7511d4ee9b9bbed67e0a4186c P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.383627 15022 leader_election.cc:302] T 590736f5f1ca4703821c409d882c2f77 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b602f6099cbb4758847e7e9550685372; no voters: c05f7620bb884392931d3243a0b47a9e, dcfdfa62edc34dab9525e2712d487582
I20240503 02:00:41.383849 16496 raft_consensus.cc:2726] T 590736f5f1ca4703821c409d882c2f77 P b602f6099cbb4758847e7e9550685372 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240503 02:00:41.384038 15217 raft_consensus.cc:3037] T de805ef7511d4ee9b9bbed67e0a4186c P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.385223 15217 raft_consensus.cc:504] T de805ef7511d4ee9b9bbed67e0a4186c P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.386803 15089 raft_consensus.cc:1226] T ca0d7d3e4d59496c9a9a59cde24b9247 P b602f6099cbb4758847e7e9550685372 [term 4 FOLLOWER]: Rejecting Update request from peer dcfdfa62edc34dab9525e2712d487582 for earlier term 3. Current term is 4. Ops: []
I20240503 02:00:41.387348 17225 consensus_queue.cc:1046] T ca0d7d3e4d59496c9a9a59cde24b9247 P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:41.387524 17225 raft_consensus.cc:3032] T ca0d7d3e4d59496c9a9a59cde24b9247 P dcfdfa62edc34dab9525e2712d487582 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 02:00:41.387563 17225 raft_consensus.cc:729] T ca0d7d3e4d59496c9a9a59cde24b9247 P dcfdfa62edc34dab9525e2712d487582 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:41.387689 17225 consensus_queue.cc:260] T ca0d7d3e4d59496c9a9a59cde24b9247 P dcfdfa62edc34dab9525e2712d487582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 3.9, Last appended by leader: 9, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.387835 17225 raft_consensus.cc:3037] T ca0d7d3e4d59496c9a9a59cde24b9247 P dcfdfa62edc34dab9525e2712d487582 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.389864 15089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de805ef7511d4ee9b9bbed67e0a4186c" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 4 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:41.390095 15089 raft_consensus.cc:3032] T de805ef7511d4ee9b9bbed67e0a4186c P b602f6099cbb4758847e7e9550685372 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 02:00:41.390187 15089 raft_consensus.cc:729] T de805ef7511d4ee9b9bbed67e0a4186c P b602f6099cbb4758847e7e9550685372 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.390403 15089 consensus_queue.cc:260] T de805ef7511d4ee9b9bbed67e0a4186c P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.7, Last appended by leader: 7, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.390563 15217 leader_election.cc:288] T de805ef7511d4ee9b9bbed67e0a4186c P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 4 election: Requested vote from peers c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
I20240503 02:00:41.391274 15219 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7b4f203f4048490ba599353a4cd92748"
dest_uuid: "620596b893d84291b88b90dc9e317612"
from {username='slave'} at 127.0.0.1:57680
I20240503 02:00:41.391397 15219 raft_consensus.cc:482] T 7b4f203f4048490ba599353a4cd92748 P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.391448 15219 raft_consensus.cc:3037] T 7b4f203f4048490ba599353a4cd92748 P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.392457 15219 raft_consensus.cc:504] T 7b4f203f4048490ba599353a4cd92748 P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.392589 15349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de805ef7511d4ee9b9bbed67e0a4186c" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 4 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.392700 15349 raft_consensus.cc:3037] T de805ef7511d4ee9b9bbed67e0a4186c P c05f7620bb884392931d3243a0b47a9e [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.394037 15349 raft_consensus.cc:2454] T de805ef7511d4ee9b9bbed67e0a4186c P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 4.
I20240503 02:00:41.394019 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4f203f4048490ba599353a4cd92748" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:41.394140 14957 raft_consensus.cc:3037] T 7b4f203f4048490ba599353a4cd92748 P dcfdfa62edc34dab9525e2712d487582 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.394347 15155 leader_election.cc:302] T de805ef7511d4ee9b9bbed67e0a4186c P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, c05f7620bb884392931d3243a0b47a9e; no voters:
I20240503 02:00:41.394551 17224 raft_consensus.cc:2781] T de805ef7511d4ee9b9bbed67e0a4186c P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 02:00:41.394615 17224 raft_consensus.cc:686] T de805ef7511d4ee9b9bbed67e0a4186c P 620596b893d84291b88b90dc9e317612 [term 4 LEADER]: Becoming Leader. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:41.394771 17224 consensus_queue.cc:237] T de805ef7511d4ee9b9bbed67e0a4186c P 620596b893d84291b88b90dc9e317612 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.395202 14957 raft_consensus.cc:2396] T 7b4f203f4048490ba599353a4cd92748 P dcfdfa62edc34dab9525e2712d487582 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 620596b893d84291b88b90dc9e317612 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 02:00:41.396351 15349 raft_consensus.cc:1226] T 6d2a1296a54a4b92a03c12954de0cc34 P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Rejecting Update request from peer dcfdfa62edc34dab9525e2712d487582 for earlier term 3. Current term is 4. Ops: []
I20240503 02:00:41.396657 15219 leader_election.cc:288] T 7b4f203f4048490ba599353a4cd92748 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 4 election: Requested vote from peers b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207), dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817)
I20240503 02:00:41.397433 15215 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "21d92d412a5247e8907c5c41d253ad2e"
dest_uuid: "620596b893d84291b88b90dc9e317612"
from {username='slave'} at 127.0.0.1:57680
I20240503 02:00:41.397581 15215 raft_consensus.cc:482] T 21d92d412a5247e8907c5c41d253ad2e P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.397639 15215 raft_consensus.cc:3037] T 21d92d412a5247e8907c5c41d253ad2e P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.397655 14811 catalog_manager.cc:5484] T de805ef7511d4ee9b9bbed67e0a4186c P 620596b893d84291b88b90dc9e317612 reported cstate change: term changed from 3 to 4, leader changed from b602f6099cbb4758847e7e9550685372 (127.0.99.130) to 620596b893d84291b88b90dc9e317612 (127.0.99.131). New cstate: current_term: 4 leader_uuid: "620596b893d84291b88b90dc9e317612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:41.398334 15349 raft_consensus.cc:1226] T 0e3bb299e61148fcb355641e0134b3df P c05f7620bb884392931d3243a0b47a9e [term 3 FOLLOWER]: Rejecting Update request from peer dcfdfa62edc34dab9525e2712d487582 for earlier term 2. Current term is 3. Ops: []
I20240503 02:00:41.398650 15215 raft_consensus.cc:504] T 21d92d412a5247e8907c5c41d253ad2e P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.399034 15215 leader_election.cc:288] T 21d92d412a5247e8907c5c41d253ad2e P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 4 election: Requested vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:41.399928 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21d92d412a5247e8907c5c41d253ad2e" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.400068 15351 raft_consensus.cc:3037] T 21d92d412a5247e8907c5c41d253ad2e P c05f7620bb884392931d3243a0b47a9e [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.400941 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21d92d412a5247e8907c5c41d253ad2e" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:41.401070 14957 raft_consensus.cc:3032] T 21d92d412a5247e8907c5c41d253ad2e P dcfdfa62edc34dab9525e2712d487582 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 02:00:41.401113 14957 raft_consensus.cc:729] T 21d92d412a5247e8907c5c41d253ad2e P dcfdfa62edc34dab9525e2712d487582 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:41.401127 15351 raft_consensus.cc:2454] T 21d92d412a5247e8907c5c41d253ad2e P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 4.
I20240503 02:00:41.401265 14957 consensus_queue.cc:260] T 21d92d412a5247e8907c5c41d253ad2e P dcfdfa62edc34dab9525e2712d487582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.401400 14957 raft_consensus.cc:3037] T 21d92d412a5247e8907c5c41d253ad2e P dcfdfa62edc34dab9525e2712d487582 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.402166 15349 raft_consensus.cc:1226] T 0989d98050744df6a1c699cb86ada505 P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Rejecting Update request from peer dcfdfa62edc34dab9525e2712d487582 for earlier term 3. Current term is 4. Ops: []
I20240503 02:00:41.402390 14957 raft_consensus.cc:2454] T 21d92d412a5247e8907c5c41d253ad2e P dcfdfa62edc34dab9525e2712d487582 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 4.
I20240503 02:00:41.402751 15214 raft_consensus.cc:1226] T 2b757c430aaa42f382d174ca57f5fcfc P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Rejecting Update request from peer dcfdfa62edc34dab9525e2712d487582 for earlier term 2. Current term is 3. Ops: []
I20240503 02:00:41.402937 15155 leader_election.cc:302] T 21d92d412a5247e8907c5c41d253ad2e P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, c05f7620bb884392931d3243a0b47a9e; no voters:
I20240503 02:00:41.403142 16264 raft_consensus.cc:2781] T 21d92d412a5247e8907c5c41d253ad2e P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 02:00:41.403218 16264 raft_consensus.cc:686] T 21d92d412a5247e8907c5c41d253ad2e P 620596b893d84291b88b90dc9e317612 [term 4 LEADER]: Becoming Leader. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:41.403352 16264 consensus_queue.cc:237] T 21d92d412a5247e8907c5c41d253ad2e P 620596b893d84291b88b90dc9e317612 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.403719 15214 raft_consensus.cc:1226] T f85a361ce0254472b37526362d4edf0e P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Rejecting Update request from peer dcfdfa62edc34dab9525e2712d487582 for earlier term 1. Current term is 2. Ops: [1.18-1.19]
I20240503 02:00:41.403944 15215 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "830da5cb4972453cb9bf2145a0839ced"
dest_uuid: "620596b893d84291b88b90dc9e317612"
from {username='slave'} at 127.0.0.1:57680
I20240503 02:00:41.404049 15215 raft_consensus.cc:482] T 830da5cb4972453cb9bf2145a0839ced P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.404096 15215 raft_consensus.cc:3037] T 830da5cb4972453cb9bf2145a0839ced P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.405174 15215 raft_consensus.cc:504] T 830da5cb4972453cb9bf2145a0839ced P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.405542 15215 leader_election.cc:288] T 830da5cb4972453cb9bf2145a0839ced P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 4 election: Requested vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
W20240503 02:00:41.405885 15311 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50904: Illegal state: replica c05f7620bb884392931d3243a0b47a9e is not leader of this config: current role FOLLOWER
I20240503 02:00:41.406733 15214 raft_consensus.cc:1226] T 21d92d412a5247e8907c5c41d253ad2e P 620596b893d84291b88b90dc9e317612 [term 4 LEADER]: Rejecting Update request from peer dcfdfa62edc34dab9525e2712d487582 for earlier term 3. Current term is 4. Ops: []
I20240503 02:00:41.407167 14816 catalog_manager.cc:5484] T 21d92d412a5247e8907c5c41d253ad2e P 620596b893d84291b88b90dc9e317612 reported cstate change: term changed from 3 to 4, leader changed from dcfdfa62edc34dab9525e2712d487582 (127.0.99.129) to 620596b893d84291b88b90dc9e317612 (127.0.99.131). New cstate: current_term: 4 leader_uuid: "620596b893d84291b88b90dc9e317612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:41.407945 14956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2a1296a54a4b92a03c12954de0cc34" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 4 candidate_status { last_received { term: 3 index: 10 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:41.408102 14956 raft_consensus.cc:3032] T 6d2a1296a54a4b92a03c12954de0cc34 P dcfdfa62edc34dab9525e2712d487582 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 02:00:41.408145 14956 raft_consensus.cc:729] T 6d2a1296a54a4b92a03c12954de0cc34 P dcfdfa62edc34dab9525e2712d487582 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:41.408278 14956 consensus_queue.cc:260] T 6d2a1296a54a4b92a03c12954de0cc34 P dcfdfa62edc34dab9525e2712d487582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 3.10, Last appended by leader: 10, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.408459 14956 raft_consensus.cc:3037] T 6d2a1296a54a4b92a03c12954de0cc34 P dcfdfa62edc34dab9525e2712d487582 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.409489 14956 raft_consensus.cc:2454] T 6d2a1296a54a4b92a03c12954de0cc34 P dcfdfa62edc34dab9525e2712d487582 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 4.
I20240503 02:00:41.409540 14954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "830da5cb4972453cb9bf2145a0839ced" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:41.409680 14954 raft_consensus.cc:3032] T 830da5cb4972453cb9bf2145a0839ced P dcfdfa62edc34dab9525e2712d487582 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:00:41.409714 14954 raft_consensus.cc:729] T 830da5cb4972453cb9bf2145a0839ced P dcfdfa62edc34dab9525e2712d487582 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:41.409848 14956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d98ba8805024eb79bf5df7a8225d784" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:41.409946 14956 raft_consensus.cc:3032] T 6d98ba8805024eb79bf5df7a8225d784 P dcfdfa62edc34dab9525e2712d487582 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:00:41.409981 14956 raft_consensus.cc:729] T 6d98ba8805024eb79bf5df7a8225d784 P dcfdfa62edc34dab9525e2712d487582 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:41.410123 14956 consensus_queue.cc:260] T 6d98ba8805024eb79bf5df7a8225d784 P dcfdfa62edc34dab9525e2712d487582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.410267 14956 raft_consensus.cc:3037] T 6d98ba8805024eb79bf5df7a8225d784 P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:41.410810 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "830da5cb4972453cb9bf2145a0839ced" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:41.411114 15215 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9e9899666b15498791f62dab6d9dda16"
dest_uuid: "620596b893d84291b88b90dc9e317612"
from {username='slave'} at 127.0.0.1:57680
I20240503 02:00:41.411196 14956 raft_consensus.cc:2454] T 6d98ba8805024eb79bf5df7a8225d784 P dcfdfa62edc34dab9525e2712d487582 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 3.
I20240503 02:00:41.411363 15215 raft_consensus.cc:482] T 9e9899666b15498791f62dab6d9dda16 P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.411459 15215 raft_consensus.cc:3037] T 9e9899666b15498791f62dab6d9dda16 P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.409854 14954 consensus_queue.cc:260] T 830da5cb4972453cb9bf2145a0839ced P dcfdfa62edc34dab9525e2712d487582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.412319 14954 raft_consensus.cc:3037] T 830da5cb4972453cb9bf2145a0839ced P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.412609 15215 raft_consensus.cc:504] T 9e9899666b15498791f62dab6d9dda16 P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:41.413548 14954 raft_consensus.cc:2454] T 830da5cb4972453cb9bf2145a0839ced P dcfdfa62edc34dab9525e2712d487582 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 4.
I20240503 02:00:41.413719 15215 leader_election.cc:288] T 9e9899666b15498791f62dab6d9dda16 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 4 election: Requested vote from peers b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:41.414119 15349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e9899666b15498791f62dab6d9dda16" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 4 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.414247 15349 raft_consensus.cc:3037] T 9e9899666b15498791f62dab6d9dda16 P c05f7620bb884392931d3243a0b47a9e [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.414861 14956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b0c8b72e1884529a8a9b83d2a734d71" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 3 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:41.414858 14954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ceeb7b3c4146cd8a2e18b3e9ae987c" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:41.414985 14956 raft_consensus.cc:3037] T 6b0c8b72e1884529a8a9b83d2a734d71 P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:41.414984 14954 raft_consensus.cc:3032] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P dcfdfa62edc34dab9525e2712d487582 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:00:41.415052 14954 raft_consensus.cc:729] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P dcfdfa62edc34dab9525e2712d487582 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:41.415201 14954 consensus_queue.cc:260] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P dcfdfa62edc34dab9525e2712d487582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:41.415334 14954 raft_consensus.cc:3037] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:41.415653 15349 raft_consensus.cc:2396] T 9e9899666b15498791f62dab6d9dda16 P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 620596b893d84291b88b90dc9e317612 for term 4 because replica has last-logged OpId of term: 3 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 8.
I20240503 02:00:41.415930 14956 raft_consensus.cc:2454] T 6b0c8b72e1884529a8a9b83d2a734d71 P dcfdfa62edc34dab9525e2712d487582 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 3.
I20240503 02:00:41.416141 14954 raft_consensus.cc:2396] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P dcfdfa62edc34dab9525e2712d487582 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b602f6099cbb4758847e7e9550685372 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20240503 02:00:41.416281 14956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0989d98050744df6a1c699cb86ada505" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:41.416460 14956 raft_consensus.cc:2396] T 0989d98050744df6a1c699cb86ada505 P dcfdfa62edc34dab9525e2712d487582 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b602f6099cbb4758847e7e9550685372 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 02:00:41.416783 14956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d805ee04df39406ba30b3b3b9e349507" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 3 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:41.416870 14956 raft_consensus.cc:3032] T d805ee04df39406ba30b3b3b9e349507 P dcfdfa62edc34dab9525e2712d487582 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:00:41.416903 14956 raft_consensus.cc:729] T d805ee04df39406ba30b3b3b9e349507 P dcfdfa62edc34dab9525e2712d487582 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:41.417057 14956 consensus_queue.cc:260] T d805ee04df39406ba30b3b3b9e349507 P dcfdfa62edc34dab9525e2712d487582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.7, Last appended by leader: 7, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.417160 14956 raft_consensus.cc:3037] T d805ee04df39406ba30b3b3b9e349507 P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:41.417831 14956 raft_consensus.cc:2396] T d805ee04df39406ba30b3b3b9e349507 P dcfdfa62edc34dab9525e2712d487582 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b602f6099cbb4758847e7e9550685372 for term 3 because replica has last-logged OpId of term: 2 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 6.
I20240503 02:00:41.418187 14956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b757c430aaa42f382d174ca57f5fcfc" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 3 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:41.418187 14954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b225db529f14e7386f21f42bf31493b" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 3 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:41.418326 14954 raft_consensus.cc:3032] T 6b225db529f14e7386f21f42bf31493b P dcfdfa62edc34dab9525e2712d487582 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:00:41.418326 14956 raft_consensus.cc:3032] T 2b757c430aaa42f382d174ca57f5fcfc P dcfdfa62edc34dab9525e2712d487582 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:00:41.418386 14954 raft_consensus.cc:729] T 6b225db529f14e7386f21f42bf31493b P dcfdfa62edc34dab9525e2712d487582 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:41.418408 14956 raft_consensus.cc:729] T 2b757c430aaa42f382d174ca57f5fcfc P dcfdfa62edc34dab9525e2712d487582 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:41.418570 14956 consensus_queue.cc:260] T 2b757c430aaa42f382d174ca57f5fcfc P dcfdfa62edc34dab9525e2712d487582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.7, Last appended by leader: 7, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.418576 14954 consensus_queue.cc:260] T 6b225db529f14e7386f21f42bf31493b P dcfdfa62edc34dab9525e2712d487582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.8, Last appended by leader: 8, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.418709 14956 raft_consensus.cc:3037] T 2b757c430aaa42f382d174ca57f5fcfc P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:41.418711 14954 raft_consensus.cc:3037] T 6b225db529f14e7386f21f42bf31493b P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:41.419018 15153 leader_election.cc:302] T 830da5cb4972453cb9bf2145a0839ced P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:41.419229 17224 raft_consensus.cc:2781] T 830da5cb4972453cb9bf2145a0839ced P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 02:00:41.419288 17224 raft_consensus.cc:686] T 830da5cb4972453cb9bf2145a0839ced P 620596b893d84291b88b90dc9e317612 [term 4 LEADER]: Becoming Leader. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:41.419410 17224 consensus_queue.cc:237] T 830da5cb4972453cb9bf2145a0839ced P 620596b893d84291b88b90dc9e317612 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.419643 14956 raft_consensus.cc:2396] T 2b757c430aaa42f382d174ca57f5fcfc P dcfdfa62edc34dab9525e2712d487582 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b602f6099cbb4758847e7e9550685372 for term 3 because replica has last-logged OpId of term: 2 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 6.
I20240503 02:00:41.419715 14954 raft_consensus.cc:2396] T 6b225db529f14e7386f21f42bf31493b P dcfdfa62edc34dab9525e2712d487582 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b602f6099cbb4758847e7e9550685372 for term 3 because replica has last-logged OpId of term: 2 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 7.
I20240503 02:00:41.420078 14954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "454a24fd6d964d73ae3d5057f90fe36d" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:41.420195 14954 raft_consensus.cc:3032] T 454a24fd6d964d73ae3d5057f90fe36d P dcfdfa62edc34dab9525e2712d487582 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:00:41.420231 14954 raft_consensus.cc:729] T 454a24fd6d964d73ae3d5057f90fe36d P dcfdfa62edc34dab9525e2712d487582 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:41.420348 14956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85a361ce0254472b37526362d4edf0e" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 2 candidate_status { last_received { term: 1 index: 19 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:41.420462 14956 raft_consensus.cc:3032] T f85a361ce0254472b37526362d4edf0e P dcfdfa62edc34dab9525e2712d487582 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:00:41.420358 14954 consensus_queue.cc:260] T 454a24fd6d964d73ae3d5057f90fe36d P dcfdfa62edc34dab9525e2712d487582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.420513 14956 raft_consensus.cc:729] T f85a361ce0254472b37526362d4edf0e P dcfdfa62edc34dab9525e2712d487582 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:41.420598 14954 raft_consensus.cc:3037] T 454a24fd6d964d73ae3d5057f90fe36d P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:41.420666 14956 consensus_queue.cc:260] T f85a361ce0254472b37526362d4edf0e P dcfdfa62edc34dab9525e2712d487582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 1.19, Last appended by leader: 19, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:41.420792 14956 raft_consensus.cc:3037] T f85a361ce0254472b37526362d4edf0e P dcfdfa62edc34dab9525e2712d487582 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:41.421689 14954 raft_consensus.cc:2396] T 454a24fd6d964d73ae3d5057f90fe36d P dcfdfa62edc34dab9525e2712d487582 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b602f6099cbb4758847e7e9550685372 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 02:00:41.421737 14956 raft_consensus.cc:2454] T f85a361ce0254472b37526362d4edf0e P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 2.
I20240503 02:00:41.421867 15219 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "50c5abd860ae4b09bb289e9c7550c84e"
dest_uuid: "620596b893d84291b88b90dc9e317612"
from {username='slave'} at 127.0.0.1:57680
I20240503 02:00:41.422011 15219 raft_consensus.cc:482] T 50c5abd860ae4b09bb289e9c7550c84e P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.422068 15219 raft_consensus.cc:3037] T 50c5abd860ae4b09bb289e9c7550c84e P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:41.422437 14816 catalog_manager.cc:5484] T 830da5cb4972453cb9bf2145a0839ced P 620596b893d84291b88b90dc9e317612 reported cstate change: term changed from 3 to 4, leader changed from b602f6099cbb4758847e7e9550685372 (127.0.99.130) to 620596b893d84291b88b90dc9e317612 (127.0.99.131). New cstate: current_term: 4 leader_uuid: "620596b893d84291b88b90dc9e317612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:41.422971 15219 raft_consensus.cc:504] T 50c5abd860ae4b09bb289e9c7550c84e P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.423447 14954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e3bb299e61148fcb355641e0134b3df" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:41.423580 14954 raft_consensus.cc:3032] T 0e3bb299e61148fcb355641e0134b3df P dcfdfa62edc34dab9525e2712d487582 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:00:41.423619 14954 raft_consensus.cc:729] T 0e3bb299e61148fcb355641e0134b3df P dcfdfa62edc34dab9525e2712d487582 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:41.424120 14954 consensus_queue.cc:260] T 0e3bb299e61148fcb355641e0134b3df P dcfdfa62edc34dab9525e2712d487582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.424293 14954 raft_consensus.cc:3037] T 0e3bb299e61148fcb355641e0134b3df P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:41.425199 14954 raft_consensus.cc:2454] T 0e3bb299e61148fcb355641e0134b3df P dcfdfa62edc34dab9525e2712d487582 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 3.
I20240503 02:00:41.425984 14954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca0d7d3e4d59496c9a9a59cde24b9247" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 4 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:41.426164 14957 raft_consensus.cc:1226] T 7b4f203f4048490ba599353a4cd92748 P dcfdfa62edc34dab9525e2712d487582 [term 4 FOLLOWER]: Rejecting Update request from peer b602f6099cbb4758847e7e9550685372 for earlier term 3. Current term is 4. Ops: []
I20240503 02:00:41.426179 15219 leader_election.cc:288] T 50c5abd860ae4b09bb289e9c7550c84e P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 3 election: Requested vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:41.426177 14954 raft_consensus.cc:2396] T ca0d7d3e4d59496c9a9a59cde24b9247 P dcfdfa62edc34dab9525e2712d487582 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b602f6099cbb4758847e7e9550685372 for term 4 because replica has last-logged OpId of term: 3 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 7.
I20240503 02:00:41.426405 15349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50c5abd860ae4b09bb289e9c7550c84e" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.426532 15349 raft_consensus.cc:3037] T 50c5abd860ae4b09bb289e9c7550c84e P c05f7620bb884392931d3243a0b47a9e [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:41.426821 15351 raft_consensus.cc:1261] T ca0d7d3e4d59496c9a9a59cde24b9247 P c05f7620bb884392931d3243a0b47a9e [term 3 FOLLOWER]: Refusing update from remote peer dcfdfa62edc34dab9525e2712d487582: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240503 02:00:41.427330 15215 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7fc3330dfd0644f6a303659b960124f4"
dest_uuid: "620596b893d84291b88b90dc9e317612"
from {username='slave'} at 127.0.0.1:57680
I20240503 02:00:41.427457 15215 raft_consensus.cc:482] T 7fc3330dfd0644f6a303659b960124f4 P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.427510 15215 raft_consensus.cc:3037] T 7fc3330dfd0644f6a303659b960124f4 P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.427522 15349 raft_consensus.cc:2454] T 50c5abd860ae4b09bb289e9c7550c84e P c05f7620bb884392931d3243a0b47a9e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 3.
I20240503 02:00:41.427821 15155 leader_election.cc:302] T 50c5abd860ae4b09bb289e9c7550c84e P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, c05f7620bb884392931d3243a0b47a9e; no voters:
I20240503 02:00:41.427986 16264 raft_consensus.cc:2781] T 50c5abd860ae4b09bb289e9c7550c84e P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:00:41.428059 16264 raft_consensus.cc:686] T 50c5abd860ae4b09bb289e9c7550c84e P 620596b893d84291b88b90dc9e317612 [term 3 LEADER]: Becoming Leader. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:41.428185 16264 consensus_queue.cc:237] T 50c5abd860ae4b09bb289e9c7550c84e P 620596b893d84291b88b90dc9e317612 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.428488 15215 raft_consensus.cc:504] T 7fc3330dfd0644f6a303659b960124f4 P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.428798 15215 leader_election.cc:288] T 7fc3330dfd0644f6a303659b960124f4 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 4 election: Requested vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:41.429010 14956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50c5abd860ae4b09bb289e9c7550c84e" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:41.429119 14956 raft_consensus.cc:3032] T 50c5abd860ae4b09bb289e9c7550c84e P dcfdfa62edc34dab9525e2712d487582 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:00:41.429157 14956 raft_consensus.cc:729] T 50c5abd860ae4b09bb289e9c7550c84e P dcfdfa62edc34dab9525e2712d487582 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
W20240503 02:00:41.391826 15048 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34032: Illegal state: replica b602f6099cbb4758847e7e9550685372 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.429347 15089 raft_consensus.cc:3037] T de805ef7511d4ee9b9bbed67e0a4186c P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.393501 15086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4f203f4048490ba599353a4cd92748" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:41.429742 15086 raft_consensus.cc:3032] T 7b4f203f4048490ba599353a4cd92748 P b602f6099cbb4758847e7e9550685372 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 02:00:41.429793 15086 raft_consensus.cc:729] T 7b4f203f4048490ba599353a4cd92748 P b602f6099cbb4758847e7e9550685372 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.429919 15086 consensus_queue.cc:260] T 7b4f203f4048490ba599353a4cd92748 P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.7, Last appended by leader: 7, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.430022 15086 raft_consensus.cc:3037] T 7b4f203f4048490ba599353a4cd92748 P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.430337 15349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fc3330dfd0644f6a303659b960124f4" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.430454 15349 raft_consensus.cc:3037] T 7fc3330dfd0644f6a303659b960124f4 P c05f7620bb884392931d3243a0b47a9e [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.430541 14956 consensus_queue.cc:260] T 50c5abd860ae4b09bb289e9c7550c84e P dcfdfa62edc34dab9525e2712d487582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.430706 14956 raft_consensus.cc:3037] T 50c5abd860ae4b09bb289e9c7550c84e P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:41.430960 15086 raft_consensus.cc:2396] T 7b4f203f4048490ba599353a4cd92748 P b602f6099cbb4758847e7e9550685372 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 620596b893d84291b88b90dc9e317612 for term 4 because replica has last-logged OpId of term: 3 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 02:00:41.431424 15349 raft_consensus.cc:2454] T 7fc3330dfd0644f6a303659b960124f4 P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 4.
I20240503 02:00:41.431622 15154 leader_election.cc:302] T 7b4f203f4048490ba599353a4cd92748 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 620596b893d84291b88b90dc9e317612; no voters: b602f6099cbb4758847e7e9550685372, dcfdfa62edc34dab9525e2712d487582
I20240503 02:00:41.431797 15155 leader_election.cc:302] T 7fc3330dfd0644f6a303659b960124f4 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, c05f7620bb884392931d3243a0b47a9e; no voters:
I20240503 02:00:41.431890 15089 raft_consensus.cc:2396] T de805ef7511d4ee9b9bbed67e0a4186c P b602f6099cbb4758847e7e9550685372 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 620596b893d84291b88b90dc9e317612 for term 4 because replica has last-logged OpId of term: 3 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 6.
I20240503 02:00:41.431877 17224 raft_consensus.cc:2726] T 7b4f203f4048490ba599353a4cd92748 P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240503 02:00:41.385916 15024 leader_election.cc:302] T f85a361ce0254472b37526362d4edf0e P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, b602f6099cbb4758847e7e9550685372; no voters:
I20240503 02:00:41.431959 16264 raft_consensus.cc:2781] T 7fc3330dfd0644f6a303659b960124f4 P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 02:00:41.432035 16264 raft_consensus.cc:686] T 7fc3330dfd0644f6a303659b960124f4 P 620596b893d84291b88b90dc9e317612 [term 4 LEADER]: Becoming Leader. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:41.432098 14956 raft_consensus.cc:2454] T 50c5abd860ae4b09bb289e9c7550c84e P dcfdfa62edc34dab9525e2712d487582 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 3.
I20240503 02:00:41.432170 16264 consensus_queue.cc:237] T 7fc3330dfd0644f6a303659b960124f4 P 620596b893d84291b88b90dc9e317612 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 12, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.404103 15083 raft_consensus.cc:1226] T 454a24fd6d964d73ae3d5057f90fe36d P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Rejecting Update request from peer dcfdfa62edc34dab9525e2712d487582 for earlier term 2. Current term is 3. Ops: []
I20240503 02:00:41.406986 15088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "830da5cb4972453cb9bf2145a0839ced" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:41.432529 15088 raft_consensus.cc:3032] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 02:00:41.432519 15217 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5736c284a85540a1bbab2be6df672fc1"
dest_uuid: "620596b893d84291b88b90dc9e317612"
from {username='slave'} at 127.0.0.1:57680
I20240503 02:00:41.432590 15088 raft_consensus.cc:729] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.432736 15217 raft_consensus.cc:482] T 5736c284a85540a1bbab2be6df672fc1 P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.432718 15088 consensus_queue.cc:260] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.432819 15217 raft_consensus.cc:3037] T 5736c284a85540a1bbab2be6df672fc1 P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:41.432858 15088 raft_consensus.cc:3037] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.432745 15219 raft_consensus.cc:1226] T 6b225db529f14e7386f21f42bf31493b P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Rejecting Update request from peer dcfdfa62edc34dab9525e2712d487582 for earlier term 2. Current term is 3. Ops: []
I20240503 02:00:41.433182 14954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fc3330dfd0644f6a303659b960124f4" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:41.433303 14954 raft_consensus.cc:3032] T 7fc3330dfd0644f6a303659b960124f4 P dcfdfa62edc34dab9525e2712d487582 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 02:00:41.433344 14954 raft_consensus.cc:729] T 7fc3330dfd0644f6a303659b960124f4 P dcfdfa62edc34dab9525e2712d487582 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:41.433483 14954 consensus_queue.cc:260] T 7fc3330dfd0644f6a303659b960124f4 P dcfdfa62edc34dab9525e2712d487582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 12, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.433771 15217 raft_consensus.cc:504] T 5736c284a85540a1bbab2be6df672fc1 P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.433820 15088 raft_consensus.cc:2396] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 620596b893d84291b88b90dc9e317612 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20240503 02:00:41.409165 15084 raft_consensus.cc:1226] T d805ee04df39406ba30b3b3b9e349507 P b602f6099cbb4758847e7e9550685372 [term 3 LEADER]: Rejecting Update request from peer dcfdfa62edc34dab9525e2712d487582 for earlier term 2. Current term is 3. Ops: []
I20240503 02:00:41.413465 15087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e9899666b15498791f62dab6d9dda16" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 4 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:41.434343 15087 raft_consensus.cc:3032] T 9e9899666b15498791f62dab6d9dda16 P b602f6099cbb4758847e7e9550685372 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 02:00:41.434405 15087 raft_consensus.cc:729] T 9e9899666b15498791f62dab6d9dda16 P b602f6099cbb4758847e7e9550685372 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.434460 15217 leader_election.cc:288] T 5736c284a85540a1bbab2be6df672fc1 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 3 election: Requested vote from peers b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:41.434578 15087 consensus_queue.cc:260] T 9e9899666b15498791f62dab6d9dda16 P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 3.9, Last appended by leader: 9, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:41.434823 15087 raft_consensus.cc:3037] T 9e9899666b15498791f62dab6d9dda16 P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.434880 15349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5736c284a85540a1bbab2be6df672fc1" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.434996 15349 raft_consensus.cc:3037] T 5736c284a85540a1bbab2be6df672fc1 P c05f7620bb884392931d3243a0b47a9e [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:41.435724 15087 raft_consensus.cc:2396] T 9e9899666b15498791f62dab6d9dda16 P b602f6099cbb4758847e7e9550685372 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 620596b893d84291b88b90dc9e317612 for term 4 because replica has last-logged OpId of term: 3 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 8.
I20240503 02:00:41.427815 15022 leader_election.cc:302] T 6d98ba8805024eb79bf5df7a8225d784 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b602f6099cbb4758847e7e9550685372, dcfdfa62edc34dab9525e2712d487582; no voters: 620596b893d84291b88b90dc9e317612
I20240503 02:00:41.435936 15349 raft_consensus.cc:2454] T 5736c284a85540a1bbab2be6df672fc1 P c05f7620bb884392931d3243a0b47a9e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 3.
I20240503 02:00:41.436321 15155 leader_election.cc:302] T 5736c284a85540a1bbab2be6df672fc1 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, c05f7620bb884392931d3243a0b47a9e; no voters:
I20240503 02:00:41.436522 16264 raft_consensus.cc:2781] T 5736c284a85540a1bbab2be6df672fc1 P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:00:41.436601 16264 raft_consensus.cc:686] T 5736c284a85540a1bbab2be6df672fc1 P 620596b893d84291b88b90dc9e317612 [term 3 LEADER]: Becoming Leader. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:41.436728 16264 consensus_queue.cc:237] T 5736c284a85540a1bbab2be6df672fc1 P 620596b893d84291b88b90dc9e317612 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
W20240503 02:00:41.436867 15176 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57610: Illegal state: replica 620596b893d84291b88b90dc9e317612 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.436298 15022 leader_election.cc:302] T 0989d98050744df6a1c699cb86ada505 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b602f6099cbb4758847e7e9550685372; no voters: c05f7620bb884392931d3243a0b47a9e, dcfdfa62edc34dab9525e2712d487582
I20240503 02:00:41.438217 15154 leader_election.cc:302] T 9e9899666b15498791f62dab6d9dda16 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 620596b893d84291b88b90dc9e317612; no voters: b602f6099cbb4758847e7e9550685372, c05f7620bb884392931d3243a0b47a9e
I20240503 02:00:41.438410 17224 raft_consensus.cc:2726] T 9e9899666b15498791f62dab6d9dda16 P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240503 02:00:41.438680 15087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5736c284a85540a1bbab2be6df672fc1" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:41.438829 15087 raft_consensus.cc:3032] T 5736c284a85540a1bbab2be6df672fc1 P b602f6099cbb4758847e7e9550685372 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:00:41.438868 15087 raft_consensus.cc:729] T 5736c284a85540a1bbab2be6df672fc1 P b602f6099cbb4758847e7e9550685372 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.439002 15087 consensus_queue.cc:260] T 5736c284a85540a1bbab2be6df672fc1 P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.439052 15022 leader_election.cc:302] T 0e3bb299e61148fcb355641e0134b3df P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b602f6099cbb4758847e7e9550685372, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:41.439146 15087 raft_consensus.cc:3037] T 5736c284a85540a1bbab2be6df672fc1 P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:41.439213 17332 raft_consensus.cc:2781] T 6d98ba8805024eb79bf5df7a8225d784 P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:00:41.439303 17332 raft_consensus.cc:686] T 6d98ba8805024eb79bf5df7a8225d784 P b602f6099cbb4758847e7e9550685372 [term 3 LEADER]: Becoming Leader. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.439417 17332 consensus_queue.cc:237] T 6d98ba8805024eb79bf5df7a8225d784 P b602f6099cbb4758847e7e9550685372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.439994 17332 raft_consensus.cc:2726] T 0989d98050744df6a1c699cb86ada505 P b602f6099cbb4758847e7e9550685372 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240503 02:00:41.440096 17176 raft_consensus.cc:2781] T 0e3bb299e61148fcb355641e0134b3df P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:00:41.440117 17332 raft_consensus.cc:2781] T f85a361ce0254472b37526362d4edf0e P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:41.440163 17176 raft_consensus.cc:686] T 0e3bb299e61148fcb355641e0134b3df P b602f6099cbb4758847e7e9550685372 [term 3 LEADER]: Becoming Leader. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.440243 15087 raft_consensus.cc:2396] T 5736c284a85540a1bbab2be6df672fc1 P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 620596b893d84291b88b90dc9e317612 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20240503 02:00:41.440294 17176 consensus_queue.cc:237] T 0e3bb299e61148fcb355641e0134b3df P b602f6099cbb4758847e7e9550685372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.440614 15217 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f85a361ce0254472b37526362d4edf0e"
dest_uuid: "620596b893d84291b88b90dc9e317612"
from {username='slave'} at 127.0.0.1:57680
I20240503 02:00:41.440758 15217 raft_consensus.cc:482] T f85a361ce0254472b37526362d4edf0e P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.440819 15217 raft_consensus.cc:3037] T f85a361ce0254472b37526362d4edf0e P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:41.440166 17332 raft_consensus.cc:686] T f85a361ce0254472b37526362d4edf0e P b602f6099cbb4758847e7e9550685372 [term 2 LEADER]: Becoming Leader. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.441030 17332 consensus_queue.cc:237] T f85a361ce0254472b37526362d4edf0e P b602f6099cbb4758847e7e9550685372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 1.19, Last appended by leader: 19, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
W20240503 02:00:41.439900 15024 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.131:42437, user_credentials={real_user=slave}} blocked reactor thread for 53989us
I20240503 02:00:41.441753 15217 raft_consensus.cc:504] T f85a361ce0254472b37526362d4edf0e P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:41.442247 15349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca0d7d3e4d59496c9a9a59cde24b9247" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 4 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.442368 15349 raft_consensus.cc:3037] T ca0d7d3e4d59496c9a9a59cde24b9247 P c05f7620bb884392931d3243a0b47a9e [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.442459 15087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85a361ce0254472b37526362d4edf0e" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 3 candidate_status { last_received { term: 1 index: 17 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:41.442591 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5da91b7af68e4cf597dc89a017ee0197" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.442581 15087 raft_consensus.cc:3032] T f85a361ce0254472b37526362d4edf0e P b602f6099cbb4758847e7e9550685372 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:00:41.442672 15087 raft_consensus.cc:729] T f85a361ce0254472b37526362d4edf0e P b602f6099cbb4758847e7e9550685372 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.442703 15351 raft_consensus.cc:3032] T 5da91b7af68e4cf597dc89a017ee0197 P c05f7620bb884392931d3243a0b47a9e [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:00:41.442744 15351 raft_consensus.cc:729] T 5da91b7af68e4cf597dc89a017ee0197 P c05f7620bb884392931d3243a0b47a9e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:41.442636 14956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85a361ce0254472b37526362d4edf0e" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 3 candidate_status { last_received { term: 1 index: 17 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:41.442798 15087 consensus_queue.cc:260] T f85a361ce0254472b37526362d4edf0e P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 2.20, Last appended by leader: 20, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:41.442875 15351 consensus_queue.cc:260] T 5da91b7af68e4cf597dc89a017ee0197 P c05f7620bb884392931d3243a0b47a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.442973 15087 raft_consensus.cc:3037] T f85a361ce0254472b37526362d4edf0e P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:41.443035 15351 raft_consensus.cc:3037] T 5da91b7af68e4cf597dc89a017ee0197 P c05f7620bb884392931d3243a0b47a9e [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:41.443138 14956 raft_consensus.cc:3037] T f85a361ce0254472b37526362d4edf0e P dcfdfa62edc34dab9525e2712d487582 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:41.443401 15349 raft_consensus.cc:2454] T ca0d7d3e4d59496c9a9a59cde24b9247 P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 4.
I20240503 02:00:41.442095 15217 leader_election.cc:288] T f85a361ce0254472b37526362d4edf0e P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 3 election: Requested vote from peers b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207), dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817)
I20240503 02:00:41.443876 15351 raft_consensus.cc:2454] T 5da91b7af68e4cf597dc89a017ee0197 P c05f7620bb884392931d3243a0b47a9e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 2.
I20240503 02:00:41.443919 15087 raft_consensus.cc:2396] T f85a361ce0254472b37526362d4edf0e P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 620596b893d84291b88b90dc9e317612 for term 3 because replica has last-logged OpId of term: 2 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 17.
I20240503 02:00:41.444172 14956 raft_consensus.cc:2396] T f85a361ce0254472b37526362d4edf0e P dcfdfa62edc34dab9525e2712d487582 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 620596b893d84291b88b90dc9e317612 for term 3 because replica has last-logged OpId of term: 1 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 17.
I20240503 02:00:41.444725 15154 leader_election.cc:302] T f85a361ce0254472b37526362d4edf0e P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 620596b893d84291b88b90dc9e317612; no voters: b602f6099cbb4758847e7e9550685372, dcfdfa62edc34dab9525e2712d487582
I20240503 02:00:41.444919 17224 raft_consensus.cc:2726] T f85a361ce0254472b37526362d4edf0e P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240503 02:00:41.445144 15217 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eade118a36da42e3ac08b96a13893a5c"
dest_uuid: "620596b893d84291b88b90dc9e317612"
from {username='slave'} at 127.0.0.1:57680
I20240503 02:00:41.445437 15024 leader_election.cc:302] T 454a24fd6d964d73ae3d5057f90fe36d P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b602f6099cbb4758847e7e9550685372, c05f7620bb884392931d3243a0b47a9e; no voters: dcfdfa62edc34dab9525e2712d487582
I20240503 02:00:41.445560 15215 raft_consensus.cc:1226] T 9e9899666b15498791f62dab6d9dda16 P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Rejecting Update request from peer b602f6099cbb4758847e7e9550685372 for earlier term 3. Current term is 4. Ops: [3.9-3.9]
I20240503 02:00:41.445814 17332 raft_consensus.cc:2781] T 454a24fd6d964d73ae3d5057f90fe36d P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:00:41.445909 17332 raft_consensus.cc:686] T 454a24fd6d964d73ae3d5057f90fe36d P b602f6099cbb4758847e7e9550685372 [term 3 LEADER]: Becoming Leader. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.445897 15217 raft_consensus.cc:482] T eade118a36da42e3ac08b96a13893a5c P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.445969 15217 raft_consensus.cc:3037] T eade118a36da42e3ac08b96a13893a5c P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:41.446038 17332 consensus_queue.cc:237] T 454a24fd6d964d73ae3d5057f90fe36d P b602f6099cbb4758847e7e9550685372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.446681 15220 raft_consensus.cc:1226] T 7b4f203f4048490ba599353a4cd92748 P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Rejecting Update request from peer b602f6099cbb4758847e7e9550685372 for earlier term 3. Current term is 4. Ops: []
I20240503 02:00:41.446985 15217 raft_consensus.cc:504] T eade118a36da42e3ac08b96a13893a5c P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:41.447371 15217 leader_election.cc:288] T eade118a36da42e3ac08b96a13893a5c P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 3 election: Requested vote from peers b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
W20240503 02:00:41.447598 14915 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48020: Illegal state: replica dcfdfa62edc34dab9525e2712d487582 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.447927 15214 raft_consensus.cc:1226] T eade118a36da42e3ac08b96a13893a5c P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Rejecting Update request from peer b602f6099cbb4758847e7e9550685372 for earlier term 2. Current term is 3. Ops: [2.20-2.20]
I20240503 02:00:41.448442 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eade118a36da42e3ac08b96a13893a5c" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 3 candidate_status { last_received { term: 2 index: 19 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.448582 15351 raft_consensus.cc:3037] T eade118a36da42e3ac08b96a13893a5c P c05f7620bb884392931d3243a0b47a9e [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:41.448814 15024 leader_election.cc:302] T ca0d7d3e4d59496c9a9a59cde24b9247 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b602f6099cbb4758847e7e9550685372, c05f7620bb884392931d3243a0b47a9e; no voters: dcfdfa62edc34dab9525e2712d487582
I20240503 02:00:41.448989 17332 raft_consensus.cc:2781] T ca0d7d3e4d59496c9a9a59cde24b9247 P b602f6099cbb4758847e7e9550685372 [term 4 FOLLOWER]: Leader election won for term 4
W20240503 02:00:41.449012 14915 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47950: Illegal state: replica dcfdfa62edc34dab9525e2712d487582 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.449148 17332 raft_consensus.cc:686] T ca0d7d3e4d59496c9a9a59cde24b9247 P b602f6099cbb4758847e7e9550685372 [term 4 LEADER]: Becoming Leader. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.449291 17332 consensus_queue.cc:237] T ca0d7d3e4d59496c9a9a59cde24b9247 P b602f6099cbb4758847e7e9550685372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.449517 15087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eade118a36da42e3ac08b96a13893a5c" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 3 candidate_status { last_received { term: 2 index: 19 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:41.449654 15087 raft_consensus.cc:3032] T eade118a36da42e3ac08b96a13893a5c P b602f6099cbb4758847e7e9550685372 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:00:41.449702 15087 raft_consensus.cc:729] T eade118a36da42e3ac08b96a13893a5c P b602f6099cbb4758847e7e9550685372 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.449791 15351 raft_consensus.cc:2396] T eade118a36da42e3ac08b96a13893a5c P c05f7620bb884392931d3243a0b47a9e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 620596b893d84291b88b90dc9e317612 for term 3 because replica has last-logged OpId of term: 2 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 19.
I20240503 02:00:41.449802 15087 consensus_queue.cc:260] T eade118a36da42e3ac08b96a13893a5c P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 2.20, Last appended by leader: 20, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:41.449904 15087 raft_consensus.cc:3037] T eade118a36da42e3ac08b96a13893a5c P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:41.450114 15219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5da91b7af68e4cf597dc89a017ee0197" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:41.450338 15219 raft_consensus.cc:3037] T 5da91b7af68e4cf597dc89a017ee0197 P 620596b893d84291b88b90dc9e317612 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:41.451181 15214 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4cdcdde158054cdaabc1856a90fe1302"
dest_uuid: "620596b893d84291b88b90dc9e317612"
from {username='slave'} at 127.0.0.1:57680
I20240503 02:00:41.451308 15214 raft_consensus.cc:482] T 4cdcdde158054cdaabc1856a90fe1302 P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.451364 15214 raft_consensus.cc:3037] T 4cdcdde158054cdaabc1856a90fe1302 P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Advancing to term 4
W20240503 02:00:41.451519 15048 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33982: Illegal state: replica b602f6099cbb4758847e7e9550685372 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.451695 15219 raft_consensus.cc:2454] T 5da91b7af68e4cf597dc89a017ee0197 P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 2.
I20240503 02:00:41.452039 15024 leader_election.cc:302] T 5da91b7af68e4cf597dc89a017ee0197 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b602f6099cbb4758847e7e9550685372, c05f7620bb884392931d3243a0b47a9e; no voters:
I20240503 02:00:41.452265 17331 raft_consensus.cc:2781] T 5da91b7af68e4cf597dc89a017ee0197 P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:41.452307 15214 raft_consensus.cc:504] T 4cdcdde158054cdaabc1856a90fe1302 P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.452347 17331 raft_consensus.cc:686] T 5da91b7af68e4cf597dc89a017ee0197 P b602f6099cbb4758847e7e9550685372 [term 2 LEADER]: Becoming Leader. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.452526 17331 consensus_queue.cc:237] T 5da91b7af68e4cf597dc89a017ee0197 P b602f6099cbb4758847e7e9550685372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.452723 15214 leader_election.cc:288] T 4cdcdde158054cdaabc1856a90fe1302 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 4 election: Requested vote from peers b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:41.452724 15351 raft_consensus.cc:1226] T eade118a36da42e3ac08b96a13893a5c P c05f7620bb884392931d3243a0b47a9e [term 3 FOLLOWER]: Rejecting Update request from peer b602f6099cbb4758847e7e9550685372 for earlier term 2. Current term is 3. Ops: []
I20240503 02:00:41.453155 15084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcdde158054cdaabc1856a90fe1302" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:41.453263 15084 raft_consensus.cc:3032] T 4cdcdde158054cdaabc1856a90fe1302 P b602f6099cbb4758847e7e9550685372 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 02:00:41.453302 15084 raft_consensus.cc:729] T 4cdcdde158054cdaabc1856a90fe1302 P b602f6099cbb4758847e7e9550685372 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.453421 15084 consensus_queue.cc:260] T 4cdcdde158054cdaabc1856a90fe1302 P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.453578 15084 raft_consensus.cc:3037] T 4cdcdde158054cdaabc1856a90fe1302 P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.453666 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcdde158054cdaabc1856a90fe1302" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.453781 15351 raft_consensus.cc:3037] T 4cdcdde158054cdaabc1856a90fe1302 P c05f7620bb884392931d3243a0b47a9e [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.454496 14811 catalog_manager.cc:5484] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372 reported cstate change: term changed from 2 to 3, leader changed from c05f7620bb884392931d3243a0b47a9e (127.0.99.132) to b602f6099cbb4758847e7e9550685372 (127.0.99.130). New cstate: current_term: 3 leader_uuid: "b602f6099cbb4758847e7e9550685372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:41.454653 15084 raft_consensus.cc:2396] T 4cdcdde158054cdaabc1856a90fe1302 P b602f6099cbb4758847e7e9550685372 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 620596b893d84291b88b90dc9e317612 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 02:00:41.454664 15351 raft_consensus.cc:2454] T 4cdcdde158054cdaabc1856a90fe1302 P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 4.
I20240503 02:00:41.454761 14811 catalog_manager.cc:5484] T ebd7cdd284a74741b338e9f4778502b5 P b602f6099cbb4758847e7e9550685372 reported cstate change: term changed from 1 to 2, leader changed from c05f7620bb884392931d3243a0b47a9e (127.0.99.132) to b602f6099cbb4758847e7e9550685372 (127.0.99.130). New cstate: current_term: 2 leader_uuid: "b602f6099cbb4758847e7e9550685372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:41.454916 15214 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6d98ba8805024eb79bf5df7a8225d784"
dest_uuid: "620596b893d84291b88b90dc9e317612"
from {username='slave'} at 127.0.0.1:57680
I20240503 02:00:41.454934 15155 leader_election.cc:302] T 4cdcdde158054cdaabc1856a90fe1302 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, c05f7620bb884392931d3243a0b47a9e; no voters:
I20240503 02:00:41.454979 14811 catalog_manager.cc:5484] T 0e3bb299e61148fcb355641e0134b3df P b602f6099cbb4758847e7e9550685372 reported cstate change: term changed from 2 to 3, leader changed from dcfdfa62edc34dab9525e2712d487582 (127.0.99.129) to b602f6099cbb4758847e7e9550685372 (127.0.99.130). New cstate: current_term: 3 leader_uuid: "b602f6099cbb4758847e7e9550685372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:41.455067 15214 raft_consensus.cc:482] T 6d98ba8805024eb79bf5df7a8225d784 P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.455127 15214 raft_consensus.cc:3037] T 6d98ba8805024eb79bf5df7a8225d784 P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:41.455160 17224 raft_consensus.cc:2781] T 4cdcdde158054cdaabc1856a90fe1302 P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 02:00:41.455247 17224 raft_consensus.cc:686] T 4cdcdde158054cdaabc1856a90fe1302 P 620596b893d84291b88b90dc9e317612 [term 4 LEADER]: Becoming Leader. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:41.455175 14811 catalog_manager.cc:5484] T d805ee04df39406ba30b3b3b9e349507 P b602f6099cbb4758847e7e9550685372 reported cstate change: term changed from 2 to 3, leader changed from dcfdfa62edc34dab9525e2712d487582 (127.0.99.129) to b602f6099cbb4758847e7e9550685372 (127.0.99.130). New cstate: current_term: 3 leader_uuid: "b602f6099cbb4758847e7e9550685372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:41.455382 17224 consensus_queue.cc:237] T 4cdcdde158054cdaabc1856a90fe1302 P 620596b893d84291b88b90dc9e317612 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.455364 14811 catalog_manager.cc:5484] T 6d98ba8805024eb79bf5df7a8225d784 P b602f6099cbb4758847e7e9550685372 reported cstate change: term changed from 2 to 3, leader changed from dcfdfa62edc34dab9525e2712d487582 (127.0.99.129) to b602f6099cbb4758847e7e9550685372 (127.0.99.130). New cstate: current_term: 3 leader_uuid: "b602f6099cbb4758847e7e9550685372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:41.455538 14811 catalog_manager.cc:5484] T 6b225db529f14e7386f21f42bf31493b P b602f6099cbb4758847e7e9550685372 reported cstate change: term changed from 2 to 3, leader changed from dcfdfa62edc34dab9525e2712d487582 (127.0.99.129) to b602f6099cbb4758847e7e9550685372 (127.0.99.130). New cstate: current_term: 3 leader_uuid: "b602f6099cbb4758847e7e9550685372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:41.455706 14811 catalog_manager.cc:5484] T 2b757c430aaa42f382d174ca57f5fcfc P b602f6099cbb4758847e7e9550685372 reported cstate change: term changed from 2 to 3, leader changed from dcfdfa62edc34dab9525e2712d487582 (127.0.99.129) to b602f6099cbb4758847e7e9550685372 (127.0.99.130). New cstate: current_term: 3 leader_uuid: "b602f6099cbb4758847e7e9550685372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:41.456233 15214 raft_consensus.cc:504] T 6d98ba8805024eb79bf5df7a8225d784 P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.456365 14816 catalog_manager.cc:5484] T 50c5abd860ae4b09bb289e9c7550c84e P 620596b893d84291b88b90dc9e317612 reported cstate change: term changed from 2 to 3, leader changed from dcfdfa62edc34dab9525e2712d487582 (127.0.99.129) to 620596b893d84291b88b90dc9e317612 (127.0.99.131). New cstate: current_term: 3 leader_uuid: "620596b893d84291b88b90dc9e317612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:41.457283 14954 raft_consensus.cc:3037] T 7fc3330dfd0644f6a303659b960124f4 P dcfdfa62edc34dab9525e2712d487582 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.458317 14954 raft_consensus.cc:2454] T 7fc3330dfd0644f6a303659b960124f4 P dcfdfa62edc34dab9525e2712d487582 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 4.
I20240503 02:00:41.459223 15214 leader_election.cc:288] T 6d98ba8805024eb79bf5df7a8225d784 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 3 election: Requested vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
W20240503 02:00:41.459386 15176 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57618: Illegal state: replica 620596b893d84291b88b90dc9e317612 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.459781 15084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d98ba8805024eb79bf5df7a8225d784" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:41.459956 15084 raft_consensus.cc:2379] T 6d98ba8805024eb79bf5df7a8225d784 P b602f6099cbb4758847e7e9550685372 [term 3 LEADER]: Leader election vote request: Denying vote to candidate 620596b893d84291b88b90dc9e317612 in current term 3: Already voted for candidate b602f6099cbb4758847e7e9550685372 in this term.
I20240503 02:00:41.460244 14954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d98ba8805024eb79bf5df7a8225d784" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:41.460433 14954 raft_consensus.cc:2379] T 6d98ba8805024eb79bf5df7a8225d784 P dcfdfa62edc34dab9525e2712d487582 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 620596b893d84291b88b90dc9e317612 in current term 3: Already voted for candidate b602f6099cbb4758847e7e9550685372 in this term.
I20240503 02:00:41.460353 15214 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "42ceeb7b3c4146cd8a2e18b3e9ae987c"
dest_uuid: "620596b893d84291b88b90dc9e317612"
from {username='slave'} at 127.0.0.1:57680
I20240503 02:00:41.460520 15214 raft_consensus.cc:482] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.460578 15214 raft_consensus.cc:3037] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.460707 15153 leader_election.cc:302] T 6d98ba8805024eb79bf5df7a8225d784 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 620596b893d84291b88b90dc9e317612; no voters: b602f6099cbb4758847e7e9550685372, dcfdfa62edc34dab9525e2712d487582
I20240503 02:00:41.460922 16264 raft_consensus.cc:2726] T 6d98ba8805024eb79bf5df7a8225d784 P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240503 02:00:41.461557 15219 raft_consensus.cc:1261] T 6798a9ed0cf441aca7b8ae1b05c001f3 P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Refusing update from remote peer dcfdfa62edc34dab9525e2712d487582: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 02:00:41.461629 15214 raft_consensus.cc:504] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:41.461864 14815 catalog_manager.cc:5484] T 7fc3330dfd0644f6a303659b960124f4 P 620596b893d84291b88b90dc9e317612 reported cstate change: term changed from 3 to 4, leader changed from dcfdfa62edc34dab9525e2712d487582 (127.0.99.129) to 620596b893d84291b88b90dc9e317612 (127.0.99.131). New cstate: current_term: 4 leader_uuid: "620596b893d84291b88b90dc9e317612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:41.462038 15214 leader_election.cc:288] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 4 election: Requested vote from peers b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207), dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817)
I20240503 02:00:41.462175 15087 raft_consensus.cc:2396] T eade118a36da42e3ac08b96a13893a5c P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 620596b893d84291b88b90dc9e317612 for term 3 because replica has last-logged OpId of term: 2 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 19.
I20240503 02:00:41.462165 14815 catalog_manager.cc:5484] T 4cdcdde158054cdaabc1856a90fe1302 P 620596b893d84291b88b90dc9e317612 reported cstate change: term changed from 3 to 4, leader changed from b602f6099cbb4758847e7e9550685372 (127.0.99.130) to 620596b893d84291b88b90dc9e317612 (127.0.99.131). New cstate: current_term: 4 leader_uuid: "620596b893d84291b88b90dc9e317612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:41.462388 14815 catalog_manager.cc:5484] T 5736c284a85540a1bbab2be6df672fc1 P 620596b893d84291b88b90dc9e317612 reported cstate change: term changed from 2 to 3, leader changed from b602f6099cbb4758847e7e9550685372 (127.0.99.130) to 620596b893d84291b88b90dc9e317612 (127.0.99.131). New cstate: current_term: 3 leader_uuid: "620596b893d84291b88b90dc9e317612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:41.462478 15154 leader_election.cc:302] T eade118a36da42e3ac08b96a13893a5c P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 620596b893d84291b88b90dc9e317612; no voters: b602f6099cbb4758847e7e9550685372, c05f7620bb884392931d3243a0b47a9e
I20240503 02:00:41.462540 15084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ceeb7b3c4146cd8a2e18b3e9ae987c" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:41.462662 15084 raft_consensus.cc:3032] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P b602f6099cbb4758847e7e9550685372 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 02:00:41.462703 15084 raft_consensus.cc:729] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P b602f6099cbb4758847e7e9550685372 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.462761 14954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ceeb7b3c4146cd8a2e18b3e9ae987c" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:41.462800 16264 raft_consensus.cc:2726] T eade118a36da42e3ac08b96a13893a5c P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240503 02:00:41.462863 14954 raft_consensus.cc:3037] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P dcfdfa62edc34dab9525e2712d487582 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.462852 15084 consensus_queue.cc:260] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:41.462985 15084 raft_consensus.cc:3037] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.463869 15084 raft_consensus.cc:2396] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P b602f6099cbb4758847e7e9550685372 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 620596b893d84291b88b90dc9e317612 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20240503 02:00:41.463869 14954 raft_consensus.cc:2396] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P dcfdfa62edc34dab9525e2712d487582 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 620596b893d84291b88b90dc9e317612 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20240503 02:00:41.464356 15214 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6b0c8b72e1884529a8a9b83d2a734d71"
dest_uuid: "620596b893d84291b88b90dc9e317612"
from {username='slave'} at 127.0.0.1:57680
I20240503 02:00:41.464419 15153 leader_election.cc:302] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 620596b893d84291b88b90dc9e317612; no voters: b602f6099cbb4758847e7e9550685372, dcfdfa62edc34dab9525e2712d487582
I20240503 02:00:41.464514 15214 raft_consensus.cc:482] T 6b0c8b72e1884529a8a9b83d2a734d71 P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.464566 15214 raft_consensus.cc:3037] T 6b0c8b72e1884529a8a9b83d2a734d71 P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.465492 15214 raft_consensus.cc:504] T 6b0c8b72e1884529a8a9b83d2a734d71 P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.466012 16264 raft_consensus.cc:2726] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240503 02:00:41.466269 15214 leader_election.cc:288] T 6b0c8b72e1884529a8a9b83d2a734d71 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 4 election: Requested vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
I20240503 02:00:41.466375 15084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b0c8b72e1884529a8a9b83d2a734d71" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 4 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:41.466495 15084 raft_consensus.cc:3032] T 6b0c8b72e1884529a8a9b83d2a734d71 P b602f6099cbb4758847e7e9550685372 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 02:00:41.466554 15084 raft_consensus.cc:729] T 6b0c8b72e1884529a8a9b83d2a734d71 P b602f6099cbb4758847e7e9550685372 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.466728 15084 consensus_queue.cc:260] T 6b0c8b72e1884529a8a9b83d2a734d71 P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.466877 15084 raft_consensus.cc:3037] T 6b0c8b72e1884529a8a9b83d2a734d71 P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.466979 15214 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9a74cca2988a4f219f6fd9b07ade42d8"
dest_uuid: "620596b893d84291b88b90dc9e317612"
from {username='slave'} at 127.0.0.1:57680
I20240503 02:00:41.467108 15214 raft_consensus.cc:482] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.467154 15214 raft_consensus.cc:3037] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Advancing to term 4
W20240503 02:00:41.467357 14915 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47996: Illegal state: replica dcfdfa62edc34dab9525e2712d487582 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.467937 15084 raft_consensus.cc:2396] T 6b0c8b72e1884529a8a9b83d2a734d71 P b602f6099cbb4758847e7e9550685372 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 620596b893d84291b88b90dc9e317612 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 5.
I20240503 02:00:41.468021 15214 raft_consensus.cc:504] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.468173 14954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b0c8b72e1884529a8a9b83d2a734d71" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 4 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:41.468292 14954 raft_consensus.cc:3037] T 6b0c8b72e1884529a8a9b83d2a734d71 P dcfdfa62edc34dab9525e2712d487582 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.468614 15214 leader_election.cc:288] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 4 election: Requested vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
W20240503 02:00:41.468899 15311 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50950: Illegal state: replica c05f7620bb884392931d3243a0b47a9e is not leader of this config: current role FOLLOWER
I20240503 02:00:41.469324 15214 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c74522fbb5c64d27b5c95f05896b0716"
dest_uuid: "620596b893d84291b88b90dc9e317612"
from {username='slave'} at 127.0.0.1:57680
I20240503 02:00:41.469450 15214 raft_consensus.cc:482] T c74522fbb5c64d27b5c95f05896b0716 P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.469331 14810 catalog_manager.cc:5484] T 5da91b7af68e4cf597dc89a017ee0197 P b602f6099cbb4758847e7e9550685372 reported cstate change: term changed from 1 to 2, leader changed from c05f7620bb884392931d3243a0b47a9e (127.0.99.132) to b602f6099cbb4758847e7e9550685372 (127.0.99.130). New cstate: current_term: 2 leader_uuid: "b602f6099cbb4758847e7e9550685372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:41.469511 15214 raft_consensus.cc:3037] T c74522fbb5c64d27b5c95f05896b0716 P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.469486 14956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a74cca2988a4f219f6fd9b07ade42d8" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 4 candidate_status { last_received { term: 3 index: 7 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:41.469633 14956 raft_consensus.cc:3032] T 9a74cca2988a4f219f6fd9b07ade42d8 P dcfdfa62edc34dab9525e2712d487582 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 02:00:41.469686 14956 raft_consensus.cc:729] T 9a74cca2988a4f219f6fd9b07ade42d8 P dcfdfa62edc34dab9525e2712d487582 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:41.469666 14810 catalog_manager.cc:5484] T 454a24fd6d964d73ae3d5057f90fe36d P b602f6099cbb4758847e7e9550685372 reported cstate change: term changed from 2 to 3, leader changed from dcfdfa62edc34dab9525e2712d487582 (127.0.99.129) to b602f6099cbb4758847e7e9550685372 (127.0.99.130). New cstate: current_term: 3 leader_uuid: "b602f6099cbb4758847e7e9550685372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:41.469843 14956 consensus_queue.cc:260] T 9a74cca2988a4f219f6fd9b07ade42d8 P dcfdfa62edc34dab9525e2712d487582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 3.7, Last appended by leader: 7, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.469869 14810 catalog_manager.cc:5484] T ca0d7d3e4d59496c9a9a59cde24b9247 P b602f6099cbb4758847e7e9550685372 reported cstate change: term changed from 3 to 4, leader changed from dcfdfa62edc34dab9525e2712d487582 (127.0.99.129) to b602f6099cbb4758847e7e9550685372 (127.0.99.130). New cstate: current_term: 4 leader_uuid: "b602f6099cbb4758847e7e9550685372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:41.470029 14956 raft_consensus.cc:3037] T 9a74cca2988a4f219f6fd9b07ade42d8 P dcfdfa62edc34dab9525e2712d487582 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.470526 14954 raft_consensus.cc:2454] T 6b0c8b72e1884529a8a9b83d2a734d71 P dcfdfa62edc34dab9525e2712d487582 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 4.
I20240503 02:00:41.470517 15214 raft_consensus.cc:504] T c74522fbb5c64d27b5c95f05896b0716 P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.470860 15153 leader_election.cc:302] T 6b0c8b72e1884529a8a9b83d2a734d71 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 620596b893d84291b88b90dc9e317612, dcfdfa62edc34dab9525e2712d487582; no voters: b602f6099cbb4758847e7e9550685372
I20240503 02:00:41.470952 14956 raft_consensus.cc:2454] T 9a74cca2988a4f219f6fd9b07ade42d8 P dcfdfa62edc34dab9525e2712d487582 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 4.
I20240503 02:00:41.471118 16264 raft_consensus.cc:2781] T 6b0c8b72e1884529a8a9b83d2a734d71 P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 02:00:41.471194 15214 leader_election.cc:288] T c74522fbb5c64d27b5c95f05896b0716 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 4 election: Requested vote from peers c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
I20240503 02:00:41.471194 16264 raft_consensus.cc:686] T 6b0c8b72e1884529a8a9b83d2a734d71 P 620596b893d84291b88b90dc9e317612 [term 4 LEADER]: Becoming Leader. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:41.471300 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a74cca2988a4f219f6fd9b07ade42d8" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 4 candidate_status { last_received { term: 3 index: 7 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.471410 15351 raft_consensus.cc:3037] T 9a74cca2988a4f219f6fd9b07ade42d8 P c05f7620bb884392931d3243a0b47a9e [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.471387 15349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c74522fbb5c64d27b5c95f05896b0716" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 4 candidate_status { last_received { term: 2 index: 25 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.471385 16264 consensus_queue.cc:237] T 6b0c8b72e1884529a8a9b83d2a734d71 P 620596b893d84291b88b90dc9e317612 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.471506 15349 raft_consensus.cc:3037] T c74522fbb5c64d27b5c95f05896b0716 P c05f7620bb884392931d3243a0b47a9e [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.472474 15349 raft_consensus.cc:2454] T c74522fbb5c64d27b5c95f05896b0716 P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 4.
I20240503 02:00:41.472568 15153 leader_election.cc:302] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:41.472589 15351 raft_consensus.cc:2454] T 9a74cca2988a4f219f6fd9b07ade42d8 P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 4.
I20240503 02:00:41.472736 16264 raft_consensus.cc:2781] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 02:00:41.472803 16264 raft_consensus.cc:686] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612 [term 4 LEADER]: Becoming Leader. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:41.472836 15084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c74522fbb5c64d27b5c95f05896b0716" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 4 candidate_status { last_received { term: 2 index: 25 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:41.472913 16264 consensus_queue.cc:237] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 3.7, Last appended by leader: 7, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.472946 15084 raft_consensus.cc:3032] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 02:00:41.473001 15084 raft_consensus.cc:729] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.473171 15084 consensus_queue.cc:260] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 3.26, Last appended by leader: 26, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.473325 15084 raft_consensus.cc:3037] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.473919 15155 leader_election.cc:302] T c74522fbb5c64d27b5c95f05896b0716 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, c05f7620bb884392931d3243a0b47a9e; no voters:
I20240503 02:00:41.474279 17224 raft_consensus.cc:2781] T c74522fbb5c64d27b5c95f05896b0716 P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 02:00:41.474349 15084 raft_consensus.cc:2396] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 620596b893d84291b88b90dc9e317612 for term 4 because replica has last-logged OpId of term: 3 index: 26, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 25.
I20240503 02:00:41.474359 17224 raft_consensus.cc:686] T c74522fbb5c64d27b5c95f05896b0716 P 620596b893d84291b88b90dc9e317612 [term 4 LEADER]: Becoming Leader. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:41.474469 17224 consensus_queue.cc:237] T c74522fbb5c64d27b5c95f05896b0716 P 620596b893d84291b88b90dc9e317612 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 2.25, Last appended by leader: 25, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.477125 15214 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2b757c430aaa42f382d174ca57f5fcfc"
dest_uuid: "620596b893d84291b88b90dc9e317612"
from {username='slave'} at 127.0.0.1:57680
I20240503 02:00:41.477411 15214 raft_consensus.cc:482] T 2b757c430aaa42f382d174ca57f5fcfc P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.477538 15214 raft_consensus.cc:3037] T 2b757c430aaa42f382d174ca57f5fcfc P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.478726 15214 raft_consensus.cc:504] T 2b757c430aaa42f382d174ca57f5fcfc P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.479091 14815 catalog_manager.cc:5484] T c74522fbb5c64d27b5c95f05896b0716 P 620596b893d84291b88b90dc9e317612 reported cstate change: term changed from 3 to 4, leader changed from b602f6099cbb4758847e7e9550685372 (127.0.99.130) to 620596b893d84291b88b90dc9e317612 (127.0.99.131). New cstate: current_term: 4 leader_uuid: "620596b893d84291b88b90dc9e317612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:41.479211 15214 leader_election.cc:288] T 2b757c430aaa42f382d174ca57f5fcfc P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 4 election: Requested vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
I20240503 02:00:41.479370 14815 catalog_manager.cc:5484] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612 reported cstate change: term changed from 3 to 4, leader changed from dcfdfa62edc34dab9525e2712d487582 (127.0.99.129) to 620596b893d84291b88b90dc9e317612 (127.0.99.131). New cstate: current_term: 4 leader_uuid: "620596b893d84291b88b90dc9e317612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:41.479595 14956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b757c430aaa42f382d174ca57f5fcfc" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 4 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:41.479591 14815 catalog_manager.cc:5484] T 6b0c8b72e1884529a8a9b83d2a734d71 P 620596b893d84291b88b90dc9e317612 reported cstate change: term changed from 3 to 4, leader changed from b602f6099cbb4758847e7e9550685372 (127.0.99.130) to 620596b893d84291b88b90dc9e317612 (127.0.99.131). New cstate: current_term: 4 leader_uuid: "620596b893d84291b88b90dc9e317612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:41.479782 14956 raft_consensus.cc:3037] T 2b757c430aaa42f382d174ca57f5fcfc P dcfdfa62edc34dab9525e2712d487582 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.480329 15214 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5d448658ed3b460fb1725a26a7c743e8"
dest_uuid: "620596b893d84291b88b90dc9e317612"
from {username='slave'} at 127.0.0.1:57680
I20240503 02:00:41.480633 15214 raft_consensus.cc:482] T 5d448658ed3b460fb1725a26a7c743e8 P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.480794 15214 raft_consensus.cc:3037] T 5d448658ed3b460fb1725a26a7c743e8 P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.481011 14956 raft_consensus.cc:2396] T 2b757c430aaa42f382d174ca57f5fcfc P dcfdfa62edc34dab9525e2712d487582 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 620596b893d84291b88b90dc9e317612 for term 4 because replica has last-logged OpId of term: 2 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 6.
I20240503 02:00:41.481113 15084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b757c430aaa42f382d174ca57f5fcfc" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 4 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:41.481254 15084 raft_consensus.cc:3032] T 2b757c430aaa42f382d174ca57f5fcfc P b602f6099cbb4758847e7e9550685372 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 02:00:41.481302 15084 raft_consensus.cc:729] T 2b757c430aaa42f382d174ca57f5fcfc P b602f6099cbb4758847e7e9550685372 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.481487 15084 consensus_queue.cc:260] T 2b757c430aaa42f382d174ca57f5fcfc P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.7, Last appended by leader: 7, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.481643 15084 raft_consensus.cc:3037] T 2b757c430aaa42f382d174ca57f5fcfc P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.482131 15214 raft_consensus.cc:504] T 5d448658ed3b460fb1725a26a7c743e8 P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.482517 15214 leader_election.cc:288] T 5d448658ed3b460fb1725a26a7c743e8 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 4 election: Requested vote from peers c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009), dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817)
I20240503 02:00:41.482641 15084 raft_consensus.cc:2396] T 2b757c430aaa42f382d174ca57f5fcfc P b602f6099cbb4758847e7e9550685372 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 620596b893d84291b88b90dc9e317612 for term 4 because replica has last-logged OpId of term: 3 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 6.
I20240503 02:00:41.482966 15154 leader_election.cc:302] T 2b757c430aaa42f382d174ca57f5fcfc P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 620596b893d84291b88b90dc9e317612; no voters: b602f6099cbb4758847e7e9550685372, dcfdfa62edc34dab9525e2712d487582
I20240503 02:00:41.483186 16264 raft_consensus.cc:2726] T 2b757c430aaa42f382d174ca57f5fcfc P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240503 02:00:41.483196 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d448658ed3b460fb1725a26a7c743e8" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.483366 14956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d448658ed3b460fb1725a26a7c743e8" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:41.483485 14956 raft_consensus.cc:3037] T 5d448658ed3b460fb1725a26a7c743e8 P dcfdfa62edc34dab9525e2712d487582 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.483755 15351 raft_consensus.cc:3037] T 5d448658ed3b460fb1725a26a7c743e8 P c05f7620bb884392931d3243a0b47a9e [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.484488 14956 raft_consensus.cc:2454] T 5d448658ed3b460fb1725a26a7c743e8 P dcfdfa62edc34dab9525e2712d487582 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 4.
I20240503 02:00:41.484717 15351 raft_consensus.cc:2454] T 5d448658ed3b460fb1725a26a7c743e8 P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 4.
I20240503 02:00:41.484820 15153 leader_election.cc:302] T 5d448658ed3b460fb1725a26a7c743e8 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:41.484925 15219 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6798a9ed0cf441aca7b8ae1b05c001f3"
dest_uuid: "620596b893d84291b88b90dc9e317612"
from {username='slave'} at 127.0.0.1:57680
I20240503 02:00:41.485003 16264 raft_consensus.cc:2781] T 5d448658ed3b460fb1725a26a7c743e8 P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 02:00:41.485050 15219 raft_consensus.cc:482] T 6798a9ed0cf441aca7b8ae1b05c001f3 P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.485091 15219 raft_consensus.cc:3037] T 6798a9ed0cf441aca7b8ae1b05c001f3 P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.485100 16264 raft_consensus.cc:686] T 5d448658ed3b460fb1725a26a7c743e8 P 620596b893d84291b88b90dc9e317612 [term 4 LEADER]: Becoming Leader. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:41.485236 16264 consensus_queue.cc:237] T 5d448658ed3b460fb1725a26a7c743e8 P 620596b893d84291b88b90dc9e317612 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.486039 15219 raft_consensus.cc:504] T 6798a9ed0cf441aca7b8ae1b05c001f3 P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.486371 15219 leader_election.cc:288] T 6798a9ed0cf441aca7b8ae1b05c001f3 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 4 election: Requested vote from peers c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009), dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817)
I20240503 02:00:41.486750 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6798a9ed0cf441aca7b8ae1b05c001f3" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.487143 15351 raft_consensus.cc:3037] T 6798a9ed0cf441aca7b8ae1b05c001f3 P c05f7620bb884392931d3243a0b47a9e [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.487071 15214 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ebd7cdd284a74741b338e9f4778502b5"
dest_uuid: "620596b893d84291b88b90dc9e317612"
from {username='slave'} at 127.0.0.1:57680
I20240503 02:00:41.487242 15214 raft_consensus.cc:482] T ebd7cdd284a74741b338e9f4778502b5 P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.487228 14815 catalog_manager.cc:5484] T 5d448658ed3b460fb1725a26a7c743e8 P 620596b893d84291b88b90dc9e317612 reported cstate change: term changed from 2 to 4, leader changed from c05f7620bb884392931d3243a0b47a9e (127.0.99.132) to 620596b893d84291b88b90dc9e317612 (127.0.99.131). New cstate: current_term: 4 leader_uuid: "620596b893d84291b88b90dc9e317612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:41.487319 15214 raft_consensus.cc:3037] T ebd7cdd284a74741b338e9f4778502b5 P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Advancing to term 3
W20240503 02:00:41.487712 15048 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34032: Illegal state: replica b602f6099cbb4758847e7e9550685372 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.488405 15214 raft_consensus.cc:504] T ebd7cdd284a74741b338e9f4778502b5 P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.488809 15214 leader_election.cc:288] T ebd7cdd284a74741b338e9f4778502b5 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 3 election: Requested vote from peers b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:41.486965 14956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6798a9ed0cf441aca7b8ae1b05c001f3" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:41.489034 14956 raft_consensus.cc:3032] T 6798a9ed0cf441aca7b8ae1b05c001f3 P dcfdfa62edc34dab9525e2712d487582 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 02:00:41.489090 14956 raft_consensus.cc:729] T 6798a9ed0cf441aca7b8ae1b05c001f3 P dcfdfa62edc34dab9525e2712d487582 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:41.489364 14956 consensus_queue.cc:260] T 6798a9ed0cf441aca7b8ae1b05c001f3 P dcfdfa62edc34dab9525e2712d487582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.489413 15349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebd7cdd284a74741b338e9f4778502b5" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.489526 15349 raft_consensus.cc:3037] T ebd7cdd284a74741b338e9f4778502b5 P c05f7620bb884392931d3243a0b47a9e [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:41.489544 14956 raft_consensus.cc:3037] T 6798a9ed0cf441aca7b8ae1b05c001f3 P dcfdfa62edc34dab9525e2712d487582 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.489579 15219 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6b225db529f14e7386f21f42bf31493b"
dest_uuid: "620596b893d84291b88b90dc9e317612"
from {username='slave'} at 127.0.0.1:57680
I20240503 02:00:41.489679 15219 raft_consensus.cc:482] T 6b225db529f14e7386f21f42bf31493b P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.489735 15219 raft_consensus.cc:3037] T 6b225db529f14e7386f21f42bf31493b P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.490459 16998 heartbeater.cc:503] Master 127.0.99.190:38675 was elected leader, sending a full tablet report...
I20240503 02:00:41.490558 15219 raft_consensus.cc:504] T 6b225db529f14e7386f21f42bf31493b P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
W20240503 02:00:41.491107 14915 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47950: Illegal state: replica dcfdfa62edc34dab9525e2712d487582 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.491240 14954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b225db529f14e7386f21f42bf31493b" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 4 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:41.491348 14954 raft_consensus.cc:3037] T 6b225db529f14e7386f21f42bf31493b P dcfdfa62edc34dab9525e2712d487582 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.491717 14956 raft_consensus.cc:2396] T 6798a9ed0cf441aca7b8ae1b05c001f3 P dcfdfa62edc34dab9525e2712d487582 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 620596b893d84291b88b90dc9e317612 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 02:00:41.492250 14954 raft_consensus.cc:2396] T 6b225db529f14e7386f21f42bf31493b P dcfdfa62edc34dab9525e2712d487582 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 620596b893d84291b88b90dc9e317612 for term 4 because replica has last-logged OpId of term: 2 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 7.
I20240503 02:00:41.492863 15087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b225db529f14e7386f21f42bf31493b" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 4 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:41.492990 15087 raft_consensus.cc:3032] T 6b225db529f14e7386f21f42bf31493b P b602f6099cbb4758847e7e9550685372 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 02:00:41.493024 15087 raft_consensus.cc:729] T 6b225db529f14e7386f21f42bf31493b P b602f6099cbb4758847e7e9550685372 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.493194 15087 consensus_queue.cc:260] T 6b225db529f14e7386f21f42bf31493b P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 3.8, Last appended by leader: 8, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.493317 15087 raft_consensus.cc:3037] T 6b225db529f14e7386f21f42bf31493b P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.494295 15087 raft_consensus.cc:2396] T 6b225db529f14e7386f21f42bf31493b P b602f6099cbb4758847e7e9550685372 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 620596b893d84291b88b90dc9e317612 for term 4 because replica has last-logged OpId of term: 3 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 7.
I20240503 02:00:41.494674 15154 leader_election.cc:302] T 6b225db529f14e7386f21f42bf31493b P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 620596b893d84291b88b90dc9e317612; no voters: b602f6099cbb4758847e7e9550685372, dcfdfa62edc34dab9525e2712d487582
I20240503 02:00:41.494814 15351 raft_consensus.cc:2396] T 6798a9ed0cf441aca7b8ae1b05c001f3 P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 620596b893d84291b88b90dc9e317612 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 02:00:41.494885 16264 raft_consensus.cc:2726] T 6b225db529f14e7386f21f42bf31493b P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240503 02:00:41.495117 15155 leader_election.cc:302] T 6798a9ed0cf441aca7b8ae1b05c001f3 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 620596b893d84291b88b90dc9e317612; no voters: c05f7620bb884392931d3243a0b47a9e, dcfdfa62edc34dab9525e2712d487582
I20240503 02:00:41.495128 15219 leader_election.cc:288] T 6b225db529f14e7386f21f42bf31493b P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 4 election: Requested vote from peers b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207), dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817)
I20240503 02:00:41.495275 15084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebd7cdd284a74741b338e9f4778502b5" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:41.495357 16264 raft_consensus.cc:2726] T 6798a9ed0cf441aca7b8ae1b05c001f3 P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240503 02:00:41.495399 15084 raft_consensus.cc:3032] T ebd7cdd284a74741b338e9f4778502b5 P b602f6099cbb4758847e7e9550685372 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:00:41.495450 15084 raft_consensus.cc:729] T ebd7cdd284a74741b338e9f4778502b5 P b602f6099cbb4758847e7e9550685372 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.495607 15349 raft_consensus.cc:2454] T ebd7cdd284a74741b338e9f4778502b5 P c05f7620bb884392931d3243a0b47a9e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 3.
I20240503 02:00:41.495615 15084 consensus_queue.cc:260] T ebd7cdd284a74741b338e9f4778502b5 P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.495755 15084 raft_consensus.cc:3037] T ebd7cdd284a74741b338e9f4778502b5 P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:41.495927 15155 leader_election.cc:302] T ebd7cdd284a74741b338e9f4778502b5 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, c05f7620bb884392931d3243a0b47a9e; no voters:
I20240503 02:00:41.496102 16264 raft_consensus.cc:2781] T ebd7cdd284a74741b338e9f4778502b5 P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:00:41.496169 16264 raft_consensus.cc:686] T ebd7cdd284a74741b338e9f4778502b5 P 620596b893d84291b88b90dc9e317612 [term 3 LEADER]: Becoming Leader. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:41.496300 16264 consensus_queue.cc:237] T ebd7cdd284a74741b338e9f4778502b5 P 620596b893d84291b88b90dc9e317612 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.496798 15084 raft_consensus.cc:2396] T ebd7cdd284a74741b338e9f4778502b5 P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 620596b893d84291b88b90dc9e317612 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20240503 02:00:41.497185 15219 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "11275e009edd4354b0807b7f015023be"
dest_uuid: "620596b893d84291b88b90dc9e317612"
from {username='slave'} at 127.0.0.1:57680
I20240503 02:00:41.497299 15219 raft_consensus.cc:482] T 11275e009edd4354b0807b7f015023be P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.497344 15219 raft_consensus.cc:3037] T 11275e009edd4354b0807b7f015023be P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.498492 14816 catalog_manager.cc:5484] T ebd7cdd284a74741b338e9f4778502b5 P 620596b893d84291b88b90dc9e317612 reported cstate change: term changed from 2 to 3, leader changed from b602f6099cbb4758847e7e9550685372 (127.0.99.130) to 620596b893d84291b88b90dc9e317612 (127.0.99.131). New cstate: current_term: 3 leader_uuid: "620596b893d84291b88b90dc9e317612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } }
W20240503 02:00:41.499868 15048 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33982: Illegal state: replica b602f6099cbb4758847e7e9550685372 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.500835 15219 raft_consensus.cc:504] T 11275e009edd4354b0807b7f015023be P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.501286 15219 leader_election.cc:288] T 11275e009edd4354b0807b7f015023be P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 4 election: Requested vote from peers c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009), dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817)
I20240503 02:00:41.501758 14954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11275e009edd4354b0807b7f015023be" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:41.501881 14954 raft_consensus.cc:3032] T 11275e009edd4354b0807b7f015023be P dcfdfa62edc34dab9525e2712d487582 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 02:00:41.501916 14954 raft_consensus.cc:729] T 11275e009edd4354b0807b7f015023be P dcfdfa62edc34dab9525e2712d487582 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:41.501952 15349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11275e009edd4354b0807b7f015023be" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.502080 15349 raft_consensus.cc:3037] T 11275e009edd4354b0807b7f015023be P c05f7620bb884392931d3243a0b47a9e [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.502074 14954 consensus_queue.cc:260] T 11275e009edd4354b0807b7f015023be P dcfdfa62edc34dab9525e2712d487582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.502207 14954 raft_consensus.cc:3037] T 11275e009edd4354b0807b7f015023be P dcfdfa62edc34dab9525e2712d487582 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.503137 15349 raft_consensus.cc:2454] T 11275e009edd4354b0807b7f015023be P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 4.
I20240503 02:00:41.503192 14954 raft_consensus.cc:2396] T 11275e009edd4354b0807b7f015023be P dcfdfa62edc34dab9525e2712d487582 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 620596b893d84291b88b90dc9e317612 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20240503 02:00:41.503556 15155 leader_election.cc:302] T 11275e009edd4354b0807b7f015023be P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 620596b893d84291b88b90dc9e317612, c05f7620bb884392931d3243a0b47a9e; no voters: dcfdfa62edc34dab9525e2712d487582
I20240503 02:00:41.503582 15351 raft_consensus.cc:1261] T 5d448658ed3b460fb1725a26a7c743e8 P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Refusing update from remote peer 620596b893d84291b88b90dc9e317612: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20240503 02:00:41.503615 15219 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5da91b7af68e4cf597dc89a017ee0197"
dest_uuid: "620596b893d84291b88b90dc9e317612"
from {username='slave'} at 127.0.0.1:57680
I20240503 02:00:41.503727 15219 raft_consensus.cc:482] T 5da91b7af68e4cf597dc89a017ee0197 P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.503773 15219 raft_consensus.cc:3037] T 5da91b7af68e4cf597dc89a017ee0197 P 620596b893d84291b88b90dc9e317612 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:41.503782 16264 raft_consensus.cc:2781] T 11275e009edd4354b0807b7f015023be P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 02:00:41.503855 16264 raft_consensus.cc:686] T 11275e009edd4354b0807b7f015023be P 620596b893d84291b88b90dc9e317612 [term 4 LEADER]: Becoming Leader. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:41.503902 14956 raft_consensus.cc:1261] T 5d448658ed3b460fb1725a26a7c743e8 P dcfdfa62edc34dab9525e2712d487582 [term 4 FOLLOWER]: Refusing update from remote peer 620596b893d84291b88b90dc9e317612: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20240503 02:00:41.503991 16264 consensus_queue.cc:237] T 11275e009edd4354b0807b7f015023be P 620596b893d84291b88b90dc9e317612 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.504398 17011 consensus_queue.cc:1035] T 5d448658ed3b460fb1725a26a7c743e8 P 620596b893d84291b88b90dc9e317612 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:41.505249 15219 raft_consensus.cc:504] T 5da91b7af68e4cf597dc89a017ee0197 P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.505901 17224 consensus_queue.cc:1035] T 5d448658ed3b460fb1725a26a7c743e8 P 620596b893d84291b88b90dc9e317612 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:41.505982 15084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5da91b7af68e4cf597dc89a017ee0197" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:41.506103 15084 raft_consensus.cc:3032] T 5da91b7af68e4cf597dc89a017ee0197 P b602f6099cbb4758847e7e9550685372 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:00:41.506140 15084 raft_consensus.cc:729] T 5da91b7af68e4cf597dc89a017ee0197 P b602f6099cbb4758847e7e9550685372 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.506319 15084 consensus_queue.cc:260] T 5da91b7af68e4cf597dc89a017ee0197 P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.506448 14815 catalog_manager.cc:5484] T 11275e009edd4354b0807b7f015023be P 620596b893d84291b88b90dc9e317612 reported cstate change: term changed from 3 to 4, leader changed from dcfdfa62edc34dab9525e2712d487582 (127.0.99.129) to 620596b893d84291b88b90dc9e317612 (127.0.99.131). New cstate: current_term: 4 leader_uuid: "620596b893d84291b88b90dc9e317612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } }
W20240503 02:00:41.507319 15176 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57618: Illegal state: replica 620596b893d84291b88b90dc9e317612 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.506465 15084 raft_consensus.cc:3037] T 5da91b7af68e4cf597dc89a017ee0197 P b602f6099cbb4758847e7e9550685372 [term 2 FOLLOWER]: Advancing to term 3
W20240503 02:00:41.508531 14915 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48020: Illegal state: replica dcfdfa62edc34dab9525e2712d487582 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.509513 15084 raft_consensus.cc:2396] T 5da91b7af68e4cf597dc89a017ee0197 P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 620596b893d84291b88b90dc9e317612 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 02:00:41.511108 15349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5da91b7af68e4cf597dc89a017ee0197" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.511246 15349 raft_consensus.cc:3037] T 5da91b7af68e4cf597dc89a017ee0197 P c05f7620bb884392931d3243a0b47a9e [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:41.512277 15349 raft_consensus.cc:2454] T 5da91b7af68e4cf597dc89a017ee0197 P c05f7620bb884392931d3243a0b47a9e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 3.
I20240503 02:00:41.512624 15155 leader_election.cc:302] T 5da91b7af68e4cf597dc89a017ee0197 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 620596b893d84291b88b90dc9e317612, c05f7620bb884392931d3243a0b47a9e; no voters: b602f6099cbb4758847e7e9550685372
I20240503 02:00:41.512801 16264 raft_consensus.cc:2781] T 5da91b7af68e4cf597dc89a017ee0197 P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:00:41.512871 16264 raft_consensus.cc:686] T 5da91b7af68e4cf597dc89a017ee0197 P 620596b893d84291b88b90dc9e317612 [term 3 LEADER]: Becoming Leader. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:41.513027 16264 consensus_queue.cc:237] T 5da91b7af68e4cf597dc89a017ee0197 P 620596b893d84291b88b90dc9e317612 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.513963 15219 leader_election.cc:288] T 5da91b7af68e4cf597dc89a017ee0197 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 3 election: Requested vote from peers b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
W20240503 02:00:41.517204 14915 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47950: Illegal state: replica dcfdfa62edc34dab9525e2712d487582 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.518436 14816 catalog_manager.cc:5484] T 5da91b7af68e4cf597dc89a017ee0197 P 620596b893d84291b88b90dc9e317612 reported cstate change: term changed from 2 to 3, leader changed from b602f6099cbb4758847e7e9550685372 (127.0.99.130) to 620596b893d84291b88b90dc9e317612 (127.0.99.131). New cstate: current_term: 3 leader_uuid: "620596b893d84291b88b90dc9e317612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:41.525465 15349 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4cdcdde158054cdaabc1856a90fe1302"
dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
from {username='slave'} at 127.0.0.1:50864
I20240503 02:00:41.525632 15349 raft_consensus.cc:482] T 4cdcdde158054cdaabc1856a90fe1302 P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.525691 15349 raft_consensus.cc:3037] T 4cdcdde158054cdaabc1856a90fe1302 P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.526738 15349 raft_consensus.cc:504] T 4cdcdde158054cdaabc1856a90fe1302 P c05f7620bb884392931d3243a0b47a9e [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.527187 15349 leader_election.cc:288] T 4cdcdde158054cdaabc1856a90fe1302 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 5 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
I20240503 02:00:41.527736 15084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcdde158054cdaabc1856a90fe1302" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:41.527863 15084 raft_consensus.cc:3037] T 4cdcdde158054cdaabc1856a90fe1302 P b602f6099cbb4758847e7e9550685372 [term 4 FOLLOWER]: Advancing to term 5
W20240503 02:00:41.528299 15311 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50950: Illegal state: replica c05f7620bb884392931d3243a0b47a9e is not leader of this config: current role FOLLOWER
W20240503 02:00:41.529482 15048 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33982: Illegal state: replica b602f6099cbb4758847e7e9550685372 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.530009 15084 raft_consensus.cc:2396] T 4cdcdde158054cdaabc1856a90fe1302 P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c05f7620bb884392931d3243a0b47a9e for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 02:00:41.530035 15219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcdde158054cdaabc1856a90fe1302" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:41.530164 15219 raft_consensus.cc:3032] T 4cdcdde158054cdaabc1856a90fe1302 P 620596b893d84291b88b90dc9e317612 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 02:00:41.530206 15219 raft_consensus.cc:729] T 4cdcdde158054cdaabc1856a90fe1302 P 620596b893d84291b88b90dc9e317612 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:41.530345 15219 consensus_queue.cc:260] T 4cdcdde158054cdaabc1856a90fe1302 P 620596b893d84291b88b90dc9e317612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.530503 15219 raft_consensus.cc:3037] T 4cdcdde158054cdaabc1856a90fe1302 P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.531637 15219 raft_consensus.cc:2396] T 4cdcdde158054cdaabc1856a90fe1302 P 620596b893d84291b88b90dc9e317612 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c05f7620bb884392931d3243a0b47a9e for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 02:00:41.531648 15349 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "de805ef7511d4ee9b9bbed67e0a4186c"
dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
from {username='slave'} at 127.0.0.1:50864
I20240503 02:00:41.531755 15349 raft_consensus.cc:482] T de805ef7511d4ee9b9bbed67e0a4186c P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.531808 15349 raft_consensus.cc:3037] T de805ef7511d4ee9b9bbed67e0a4186c P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.532011 15286 leader_election.cc:302] T 4cdcdde158054cdaabc1856a90fe1302 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c05f7620bb884392931d3243a0b47a9e; no voters: 620596b893d84291b88b90dc9e317612, b602f6099cbb4758847e7e9550685372
I20240503 02:00:41.532274 17163 raft_consensus.cc:2726] T 4cdcdde158054cdaabc1856a90fe1302 P c05f7620bb884392931d3243a0b47a9e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240503 02:00:41.532819 15349 raft_consensus.cc:504] T de805ef7511d4ee9b9bbed67e0a4186c P c05f7620bb884392931d3243a0b47a9e [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.533210 15349 leader_election.cc:288] T de805ef7511d4ee9b9bbed67e0a4186c P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 5 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
I20240503 02:00:41.533643 15084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de805ef7511d4ee9b9bbed67e0a4186c" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 5 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:41.533782 15084 raft_consensus.cc:3037] T de805ef7511d4ee9b9bbed67e0a4186c P b602f6099cbb4758847e7e9550685372 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.534047 15349 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "21d92d412a5247e8907c5c41d253ad2e"
dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
from {username='slave'} at 127.0.0.1:50864
I20240503 02:00:41.534161 15349 raft_consensus.cc:482] T 21d92d412a5247e8907c5c41d253ad2e P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.534217 15349 raft_consensus.cc:3037] T 21d92d412a5247e8907c5c41d253ad2e P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.534862 15084 raft_consensus.cc:2396] T de805ef7511d4ee9b9bbed67e0a4186c P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c05f7620bb884392931d3243a0b47a9e for term 5 because replica has last-logged OpId of term: 3 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 6.
I20240503 02:00:41.535104 15349 raft_consensus.cc:504] T 21d92d412a5247e8907c5c41d253ad2e P c05f7620bb884392931d3243a0b47a9e [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.535319 15219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de805ef7511d4ee9b9bbed67e0a4186c" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 5 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:41.535449 15219 raft_consensus.cc:3032] T de805ef7511d4ee9b9bbed67e0a4186c P 620596b893d84291b88b90dc9e317612 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 02:00:41.535499 15219 raft_consensus.cc:729] T de805ef7511d4ee9b9bbed67e0a4186c P 620596b893d84291b88b90dc9e317612 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:41.535671 15219 consensus_queue.cc:260] T de805ef7511d4ee9b9bbed67e0a4186c P 620596b893d84291b88b90dc9e317612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.535826 15219 raft_consensus.cc:3037] T de805ef7511d4ee9b9bbed67e0a4186c P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.535840 14954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21d92d412a5247e8907c5c41d253ad2e" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:41.535976 14954 raft_consensus.cc:3037] T 21d92d412a5247e8907c5c41d253ad2e P dcfdfa62edc34dab9525e2712d487582 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.536118 15349 leader_election.cc:288] T 21d92d412a5247e8907c5c41d253ad2e P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 5 election: Requested vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437)
I20240503 02:00:41.536823 15349 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9e9899666b15498791f62dab6d9dda16"
dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
from {username='slave'} at 127.0.0.1:50864
I20240503 02:00:41.536947 15219 raft_consensus.cc:2396] T de805ef7511d4ee9b9bbed67e0a4186c P 620596b893d84291b88b90dc9e317612 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c05f7620bb884392931d3243a0b47a9e for term 5 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 6.
I20240503 02:00:41.536942 15349 raft_consensus.cc:482] T 9e9899666b15498791f62dab6d9dda16 P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.536993 15349 raft_consensus.cc:3037] T 9e9899666b15498791f62dab6d9dda16 P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.537166 14954 raft_consensus.cc:2454] T 21d92d412a5247e8907c5c41d253ad2e P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 5.
I20240503 02:00:41.537282 15286 leader_election.cc:302] T de805ef7511d4ee9b9bbed67e0a4186c P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c05f7620bb884392931d3243a0b47a9e; no voters: 620596b893d84291b88b90dc9e317612, b602f6099cbb4758847e7e9550685372
I20240503 02:00:41.537454 17163 raft_consensus.cc:2726] T de805ef7511d4ee9b9bbed67e0a4186c P c05f7620bb884392931d3243a0b47a9e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240503 02:00:41.537604 15284 leader_election.cc:302] T 21d92d412a5247e8907c5c41d253ad2e P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c05f7620bb884392931d3243a0b47a9e, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:41.537761 17163 raft_consensus.cc:2781] T 21d92d412a5247e8907c5c41d253ad2e P c05f7620bb884392931d3243a0b47a9e [term 5 FOLLOWER]: Leader election won for term 5
I20240503 02:00:41.537827 17163 raft_consensus.cc:686] T 21d92d412a5247e8907c5c41d253ad2e P c05f7620bb884392931d3243a0b47a9e [term 5 LEADER]: Becoming Leader. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:41.537953 17163 consensus_queue.cc:237] T 21d92d412a5247e8907c5c41d253ad2e P c05f7620bb884392931d3243a0b47a9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.538642 15349 raft_consensus.cc:504] T 9e9899666b15498791f62dab6d9dda16 P c05f7620bb884392931d3243a0b47a9e [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:41.539646 15214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21d92d412a5247e8907c5c41d253ad2e" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:41.539700 15084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e9899666b15498791f62dab6d9dda16" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 5 candidate_status { last_received { term: 3 index: 9 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:41.539781 15214 raft_consensus.cc:3032] T 21d92d412a5247e8907c5c41d253ad2e P 620596b893d84291b88b90dc9e317612 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 02:00:41.539820 15084 raft_consensus.cc:3037] T 9e9899666b15498791f62dab6d9dda16 P b602f6099cbb4758847e7e9550685372 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.539821 15214 raft_consensus.cc:729] T 21d92d412a5247e8907c5c41d253ad2e P 620596b893d84291b88b90dc9e317612 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:41.539979 15219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e9899666b15498791f62dab6d9dda16" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 5 candidate_status { last_received { term: 3 index: 9 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:41.540066 15219 raft_consensus.cc:3037] T 9e9899666b15498791f62dab6d9dda16 P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.539943 14816 catalog_manager.cc:5484] T 21d92d412a5247e8907c5c41d253ad2e P c05f7620bb884392931d3243a0b47a9e reported cstate change: term changed from 4 to 5, leader changed from 620596b893d84291b88b90dc9e317612 (127.0.99.131) to c05f7620bb884392931d3243a0b47a9e (127.0.99.132). New cstate: current_term: 5 leader_uuid: "c05f7620bb884392931d3243a0b47a9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:41.541002 15084 raft_consensus.cc:2454] T 9e9899666b15498791f62dab6d9dda16 P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 5.
I20240503 02:00:41.541028 15219 raft_consensus.cc:2454] T 9e9899666b15498791f62dab6d9dda16 P 620596b893d84291b88b90dc9e317612 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 5.
I20240503 02:00:41.541319 15286 leader_election.cc:302] T 9e9899666b15498791f62dab6d9dda16 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, c05f7620bb884392931d3243a0b47a9e; no voters:
I20240503 02:00:41.541210 15349 leader_election.cc:288] T 9e9899666b15498791f62dab6d9dda16 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 5 election: Requested vote from peers b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207), 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437)
I20240503 02:00:41.541503 17163 raft_consensus.cc:2781] T 9e9899666b15498791f62dab6d9dda16 P c05f7620bb884392931d3243a0b47a9e [term 5 FOLLOWER]: Leader election won for term 5
I20240503 02:00:41.541581 17163 raft_consensus.cc:686] T 9e9899666b15498791f62dab6d9dda16 P c05f7620bb884392931d3243a0b47a9e [term 5 LEADER]: Becoming Leader. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:41.541709 17163 consensus_queue.cc:237] T 9e9899666b15498791f62dab6d9dda16 P c05f7620bb884392931d3243a0b47a9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 3.9, Last appended by leader: 9, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:41.541714 15214 consensus_queue.cc:260] T 21d92d412a5247e8907c5c41d253ad2e P 620596b893d84291b88b90dc9e317612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.541860 15214 raft_consensus.cc:3037] T 21d92d412a5247e8907c5c41d253ad2e P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Advancing to term 5
W20240503 02:00:41.542364 15176 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57618: Illegal state: replica 620596b893d84291b88b90dc9e317612 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.542927 15214 raft_consensus.cc:2396] T 21d92d412a5247e8907c5c41d253ad2e P 620596b893d84291b88b90dc9e317612 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c05f7620bb884392931d3243a0b47a9e for term 5 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20240503 02:00:41.545356 14816 catalog_manager.cc:5484] T 9e9899666b15498791f62dab6d9dda16 P c05f7620bb884392931d3243a0b47a9e reported cstate change: term changed from 3 to 5, leader changed from b602f6099cbb4758847e7e9550685372 (127.0.99.130) to c05f7620bb884392931d3243a0b47a9e (127.0.99.132). New cstate: current_term: 5 leader_uuid: "c05f7620bb884392931d3243a0b47a9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:41.546183 15349 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "50c5abd860ae4b09bb289e9c7550c84e"
dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
from {username='slave'} at 127.0.0.1:50864
I20240503 02:00:41.546331 15349 raft_consensus.cc:482] T 50c5abd860ae4b09bb289e9c7550c84e P c05f7620bb884392931d3243a0b47a9e [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.546391 15349 raft_consensus.cc:3037] T 50c5abd860ae4b09bb289e9c7550c84e P c05f7620bb884392931d3243a0b47a9e [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.547459 15349 raft_consensus.cc:504] T 50c5abd860ae4b09bb289e9c7550c84e P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.548313 14954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50c5abd860ae4b09bb289e9c7550c84e" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 4 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:41.548478 14954 raft_consensus.cc:3037] T 50c5abd860ae4b09bb289e9c7550c84e P dcfdfa62edc34dab9525e2712d487582 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.548838 15349 leader_election.cc:288] T 50c5abd860ae4b09bb289e9c7550c84e P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 4 election: Requested vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437)
W20240503 02:00:41.548962 15048 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34032: Illegal state: replica b602f6099cbb4758847e7e9550685372 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.549494 14954 raft_consensus.cc:2454] T 50c5abd860ae4b09bb289e9c7550c84e P dcfdfa62edc34dab9525e2712d487582 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 4.
I20240503 02:00:41.550271 15284 leader_election.cc:302] T 50c5abd860ae4b09bb289e9c7550c84e P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c05f7620bb884392931d3243a0b47a9e, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:41.550266 15349 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7fc3330dfd0644f6a303659b960124f4"
dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
from {username='slave'} at 127.0.0.1:50864
I20240503 02:00:41.550485 15349 raft_consensus.cc:482] T 7fc3330dfd0644f6a303659b960124f4 P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.550541 15349 raft_consensus.cc:3037] T 7fc3330dfd0644f6a303659b960124f4 P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.550733 16986 raft_consensus.cc:2781] T 50c5abd860ae4b09bb289e9c7550c84e P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Leader election won for term 4
I20240503 02:00:41.550871 16986 raft_consensus.cc:686] T 50c5abd860ae4b09bb289e9c7550c84e P c05f7620bb884392931d3243a0b47a9e [term 4 LEADER]: Becoming Leader. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:41.551079 16986 consensus_queue.cc:237] T 50c5abd860ae4b09bb289e9c7550c84e P c05f7620bb884392931d3243a0b47a9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.551612 15349 raft_consensus.cc:504] T 7fc3330dfd0644f6a303659b960124f4 P c05f7620bb884392931d3243a0b47a9e [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.551950 15349 leader_election.cc:288] T 7fc3330dfd0644f6a303659b960124f4 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 5 election: Requested vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437)
I20240503 02:00:41.553246 15219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fc3330dfd0644f6a303659b960124f4" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 5 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:41.553396 15219 raft_consensus.cc:3032] T 7fc3330dfd0644f6a303659b960124f4 P 620596b893d84291b88b90dc9e317612 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 02:00:41.553445 15219 raft_consensus.cc:729] T 7fc3330dfd0644f6a303659b960124f4 P 620596b893d84291b88b90dc9e317612 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:41.553618 15219 consensus_queue.cc:260] T 7fc3330dfd0644f6a303659b960124f4 P 620596b893d84291b88b90dc9e317612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 4.13, Last appended by leader: 13, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.553817 15219 raft_consensus.cc:3037] T 7fc3330dfd0644f6a303659b960124f4 P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.555051 15219 raft_consensus.cc:2396] T 7fc3330dfd0644f6a303659b960124f4 P 620596b893d84291b88b90dc9e317612 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c05f7620bb884392931d3243a0b47a9e for term 5 because replica has last-logged OpId of term: 4 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 12.
I20240503 02:00:41.556027 14954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fc3330dfd0644f6a303659b960124f4" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 5 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:41.556175 14954 raft_consensus.cc:3037] T 7fc3330dfd0644f6a303659b960124f4 P dcfdfa62edc34dab9525e2712d487582 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.557296 14954 raft_consensus.cc:2454] T 7fc3330dfd0644f6a303659b960124f4 P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 5.
I20240503 02:00:41.558163 15284 leader_election.cc:302] T 7fc3330dfd0644f6a303659b960124f4 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c05f7620bb884392931d3243a0b47a9e, dcfdfa62edc34dab9525e2712d487582; no voters: 620596b893d84291b88b90dc9e317612
I20240503 02:00:41.558182 15349 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c74522fbb5c64d27b5c95f05896b0716"
dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
from {username='slave'} at 127.0.0.1:50864
I20240503 02:00:41.558354 15349 raft_consensus.cc:482] T c74522fbb5c64d27b5c95f05896b0716 P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.558404 15349 raft_consensus.cc:3037] T c74522fbb5c64d27b5c95f05896b0716 P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.558451 17163 raft_consensus.cc:2781] T 7fc3330dfd0644f6a303659b960124f4 P c05f7620bb884392931d3243a0b47a9e [term 5 FOLLOWER]: Leader election won for term 5
I20240503 02:00:41.558535 17163 raft_consensus.cc:686] T 7fc3330dfd0644f6a303659b960124f4 P c05f7620bb884392931d3243a0b47a9e [term 5 LEADER]: Becoming Leader. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:41.558670 17163 consensus_queue.cc:237] T 7fc3330dfd0644f6a303659b960124f4 P c05f7620bb884392931d3243a0b47a9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 3.12, Last appended by leader: 12, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.559383 15349 raft_consensus.cc:504] T c74522fbb5c64d27b5c95f05896b0716 P c05f7620bb884392931d3243a0b47a9e [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.559716 15349 leader_election.cc:288] T c74522fbb5c64d27b5c95f05896b0716 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 5 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
W20240503 02:00:41.559717 14915 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47950: Illegal state: replica dcfdfa62edc34dab9525e2712d487582 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.560211 15084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c74522fbb5c64d27b5c95f05896b0716" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 5 candidate_status { last_received { term: 2 index: 25 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:41.560344 15084 raft_consensus.cc:3037] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.561383 15084 raft_consensus.cc:2396] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c05f7620bb884392931d3243a0b47a9e for term 5 because replica has last-logged OpId of term: 3 index: 26, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 25.
I20240503 02:00:41.562047 15349 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6d2a1296a54a4b92a03c12954de0cc34"
dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
from {username='slave'} at 127.0.0.1:50864
I20240503 02:00:41.562191 15349 raft_consensus.cc:482] T 6d2a1296a54a4b92a03c12954de0cc34 P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.562249 15349 raft_consensus.cc:3037] T 6d2a1296a54a4b92a03c12954de0cc34 P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.562301 15219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c74522fbb5c64d27b5c95f05896b0716" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 5 candidate_status { last_received { term: 2 index: 25 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:41.562422 15219 raft_consensus.cc:3032] T c74522fbb5c64d27b5c95f05896b0716 P 620596b893d84291b88b90dc9e317612 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 02:00:41.562455 15219 raft_consensus.cc:729] T c74522fbb5c64d27b5c95f05896b0716 P 620596b893d84291b88b90dc9e317612 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:41.562589 15219 consensus_queue.cc:260] T c74522fbb5c64d27b5c95f05896b0716 P 620596b893d84291b88b90dc9e317612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 4.26, Last appended by leader: 26, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.562716 15219 raft_consensus.cc:3037] T c74522fbb5c64d27b5c95f05896b0716 P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.563287 15349 raft_consensus.cc:504] T 6d2a1296a54a4b92a03c12954de0cc34 P c05f7620bb884392931d3243a0b47a9e [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.563642 15219 raft_consensus.cc:2396] T c74522fbb5c64d27b5c95f05896b0716 P 620596b893d84291b88b90dc9e317612 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c05f7620bb884392931d3243a0b47a9e for term 5 because replica has last-logged OpId of term: 4 index: 26, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 25.
I20240503 02:00:41.563994 15286 leader_election.cc:302] T c74522fbb5c64d27b5c95f05896b0716 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c05f7620bb884392931d3243a0b47a9e; no voters: 620596b893d84291b88b90dc9e317612, b602f6099cbb4758847e7e9550685372
I20240503 02:00:41.564185 16986 raft_consensus.cc:2726] T c74522fbb5c64d27b5c95f05896b0716 P c05f7620bb884392931d3243a0b47a9e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240503 02:00:41.564837 14954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2a1296a54a4b92a03c12954de0cc34" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 5 candidate_status { last_received { term: 3 index: 10 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:41.564870 15084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2a1296a54a4b92a03c12954de0cc34" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 5 candidate_status { last_received { term: 3 index: 10 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:41.564972 14954 raft_consensus.cc:3037] T 6d2a1296a54a4b92a03c12954de0cc34 P dcfdfa62edc34dab9525e2712d487582 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.564992 15084 raft_consensus.cc:3032] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 02:00:41.565037 15084 raft_consensus.cc:729] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.565189 15084 consensus_queue.cc:260] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 4.11, Last appended by leader: 11, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.565330 15084 raft_consensus.cc:3037] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.565940 14954 raft_consensus.cc:2454] T 6d2a1296a54a4b92a03c12954de0cc34 P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 5.
I20240503 02:00:41.566299 15084 raft_consensus.cc:2396] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c05f7620bb884392931d3243a0b47a9e for term 5 because replica has last-logged OpId of term: 4 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 10.
I20240503 02:00:41.566512 15284 leader_election.cc:302] T 6d2a1296a54a4b92a03c12954de0cc34 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c05f7620bb884392931d3243a0b47a9e, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:41.566879 16986 raft_consensus.cc:2781] T 6d2a1296a54a4b92a03c12954de0cc34 P c05f7620bb884392931d3243a0b47a9e [term 5 FOLLOWER]: Leader election won for term 5
I20240503 02:00:41.566991 16986 raft_consensus.cc:686] T 6d2a1296a54a4b92a03c12954de0cc34 P c05f7620bb884392931d3243a0b47a9e [term 5 LEADER]: Becoming Leader. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:41.567157 16986 consensus_queue.cc:237] T 6d2a1296a54a4b92a03c12954de0cc34 P c05f7620bb884392931d3243a0b47a9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 3.10, Last appended by leader: 10, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.567730 14816 catalog_manager.cc:5484] T 50c5abd860ae4b09bb289e9c7550c84e P c05f7620bb884392931d3243a0b47a9e reported cstate change: term changed from 3 to 4, leader changed from 620596b893d84291b88b90dc9e317612 (127.0.99.131) to c05f7620bb884392931d3243a0b47a9e (127.0.99.132). New cstate: current_term: 4 leader_uuid: "c05f7620bb884392931d3243a0b47a9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:41.568360 15349 leader_election.cc:288] T 6d2a1296a54a4b92a03c12954de0cc34 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 5 election: Requested vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
I20240503 02:00:41.569160 15349 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5736c284a85540a1bbab2be6df672fc1"
dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
from {username='slave'} at 127.0.0.1:50864
I20240503 02:00:41.569300 15349 raft_consensus.cc:482] T 5736c284a85540a1bbab2be6df672fc1 P c05f7620bb884392931d3243a0b47a9e [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.569358 15349 raft_consensus.cc:3037] T 5736c284a85540a1bbab2be6df672fc1 P c05f7620bb884392931d3243a0b47a9e [term 3 FOLLOWER]: Advancing to term 4
W20240503 02:00:41.570195 14915 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48020: Illegal state: replica dcfdfa62edc34dab9525e2712d487582 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.570406 15349 raft_consensus.cc:504] T 5736c284a85540a1bbab2be6df672fc1 P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.571089 15214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50c5abd860ae4b09bb289e9c7550c84e" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 4 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:41.571166 15219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5736c284a85540a1bbab2be6df672fc1" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:41.571230 15214 raft_consensus.cc:3032] T 50c5abd860ae4b09bb289e9c7550c84e P 620596b893d84291b88b90dc9e317612 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 02:00:41.571271 15214 raft_consensus.cc:729] T 50c5abd860ae4b09bb289e9c7550c84e P 620596b893d84291b88b90dc9e317612 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:41.571280 15219 raft_consensus.cc:3032] T 5736c284a85540a1bbab2be6df672fc1 P 620596b893d84291b88b90dc9e317612 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 02:00:41.571323 15219 raft_consensus.cc:729] T 5736c284a85540a1bbab2be6df672fc1 P 620596b893d84291b88b90dc9e317612 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:41.571420 15214 consensus_queue.cc:260] T 50c5abd860ae4b09bb289e9c7550c84e P 620596b893d84291b88b90dc9e317612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.7, Last appended by leader: 7, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.571447 15219 consensus_queue.cc:260] T 5736c284a85540a1bbab2be6df672fc1 P 620596b893d84291b88b90dc9e317612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.571568 15214 raft_consensus.cc:3037] T 50c5abd860ae4b09bb289e9c7550c84e P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.571592 15219 raft_consensus.cc:3037] T 5736c284a85540a1bbab2be6df672fc1 P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.572762 15219 raft_consensus.cc:2396] T 5736c284a85540a1bbab2be6df672fc1 P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate c05f7620bb884392931d3243a0b47a9e for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20240503 02:00:41.573063 15349 leader_election.cc:288] T 5736c284a85540a1bbab2be6df672fc1 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 4 election: Requested vote from peers b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207), 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437)
I20240503 02:00:41.573258 15214 raft_consensus.cc:2396] T 50c5abd860ae4b09bb289e9c7550c84e P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate c05f7620bb884392931d3243a0b47a9e for term 4 because replica has last-logged OpId of term: 3 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 6.
W20240503 02:00:41.573626 15048 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33982: Illegal state: replica b602f6099cbb4758847e7e9550685372 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.573665 15351 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eade118a36da42e3ac08b96a13893a5c"
dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
from {username='slave'} at 127.0.0.1:50864
I20240503 02:00:41.573793 15351 raft_consensus.cc:482] T eade118a36da42e3ac08b96a13893a5c P c05f7620bb884392931d3243a0b47a9e [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.573853 15351 raft_consensus.cc:3037] T eade118a36da42e3ac08b96a13893a5c P c05f7620bb884392931d3243a0b47a9e [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.573894 14816 catalog_manager.cc:5484] T 7fc3330dfd0644f6a303659b960124f4 P c05f7620bb884392931d3243a0b47a9e reported cstate change: term changed from 4 to 5, leader changed from 620596b893d84291b88b90dc9e317612 (127.0.99.131) to c05f7620bb884392931d3243a0b47a9e (127.0.99.132). New cstate: current_term: 5 leader_uuid: "c05f7620bb884392931d3243a0b47a9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:41.574154 14816 catalog_manager.cc:5484] T 6d2a1296a54a4b92a03c12954de0cc34 P c05f7620bb884392931d3243a0b47a9e reported cstate change: term changed from 4 to 5, leader changed from b602f6099cbb4758847e7e9550685372 (127.0.99.130) to c05f7620bb884392931d3243a0b47a9e (127.0.99.132). New cstate: current_term: 5 leader_uuid: "c05f7620bb884392931d3243a0b47a9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:41.574561 15084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5736c284a85540a1bbab2be6df672fc1" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:41.574690 15084 raft_consensus.cc:3037] T 5736c284a85540a1bbab2be6df672fc1 P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.574949 15351 raft_consensus.cc:504] T eade118a36da42e3ac08b96a13893a5c P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:41.575364 15351 leader_election.cc:288] T eade118a36da42e3ac08b96a13893a5c P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 4 election: Requested vote from peers b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207), 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437)
I20240503 02:00:41.575989 15087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eade118a36da42e3ac08b96a13893a5c" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 4 candidate_status { last_received { term: 2 index: 20 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:41.576131 15087 raft_consensus.cc:3037] T eade118a36da42e3ac08b96a13893a5c P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.576552 15214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eade118a36da42e3ac08b96a13893a5c" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 4 candidate_status { last_received { term: 2 index: 20 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:41.576753 15214 raft_consensus.cc:3037] T eade118a36da42e3ac08b96a13893a5c P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.576776 15351 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9a74cca2988a4f219f6fd9b07ade42d8"
dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
from {username='slave'} at 127.0.0.1:50864
I20240503 02:00:41.576895 15351 raft_consensus.cc:482] T 9a74cca2988a4f219f6fd9b07ade42d8 P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.576944 15351 raft_consensus.cc:3037] T 9a74cca2988a4f219f6fd9b07ade42d8 P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.577262 15087 raft_consensus.cc:2454] T eade118a36da42e3ac08b96a13893a5c P b602f6099cbb4758847e7e9550685372 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 4.
I20240503 02:00:41.577740 15214 raft_consensus.cc:2454] T eade118a36da42e3ac08b96a13893a5c P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 4.
I20240503 02:00:41.577797 15285 leader_election.cc:302] T eade118a36da42e3ac08b96a13893a5c P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b602f6099cbb4758847e7e9550685372, c05f7620bb884392931d3243a0b47a9e; no voters:
I20240503 02:00:41.577824 15351 raft_consensus.cc:504] T 9a74cca2988a4f219f6fd9b07ade42d8 P c05f7620bb884392931d3243a0b47a9e [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.577968 17163 raft_consensus.cc:2781] T eade118a36da42e3ac08b96a13893a5c P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Leader election won for term 4
I20240503 02:00:41.578054 17163 raft_consensus.cc:686] T eade118a36da42e3ac08b96a13893a5c P c05f7620bb884392931d3243a0b47a9e [term 4 LEADER]: Becoming Leader. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:41.578189 17163 consensus_queue.cc:237] T eade118a36da42e3ac08b96a13893a5c P c05f7620bb884392931d3243a0b47a9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 2.20, Last appended by leader: 20, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:41.578259 15351 leader_election.cc:288] T 9a74cca2988a4f219f6fd9b07ade42d8 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 5 election: Requested vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437)
I20240503 02:00:41.579422 15214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a74cca2988a4f219f6fd9b07ade42d8" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 5 candidate_status { last_received { term: 3 index: 7 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:41.579504 14954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a74cca2988a4f219f6fd9b07ade42d8" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 5 candidate_status { last_received { term: 3 index: 7 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:41.579566 15214 raft_consensus.cc:3032] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 02:00:41.579607 15214 raft_consensus.cc:729] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:41.579643 14954 raft_consensus.cc:3037] T 9a74cca2988a4f219f6fd9b07ade42d8 P dcfdfa62edc34dab9525e2712d487582 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.579752 15214 consensus_queue.cc:260] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.8, Last appended by leader: 8, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.579896 15214 raft_consensus.cc:3037] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.576009 15084 raft_consensus.cc:2396] T 5736c284a85540a1bbab2be6df672fc1 P b602f6099cbb4758847e7e9550685372 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate c05f7620bb884392931d3243a0b47a9e for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20240503 02:00:41.580399 15285 leader_election.cc:302] T 5736c284a85540a1bbab2be6df672fc1 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c05f7620bb884392931d3243a0b47a9e; no voters: 620596b893d84291b88b90dc9e317612, b602f6099cbb4758847e7e9550685372
I20240503 02:00:41.580623 15351 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d805ee04df39406ba30b3b3b9e349507"
dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
from {username='slave'} at 127.0.0.1:50864
I20240503 02:00:41.580674 16986 raft_consensus.cc:2726] T 5736c284a85540a1bbab2be6df672fc1 P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240503 02:00:41.580752 15351 raft_consensus.cc:482] T d805ee04df39406ba30b3b3b9e349507 P c05f7620bb884392931d3243a0b47a9e [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.580811 15351 raft_consensus.cc:3037] T d805ee04df39406ba30b3b3b9e349507 P c05f7620bb884392931d3243a0b47a9e [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.580981 15214 raft_consensus.cc:2396] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c05f7620bb884392931d3243a0b47a9e for term 5 because replica has last-logged OpId of term: 4 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 7.
I20240503 02:00:41.581293 14816 catalog_manager.cc:5484] T eade118a36da42e3ac08b96a13893a5c P c05f7620bb884392931d3243a0b47a9e reported cstate change: term changed from 2 to 4, leader changed from b602f6099cbb4758847e7e9550685372 (127.0.99.130) to c05f7620bb884392931d3243a0b47a9e (127.0.99.132). New cstate: current_term: 4 leader_uuid: "c05f7620bb884392931d3243a0b47a9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:41.581825 15351 raft_consensus.cc:504] T d805ee04df39406ba30b3b3b9e349507 P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.582682 15084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d805ee04df39406ba30b3b3b9e349507" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 4 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:41.582942 14956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d805ee04df39406ba30b3b3b9e349507" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 4 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:41.583083 14956 raft_consensus.cc:3037] T d805ee04df39406ba30b3b3b9e349507 P dcfdfa62edc34dab9525e2712d487582 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.583274 14954 raft_consensus.cc:2454] T 9a74cca2988a4f219f6fd9b07ade42d8 P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 5.
I20240503 02:00:41.583583 15284 leader_election.cc:302] T 9a74cca2988a4f219f6fd9b07ade42d8 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c05f7620bb884392931d3243a0b47a9e, dcfdfa62edc34dab9525e2712d487582; no voters: 620596b893d84291b88b90dc9e317612
I20240503 02:00:41.583997 15351 leader_election.cc:288] T d805ee04df39406ba30b3b3b9e349507 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 4 election: Requested vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
I20240503 02:00:41.584187 14956 raft_consensus.cc:2396] T d805ee04df39406ba30b3b3b9e349507 P dcfdfa62edc34dab9525e2712d487582 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate c05f7620bb884392931d3243a0b47a9e for term 4 because replica has last-logged OpId of term: 2 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 6.
I20240503 02:00:41.584420 16986 raft_consensus.cc:2781] T 9a74cca2988a4f219f6fd9b07ade42d8 P c05f7620bb884392931d3243a0b47a9e [term 5 FOLLOWER]: Leader election won for term 5
I20240503 02:00:41.584507 16986 raft_consensus.cc:686] T 9a74cca2988a4f219f6fd9b07ade42d8 P c05f7620bb884392931d3243a0b47a9e [term 5 LEADER]: Becoming Leader. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:41.584642 16986 consensus_queue.cc:237] T 9a74cca2988a4f219f6fd9b07ade42d8 P c05f7620bb884392931d3243a0b47a9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 3.7, Last appended by leader: 7, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.584676 15351 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6798a9ed0cf441aca7b8ae1b05c001f3"
dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
from {username='slave'} at 127.0.0.1:50864
I20240503 02:00:41.584790 15351 raft_consensus.cc:482] T 6798a9ed0cf441aca7b8ae1b05c001f3 P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.584836 15351 raft_consensus.cc:3037] T 6798a9ed0cf441aca7b8ae1b05c001f3 P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.585791 15351 raft_consensus.cc:504] T 6798a9ed0cf441aca7b8ae1b05c001f3 P c05f7620bb884392931d3243a0b47a9e [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.586319 15351 leader_election.cc:288] T 6798a9ed0cf441aca7b8ae1b05c001f3 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 5 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817)
W20240503 02:00:41.587489 15176 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57618: Illegal state: replica 620596b893d84291b88b90dc9e317612 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.588110 14816 catalog_manager.cc:5484] T 9a74cca2988a4f219f6fd9b07ade42d8 P c05f7620bb884392931d3243a0b47a9e reported cstate change: term changed from 4 to 5, leader changed from 620596b893d84291b88b90dc9e317612 (127.0.99.131) to c05f7620bb884392931d3243a0b47a9e (127.0.99.132). New cstate: current_term: 5 leader_uuid: "c05f7620bb884392931d3243a0b47a9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:41.589128 14956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6798a9ed0cf441aca7b8ae1b05c001f3" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:41.589260 14956 raft_consensus.cc:3037] T 6798a9ed0cf441aca7b8ae1b05c001f3 P dcfdfa62edc34dab9525e2712d487582 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.589270 15084 raft_consensus.cc:3032] T d805ee04df39406ba30b3b3b9e349507 P b602f6099cbb4758847e7e9550685372 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 02:00:41.589332 15084 raft_consensus.cc:729] T d805ee04df39406ba30b3b3b9e349507 P b602f6099cbb4758847e7e9550685372 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.589494 15084 consensus_queue.cc:260] T d805ee04df39406ba30b3b3b9e349507 P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.7, Last appended by leader: 7, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.589694 15084 raft_consensus.cc:3037] T d805ee04df39406ba30b3b3b9e349507 P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Advancing to term 4
W20240503 02:00:41.585943 15048 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34060: Illegal state: replica b602f6099cbb4758847e7e9550685372 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.590370 14956 raft_consensus.cc:2396] T 6798a9ed0cf441aca7b8ae1b05c001f3 P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c05f7620bb884392931d3243a0b47a9e for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 02:00:41.590996 15349 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ebd7cdd284a74741b338e9f4778502b5"
dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
from {username='slave'} at 127.0.0.1:50864
I20240503 02:00:41.591152 15349 raft_consensus.cc:482] T ebd7cdd284a74741b338e9f4778502b5 P c05f7620bb884392931d3243a0b47a9e [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.591225 15349 raft_consensus.cc:3037] T ebd7cdd284a74741b338e9f4778502b5 P c05f7620bb884392931d3243a0b47a9e [term 3 FOLLOWER]: Advancing to term 4
W20240503 02:00:41.591507 15311 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50950: Illegal state: replica c05f7620bb884392931d3243a0b47a9e is not leader of this config: current role FOLLOWER
I20240503 02:00:41.593040 15349 raft_consensus.cc:504] T ebd7cdd284a74741b338e9f4778502b5 P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.593829 15219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebd7cdd284a74741b338e9f4778502b5" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:41.593953 15219 raft_consensus.cc:3032] T ebd7cdd284a74741b338e9f4778502b5 P 620596b893d84291b88b90dc9e317612 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 02:00:41.593991 15219 raft_consensus.cc:729] T ebd7cdd284a74741b338e9f4778502b5 P 620596b893d84291b88b90dc9e317612 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:41.594157 15219 consensus_queue.cc:260] T ebd7cdd284a74741b338e9f4778502b5 P 620596b893d84291b88b90dc9e317612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.594270 15219 raft_consensus.cc:3037] T ebd7cdd284a74741b338e9f4778502b5 P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.594702 15087 raft_consensus.cc:1261] T ebd7cdd284a74741b338e9f4778502b5 P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Refusing update from remote peer 620596b893d84291b88b90dc9e317612: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20240503 02:00:41.595346 15219 raft_consensus.cc:2396] T ebd7cdd284a74741b338e9f4778502b5 P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate c05f7620bb884392931d3243a0b47a9e for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20240503 02:00:41.595772 15084 raft_consensus.cc:2396] T d805ee04df39406ba30b3b3b9e349507 P b602f6099cbb4758847e7e9550685372 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate c05f7620bb884392931d3243a0b47a9e for term 4 because replica has last-logged OpId of term: 3 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 6.
I20240503 02:00:41.596165 15351 raft_consensus.cc:1226] T ebd7cdd284a74741b338e9f4778502b5 P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Rejecting Update request from peer 620596b893d84291b88b90dc9e317612 for earlier term 3. Current term is 4. Ops: []
I20240503 02:00:41.596607 15285 leader_election.cc:302] T d805ee04df39406ba30b3b3b9e349507 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c05f7620bb884392931d3243a0b47a9e; no voters: b602f6099cbb4758847e7e9550685372, dcfdfa62edc34dab9525e2712d487582
I20240503 02:00:41.596896 17163 raft_consensus.cc:2726] T d805ee04df39406ba30b3b3b9e349507 P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240503 02:00:41.597084 15084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebd7cdd284a74741b338e9f4778502b5" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:41.597215 15084 raft_consensus.cc:3037] T ebd7cdd284a74741b338e9f4778502b5 P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.598302 15084 raft_consensus.cc:2396] T ebd7cdd284a74741b338e9f4778502b5 P b602f6099cbb4758847e7e9550685372 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate c05f7620bb884392931d3243a0b47a9e for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20240503 02:00:41.598690 15285 leader_election.cc:302] T ebd7cdd284a74741b338e9f4778502b5 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c05f7620bb884392931d3243a0b47a9e; no voters: 620596b893d84291b88b90dc9e317612, b602f6099cbb4758847e7e9550685372
I20240503 02:00:41.598902 17163 raft_consensus.cc:2726] T ebd7cdd284a74741b338e9f4778502b5 P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240503 02:00:41.599050 15349 leader_election.cc:288] T ebd7cdd284a74741b338e9f4778502b5 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 4 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
I20240503 02:00:41.599879 15349 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5d448658ed3b460fb1725a26a7c743e8"
dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
from {username='slave'} at 127.0.0.1:50864
I20240503 02:00:41.600023 15349 raft_consensus.cc:482] T 5d448658ed3b460fb1725a26a7c743e8 P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.600085 15349 raft_consensus.cc:3037] T 5d448658ed3b460fb1725a26a7c743e8 P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.601318 15349 raft_consensus.cc:504] T 5d448658ed3b460fb1725a26a7c743e8 P c05f7620bb884392931d3243a0b47a9e [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.601991 15214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6798a9ed0cf441aca7b8ae1b05c001f3" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:41.602082 15219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d448658ed3b460fb1725a26a7c743e8" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 5 candidate_status { last_received { term: 4 index: 9 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:41.602216 15219 raft_consensus.cc:3032] T 5d448658ed3b460fb1725a26a7c743e8 P 620596b893d84291b88b90dc9e317612 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 02:00:41.602264 15219 raft_consensus.cc:729] T 5d448658ed3b460fb1725a26a7c743e8 P 620596b893d84291b88b90dc9e317612 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:41.602443 15219 consensus_queue.cc:260] T 5d448658ed3b460fb1725a26a7c743e8 P 620596b893d84291b88b90dc9e317612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 4.9, Last appended by leader: 9, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.602644 15219 raft_consensus.cc:3037] T 5d448658ed3b460fb1725a26a7c743e8 P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.603788 15219 raft_consensus.cc:2454] T 5d448658ed3b460fb1725a26a7c743e8 P 620596b893d84291b88b90dc9e317612 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 5.
I20240503 02:00:41.604153 15286 leader_election.cc:302] T 5d448658ed3b460fb1725a26a7c743e8 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, c05f7620bb884392931d3243a0b47a9e; no voters:
I20240503 02:00:41.604393 17163 raft_consensus.cc:2781] T 5d448658ed3b460fb1725a26a7c743e8 P c05f7620bb884392931d3243a0b47a9e [term 5 FOLLOWER]: Leader election won for term 5
I20240503 02:00:41.604483 17163 raft_consensus.cc:686] T 5d448658ed3b460fb1725a26a7c743e8 P c05f7620bb884392931d3243a0b47a9e [term 5 LEADER]: Becoming Leader. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:41.604633 17163 consensus_queue.cc:237] T 5d448658ed3b460fb1725a26a7c743e8 P c05f7620bb884392931d3243a0b47a9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 4.9, Last appended by leader: 9, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.602135 15214 raft_consensus.cc:3037] T 6798a9ed0cf441aca7b8ae1b05c001f3 P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.606528 14811 catalog_manager.cc:5484] T 5d448658ed3b460fb1725a26a7c743e8 P c05f7620bb884392931d3243a0b47a9e reported cstate change: term changed from 4 to 5, leader changed from 620596b893d84291b88b90dc9e317612 (127.0.99.131) to c05f7620bb884392931d3243a0b47a9e (127.0.99.132). New cstate: current_term: 5 leader_uuid: "c05f7620bb884392931d3243a0b47a9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:41.607117 15214 raft_consensus.cc:2454] T 6798a9ed0cf441aca7b8ae1b05c001f3 P 620596b893d84291b88b90dc9e317612 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 5.
W20240503 02:00:41.607337 14915 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47950: Illegal state: replica dcfdfa62edc34dab9525e2712d487582 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.607461 15286 leader_election.cc:302] T 6798a9ed0cf441aca7b8ae1b05c001f3 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 620596b893d84291b88b90dc9e317612, c05f7620bb884392931d3243a0b47a9e; no voters: dcfdfa62edc34dab9525e2712d487582
I20240503 02:00:41.607751 17163 raft_consensus.cc:2781] T 6798a9ed0cf441aca7b8ae1b05c001f3 P c05f7620bb884392931d3243a0b47a9e [term 5 FOLLOWER]: Leader election won for term 5
I20240503 02:00:41.607851 17163 raft_consensus.cc:686] T 6798a9ed0cf441aca7b8ae1b05c001f3 P c05f7620bb884392931d3243a0b47a9e [term 5 LEADER]: Becoming Leader. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:41.607985 17163 consensus_queue.cc:237] T 6798a9ed0cf441aca7b8ae1b05c001f3 P c05f7620bb884392931d3243a0b47a9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.608597 15349 leader_election.cc:288] T 5d448658ed3b460fb1725a26a7c743e8 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 5 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817)
I20240503 02:00:41.609064 14956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d448658ed3b460fb1725a26a7c743e8" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 5 candidate_status { last_received { term: 4 index: 9 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:41.609192 14956 raft_consensus.cc:3037] T 5d448658ed3b460fb1725a26a7c743e8 P dcfdfa62edc34dab9525e2712d487582 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.610323 14956 raft_consensus.cc:2454] T 5d448658ed3b460fb1725a26a7c743e8 P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 5.
I20240503 02:00:41.611765 14812 catalog_manager.cc:5484] T 6798a9ed0cf441aca7b8ae1b05c001f3 P c05f7620bb884392931d3243a0b47a9e reported cstate change: term changed from 3 to 5, leader changed from dcfdfa62edc34dab9525e2712d487582 (127.0.99.129) to c05f7620bb884392931d3243a0b47a9e (127.0.99.132). New cstate: current_term: 5 leader_uuid: "c05f7620bb884392931d3243a0b47a9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } }
W20240503 02:00:41.614037 15048 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34032: Illegal state: replica b602f6099cbb4758847e7e9550685372 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.615554 15349 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "454a24fd6d964d73ae3d5057f90fe36d"
dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
from {username='slave'} at 127.0.0.1:50864
I20240503 02:00:41.617476 15349 raft_consensus.cc:482] T 454a24fd6d964d73ae3d5057f90fe36d P c05f7620bb884392931d3243a0b47a9e [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.617578 15349 raft_consensus.cc:3037] T 454a24fd6d964d73ae3d5057f90fe36d P c05f7620bb884392931d3243a0b47a9e [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.618733 15349 raft_consensus.cc:504] T 454a24fd6d964d73ae3d5057f90fe36d P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.619488 15349 leader_election.cc:288] T 454a24fd6d964d73ae3d5057f90fe36d P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 4 election: Requested vote from peers b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207), dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817)
I20240503 02:00:41.619612 14956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "454a24fd6d964d73ae3d5057f90fe36d" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:41.619639 15084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "454a24fd6d964d73ae3d5057f90fe36d" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:41.619763 14956 raft_consensus.cc:3037] T 454a24fd6d964d73ae3d5057f90fe36d P dcfdfa62edc34dab9525e2712d487582 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.619774 15084 raft_consensus.cc:3032] T 454a24fd6d964d73ae3d5057f90fe36d P b602f6099cbb4758847e7e9550685372 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 02:00:41.620040 15084 raft_consensus.cc:729] T 454a24fd6d964d73ae3d5057f90fe36d P b602f6099cbb4758847e7e9550685372 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.620262 15084 consensus_queue.cc:260] T 454a24fd6d964d73ae3d5057f90fe36d P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.620266 15349 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0e3bb299e61148fcb355641e0134b3df"
dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
from {username='slave'} at 127.0.0.1:50864
I20240503 02:00:41.620432 15349 raft_consensus.cc:482] T 0e3bb299e61148fcb355641e0134b3df P c05f7620bb884392931d3243a0b47a9e [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.620509 15349 raft_consensus.cc:3037] T 0e3bb299e61148fcb355641e0134b3df P c05f7620bb884392931d3243a0b47a9e [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.620532 15084 raft_consensus.cc:3037] T 454a24fd6d964d73ae3d5057f90fe36d P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.621305 14956 raft_consensus.cc:2396] T 454a24fd6d964d73ae3d5057f90fe36d P dcfdfa62edc34dab9525e2712d487582 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate c05f7620bb884392931d3243a0b47a9e for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 02:00:41.621569 15349 raft_consensus.cc:504] T 0e3bb299e61148fcb355641e0134b3df P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.621927 15084 raft_consensus.cc:2396] T 454a24fd6d964d73ae3d5057f90fe36d P b602f6099cbb4758847e7e9550685372 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate c05f7620bb884392931d3243a0b47a9e for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 02:00:41.622413 15087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e3bb299e61148fcb355641e0134b3df" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:41.622526 15087 raft_consensus.cc:3032] T 0e3bb299e61148fcb355641e0134b3df P b602f6099cbb4758847e7e9550685372 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 02:00:41.622576 15087 raft_consensus.cc:729] T 0e3bb299e61148fcb355641e0134b3df P b602f6099cbb4758847e7e9550685372 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.622545 15285 leader_election.cc:302] T 454a24fd6d964d73ae3d5057f90fe36d P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c05f7620bb884392931d3243a0b47a9e; no voters: b602f6099cbb4758847e7e9550685372, dcfdfa62edc34dab9525e2712d487582
I20240503 02:00:41.622720 15087 consensus_queue.cc:260] T 0e3bb299e61148fcb355641e0134b3df P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 3.8, Last appended by leader: 8, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.622789 17163 raft_consensus.cc:2726] T 454a24fd6d964d73ae3d5057f90fe36d P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240503 02:00:41.622856 15087 raft_consensus.cc:3037] T 0e3bb299e61148fcb355641e0134b3df P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.623095 15349 leader_election.cc:288] T 0e3bb299e61148fcb355641e0134b3df P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 4 election: Requested vote from peers b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207), dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817)
I20240503 02:00:41.623858 15087 raft_consensus.cc:2396] T 0e3bb299e61148fcb355641e0134b3df P b602f6099cbb4758847e7e9550685372 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate c05f7620bb884392931d3243a0b47a9e for term 4 because replica has last-logged OpId of term: 3 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20240503 02:00:41.623857 15349 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "590736f5f1ca4703821c409d882c2f77"
dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
from {username='slave'} at 127.0.0.1:50864
I20240503 02:00:41.624208 15349 raft_consensus.cc:482] T 590736f5f1ca4703821c409d882c2f77 P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.624328 15349 raft_consensus.cc:3037] T 590736f5f1ca4703821c409d882c2f77 P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.624331 14956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e3bb299e61148fcb355641e0134b3df" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:41.624519 14956 raft_consensus.cc:3037] T 0e3bb299e61148fcb355641e0134b3df P dcfdfa62edc34dab9525e2712d487582 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.625526 14956 raft_consensus.cc:2396] T 0e3bb299e61148fcb355641e0134b3df P dcfdfa62edc34dab9525e2712d487582 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate c05f7620bb884392931d3243a0b47a9e for term 4 because replica has last-logged OpId of term: 2 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20240503 02:00:41.625862 15284 leader_election.cc:302] T 0e3bb299e61148fcb355641e0134b3df P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c05f7620bb884392931d3243a0b47a9e; no voters: b602f6099cbb4758847e7e9550685372, dcfdfa62edc34dab9525e2712d487582
W20240503 02:00:41.625931 15048 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33982: Illegal state: replica b602f6099cbb4758847e7e9550685372 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.626050 17163 raft_consensus.cc:2726] T 0e3bb299e61148fcb355641e0134b3df P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240503 02:00:41.626314 15349 raft_consensus.cc:504] T 590736f5f1ca4703821c409d882c2f77 P c05f7620bb884392931d3243a0b47a9e [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.627421 14956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "590736f5f1ca4703821c409d882c2f77" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:41.627640 14956 raft_consensus.cc:3037] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.629175 14956 raft_consensus.cc:2396] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c05f7620bb884392931d3243a0b47a9e for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 02:00:41.632596 15349 leader_election.cc:288] T 590736f5f1ca4703821c409d882c2f77 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 5 election: Requested vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
I20240503 02:00:41.633147 15087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "590736f5f1ca4703821c409d882c2f77" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:41.633303 15087 raft_consensus.cc:3037] T 590736f5f1ca4703821c409d882c2f77 P b602f6099cbb4758847e7e9550685372 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.634274 15087 raft_consensus.cc:2454] T 590736f5f1ca4703821c409d882c2f77 P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 5.
I20240503 02:00:41.634766 15285 leader_election.cc:302] T 590736f5f1ca4703821c409d882c2f77 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b602f6099cbb4758847e7e9550685372, c05f7620bb884392931d3243a0b47a9e; no voters: dcfdfa62edc34dab9525e2712d487582
I20240503 02:00:41.635036 17163 raft_consensus.cc:2781] T 590736f5f1ca4703821c409d882c2f77 P c05f7620bb884392931d3243a0b47a9e [term 5 FOLLOWER]: Leader election won for term 5
I20240503 02:00:41.635149 17163 raft_consensus.cc:686] T 590736f5f1ca4703821c409d882c2f77 P c05f7620bb884392931d3243a0b47a9e [term 5 LEADER]: Becoming Leader. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:41.635298 17163 consensus_queue.cc:237] T 590736f5f1ca4703821c409d882c2f77 P c05f7620bb884392931d3243a0b47a9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.635458 15349 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "11275e009edd4354b0807b7f015023be"
dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
from {username='slave'} at 127.0.0.1:50864
I20240503 02:00:41.635586 15349 raft_consensus.cc:482] T 11275e009edd4354b0807b7f015023be P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.635634 15349 raft_consensus.cc:3037] T 11275e009edd4354b0807b7f015023be P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.636802 15349 raft_consensus.cc:504] T 11275e009edd4354b0807b7f015023be P c05f7620bb884392931d3243a0b47a9e [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.637197 15349 leader_election.cc:288] T 11275e009edd4354b0807b7f015023be P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 5 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817)
I20240503 02:00:41.638226 14816 catalog_manager.cc:5484] T 590736f5f1ca4703821c409d882c2f77 P c05f7620bb884392931d3243a0b47a9e reported cstate change: term changed from 3 to 5, leader changed from dcfdfa62edc34dab9525e2712d487582 (127.0.99.129) to c05f7620bb884392931d3243a0b47a9e (127.0.99.132). New cstate: current_term: 5 leader_uuid: "c05f7620bb884392931d3243a0b47a9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:41.638442 14956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11275e009edd4354b0807b7f015023be" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:41.638571 14956 raft_consensus.cc:3037] T 11275e009edd4354b0807b7f015023be P dcfdfa62edc34dab9525e2712d487582 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.638836 15349 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ca0d7d3e4d59496c9a9a59cde24b9247"
dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
from {username='slave'} at 127.0.0.1:50864
I20240503 02:00:41.638952 15349 raft_consensus.cc:482] T ca0d7d3e4d59496c9a9a59cde24b9247 P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.639007 15349 raft_consensus.cc:3037] T ca0d7d3e4d59496c9a9a59cde24b9247 P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.639750 14956 raft_consensus.cc:2396] T 11275e009edd4354b0807b7f015023be P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c05f7620bb884392931d3243a0b47a9e for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20240503 02:00:41.640056 15349 raft_consensus.cc:504] T ca0d7d3e4d59496c9a9a59cde24b9247 P c05f7620bb884392931d3243a0b47a9e [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.640591 15349 leader_election.cc:288] T ca0d7d3e4d59496c9a9a59cde24b9247 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 5 election: Requested vote from peers b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207), dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817)
I20240503 02:00:41.641479 15349 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5da91b7af68e4cf597dc89a017ee0197"
dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
from {username='slave'} at 127.0.0.1:50864
I20240503 02:00:41.641603 15349 raft_consensus.cc:482] T 5da91b7af68e4cf597dc89a017ee0197 P c05f7620bb884392931d3243a0b47a9e [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.641659 15349 raft_consensus.cc:3037] T 5da91b7af68e4cf597dc89a017ee0197 P c05f7620bb884392931d3243a0b47a9e [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.641908 14956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca0d7d3e4d59496c9a9a59cde24b9247" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 5 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:41.642038 14956 raft_consensus.cc:3037] T ca0d7d3e4d59496c9a9a59cde24b9247 P dcfdfa62edc34dab9525e2712d487582 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.642431 17064 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:41.642663 15349 raft_consensus.cc:504] T 5da91b7af68e4cf597dc89a017ee0197 P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.643056 15349 leader_election.cc:288] T 5da91b7af68e4cf597dc89a017ee0197 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 4 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
I20240503 02:00:41.643067 14956 raft_consensus.cc:2396] T ca0d7d3e4d59496c9a9a59cde24b9247 P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c05f7620bb884392931d3243a0b47a9e for term 5 because replica has last-logged OpId of term: 3 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 7.
I20240503 02:00:41.643396 15219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5da91b7af68e4cf597dc89a017ee0197" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:41.643523 15219 raft_consensus.cc:3032] T 5da91b7af68e4cf597dc89a017ee0197 P 620596b893d84291b88b90dc9e317612 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 02:00:41.643565 15219 raft_consensus.cc:729] T 5da91b7af68e4cf597dc89a017ee0197 P 620596b893d84291b88b90dc9e317612 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:41.643718 15219 consensus_queue.cc:260] T 5da91b7af68e4cf597dc89a017ee0197 P 620596b893d84291b88b90dc9e317612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.643854 15219 raft_consensus.cc:3037] T 5da91b7af68e4cf597dc89a017ee0197 P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Advancing to term 4
W20240503 02:00:41.643903 14915 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48020: Illegal state: replica dcfdfa62edc34dab9525e2712d487582 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.644562 15087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca0d7d3e4d59496c9a9a59cde24b9247" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 5 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:41.644695 15087 raft_consensus.cc:3032] T ca0d7d3e4d59496c9a9a59cde24b9247 P b602f6099cbb4758847e7e9550685372 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 02:00:41.644739 15087 raft_consensus.cc:729] T ca0d7d3e4d59496c9a9a59cde24b9247 P b602f6099cbb4758847e7e9550685372 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.644881 15087 consensus_queue.cc:260] T ca0d7d3e4d59496c9a9a59cde24b9247 P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.8, Last appended by leader: 8, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.644937 15219 raft_consensus.cc:2396] T 5da91b7af68e4cf597dc89a017ee0197 P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate c05f7620bb884392931d3243a0b47a9e for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 02:00:41.645023 15087 raft_consensus.cc:3037] T ca0d7d3e4d59496c9a9a59cde24b9247 P b602f6099cbb4758847e7e9550685372 [term 4 FOLLOWER]: Advancing to term 5
W20240503 02:00:41.645138 15176 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57618: Illegal state: replica 620596b893d84291b88b90dc9e317612 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.645687 15214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11275e009edd4354b0807b7f015023be" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:41.645795 15214 raft_consensus.cc:3032] T 11275e009edd4354b0807b7f015023be P 620596b893d84291b88b90dc9e317612 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 02:00:41.645833 15214 raft_consensus.cc:729] T 11275e009edd4354b0807b7f015023be P 620596b893d84291b88b90dc9e317612 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:41.646036 15087 raft_consensus.cc:2396] T ca0d7d3e4d59496c9a9a59cde24b9247 P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c05f7620bb884392931d3243a0b47a9e for term 5 because replica has last-logged OpId of term: 4 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 7.
I20240503 02:00:41.646004 15214 consensus_queue.cc:260] T 11275e009edd4354b0807b7f015023be P 620596b893d84291b88b90dc9e317612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.646440 15285 leader_election.cc:302] T ca0d7d3e4d59496c9a9a59cde24b9247 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c05f7620bb884392931d3243a0b47a9e; no voters: b602f6099cbb4758847e7e9550685372, dcfdfa62edc34dab9525e2712d487582
I20240503 02:00:41.646718 16986 raft_consensus.cc:2726] T ca0d7d3e4d59496c9a9a59cde24b9247 P c05f7620bb884392931d3243a0b47a9e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240503 02:00:41.646219 15214 raft_consensus.cc:3037] T 11275e009edd4354b0807b7f015023be P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.647166 15087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5da91b7af68e4cf597dc89a017ee0197" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:41.647279 15087 raft_consensus.cc:3037] T 5da91b7af68e4cf597dc89a017ee0197 P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.647981 15214 raft_consensus.cc:2396] T 11275e009edd4354b0807b7f015023be P 620596b893d84291b88b90dc9e317612 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c05f7620bb884392931d3243a0b47a9e for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20240503 02:00:41.648011 15087 raft_consensus.cc:2396] T 5da91b7af68e4cf597dc89a017ee0197 P b602f6099cbb4758847e7e9550685372 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate c05f7620bb884392931d3243a0b47a9e for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 02:00:41.648211 15349 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0989d98050744df6a1c699cb86ada505"
dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
from {username='slave'} at 127.0.0.1:50864
I20240503 02:00:41.648356 15349 raft_consensus.cc:482] T 0989d98050744df6a1c699cb86ada505 P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.648437 15285 leader_election.cc:302] T 5da91b7af68e4cf597dc89a017ee0197 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c05f7620bb884392931d3243a0b47a9e; no voters: 620596b893d84291b88b90dc9e317612, b602f6099cbb4758847e7e9550685372
I20240503 02:00:41.648481 15349 raft_consensus.cc:3037] T 0989d98050744df6a1c699cb86ada505 P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.648674 15286 leader_election.cc:302] T 11275e009edd4354b0807b7f015023be P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c05f7620bb884392931d3243a0b47a9e; no voters: 620596b893d84291b88b90dc9e317612, dcfdfa62edc34dab9525e2712d487582
I20240503 02:00:41.648748 16986 raft_consensus.cc:2726] T 5da91b7af68e4cf597dc89a017ee0197 P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240503 02:00:41.648816 16986 raft_consensus.cc:2726] T 11275e009edd4354b0807b7f015023be P c05f7620bb884392931d3243a0b47a9e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240503 02:00:41.649626 15349 raft_consensus.cc:504] T 0989d98050744df6a1c699cb86ada505 P c05f7620bb884392931d3243a0b47a9e [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.650153 15349 leader_election.cc:288] T 0989d98050744df6a1c699cb86ada505 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 5 election: Requested vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
I20240503 02:00:41.651033 14956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0989d98050744df6a1c699cb86ada505" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:41.651165 14956 raft_consensus.cc:3037] T 0989d98050744df6a1c699cb86ada505 P dcfdfa62edc34dab9525e2712d487582 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.652029 14956 raft_consensus.cc:2396] T 0989d98050744df6a1c699cb86ada505 P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c05f7620bb884392931d3243a0b47a9e for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 02:00:41.654809 15200 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:00:41.657527 14956 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7b4f203f4048490ba599353a4cd92748"
dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
from {username='slave'} at 127.0.0.1:47980
I20240503 02:00:41.657701 14956 raft_consensus.cc:482] T 7b4f203f4048490ba599353a4cd92748 P dcfdfa62edc34dab9525e2712d487582 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.657773 14956 raft_consensus.cc:3037] T 7b4f203f4048490ba599353a4cd92748 P dcfdfa62edc34dab9525e2712d487582 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.659545 14956 raft_consensus.cc:504] T 7b4f203f4048490ba599353a4cd92748 P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.660339 15087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0989d98050744df6a1c699cb86ada505" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:41.660404 15084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4f203f4048490ba599353a4cd92748" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:41.660496 15087 raft_consensus.cc:3037] T 0989d98050744df6a1c699cb86ada505 P b602f6099cbb4758847e7e9550685372 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.660534 15084 raft_consensus.cc:3037] T 7b4f203f4048490ba599353a4cd92748 P b602f6099cbb4758847e7e9550685372 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.661502 15084 raft_consensus.cc:2396] T 7b4f203f4048490ba599353a4cd92748 P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate dcfdfa62edc34dab9525e2712d487582 for term 5 because replica has last-logged OpId of term: 3 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20240503 02:00:41.661595 15087 raft_consensus.cc:2454] T 0989d98050744df6a1c699cb86ada505 P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 5.
I20240503 02:00:41.661993 14956 leader_election.cc:288] T 7b4f203f4048490ba599353a4cd92748 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 5 election: Requested vote from peers b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207), 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437)
I20240503 02:00:41.662590 15214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4f203f4048490ba599353a4cd92748" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:41.662750 15214 raft_consensus.cc:3037] T 7b4f203f4048490ba599353a4cd92748 P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.662952 14956 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "21d92d412a5247e8907c5c41d253ad2e"
dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
from {username='slave'} at 127.0.0.1:47980
I20240503 02:00:41.663139 14956 raft_consensus.cc:482] T 21d92d412a5247e8907c5c41d253ad2e P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.663245 14956 raft_consensus.cc:3037] T 21d92d412a5247e8907c5c41d253ad2e P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.663836 15214 raft_consensus.cc:2454] T 7b4f203f4048490ba599353a4cd92748 P 620596b893d84291b88b90dc9e317612 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 5.
I20240503 02:00:41.664477 14956 raft_consensus.cc:504] T 21d92d412a5247e8907c5c41d253ad2e P dcfdfa62edc34dab9525e2712d487582 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.665062 14956 leader_election.cc:288] T 21d92d412a5247e8907c5c41d253ad2e P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 6 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:41.666270 15285 leader_election.cc:302] T 0989d98050744df6a1c699cb86ada505 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b602f6099cbb4758847e7e9550685372, c05f7620bb884392931d3243a0b47a9e; no voters: dcfdfa62edc34dab9525e2712d487582
I20240503 02:00:41.666584 17163 raft_consensus.cc:2781] T 0989d98050744df6a1c699cb86ada505 P c05f7620bb884392931d3243a0b47a9e [term 5 FOLLOWER]: Leader election won for term 5
I20240503 02:00:41.666659 17163 raft_consensus.cc:686] T 0989d98050744df6a1c699cb86ada505 P c05f7620bb884392931d3243a0b47a9e [term 5 LEADER]: Becoming Leader. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:41.666747 14893 leader_election.cc:302] T 7b4f203f4048490ba599353a4cd92748 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 620596b893d84291b88b90dc9e317612, dcfdfa62edc34dab9525e2712d487582; no voters: b602f6099cbb4758847e7e9550685372
I20240503 02:00:41.666795 17163 consensus_queue.cc:237] T 0989d98050744df6a1c699cb86ada505 P c05f7620bb884392931d3243a0b47a9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.666927 17225 raft_consensus.cc:2781] T 7b4f203f4048490ba599353a4cd92748 P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 02:00:41.667004 17225 raft_consensus.cc:686] T 7b4f203f4048490ba599353a4cd92748 P dcfdfa62edc34dab9525e2712d487582 [term 5 LEADER]: Becoming Leader. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:41.667130 17225 consensus_queue.cc:237] T 7b4f203f4048490ba599353a4cd92748 P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.668730 15214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21d92d412a5247e8907c5c41d253ad2e" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:41.668887 15214 raft_consensus.cc:3037] T 21d92d412a5247e8907c5c41d253ad2e P 620596b893d84291b88b90dc9e317612 [term 5 FOLLOWER]: Advancing to term 6
W20240503 02:00:41.668932 14915 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47950: Illegal state: replica dcfdfa62edc34dab9525e2712d487582 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.669911 14816 catalog_manager.cc:5484] T 0989d98050744df6a1c699cb86ada505 P c05f7620bb884392931d3243a0b47a9e reported cstate change: term changed from 3 to 5, leader changed from dcfdfa62edc34dab9525e2712d487582 (127.0.99.129) to c05f7620bb884392931d3243a0b47a9e (127.0.99.132). New cstate: current_term: 5 leader_uuid: "c05f7620bb884392931d3243a0b47a9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:41.670104 15214 raft_consensus.cc:2396] T 21d92d412a5247e8907c5c41d253ad2e P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate dcfdfa62edc34dab9525e2712d487582 for term 6 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20240503 02:00:41.671088 14811 catalog_manager.cc:5484] T 7b4f203f4048490ba599353a4cd92748 P dcfdfa62edc34dab9525e2712d487582 reported cstate change: term changed from 3 to 5, leader changed from b602f6099cbb4758847e7e9550685372 (127.0.99.130) to dcfdfa62edc34dab9525e2712d487582 (127.0.99.129). New cstate: current_term: 5 leader_uuid: "dcfdfa62edc34dab9525e2712d487582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:41.672498 15069 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:41.675524 15349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21d92d412a5247e8907c5c41d253ad2e" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.675699 15349 raft_consensus.cc:3032] T 21d92d412a5247e8907c5c41d253ad2e P c05f7620bb884392931d3243a0b47a9e [term 5 LEADER]: Stepping down as leader of term 5
I20240503 02:00:41.675745 15349 raft_consensus.cc:729] T 21d92d412a5247e8907c5c41d253ad2e P c05f7620bb884392931d3243a0b47a9e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:41.675927 15349 consensus_queue.cc:260] T 21d92d412a5247e8907c5c41d253ad2e P c05f7620bb884392931d3243a0b47a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.676082 15349 raft_consensus.cc:3037] T 21d92d412a5247e8907c5c41d253ad2e P c05f7620bb884392931d3243a0b47a9e [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.676723 15087 raft_consensus.cc:1261] T 590736f5f1ca4703821c409d882c2f77 P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Refusing update from remote peer c05f7620bb884392931d3243a0b47a9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240503 02:00:41.677152 17163 consensus_queue.cc:1035] T 590736f5f1ca4703821c409d882c2f77 P c05f7620bb884392931d3243a0b47a9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:41.678251 14956 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "830da5cb4972453cb9bf2145a0839ced"
dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
from {username='slave'} at 127.0.0.1:47980
I20240503 02:00:41.678400 14956 raft_consensus.cc:482] T 830da5cb4972453cb9bf2145a0839ced P dcfdfa62edc34dab9525e2712d487582 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.678457 14956 raft_consensus.cc:3037] T 830da5cb4972453cb9bf2145a0839ced P dcfdfa62edc34dab9525e2712d487582 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.680028 15349 raft_consensus.cc:2396] T 21d92d412a5247e8907c5c41d253ad2e P c05f7620bb884392931d3243a0b47a9e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate dcfdfa62edc34dab9525e2712d487582 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20240503 02:00:41.680917 17130 heartbeater.cc:503] Master 127.0.99.190:38675 was elected leader, sending a full tablet report...
I20240503 02:00:41.681486 14954 raft_consensus.cc:1261] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Refusing update from remote peer c05f7620bb884392931d3243a0b47a9e: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240503 02:00:41.681861 14956 raft_consensus.cc:504] T 830da5cb4972453cb9bf2145a0839ced P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.682085 14893 leader_election.cc:302] T 21d92d412a5247e8907c5c41d253ad2e P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dcfdfa62edc34dab9525e2712d487582; no voters: 620596b893d84291b88b90dc9e317612, c05f7620bb884392931d3243a0b47a9e
I20240503 02:00:41.682416 17308 raft_consensus.cc:2726] T 21d92d412a5247e8907c5c41d253ad2e P dcfdfa62edc34dab9525e2712d487582 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240503 02:00:41.682600 15087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "830da5cb4972453cb9bf2145a0839ced" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:41.682722 15087 raft_consensus.cc:3037] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.682819 17163 consensus_queue.cc:1035] T 590736f5f1ca4703821c409d882c2f77 P c05f7620bb884392931d3243a0b47a9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:41.682931 17163 consensus_queue.cc:1225] T 590736f5f1ca4703821c409d882c2f77 P c05f7620bb884392931d3243a0b47a9e [LEADER]: Peer dcfdfa62edc34dab9525e2712d487582 log is divergent from this leader: its last log entry 3.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240503 02:00:41.683755 14954 pending_rounds.cc:76] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582: Aborting all ops after (but not including) 3
I20240503 02:00:41.683820 14954 pending_rounds.cc:97] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582: Aborting uncommitted NO_OP operation due to leader change: 3.4
I20240503 02:00:41.683884 14954 raft_consensus.cc:2866] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 02:00:41.683899 15214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "830da5cb4972453cb9bf2145a0839ced" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:41.684043 15214 raft_consensus.cc:3032] T 830da5cb4972453cb9bf2145a0839ced P 620596b893d84291b88b90dc9e317612 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 02:00:41.684092 15214 raft_consensus.cc:729] T 830da5cb4972453cb9bf2145a0839ced P 620596b893d84291b88b90dc9e317612 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:41.684284 15214 consensus_queue.cc:260] T 830da5cb4972453cb9bf2145a0839ced P 620596b893d84291b88b90dc9e317612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.684466 15214 raft_consensus.cc:3037] T 830da5cb4972453cb9bf2145a0839ced P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.685911 15214 raft_consensus.cc:2396] T 830da5cb4972453cb9bf2145a0839ced P 620596b893d84291b88b90dc9e317612 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate dcfdfa62edc34dab9525e2712d487582 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20240503 02:00:41.686468 15087 raft_consensus.cc:2396] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate dcfdfa62edc34dab9525e2712d487582 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20240503 02:00:41.686810 14892 leader_election.cc:302] T 830da5cb4972453cb9bf2145a0839ced P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dcfdfa62edc34dab9525e2712d487582; no voters: 620596b893d84291b88b90dc9e317612, b602f6099cbb4758847e7e9550685372
I20240503 02:00:41.687511 17308 raft_consensus.cc:2726] T 830da5cb4972453cb9bf2145a0839ced P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240503 02:00:41.689056 16931 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:41.689062 14956 leader_election.cc:288] T 830da5cb4972453cb9bf2145a0839ced P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 5 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
W20240503 02:00:41.689884 15048 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33982: Illegal state: replica b602f6099cbb4758847e7e9550685372 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.689862 14956 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "50c5abd860ae4b09bb289e9c7550c84e"
dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
from {username='slave'} at 127.0.0.1:47980
I20240503 02:00:41.690064 14956 raft_consensus.cc:482] T 50c5abd860ae4b09bb289e9c7550c84e P dcfdfa62edc34dab9525e2712d487582 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.690130 14956 raft_consensus.cc:3037] T 50c5abd860ae4b09bb289e9c7550c84e P dcfdfa62edc34dab9525e2712d487582 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.692909 14956 raft_consensus.cc:504] T 50c5abd860ae4b09bb289e9c7550c84e P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.693361 14956 leader_election.cc:288] T 50c5abd860ae4b09bb289e9c7550c84e P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 5 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:41.694473 14954 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6d2a1296a54a4b92a03c12954de0cc34"
dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
from {username='slave'} at 127.0.0.1:47980
I20240503 02:00:41.694602 14954 raft_consensus.cc:482] T 6d2a1296a54a4b92a03c12954de0cc34 P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.694658 14954 raft_consensus.cc:3037] T 6d2a1296a54a4b92a03c12954de0cc34 P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.695135 15349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50c5abd860ae4b09bb289e9c7550c84e" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 5 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.695401 15349 raft_consensus.cc:3032] T 50c5abd860ae4b09bb289e9c7550c84e P c05f7620bb884392931d3243a0b47a9e [term 4 LEADER]: Stepping down as leader of term 4
I20240503 02:00:41.695420 16800 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:41.695463 15349 raft_consensus.cc:729] T 50c5abd860ae4b09bb289e9c7550c84e P c05f7620bb884392931d3243a0b47a9e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:41.695636 15349 consensus_queue.cc:260] T 50c5abd860ae4b09bb289e9c7550c84e P c05f7620bb884392931d3243a0b47a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.695685 14954 raft_consensus.cc:504] T 6d2a1296a54a4b92a03c12954de0cc34 P dcfdfa62edc34dab9525e2712d487582 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.695811 15349 raft_consensus.cc:3037] T 50c5abd860ae4b09bb289e9c7550c84e P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.696056 14954 leader_election.cc:288] T 6d2a1296a54a4b92a03c12954de0cc34 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 6 election: Requested vote from peers c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
I20240503 02:00:41.697168 15349 raft_consensus.cc:2396] T 50c5abd860ae4b09bb289e9c7550c84e P c05f7620bb884392931d3243a0b47a9e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate dcfdfa62edc34dab9525e2712d487582 for term 5 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 6.
I20240503 02:00:41.697789 14954 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7fc3330dfd0644f6a303659b960124f4"
dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
from {username='slave'} at 127.0.0.1:47980
I20240503 02:00:41.697950 14954 raft_consensus.cc:482] T 7fc3330dfd0644f6a303659b960124f4 P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.698006 14954 raft_consensus.cc:3037] T 7fc3330dfd0644f6a303659b960124f4 P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Advancing to term 6
W20240503 02:00:41.698765 14915 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48020: Illegal state: replica dcfdfa62edc34dab9525e2712d487582 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.699069 14954 raft_consensus.cc:504] T 7fc3330dfd0644f6a303659b960124f4 P dcfdfa62edc34dab9525e2712d487582 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.699306 15214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50c5abd860ae4b09bb289e9c7550c84e" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 5 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:41.699514 15214 raft_consensus.cc:3037] T 50c5abd860ae4b09bb289e9c7550c84e P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.699728 15087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2a1296a54a4b92a03c12954de0cc34" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 6 candidate_status { last_received { term: 3 index: 10 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:41.699892 15087 raft_consensus.cc:3037] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.701015 15214 raft_consensus.cc:2396] T 50c5abd860ae4b09bb289e9c7550c84e P 620596b893d84291b88b90dc9e317612 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate dcfdfa62edc34dab9525e2712d487582 for term 5 because replica has last-logged OpId of term: 3 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 6.
I20240503 02:00:41.701491 14893 leader_election.cc:302] T 50c5abd860ae4b09bb289e9c7550c84e P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dcfdfa62edc34dab9525e2712d487582; no voters: 620596b893d84291b88b90dc9e317612, c05f7620bb884392931d3243a0b47a9e
I20240503 02:00:41.701859 17308 raft_consensus.cc:2726] T 50c5abd860ae4b09bb289e9c7550c84e P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240503 02:00:41.702332 15349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2a1296a54a4b92a03c12954de0cc34" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 6 candidate_status { last_received { term: 3 index: 10 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.702581 15349 raft_consensus.cc:3032] T 6d2a1296a54a4b92a03c12954de0cc34 P c05f7620bb884392931d3243a0b47a9e [term 5 LEADER]: Stepping down as leader of term 5
I20240503 02:00:41.702674 15349 raft_consensus.cc:729] T 6d2a1296a54a4b92a03c12954de0cc34 P c05f7620bb884392931d3243a0b47a9e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:41.703014 15349 consensus_queue.cc:260] T 6d2a1296a54a4b92a03c12954de0cc34 P c05f7620bb884392931d3243a0b47a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 5.11, Last appended by leader: 11, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.703316 15349 raft_consensus.cc:3037] T 6d2a1296a54a4b92a03c12954de0cc34 P c05f7620bb884392931d3243a0b47a9e [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.705574 15214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fc3330dfd0644f6a303659b960124f4" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 6 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:41.705915 15214 raft_consensus.cc:3037] T 7fc3330dfd0644f6a303659b960124f4 P 620596b893d84291b88b90dc9e317612 [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.707209 15214 raft_consensus.cc:2396] T 7fc3330dfd0644f6a303659b960124f4 P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate dcfdfa62edc34dab9525e2712d487582 for term 6 because replica has last-logged OpId of term: 4 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 12.
I20240503 02:00:41.707264 14954 leader_election.cc:288] T 7fc3330dfd0644f6a303659b960124f4 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 6 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:41.707517 15349 raft_consensus.cc:2396] T 6d2a1296a54a4b92a03c12954de0cc34 P c05f7620bb884392931d3243a0b47a9e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate dcfdfa62edc34dab9525e2712d487582 for term 6 because replica has last-logged OpId of term: 5 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 10.
I20240503 02:00:41.707774 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fc3330dfd0644f6a303659b960124f4" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 6 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.707896 15351 raft_consensus.cc:3032] T 7fc3330dfd0644f6a303659b960124f4 P c05f7620bb884392931d3243a0b47a9e [term 5 LEADER]: Stepping down as leader of term 5
I20240503 02:00:41.707945 15351 raft_consensus.cc:729] T 7fc3330dfd0644f6a303659b960124f4 P c05f7620bb884392931d3243a0b47a9e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:41.708120 15351 consensus_queue.cc:260] T 7fc3330dfd0644f6a303659b960124f4 P c05f7620bb884392931d3243a0b47a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 5.13, Last appended by leader: 13, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.708179 14957 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6798a9ed0cf441aca7b8ae1b05c001f3"
dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
from {username='slave'} at 127.0.0.1:47980
I20240503 02:00:41.708298 15351 raft_consensus.cc:3037] T 7fc3330dfd0644f6a303659b960124f4 P c05f7620bb884392931d3243a0b47a9e [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.708328 14957 raft_consensus.cc:482] T 6798a9ed0cf441aca7b8ae1b05c001f3 P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.708413 14957 raft_consensus.cc:3037] T 6798a9ed0cf441aca7b8ae1b05c001f3 P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.709200 15351 raft_consensus.cc:2396] T 7fc3330dfd0644f6a303659b960124f4 P c05f7620bb884392931d3243a0b47a9e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate dcfdfa62edc34dab9525e2712d487582 for term 6 because replica has last-logged OpId of term: 5 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 12.
I20240503 02:00:41.709412 14957 raft_consensus.cc:504] T 6798a9ed0cf441aca7b8ae1b05c001f3 P dcfdfa62edc34dab9525e2712d487582 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
W20240503 02:00:41.707425 15176 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57688: Illegal state: replica 620596b893d84291b88b90dc9e317612 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.709636 14893 leader_election.cc:302] T 7fc3330dfd0644f6a303659b960124f4 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dcfdfa62edc34dab9525e2712d487582; no voters: 620596b893d84291b88b90dc9e317612, c05f7620bb884392931d3243a0b47a9e
I20240503 02:00:41.709817 17308 raft_consensus.cc:2726] T 7fc3330dfd0644f6a303659b960124f4 P dcfdfa62edc34dab9525e2712d487582 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240503 02:00:41.709846 14957 leader_election.cc:288] T 6798a9ed0cf441aca7b8ae1b05c001f3 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 6 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:41.710529 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6798a9ed0cf441aca7b8ae1b05c001f3" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.710647 15351 raft_consensus.cc:3032] T 6798a9ed0cf441aca7b8ae1b05c001f3 P c05f7620bb884392931d3243a0b47a9e [term 5 LEADER]: Stepping down as leader of term 5
I20240503 02:00:41.710686 15351 raft_consensus.cc:729] T 6798a9ed0cf441aca7b8ae1b05c001f3 P c05f7620bb884392931d3243a0b47a9e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:41.710703 14957 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0989d98050744df6a1c699cb86ada505"
dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
from {username='slave'} at 127.0.0.1:47980
I20240503 02:00:41.710829 14957 raft_consensus.cc:482] T 0989d98050744df6a1c699cb86ada505 P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.710827 15351 consensus_queue.cc:260] T 6798a9ed0cf441aca7b8ae1b05c001f3 P c05f7620bb884392931d3243a0b47a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.710882 14957 raft_consensus.cc:3037] T 0989d98050744df6a1c699cb86ada505 P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.710955 15351 raft_consensus.cc:3037] T 6798a9ed0cf441aca7b8ae1b05c001f3 P c05f7620bb884392931d3243a0b47a9e [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.712013 15351 raft_consensus.cc:2396] T 6798a9ed0cf441aca7b8ae1b05c001f3 P c05f7620bb884392931d3243a0b47a9e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate dcfdfa62edc34dab9525e2712d487582 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240503 02:00:41.711956 14957 raft_consensus.cc:504] T 0989d98050744df6a1c699cb86ada505 P dcfdfa62edc34dab9525e2712d487582 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.712371 14957 leader_election.cc:288] T 0989d98050744df6a1c699cb86ada505 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 6 election: Requested vote from peers c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
I20240503 02:00:41.712733 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0989d98050744df6a1c699cb86ada505" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.712834 15351 raft_consensus.cc:3032] T 0989d98050744df6a1c699cb86ada505 P c05f7620bb884392931d3243a0b47a9e [term 5 LEADER]: Stepping down as leader of term 5
I20240503 02:00:41.712874 15351 raft_consensus.cc:729] T 0989d98050744df6a1c699cb86ada505 P c05f7620bb884392931d3243a0b47a9e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:41.713048 15351 consensus_queue.cc:260] T 0989d98050744df6a1c699cb86ada505 P c05f7620bb884392931d3243a0b47a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.713184 15351 raft_consensus.cc:3037] T 0989d98050744df6a1c699cb86ada505 P c05f7620bb884392931d3243a0b47a9e [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.713271 15088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0989d98050744df6a1c699cb86ada505" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:41.713431 15088 raft_consensus.cc:3037] T 0989d98050744df6a1c699cb86ada505 P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.713881 14957 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d805ee04df39406ba30b3b3b9e349507"
dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
from {username='slave'} at 127.0.0.1:47980
I20240503 02:00:41.714051 14957 raft_consensus.cc:482] T d805ee04df39406ba30b3b3b9e349507 P dcfdfa62edc34dab9525e2712d487582 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.714105 14957 raft_consensus.cc:3037] T d805ee04df39406ba30b3b3b9e349507 P dcfdfa62edc34dab9525e2712d487582 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.714269 15351 raft_consensus.cc:2396] T 0989d98050744df6a1c699cb86ada505 P c05f7620bb884392931d3243a0b47a9e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate dcfdfa62edc34dab9525e2712d487582 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
W20240503 02:00:41.714440 15048 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34032: Illegal state: replica b602f6099cbb4758847e7e9550685372 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.714769 15214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6798a9ed0cf441aca7b8ae1b05c001f3" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:41.714897 15214 raft_consensus.cc:3037] T 6798a9ed0cf441aca7b8ae1b05c001f3 P 620596b893d84291b88b90dc9e317612 [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.715080 14957 raft_consensus.cc:504] T d805ee04df39406ba30b3b3b9e349507 P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.715807 15089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d805ee04df39406ba30b3b3b9e349507" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 5 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:41.715844 15214 raft_consensus.cc:2454] T 6798a9ed0cf441aca7b8ae1b05c001f3 P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 6.
I20240503 02:00:41.715932 15089 raft_consensus.cc:3037] T d805ee04df39406ba30b3b3b9e349507 P b602f6099cbb4758847e7e9550685372 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.716012 14957 leader_election.cc:288] T d805ee04df39406ba30b3b3b9e349507 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 5 election: Requested vote from peers c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
I20240503 02:00:41.716135 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d805ee04df39406ba30b3b3b9e349507" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 5 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.716236 15351 raft_consensus.cc:3037] T d805ee04df39406ba30b3b3b9e349507 P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.716755 14958 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2b757c430aaa42f382d174ca57f5fcfc"
dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
from {username='slave'} at 127.0.0.1:47980
I20240503 02:00:41.716877 14958 raft_consensus.cc:482] T 2b757c430aaa42f382d174ca57f5fcfc P dcfdfa62edc34dab9525e2712d487582 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.716926 14958 raft_consensus.cc:3037] T 2b757c430aaa42f382d174ca57f5fcfc P dcfdfa62edc34dab9525e2712d487582 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.717172 15351 raft_consensus.cc:2454] T d805ee04df39406ba30b3b3b9e349507 P c05f7620bb884392931d3243a0b47a9e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 5.
I20240503 02:00:41.717864 14958 raft_consensus.cc:504] T 2b757c430aaa42f382d174ca57f5fcfc P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.718158 15088 raft_consensus.cc:2454] T 0989d98050744df6a1c699cb86ada505 P b602f6099cbb4758847e7e9550685372 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 6.
I20240503 02:00:41.718250 14958 leader_election.cc:288] T 2b757c430aaa42f382d174ca57f5fcfc P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 5 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
I20240503 02:00:41.718724 14892 leader_election.cc:302] T 0989d98050744df6a1c699cb86ada505 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b602f6099cbb4758847e7e9550685372, dcfdfa62edc34dab9525e2712d487582; no voters: c05f7620bb884392931d3243a0b47a9e
I20240503 02:00:41.719031 17308 raft_consensus.cc:2781] T 0989d98050744df6a1c699cb86ada505 P dcfdfa62edc34dab9525e2712d487582 [term 6 FOLLOWER]: Leader election won for term 6
I20240503 02:00:41.719204 17308 raft_consensus.cc:686] T 0989d98050744df6a1c699cb86ada505 P dcfdfa62edc34dab9525e2712d487582 [term 6 LEADER]: Becoming Leader. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:41.719260 15089 raft_consensus.cc:2396] T d805ee04df39406ba30b3b3b9e349507 P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate dcfdfa62edc34dab9525e2712d487582 for term 5 because replica has last-logged OpId of term: 3 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 7.
I20240503 02:00:41.719355 17308 consensus_queue.cc:237] T 0989d98050744df6a1c699cb86ada505 P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
W20240503 02:00:41.717144 15176 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57618: Illegal state: replica 620596b893d84291b88b90dc9e317612 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.719449 14958 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5d448658ed3b460fb1725a26a7c743e8"
dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
from {username='slave'} at 127.0.0.1:47980
I20240503 02:00:41.719600 14958 raft_consensus.cc:482] T 5d448658ed3b460fb1725a26a7c743e8 P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.719645 14958 raft_consensus.cc:3037] T 5d448658ed3b460fb1725a26a7c743e8 P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.720320 15088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b757c430aaa42f382d174ca57f5fcfc" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 5 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:41.720479 15088 raft_consensus.cc:3037] T 2b757c430aaa42f382d174ca57f5fcfc P b602f6099cbb4758847e7e9550685372 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.720698 14958 raft_consensus.cc:504] T 5d448658ed3b460fb1725a26a7c743e8 P dcfdfa62edc34dab9525e2712d487582 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.721555 15088 raft_consensus.cc:2396] T 2b757c430aaa42f382d174ca57f5fcfc P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate dcfdfa62edc34dab9525e2712d487582 for term 5 because replica has last-logged OpId of term: 3 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 7.
I20240503 02:00:41.721033 14958 leader_election.cc:288] T 5d448658ed3b460fb1725a26a7c743e8 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 6 election: Requested vote from peers c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009), 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437)
I20240503 02:00:41.723357 14958 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ca0d7d3e4d59496c9a9a59cde24b9247"
dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
from {username='slave'} at 127.0.0.1:47980
I20240503 02:00:41.723496 14958 raft_consensus.cc:482] T ca0d7d3e4d59496c9a9a59cde24b9247 P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.723556 14958 raft_consensus.cc:3037] T ca0d7d3e4d59496c9a9a59cde24b9247 P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.724964 14958 raft_consensus.cc:504] T ca0d7d3e4d59496c9a9a59cde24b9247 P dcfdfa62edc34dab9525e2712d487582 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.725383 14958 leader_election.cc:288] T ca0d7d3e4d59496c9a9a59cde24b9247 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 6 election: Requested vote from peers c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
I20240503 02:00:41.727221 15214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b757c430aaa42f382d174ca57f5fcfc" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 5 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:41.727386 15214 raft_consensus.cc:3037] T 2b757c430aaa42f382d174ca57f5fcfc P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.728024 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d448658ed3b460fb1725a26a7c743e8" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 6 candidate_status { last_received { term: 4 index: 9 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.728513 15351 raft_consensus.cc:3032] T 5d448658ed3b460fb1725a26a7c743e8 P c05f7620bb884392931d3243a0b47a9e [term 5 LEADER]: Stepping down as leader of term 5
I20240503 02:00:41.728590 15351 raft_consensus.cc:729] T 5d448658ed3b460fb1725a26a7c743e8 P c05f7620bb884392931d3243a0b47a9e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:41.728753 15351 consensus_queue.cc:260] T 5d448658ed3b460fb1725a26a7c743e8 P c05f7620bb884392931d3243a0b47a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 5.10, Last appended by leader: 10, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.728950 15351 raft_consensus.cc:3037] T 5d448658ed3b460fb1725a26a7c743e8 P c05f7620bb884392931d3243a0b47a9e [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.729028 15214 raft_consensus.cc:2454] T 2b757c430aaa42f382d174ca57f5fcfc P 620596b893d84291b88b90dc9e317612 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 5.
I20240503 02:00:41.730811 15351 raft_consensus.cc:2396] T 5d448658ed3b460fb1725a26a7c743e8 P c05f7620bb884392931d3243a0b47a9e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate dcfdfa62edc34dab9525e2712d487582 for term 6 because replica has last-logged OpId of term: 5 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 9.
I20240503 02:00:41.731665 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca0d7d3e4d59496c9a9a59cde24b9247" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 6 candidate_status { last_received { term: 3 index: 9 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.731796 15351 raft_consensus.cc:3037] T ca0d7d3e4d59496c9a9a59cde24b9247 P c05f7620bb884392931d3243a0b47a9e [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.731781 15214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d448658ed3b460fb1725a26a7c743e8" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 6 candidate_status { last_received { term: 4 index: 9 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:41.732139 15214 raft_consensus.cc:3037] T 5d448658ed3b460fb1725a26a7c743e8 P 620596b893d84291b88b90dc9e317612 [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.732843 15351 raft_consensus.cc:2454] T ca0d7d3e4d59496c9a9a59cde24b9247 P c05f7620bb884392931d3243a0b47a9e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 6.
I20240503 02:00:41.733222 14893 leader_election.cc:302] T 6798a9ed0cf441aca7b8ae1b05c001f3 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 620596b893d84291b88b90dc9e317612, dcfdfa62edc34dab9525e2712d487582; no voters: c05f7620bb884392931d3243a0b47a9e
I20240503 02:00:41.733521 14893 leader_election.cc:302] T 2b757c430aaa42f382d174ca57f5fcfc P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 620596b893d84291b88b90dc9e317612, dcfdfa62edc34dab9525e2712d487582; no voters: b602f6099cbb4758847e7e9550685372
I20240503 02:00:41.733597 17308 raft_consensus.cc:2781] T 6798a9ed0cf441aca7b8ae1b05c001f3 P dcfdfa62edc34dab9525e2712d487582 [term 6 FOLLOWER]: Leader election won for term 6
I20240503 02:00:41.733668 17225 raft_consensus.cc:2781] T 2b757c430aaa42f382d174ca57f5fcfc P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 02:00:41.733698 17308 raft_consensus.cc:686] T 6798a9ed0cf441aca7b8ae1b05c001f3 P dcfdfa62edc34dab9525e2712d487582 [term 6 LEADER]: Becoming Leader. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:41.733712 14893 leader_election.cc:302] T d805ee04df39406ba30b3b3b9e349507 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c05f7620bb884392931d3243a0b47a9e, dcfdfa62edc34dab9525e2712d487582; no voters: b602f6099cbb4758847e7e9550685372
I20240503 02:00:41.733800 17225 raft_consensus.cc:686] T 2b757c430aaa42f382d174ca57f5fcfc P dcfdfa62edc34dab9525e2712d487582 [term 5 LEADER]: Becoming Leader. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:41.733850 17308 consensus_queue.cc:237] T 6798a9ed0cf441aca7b8ae1b05c001f3 P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.733911 17225 consensus_queue.cc:237] T 2b757c430aaa42f382d174ca57f5fcfc P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.7, Last appended by leader: 7, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.733991 14893 leader_election.cc:302] T ca0d7d3e4d59496c9a9a59cde24b9247 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c05f7620bb884392931d3243a0b47a9e, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:41.734210 14958 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6d98ba8805024eb79bf5df7a8225d784"
dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
from {username='slave'} at 127.0.0.1:47980
I20240503 02:00:41.734318 14958 raft_consensus.cc:482] T 6d98ba8805024eb79bf5df7a8225d784 P dcfdfa62edc34dab9525e2712d487582 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.734366 14958 raft_consensus.cc:3037] T 6d98ba8805024eb79bf5df7a8225d784 P dcfdfa62edc34dab9525e2712d487582 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.734445 17225 raft_consensus.cc:2781] T ca0d7d3e4d59496c9a9a59cde24b9247 P dcfdfa62edc34dab9525e2712d487582 [term 6 FOLLOWER]: Leader election won for term 6
I20240503 02:00:41.734534 17225 raft_consensus.cc:686] T ca0d7d3e4d59496c9a9a59cde24b9247 P dcfdfa62edc34dab9525e2712d487582 [term 6 LEADER]: Becoming Leader. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:41.734642 17225 consensus_queue.cc:237] T ca0d7d3e4d59496c9a9a59cde24b9247 P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 3.9, Last appended by leader: 9, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.734445 17308 raft_consensus.cc:2781] T d805ee04df39406ba30b3b3b9e349507 P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 02:00:41.735402 17308 raft_consensus.cc:686] T d805ee04df39406ba30b3b3b9e349507 P dcfdfa62edc34dab9525e2712d487582 [term 5 LEADER]: Becoming Leader. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:41.735509 17308 consensus_queue.cc:237] T d805ee04df39406ba30b3b3b9e349507 P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.7, Last appended by leader: 7, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.735738 14811 catalog_manager.cc:5484] T 0989d98050744df6a1c699cb86ada505 P dcfdfa62edc34dab9525e2712d487582 reported cstate change: term changed from 5 to 6, leader changed from c05f7620bb884392931d3243a0b47a9e (127.0.99.132) to dcfdfa62edc34dab9525e2712d487582 (127.0.99.129). New cstate: current_term: 6 leader_uuid: "dcfdfa62edc34dab9525e2712d487582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:41.735309 14958 raft_consensus.cc:504] T 6d98ba8805024eb79bf5df7a8225d784 P dcfdfa62edc34dab9525e2712d487582 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.736826 14958 leader_election.cc:288] T 6d98ba8805024eb79bf5df7a8225d784 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 4 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
W20240503 02:00:41.737001 15176 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57688: Illegal state: replica 620596b893d84291b88b90dc9e317612 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.737424 15088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca0d7d3e4d59496c9a9a59cde24b9247" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 6 candidate_status { last_received { term: 3 index: 9 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:41.737550 15088 raft_consensus.cc:3037] T ca0d7d3e4d59496c9a9a59cde24b9247 P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.738267 15219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d98ba8805024eb79bf5df7a8225d784" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 4 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:41.738394 15219 raft_consensus.cc:3037] T 6d98ba8805024eb79bf5df7a8225d784 P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.738718 15088 raft_consensus.cc:2396] T ca0d7d3e4d59496c9a9a59cde24b9247 P b602f6099cbb4758847e7e9550685372 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate dcfdfa62edc34dab9525e2712d487582 for term 6 because replica has last-logged OpId of term: 4 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 9.
I20240503 02:00:41.739428 15219 raft_consensus.cc:2454] T 6d98ba8805024eb79bf5df7a8225d784 P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 4.
W20240503 02:00:41.740309 14915 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47950: Illegal state: replica dcfdfa62edc34dab9525e2712d487582 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.740856 14893 leader_election.cc:302] T 6d98ba8805024eb79bf5df7a8225d784 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:41.733198 15214 raft_consensus.cc:2454] T 5d448658ed3b460fb1725a26a7c743e8 P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 6.
I20240503 02:00:41.741281 14893 leader_election.cc:302] T 5d448658ed3b460fb1725a26a7c743e8 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 620596b893d84291b88b90dc9e317612, dcfdfa62edc34dab9525e2712d487582; no voters: c05f7620bb884392931d3243a0b47a9e
I20240503 02:00:41.741390 14937 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:00:41.741775 17308 raft_consensus.cc:2781] T 6d98ba8805024eb79bf5df7a8225d784 P dcfdfa62edc34dab9525e2712d487582 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 02:00:41.741868 17308 raft_consensus.cc:686] T 6d98ba8805024eb79bf5df7a8225d784 P dcfdfa62edc34dab9525e2712d487582 [term 4 LEADER]: Becoming Leader. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:41.742002 17308 consensus_queue.cc:237] T 6d98ba8805024eb79bf5df7a8225d784 P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.742575 17308 raft_consensus.cc:2781] T 5d448658ed3b460fb1725a26a7c743e8 P dcfdfa62edc34dab9525e2712d487582 [term 6 FOLLOWER]: Leader election won for term 6
I20240503 02:00:41.742651 17308 raft_consensus.cc:686] T 5d448658ed3b460fb1725a26a7c743e8 P dcfdfa62edc34dab9525e2712d487582 [term 6 LEADER]: Becoming Leader. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:41.742758 17308 consensus_queue.cc:237] T 5d448658ed3b460fb1725a26a7c743e8 P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 4.9, Last appended by leader: 9, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.742761 14816 catalog_manager.cc:5484] T 2b757c430aaa42f382d174ca57f5fcfc P dcfdfa62edc34dab9525e2712d487582 reported cstate change: term changed from 3 to 5, leader changed from b602f6099cbb4758847e7e9550685372 (127.0.99.130) to dcfdfa62edc34dab9525e2712d487582 (127.0.99.129). New cstate: current_term: 5 leader_uuid: "dcfdfa62edc34dab9525e2712d487582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:41.744071 14816 catalog_manager.cc:5484] T d805ee04df39406ba30b3b3b9e349507 P dcfdfa62edc34dab9525e2712d487582 reported cstate change: term changed from 3 to 5, leader changed from b602f6099cbb4758847e7e9550685372 (127.0.99.130) to dcfdfa62edc34dab9525e2712d487582 (127.0.99.129). New cstate: current_term: 5 leader_uuid: "dcfdfa62edc34dab9525e2712d487582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:41.744300 14816 catalog_manager.cc:5484] T 6798a9ed0cf441aca7b8ae1b05c001f3 P dcfdfa62edc34dab9525e2712d487582 reported cstate change: term changed from 5 to 6, leader changed from c05f7620bb884392931d3243a0b47a9e (127.0.99.132) to dcfdfa62edc34dab9525e2712d487582 (127.0.99.129). New cstate: current_term: 6 leader_uuid: "dcfdfa62edc34dab9525e2712d487582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:41.744530 14816 catalog_manager.cc:5484] T ca0d7d3e4d59496c9a9a59cde24b9247 P dcfdfa62edc34dab9525e2712d487582 reported cstate change: term changed from 4 to 6, leader changed from b602f6099cbb4758847e7e9550685372 (127.0.99.130) to dcfdfa62edc34dab9525e2712d487582 (127.0.99.129). New cstate: current_term: 6 leader_uuid: "dcfdfa62edc34dab9525e2712d487582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:41.745054 15088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d98ba8805024eb79bf5df7a8225d784" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 4 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:41.745224 15088 raft_consensus.cc:3032] T 6d98ba8805024eb79bf5df7a8225d784 P b602f6099cbb4758847e7e9550685372 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 02:00:41.745321 15088 raft_consensus.cc:729] T 6d98ba8805024eb79bf5df7a8225d784 P b602f6099cbb4758847e7e9550685372 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.745518 15088 consensus_queue.cc:260] T 6d98ba8805024eb79bf5df7a8225d784 P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.7, Last appended by leader: 7, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.745695 15088 raft_consensus.cc:3037] T 6d98ba8805024eb79bf5df7a8225d784 P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.747709 15088 raft_consensus.cc:2396] T 6d98ba8805024eb79bf5df7a8225d784 P b602f6099cbb4758847e7e9550685372 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate dcfdfa62edc34dab9525e2712d487582 for term 4 because replica has last-logged OpId of term: 3 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 6.
I20240503 02:00:41.748116 14958 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "42ceeb7b3c4146cd8a2e18b3e9ae987c"
dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
from {username='slave'} at 127.0.0.1:47980
I20240503 02:00:41.748291 14958 raft_consensus.cc:482] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P dcfdfa62edc34dab9525e2712d487582 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.748414 14958 raft_consensus.cc:3037] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P dcfdfa62edc34dab9525e2712d487582 [term 4 FOLLOWER]: Advancing to term 5
W20240503 02:00:41.748654 14915 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48020: Illegal state: replica dcfdfa62edc34dab9525e2712d487582 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.749183 14816 catalog_manager.cc:5484] T 5d448658ed3b460fb1725a26a7c743e8 P dcfdfa62edc34dab9525e2712d487582 reported cstate change: term changed from 5 to 6, leader changed from c05f7620bb884392931d3243a0b47a9e (127.0.99.132) to dcfdfa62edc34dab9525e2712d487582 (127.0.99.129). New cstate: current_term: 6 leader_uuid: "dcfdfa62edc34dab9525e2712d487582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:41.749598 14816 catalog_manager.cc:5484] T 6d98ba8805024eb79bf5df7a8225d784 P dcfdfa62edc34dab9525e2712d487582 reported cstate change: term changed from 3 to 4, leader changed from b602f6099cbb4758847e7e9550685372 (127.0.99.130) to dcfdfa62edc34dab9525e2712d487582 (127.0.99.129). New cstate: current_term: 4 leader_uuid: "dcfdfa62edc34dab9525e2712d487582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:41.750474 14958 raft_consensus.cc:504] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:41.750918 14958 leader_election.cc:288] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 5 election: Requested vote from peers b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207), 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437)
I20240503 02:00:41.752487 15214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ceeb7b3c4146cd8a2e18b3e9ae987c" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:41.752676 15214 raft_consensus.cc:3037] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.753819 15214 raft_consensus.cc:2454] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P 620596b893d84291b88b90dc9e317612 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 5.
I20240503 02:00:41.754105 14893 leader_election.cc:302] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:41.754285 17225 raft_consensus.cc:2781] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 02:00:41.754369 17225 raft_consensus.cc:686] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P dcfdfa62edc34dab9525e2712d487582 [term 5 LEADER]: Becoming Leader. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:41.754523 17225 consensus_queue.cc:237] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:41.755640 15329 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:41.755725 16669 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:41.756930 14816 catalog_manager.cc:5484] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P dcfdfa62edc34dab9525e2712d487582 reported cstate change: term changed from 3 to 5, leader changed from b602f6099cbb4758847e7e9550685372 (127.0.99.130) to dcfdfa62edc34dab9525e2712d487582 (127.0.99.129). New cstate: current_term: 5 leader_uuid: "dcfdfa62edc34dab9525e2712d487582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:41.759537 15088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ceeb7b3c4146cd8a2e18b3e9ae987c" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:41.759784 15088 raft_consensus.cc:3037] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P b602f6099cbb4758847e7e9550685372 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.760972 15088 raft_consensus.cc:2396] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate dcfdfa62edc34dab9525e2712d487582 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20240503 02:00:41.762173 14958 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6b0c8b72e1884529a8a9b83d2a734d71"
dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
from {username='slave'} at 127.0.0.1:47980
I20240503 02:00:41.762331 14958 raft_consensus.cc:482] T 6b0c8b72e1884529a8a9b83d2a734d71 P dcfdfa62edc34dab9525e2712d487582 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.762383 14958 raft_consensus.cc:3037] T 6b0c8b72e1884529a8a9b83d2a734d71 P dcfdfa62edc34dab9525e2712d487582 [term 4 FOLLOWER]: Advancing to term 5
W20240503 02:00:41.763697 15048 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34032: Illegal state: replica b602f6099cbb4758847e7e9550685372 is not leader of this config: current role FOLLOWER
W20240503 02:00:41.765157 15048 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33982: Illegal state: replica b602f6099cbb4758847e7e9550685372 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.768219 14958 raft_consensus.cc:504] T 6b0c8b72e1884529a8a9b83d2a734d71 P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.769119 15214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b0c8b72e1884529a8a9b83d2a734d71" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:41.769325 15214 raft_consensus.cc:3032] T 6b0c8b72e1884529a8a9b83d2a734d71 P 620596b893d84291b88b90dc9e317612 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 02:00:41.769407 15214 raft_consensus.cc:729] T 6b0c8b72e1884529a8a9b83d2a734d71 P 620596b893d84291b88b90dc9e317612 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:41.769688 15214 consensus_queue.cc:260] T 6b0c8b72e1884529a8a9b83d2a734d71 P 620596b893d84291b88b90dc9e317612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.769925 15214 raft_consensus.cc:3037] T 6b0c8b72e1884529a8a9b83d2a734d71 P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.771132 15214 raft_consensus.cc:2396] T 6b0c8b72e1884529a8a9b83d2a734d71 P 620596b893d84291b88b90dc9e317612 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate dcfdfa62edc34dab9525e2712d487582 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20240503 02:00:41.771458 14958 leader_election.cc:288] T 6b0c8b72e1884529a8a9b83d2a734d71 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 5 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
I20240503 02:00:41.772158 15088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b0c8b72e1884529a8a9b83d2a734d71" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:41.772290 15088 raft_consensus.cc:3037] T 6b0c8b72e1884529a8a9b83d2a734d71 P b602f6099cbb4758847e7e9550685372 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.773365 15088 raft_consensus.cc:2396] T 6b0c8b72e1884529a8a9b83d2a734d71 P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate dcfdfa62edc34dab9525e2712d487582 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20240503 02:00:41.774068 14892 leader_election.cc:302] T 6b0c8b72e1884529a8a9b83d2a734d71 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dcfdfa62edc34dab9525e2712d487582; no voters: 620596b893d84291b88b90dc9e317612, b602f6099cbb4758847e7e9550685372
I20240503 02:00:41.774358 17225 raft_consensus.cc:2726] T 6b0c8b72e1884529a8a9b83d2a734d71 P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240503 02:00:41.774689 14958 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9a74cca2988a4f219f6fd9b07ade42d8"
dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
from {username='slave'} at 127.0.0.1:47980
I20240503 02:00:41.774892 14958 raft_consensus.cc:482] T 9a74cca2988a4f219f6fd9b07ade42d8 P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.775415 15087 raft_consensus.cc:2396] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate dcfdfa62edc34dab9525e2712d487582 for term 6 because replica has last-logged OpId of term: 4 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 10.
I20240503 02:00:41.775776 14892 leader_election.cc:302] T 6d2a1296a54a4b92a03c12954de0cc34 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dcfdfa62edc34dab9525e2712d487582; no voters: b602f6099cbb4758847e7e9550685372, c05f7620bb884392931d3243a0b47a9e
I20240503 02:00:41.776016 17225 raft_consensus.cc:2726] T 6d2a1296a54a4b92a03c12954de0cc34 P dcfdfa62edc34dab9525e2712d487582 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240503 02:00:41.776155 14958 raft_consensus.cc:3037] T 9a74cca2988a4f219f6fd9b07ade42d8 P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Advancing to term 6
W20240503 02:00:41.776187 15176 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57688: Illegal state: replica 620596b893d84291b88b90dc9e317612 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.777482 14958 raft_consensus.cc:504] T 9a74cca2988a4f219f6fd9b07ade42d8 P dcfdfa62edc34dab9525e2712d487582 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.779029 14958 leader_election.cc:288] T 9a74cca2988a4f219f6fd9b07ade42d8 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 6 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:41.779654 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a74cca2988a4f219f6fd9b07ade42d8" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 6 candidate_status { last_received { term: 3 index: 7 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.779808 15351 raft_consensus.cc:3032] T 9a74cca2988a4f219f6fd9b07ade42d8 P c05f7620bb884392931d3243a0b47a9e [term 5 LEADER]: Stepping down as leader of term 5
I20240503 02:00:41.779856 15351 raft_consensus.cc:729] T 9a74cca2988a4f219f6fd9b07ade42d8 P c05f7620bb884392931d3243a0b47a9e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:41.779883 14958 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6b225db529f14e7386f21f42bf31493b"
dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
from {username='slave'} at 127.0.0.1:47980
I20240503 02:00:41.780011 14958 raft_consensus.cc:482] T 6b225db529f14e7386f21f42bf31493b P dcfdfa62edc34dab9525e2712d487582 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.780009 15351 consensus_queue.cc:260] T 9a74cca2988a4f219f6fd9b07ade42d8 P c05f7620bb884392931d3243a0b47a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.8, Last appended by leader: 8, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.780082 14958 raft_consensus.cc:3037] T 6b225db529f14e7386f21f42bf31493b P dcfdfa62edc34dab9525e2712d487582 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.780153 15351 raft_consensus.cc:3037] T 9a74cca2988a4f219f6fd9b07ade42d8 P c05f7620bb884392931d3243a0b47a9e [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.781141 14958 raft_consensus.cc:504] T 6b225db529f14e7386f21f42bf31493b P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.781594 15214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a74cca2988a4f219f6fd9b07ade42d8" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 6 candidate_status { last_received { term: 3 index: 7 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:41.781716 15214 raft_consensus.cc:3037] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612 [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.781853 15087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b225db529f14e7386f21f42bf31493b" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 5 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:41.782007 15087 raft_consensus.cc:3037] T 6b225db529f14e7386f21f42bf31493b P b602f6099cbb4758847e7e9550685372 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.782898 15214 raft_consensus.cc:2396] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate dcfdfa62edc34dab9525e2712d487582 for term 6 because replica has last-logged OpId of term: 4 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 7.
I20240503 02:00:41.781188 15351 raft_consensus.cc:2396] T 9a74cca2988a4f219f6fd9b07ade42d8 P c05f7620bb884392931d3243a0b47a9e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate dcfdfa62edc34dab9525e2712d487582 for term 6 because replica has last-logged OpId of term: 5 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 7.
I20240503 02:00:41.783151 15087 raft_consensus.cc:2396] T 6b225db529f14e7386f21f42bf31493b P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate dcfdfa62edc34dab9525e2712d487582 for term 5 because replica has last-logged OpId of term: 3 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 8.
I20240503 02:00:41.783432 14958 leader_election.cc:288] T 6b225db529f14e7386f21f42bf31493b P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 5 election: Requested vote from peers b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207), 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437)
I20240503 02:00:41.783483 14893 leader_election.cc:302] T 9a74cca2988a4f219f6fd9b07ade42d8 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dcfdfa62edc34dab9525e2712d487582; no voters: 620596b893d84291b88b90dc9e317612, c05f7620bb884392931d3243a0b47a9e
I20240503 02:00:41.783687 17397 raft_consensus.cc:2726] T 9a74cca2988a4f219f6fd9b07ade42d8 P dcfdfa62edc34dab9525e2712d487582 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240503 02:00:41.784564 14958 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f85a361ce0254472b37526362d4edf0e"
dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
from {username='slave'} at 127.0.0.1:47980
I20240503 02:00:41.784615 15214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b225db529f14e7386f21f42bf31493b" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 5 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:41.784727 15214 raft_consensus.cc:3037] T 6b225db529f14e7386f21f42bf31493b P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.784840 14958 raft_consensus.cc:482] T f85a361ce0254472b37526362d4edf0e P dcfdfa62edc34dab9525e2712d487582 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.784940 14958 raft_consensus.cc:3037] T f85a361ce0254472b37526362d4edf0e P dcfdfa62edc34dab9525e2712d487582 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.785682 15214 raft_consensus.cc:2454] T 6b225db529f14e7386f21f42bf31493b P 620596b893d84291b88b90dc9e317612 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 5.
I20240503 02:00:41.785946 14893 leader_election.cc:302] T 6b225db529f14e7386f21f42bf31493b P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 620596b893d84291b88b90dc9e317612, dcfdfa62edc34dab9525e2712d487582; no voters: b602f6099cbb4758847e7e9550685372
I20240503 02:00:41.786134 17397 raft_consensus.cc:2781] T 6b225db529f14e7386f21f42bf31493b P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Leader election won for term 5
W20240503 02:00:41.786875 14915 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48020: Illegal state: replica dcfdfa62edc34dab9525e2712d487582 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.786906 14958 raft_consensus.cc:504] T f85a361ce0254472b37526362d4edf0e P dcfdfa62edc34dab9525e2712d487582 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
W20240503 02:00:41.787176 15176 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57618: Illegal state: replica 620596b893d84291b88b90dc9e317612 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.786219 17397 raft_consensus.cc:686] T 6b225db529f14e7386f21f42bf31493b P dcfdfa62edc34dab9525e2712d487582 [term 5 LEADER]: Becoming Leader. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:41.787405 17397 consensus_queue.cc:237] T 6b225db529f14e7386f21f42bf31493b P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.8, Last appended by leader: 8, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.788057 15087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85a361ce0254472b37526362d4edf0e" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 4 candidate_status { last_received { term: 1 index: 19 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:41.788190 15087 raft_consensus.cc:3037] T f85a361ce0254472b37526362d4edf0e P b602f6099cbb4758847e7e9550685372 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.789311 15087 raft_consensus.cc:2396] T f85a361ce0254472b37526362d4edf0e P b602f6099cbb4758847e7e9550685372 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate dcfdfa62edc34dab9525e2712d487582 for term 4 because replica has last-logged OpId of term: 2 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 19.
I20240503 02:00:41.789685 14816 catalog_manager.cc:5484] T 6b225db529f14e7386f21f42bf31493b P dcfdfa62edc34dab9525e2712d487582 reported cstate change: term changed from 3 to 5, leader changed from b602f6099cbb4758847e7e9550685372 (127.0.99.130) to dcfdfa62edc34dab9525e2712d487582 (127.0.99.129). New cstate: current_term: 5 leader_uuid: "dcfdfa62edc34dab9525e2712d487582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:41.790140 14958 leader_election.cc:288] T f85a361ce0254472b37526362d4edf0e P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 4 election: Requested vote from peers b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207), 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437)
I20240503 02:00:41.790696 15214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85a361ce0254472b37526362d4edf0e" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 4 candidate_status { last_received { term: 1 index: 19 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:41.790824 15214 raft_consensus.cc:3037] T f85a361ce0254472b37526362d4edf0e P 620596b893d84291b88b90dc9e317612 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:00:41.790923 14958 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "454a24fd6d964d73ae3d5057f90fe36d"
dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
from {username='slave'} at 127.0.0.1:47980
I20240503 02:00:41.791105 14958 raft_consensus.cc:482] T 454a24fd6d964d73ae3d5057f90fe36d P dcfdfa62edc34dab9525e2712d487582 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.791208 14958 raft_consensus.cc:3037] T 454a24fd6d964d73ae3d5057f90fe36d P dcfdfa62edc34dab9525e2712d487582 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.792177 15214 raft_consensus.cc:2454] T f85a361ce0254472b37526362d4edf0e P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 4.
I20240503 02:00:41.792171 14958 raft_consensus.cc:504] T 454a24fd6d964d73ae3d5057f90fe36d P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.792517 14893 leader_election.cc:302] T f85a361ce0254472b37526362d4edf0e P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 620596b893d84291b88b90dc9e317612, dcfdfa62edc34dab9525e2712d487582; no voters: b602f6099cbb4758847e7e9550685372
I20240503 02:00:41.792726 14958 leader_election.cc:288] T 454a24fd6d964d73ae3d5057f90fe36d P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 5 election: Requested vote from peers b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:41.793045 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "454a24fd6d964d73ae3d5057f90fe36d" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.793174 15351 raft_consensus.cc:3037] T 454a24fd6d964d73ae3d5057f90fe36d P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.793221 15087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "454a24fd6d964d73ae3d5057f90fe36d" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:41.793342 15087 raft_consensus.cc:3037] T 454a24fd6d964d73ae3d5057f90fe36d P b602f6099cbb4758847e7e9550685372 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.793519 14956 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0e3bb299e61148fcb355641e0134b3df"
dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
from {username='slave'} at 127.0.0.1:47980
I20240503 02:00:41.793624 14956 raft_consensus.cc:482] T 0e3bb299e61148fcb355641e0134b3df P dcfdfa62edc34dab9525e2712d487582 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.793675 14956 raft_consensus.cc:3037] T 0e3bb299e61148fcb355641e0134b3df P dcfdfa62edc34dab9525e2712d487582 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.794019 15351 raft_consensus.cc:2454] T 454a24fd6d964d73ae3d5057f90fe36d P c05f7620bb884392931d3243a0b47a9e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 5.
I20240503 02:00:41.794310 14893 leader_election.cc:302] T 454a24fd6d964d73ae3d5057f90fe36d P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c05f7620bb884392931d3243a0b47a9e, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:41.794469 17396 raft_consensus.cc:2781] T 454a24fd6d964d73ae3d5057f90fe36d P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 02:00:41.794456 17397 raft_consensus.cc:2781] T f85a361ce0254472b37526362d4edf0e P dcfdfa62edc34dab9525e2712d487582 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 02:00:41.794618 17396 raft_consensus.cc:686] T 454a24fd6d964d73ae3d5057f90fe36d P dcfdfa62edc34dab9525e2712d487582 [term 5 LEADER]: Becoming Leader. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:41.794587 14956 raft_consensus.cc:504] T 0e3bb299e61148fcb355641e0134b3df P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.794790 17396 consensus_queue.cc:237] T 454a24fd6d964d73ae3d5057f90fe36d P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.794903 17397 raft_consensus.cc:686] T f85a361ce0254472b37526362d4edf0e P dcfdfa62edc34dab9525e2712d487582 [term 4 LEADER]: Becoming Leader. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:41.795141 14956 leader_election.cc:288] T 0e3bb299e61148fcb355641e0134b3df P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 5 election: Requested vote from peers c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
I20240503 02:00:41.795702 15084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e3bb299e61148fcb355641e0134b3df" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 5 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:41.795071 17397 consensus_queue.cc:237] T f85a361ce0254472b37526362d4edf0e P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 1.19, Last appended by leader: 19, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:41.795833 15084 raft_consensus.cc:3037] T 0e3bb299e61148fcb355641e0134b3df P b602f6099cbb4758847e7e9550685372 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.796182 14956 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "590736f5f1ca4703821c409d882c2f77"
dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
from {username='slave'} at 127.0.0.1:47980
I20240503 02:00:41.796303 14956 raft_consensus.cc:482] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.796319 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e3bb299e61148fcb355641e0134b3df" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 5 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.796363 14956 raft_consensus.cc:3037] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.796464 15351 raft_consensus.cc:3037] T 0e3bb299e61148fcb355641e0134b3df P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.797078 15084 raft_consensus.cc:2396] T 0e3bb299e61148fcb355641e0134b3df P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate dcfdfa62edc34dab9525e2712d487582 for term 5 because replica has last-logged OpId of term: 3 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 7.
I20240503 02:00:41.797302 14956 raft_consensus.cc:504] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.797431 15351 raft_consensus.cc:2454] T 0e3bb299e61148fcb355641e0134b3df P c05f7620bb884392931d3243a0b47a9e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 5.
I20240503 02:00:41.797667 14956 leader_election.cc:288] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 6 election: Requested vote from peers c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
W20240503 02:00:41.797906 15048 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34032: Illegal state: replica b602f6099cbb4758847e7e9550685372 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.798502 14893 leader_election.cc:302] T 0e3bb299e61148fcb355641e0134b3df P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c05f7620bb884392931d3243a0b47a9e, dcfdfa62edc34dab9525e2712d487582; no voters: b602f6099cbb4758847e7e9550685372
I20240503 02:00:41.798671 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "590736f5f1ca4703821c409d882c2f77" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.798777 15084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "590736f5f1ca4703821c409d882c2f77" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:41.798825 15351 raft_consensus.cc:3032] T 590736f5f1ca4703821c409d882c2f77 P c05f7620bb884392931d3243a0b47a9e [term 5 LEADER]: Stepping down as leader of term 5
I20240503 02:00:41.798868 15351 raft_consensus.cc:729] T 590736f5f1ca4703821c409d882c2f77 P c05f7620bb884392931d3243a0b47a9e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:41.798882 15084 raft_consensus.cc:3037] T 590736f5f1ca4703821c409d882c2f77 P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.798928 14956 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "11275e009edd4354b0807b7f015023be"
dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
from {username='slave'} at 127.0.0.1:47980
I20240503 02:00:41.799027 14956 raft_consensus.cc:482] T 11275e009edd4354b0807b7f015023be P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.799005 15351 consensus_queue.cc:260] T 590736f5f1ca4703821c409d882c2f77 P c05f7620bb884392931d3243a0b47a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.799075 14956 raft_consensus.cc:3037] T 11275e009edd4354b0807b7f015023be P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.799182 15351 raft_consensus.cc:3037] T 590736f5f1ca4703821c409d882c2f77 P c05f7620bb884392931d3243a0b47a9e [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.799268 17396 raft_consensus.cc:2781] T 0e3bb299e61148fcb355641e0134b3df P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 02:00:41.799337 17396 raft_consensus.cc:686] T 0e3bb299e61148fcb355641e0134b3df P dcfdfa62edc34dab9525e2712d487582 [term 5 LEADER]: Becoming Leader. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:41.799500 17396 consensus_queue.cc:237] T 0e3bb299e61148fcb355641e0134b3df P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.799868 15084 raft_consensus.cc:2454] T 590736f5f1ca4703821c409d882c2f77 P b602f6099cbb4758847e7e9550685372 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 6.
I20240503 02:00:41.799963 14956 raft_consensus.cc:504] T 11275e009edd4354b0807b7f015023be P dcfdfa62edc34dab9525e2712d487582 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.800158 15351 raft_consensus.cc:2454] T 590736f5f1ca4703821c409d882c2f77 P c05f7620bb884392931d3243a0b47a9e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 6.
I20240503 02:00:41.800305 14956 leader_election.cc:288] T 11275e009edd4354b0807b7f015023be P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 6 election: Requested vote from peers c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009), 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437)
I20240503 02:00:41.800499 14892 leader_election.cc:302] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b602f6099cbb4758847e7e9550685372, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:41.800871 15214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11275e009edd4354b0807b7f015023be" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:41.800992 15214 raft_consensus.cc:3037] T 11275e009edd4354b0807b7f015023be P 620596b893d84291b88b90dc9e317612 [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.801376 15087 raft_consensus.cc:2396] T 454a24fd6d964d73ae3d5057f90fe36d P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate dcfdfa62edc34dab9525e2712d487582 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 02:00:41.801954 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11275e009edd4354b0807b7f015023be" candidate_uuid: "dcfdfa62edc34dab9525e2712d487582" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.802074 15351 raft_consensus.cc:3037] T 11275e009edd4354b0807b7f015023be P c05f7620bb884392931d3243a0b47a9e [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.802345 14816 catalog_manager.cc:5484] T 454a24fd6d964d73ae3d5057f90fe36d P dcfdfa62edc34dab9525e2712d487582 reported cstate change: term changed from 3 to 5, leader changed from b602f6099cbb4758847e7e9550685372 (127.0.99.130) to dcfdfa62edc34dab9525e2712d487582 (127.0.99.129). New cstate: current_term: 5 leader_uuid: "dcfdfa62edc34dab9525e2712d487582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:41.802842 15351 raft_consensus.cc:2454] T 11275e009edd4354b0807b7f015023be P c05f7620bb884392931d3243a0b47a9e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcfdfa62edc34dab9525e2712d487582 in term 6.
I20240503 02:00:41.803174 14893 leader_election.cc:302] T 11275e009edd4354b0807b7f015023be P dcfdfa62edc34dab9525e2712d487582 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c05f7620bb884392931d3243a0b47a9e, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:41.803359 17395 raft_consensus.cc:2781] T 11275e009edd4354b0807b7f015023be P dcfdfa62edc34dab9525e2712d487582 [term 6 FOLLOWER]: Leader election won for term 6
I20240503 02:00:41.803458 17395 raft_consensus.cc:686] T 11275e009edd4354b0807b7f015023be P dcfdfa62edc34dab9525e2712d487582 [term 6 LEADER]: Becoming Leader. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:41.803601 15214 raft_consensus.cc:2396] T 11275e009edd4354b0807b7f015023be P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate dcfdfa62edc34dab9525e2712d487582 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20240503 02:00:41.803611 17395 consensus_queue.cc:237] T 11275e009edd4354b0807b7f015023be P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.803797 17396 raft_consensus.cc:2781] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582 [term 6 FOLLOWER]: Leader election won for term 6
I20240503 02:00:41.803927 17396 raft_consensus.cc:686] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582 [term 6 LEADER]: Becoming Leader. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:41.804051 17396 consensus_queue.cc:237] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.807122 14816 catalog_manager.cc:5484] T 0e3bb299e61148fcb355641e0134b3df P dcfdfa62edc34dab9525e2712d487582 reported cstate change: term changed from 3 to 5, leader changed from b602f6099cbb4758847e7e9550685372 (127.0.99.130) to dcfdfa62edc34dab9525e2712d487582 (127.0.99.129). New cstate: current_term: 5 leader_uuid: "dcfdfa62edc34dab9525e2712d487582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:41.807418 14816 catalog_manager.cc:5484] T f85a361ce0254472b37526362d4edf0e P dcfdfa62edc34dab9525e2712d487582 reported cstate change: term changed from 1 to 4. New cstate: current_term: 4 leader_uuid: "dcfdfa62edc34dab9525e2712d487582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:41.807729 14816 catalog_manager.cc:5484] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582 reported cstate change: term changed from 5 to 6, leader changed from c05f7620bb884392931d3243a0b47a9e (127.0.99.132) to dcfdfa62edc34dab9525e2712d487582 (127.0.99.129). New cstate: current_term: 6 leader_uuid: "dcfdfa62edc34dab9525e2712d487582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:41.807937 14816 catalog_manager.cc:5484] T 11275e009edd4354b0807b7f015023be P dcfdfa62edc34dab9525e2712d487582 reported cstate change: term changed from 4 to 6, leader changed from 620596b893d84291b88b90dc9e317612 (127.0.99.131) to dcfdfa62edc34dab9525e2712d487582 (127.0.99.129). New cstate: current_term: 6 leader_uuid: "dcfdfa62edc34dab9525e2712d487582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: HEALTHY } } }
W20240503 02:00:41.809635 15176 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57688: Illegal state: replica 620596b893d84291b88b90dc9e317612 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.813246 15214 raft_consensus.cc:1261] T f85a361ce0254472b37526362d4edf0e P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Refusing update from remote peer dcfdfa62edc34dab9525e2712d487582: Log matching property violated. Preceding OpId in replica: term: 1 index: 17. Preceding OpId from leader: term: 4 index: 21. (index mismatch)
I20240503 02:00:41.813369 15087 raft_consensus.cc:1261] T f85a361ce0254472b37526362d4edf0e P b602f6099cbb4758847e7e9550685372 [term 4 FOLLOWER]: Refusing update from remote peer dcfdfa62edc34dab9525e2712d487582: Log matching property violated. Preceding OpId in replica: term: 2 index: 20. Preceding OpId from leader: term: 4 index: 21. (index mismatch)
I20240503 02:00:41.813720 17396 consensus_queue.cc:1035] T f85a361ce0254472b37526362d4edf0e P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Connected to new peer: Peer: permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 16, Time since last communication: 0.000s
I20240503 02:00:41.813926 17395 consensus_queue.cc:1035] T f85a361ce0254472b37526362d4edf0e P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20240503 02:00:41.814036 17395 consensus_queue.cc:1225] T f85a361ce0254472b37526362d4edf0e P dcfdfa62edc34dab9525e2712d487582 [LEADER]: Peer b602f6099cbb4758847e7e9550685372 log is divergent from this leader: its last log entry 2.20 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 19
I20240503 02:00:41.814939 15087 pending_rounds.cc:76] T f85a361ce0254472b37526362d4edf0e P b602f6099cbb4758847e7e9550685372: Aborting all ops after (but not including) 19
I20240503 02:00:41.815006 15087 pending_rounds.cc:97] T f85a361ce0254472b37526362d4edf0e P b602f6099cbb4758847e7e9550685372: Aborting uncommitted NO_OP operation due to leader change: 2.20
I20240503 02:00:41.815083 15087 raft_consensus.cc:2866] T f85a361ce0254472b37526362d4edf0e P b602f6099cbb4758847e7e9550685372 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 02:00:41.818944 15084 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "de805ef7511d4ee9b9bbed67e0a4186c"
dest_uuid: "b602f6099cbb4758847e7e9550685372"
from {username='slave'} at 127.0.0.1:34018
I20240503 02:00:41.819120 15084 raft_consensus.cc:482] T de805ef7511d4ee9b9bbed67e0a4186c P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.819176 15084 raft_consensus.cc:3037] T de805ef7511d4ee9b9bbed67e0a4186c P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Advancing to term 6
W20240503 02:00:41.819823 15048 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33982: Illegal state: replica b602f6099cbb4758847e7e9550685372 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.820132 15084 raft_consensus.cc:504] T de805ef7511d4ee9b9bbed67e0a4186c P b602f6099cbb4758847e7e9550685372 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.820569 15084 leader_election.cc:288] T de805ef7511d4ee9b9bbed67e0a4186c P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 6 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:41.820895 15214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de805ef7511d4ee9b9bbed67e0a4186c" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 6 candidate_status { last_received { term: 3 index: 7 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:41.821036 15214 raft_consensus.cc:3037] T de805ef7511d4ee9b9bbed67e0a4186c P 620596b893d84291b88b90dc9e317612 [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.821643 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de805ef7511d4ee9b9bbed67e0a4186c" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 6 candidate_status { last_received { term: 3 index: 7 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.821785 15351 raft_consensus.cc:3037] T de805ef7511d4ee9b9bbed67e0a4186c P c05f7620bb884392931d3243a0b47a9e [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.822139 15214 raft_consensus.cc:2396] T de805ef7511d4ee9b9bbed67e0a4186c P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b602f6099cbb4758847e7e9550685372 for term 6 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 7.
I20240503 02:00:41.822842 15351 raft_consensus.cc:2454] T de805ef7511d4ee9b9bbed67e0a4186c P c05f7620bb884392931d3243a0b47a9e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 6.
I20240503 02:00:41.823256 15024 leader_election.cc:302] T de805ef7511d4ee9b9bbed67e0a4186c P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b602f6099cbb4758847e7e9550685372, c05f7620bb884392931d3243a0b47a9e; no voters:
I20240503 02:00:41.823441 17331 raft_consensus.cc:2781] T de805ef7511d4ee9b9bbed67e0a4186c P b602f6099cbb4758847e7e9550685372 [term 6 FOLLOWER]: Leader election won for term 6
I20240503 02:00:41.823516 17331 raft_consensus.cc:686] T de805ef7511d4ee9b9bbed67e0a4186c P b602f6099cbb4758847e7e9550685372 [term 6 LEADER]: Becoming Leader. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.823509 15087 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7b4f203f4048490ba599353a4cd92748"
dest_uuid: "b602f6099cbb4758847e7e9550685372"
from {username='slave'} at 127.0.0.1:34018
I20240503 02:00:41.823653 15087 raft_consensus.cc:482] T 7b4f203f4048490ba599353a4cd92748 P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.823711 15087 raft_consensus.cc:3037] T 7b4f203f4048490ba599353a4cd92748 P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.824710 15087 raft_consensus.cc:504] T 7b4f203f4048490ba599353a4cd92748 P b602f6099cbb4758847e7e9550685372 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.825109 15087 leader_election.cc:288] T 7b4f203f4048490ba599353a4cd92748 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 6 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817)
I20240503 02:00:41.823658 17331 consensus_queue.cc:237] T de805ef7511d4ee9b9bbed67e0a4186c P b602f6099cbb4758847e7e9550685372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.825740 15219 raft_consensus.cc:1261] T eade118a36da42e3ac08b96a13893a5c P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Refusing update from remote peer c05f7620bb884392931d3243a0b47a9e: Log matching property violated. Preceding OpId in replica: term: 2 index: 19. Preceding OpId from leader: term: 4 index: 22. (index mismatch)
W20240503 02:00:41.826069 14915 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48020: Illegal state: replica dcfdfa62edc34dab9525e2712d487582 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.826396 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4f203f4048490ba599353a4cd92748" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 6 candidate_status { last_received { term: 3 index: 7 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:41.826506 14958 raft_consensus.cc:3032] T 7b4f203f4048490ba599353a4cd92748 P dcfdfa62edc34dab9525e2712d487582 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 02:00:41.826545 14958 raft_consensus.cc:729] T 7b4f203f4048490ba599353a4cd92748 P dcfdfa62edc34dab9525e2712d487582 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:41.826673 14958 consensus_queue.cc:260] T 7b4f203f4048490ba599353a4cd92748 P dcfdfa62edc34dab9525e2712d487582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.826810 14958 raft_consensus.cc:3037] T 7b4f203f4048490ba599353a4cd92748 P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.826897 16986 consensus_queue.cc:1035] T eade118a36da42e3ac08b96a13893a5c P c05f7620bb884392931d3243a0b47a9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.000s
I20240503 02:00:41.827306 15087 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "590736f5f1ca4703821c409d882c2f77"
dest_uuid: "b602f6099cbb4758847e7e9550685372"
from {username='slave'} at 127.0.0.1:34018
I20240503 02:00:41.827415 15087 raft_consensus.cc:482] T 590736f5f1ca4703821c409d882c2f77 P b602f6099cbb4758847e7e9550685372 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.827471 15087 raft_consensus.cc:3037] T 590736f5f1ca4703821c409d882c2f77 P b602f6099cbb4758847e7e9550685372 [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:41.827864 14958 raft_consensus.cc:2396] T 7b4f203f4048490ba599353a4cd92748 P dcfdfa62edc34dab9525e2712d487582 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b602f6099cbb4758847e7e9550685372 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 7.
I20240503 02:00:41.828049 14811 catalog_manager.cc:5484] T de805ef7511d4ee9b9bbed67e0a4186c P b602f6099cbb4758847e7e9550685372 reported cstate change: term changed from 4 to 6, leader changed from 620596b893d84291b88b90dc9e317612 (127.0.99.131) to b602f6099cbb4758847e7e9550685372 (127.0.99.130). New cstate: current_term: 6 leader_uuid: "b602f6099cbb4758847e7e9550685372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:41.828502 15087 raft_consensus.cc:504] T 590736f5f1ca4703821c409d882c2f77 P b602f6099cbb4758847e7e9550685372 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.828912 15087 leader_election.cc:288] T 590736f5f1ca4703821c409d882c2f77 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 7 election: Requested vote from peers c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009), dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817)
I20240503 02:00:41.829360 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "590736f5f1ca4703821c409d882c2f77" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.829433 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "590736f5f1ca4703821c409d882c2f77" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:41.829484 15351 raft_consensus.cc:3037] T 590736f5f1ca4703821c409d882c2f77 P c05f7620bb884392931d3243a0b47a9e [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:41.829572 14958 raft_consensus.cc:3032] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582 [term 6 LEADER]: Stepping down as leader of term 6
I20240503 02:00:41.829622 14958 raft_consensus.cc:729] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:41.829685 15087 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "830da5cb4972453cb9bf2145a0839ced"
dest_uuid: "b602f6099cbb4758847e7e9550685372"
from {username='slave'} at 127.0.0.1:34018
I20240503 02:00:41.829784 15087 raft_consensus.cc:482] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.829758 14958 consensus_queue.cc:260] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.829830 15087 raft_consensus.cc:3037] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.829893 14958 raft_consensus.cc:3037] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582 [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:41.830528 15351 raft_consensus.cc:2454] T 590736f5f1ca4703821c409d882c2f77 P c05f7620bb884392931d3243a0b47a9e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 7.
I20240503 02:00:41.830753 14958 raft_consensus.cc:2396] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b602f6099cbb4758847e7e9550685372 for term 7 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20240503 02:00:41.830731 15087 raft_consensus.cc:504] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.831429 15087 leader_election.cc:288] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 6 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817)
I20240503 02:00:41.831429 15219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "830da5cb4972453cb9bf2145a0839ced" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:41.831562 15219 raft_consensus.cc:3037] T 830da5cb4972453cb9bf2145a0839ced P 620596b893d84291b88b90dc9e317612 [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.832190 15088 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9e9899666b15498791f62dab6d9dda16"
dest_uuid: "b602f6099cbb4758847e7e9550685372"
from {username='slave'} at 127.0.0.1:34018
I20240503 02:00:41.832317 15088 raft_consensus.cc:482] T 9e9899666b15498791f62dab6d9dda16 P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.832463 15088 raft_consensus.cc:3037] T 9e9899666b15498791f62dab6d9dda16 P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.832722 15219 raft_consensus.cc:2396] T 830da5cb4972453cb9bf2145a0839ced P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b602f6099cbb4758847e7e9550685372 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20240503 02:00:41.833328 15088 raft_consensus.cc:504] T 9e9899666b15498791f62dab6d9dda16 P b602f6099cbb4758847e7e9550685372 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:41.833709 15088 leader_election.cc:288] T 9e9899666b15498791f62dab6d9dda16 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 6 election: Requested vote from peers c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009), 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437)
I20240503 02:00:41.834414 15088 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6d2a1296a54a4b92a03c12954de0cc34"
dest_uuid: "b602f6099cbb4758847e7e9550685372"
from {username='slave'} at 127.0.0.1:34018
I20240503 02:00:41.834460 15219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e9899666b15498791f62dab6d9dda16" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 6 candidate_status { last_received { term: 3 index: 9 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:41.834546 15088 raft_consensus.cc:482] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.834605 15088 raft_consensus.cc:3037] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372 [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:41.834954 15024 leader_election.cc:302] T 590736f5f1ca4703821c409d882c2f77 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b602f6099cbb4758847e7e9550685372, c05f7620bb884392931d3243a0b47a9e; no voters:
I20240503 02:00:41.835134 17331 raft_consensus.cc:2781] T 590736f5f1ca4703821c409d882c2f77 P b602f6099cbb4758847e7e9550685372 [term 7 FOLLOWER]: Leader election won for term 7
I20240503 02:00:41.835206 17331 raft_consensus.cc:686] T 590736f5f1ca4703821c409d882c2f77 P b602f6099cbb4758847e7e9550685372 [term 7 LEADER]: Becoming Leader. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.834561 15219 raft_consensus.cc:3037] T 9e9899666b15498791f62dab6d9dda16 P 620596b893d84291b88b90dc9e317612 [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.835347 17331 consensus_queue.cc:237] T 590736f5f1ca4703821c409d882c2f77 P b602f6099cbb4758847e7e9550685372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.835582 15088 raft_consensus.cc:504] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.835884 15088 leader_election.cc:288] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 7 election: Requested vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:41.836328 15349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2a1296a54a4b92a03c12954de0cc34" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 7 candidate_status { last_received { term: 4 index: 11 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.836474 15219 raft_consensus.cc:2454] T 9e9899666b15498791f62dab6d9dda16 P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 6.
I20240503 02:00:41.836496 15349 raft_consensus.cc:3037] T 6d2a1296a54a4b92a03c12954de0cc34 P c05f7620bb884392931d3243a0b47a9e [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:41.836988 15087 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5736c284a85540a1bbab2be6df672fc1"
dest_uuid: "b602f6099cbb4758847e7e9550685372"
from {username='slave'} at 127.0.0.1:34018
I20240503 02:00:41.837175 15024 leader_election.cc:302] T 9e9899666b15498791f62dab6d9dda16 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, b602f6099cbb4758847e7e9550685372; no voters:
I20240503 02:00:41.837188 15087 raft_consensus.cc:482] T 5736c284a85540a1bbab2be6df672fc1 P b602f6099cbb4758847e7e9550685372 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.837365 15087 raft_consensus.cc:3037] T 5736c284a85540a1bbab2be6df672fc1 P b602f6099cbb4758847e7e9550685372 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.837543 15349 raft_consensus.cc:2396] T 6d2a1296a54a4b92a03c12954de0cc34 P c05f7620bb884392931d3243a0b47a9e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b602f6099cbb4758847e7e9550685372 for term 7 because replica has last-logged OpId of term: 5 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 11.
I20240503 02:00:41.837738 17347 raft_consensus.cc:2781] T 9e9899666b15498791f62dab6d9dda16 P b602f6099cbb4758847e7e9550685372 [term 6 FOLLOWER]: Leader election won for term 6
I20240503 02:00:41.837818 17347 raft_consensus.cc:686] T 9e9899666b15498791f62dab6d9dda16 P b602f6099cbb4758847e7e9550685372 [term 6 LEADER]: Becoming Leader. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.837947 17347 consensus_queue.cc:237] T 9e9899666b15498791f62dab6d9dda16 P b602f6099cbb4758847e7e9550685372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 3.9, Last appended by leader: 9, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:41.838711 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "830da5cb4972453cb9bf2145a0839ced" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:41.839283 14958 raft_consensus.cc:3037] T 830da5cb4972453cb9bf2145a0839ced P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.839358 15084 raft_consensus.cc:1261] T eade118a36da42e3ac08b96a13893a5c P b602f6099cbb4758847e7e9550685372 [term 4 FOLLOWER]: Refusing update from remote peer c05f7620bb884392931d3243a0b47a9e: Log matching property violated. Preceding OpId in replica: term: 2 index: 20. Preceding OpId from leader: term: 4 index: 22. (index mismatch)
I20240503 02:00:41.839778 17163 consensus_queue.cc:1035] T eade118a36da42e3ac08b96a13893a5c P c05f7620bb884392931d3243a0b47a9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20240503 02:00:41.840371 14958 raft_consensus.cc:2454] T 830da5cb4972453cb9bf2145a0839ced P dcfdfa62edc34dab9525e2712d487582 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 6.
I20240503 02:00:41.841356 15214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4f203f4048490ba599353a4cd92748" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 6 candidate_status { last_received { term: 3 index: 7 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:41.841517 15214 raft_consensus.cc:3037] T 7b4f203f4048490ba599353a4cd92748 P 620596b893d84291b88b90dc9e317612 [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.841506 15022 leader_election.cc:302] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b602f6099cbb4758847e7e9550685372, dcfdfa62edc34dab9525e2712d487582; no voters: 620596b893d84291b88b90dc9e317612
I20240503 02:00:41.841918 17331 raft_consensus.cc:2781] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372 [term 6 FOLLOWER]: Leader election won for term 6
I20240503 02:00:41.842007 17331 raft_consensus.cc:686] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372 [term 6 LEADER]: Becoming Leader. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.842136 17331 consensus_queue.cc:237] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.842274 14812 catalog_manager.cc:5484] T 590736f5f1ca4703821c409d882c2f77 P b602f6099cbb4758847e7e9550685372 reported cstate change: term changed from 6 to 7, leader changed from dcfdfa62edc34dab9525e2712d487582 (127.0.99.129) to b602f6099cbb4758847e7e9550685372 (127.0.99.130). New cstate: current_term: 7 leader_uuid: "b602f6099cbb4758847e7e9550685372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:41.842608 15214 raft_consensus.cc:2454] T 7b4f203f4048490ba599353a4cd92748 P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 6.
I20240503 02:00:41.843036 14956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2a1296a54a4b92a03c12954de0cc34" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 7 candidate_status { last_received { term: 4 index: 11 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:41.843200 14956 raft_consensus.cc:3037] T 6d2a1296a54a4b92a03c12954de0cc34 P dcfdfa62edc34dab9525e2712d487582 [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:41.843237 15024 leader_election.cc:302] T 7b4f203f4048490ba599353a4cd92748 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 620596b893d84291b88b90dc9e317612, b602f6099cbb4758847e7e9550685372; no voters: dcfdfa62edc34dab9525e2712d487582
I20240503 02:00:41.843513 17347 raft_consensus.cc:2781] T 7b4f203f4048490ba599353a4cd92748 P b602f6099cbb4758847e7e9550685372 [term 6 FOLLOWER]: Leader election won for term 6
I20240503 02:00:41.843590 17347 raft_consensus.cc:686] T 7b4f203f4048490ba599353a4cd92748 P b602f6099cbb4758847e7e9550685372 [term 6 LEADER]: Becoming Leader. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.843708 17347 consensus_queue.cc:237] T 7b4f203f4048490ba599353a4cd92748 P b602f6099cbb4758847e7e9550685372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.844216 14956 raft_consensus.cc:2454] T 6d2a1296a54a4b92a03c12954de0cc34 P dcfdfa62edc34dab9525e2712d487582 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 7.
W20240503 02:00:41.844245 15311 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50880: Illegal state: replica c05f7620bb884392931d3243a0b47a9e is not leader of this config: current role FOLLOWER
I20240503 02:00:41.844834 15022 leader_election.cc:302] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b602f6099cbb4758847e7e9550685372, dcfdfa62edc34dab9525e2712d487582; no voters: c05f7620bb884392931d3243a0b47a9e
I20240503 02:00:41.845080 17347 raft_consensus.cc:2781] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372 [term 7 FOLLOWER]: Leader election won for term 7
I20240503 02:00:41.845162 17347 raft_consensus.cc:686] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372 [term 7 LEADER]: Becoming Leader. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.845265 17347 consensus_queue.cc:237] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 4.11, Last appended by leader: 11, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.845913 15087 raft_consensus.cc:504] T 5736c284a85540a1bbab2be6df672fc1 P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.846386 15219 raft_consensus.cc:1261] T 830da5cb4972453cb9bf2145a0839ced P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Refusing update from remote peer b602f6099cbb4758847e7e9550685372: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20240503 02:00:41.846618 14956 raft_consensus.cc:1261] T 830da5cb4972453cb9bf2145a0839ced P dcfdfa62edc34dab9525e2712d487582 [term 6 FOLLOWER]: Refusing update from remote peer b602f6099cbb4758847e7e9550685372: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20240503 02:00:41.847128 17176 consensus_queue.cc:1035] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:41.847239 17176 consensus_queue.cc:1225] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372 [LEADER]: Peer 620596b893d84291b88b90dc9e317612 log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20240503 02:00:41.847707 15219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5736c284a85540a1bbab2be6df672fc1" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:41.847821 15219 raft_consensus.cc:3037] T 5736c284a85540a1bbab2be6df672fc1 P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.847839 15217 pending_rounds.cc:76] T 830da5cb4972453cb9bf2145a0839ced P 620596b893d84291b88b90dc9e317612: Aborting all ops after (but not including) 5
I20240503 02:00:41.847877 15217 pending_rounds.cc:97] T 830da5cb4972453cb9bf2145a0839ced P 620596b893d84291b88b90dc9e317612: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20240503 02:00:41.847944 15217 raft_consensus.cc:2866] T 830da5cb4972453cb9bf2145a0839ced P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 02:00:41.847877 15349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5736c284a85540a1bbab2be6df672fc1" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.848029 15349 raft_consensus.cc:3037] T 5736c284a85540a1bbab2be6df672fc1 P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.848099 17176 consensus_queue.cc:1035] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:41.847249 15087 leader_election.cc:288] T 5736c284a85540a1bbab2be6df672fc1 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 5 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:41.849198 15349 raft_consensus.cc:2454] T 5736c284a85540a1bbab2be6df672fc1 P c05f7620bb884392931d3243a0b47a9e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 5.
I20240503 02:00:41.849346 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e9899666b15498791f62dab6d9dda16" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 6 candidate_status { last_received { term: 3 index: 9 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.849447 15351 raft_consensus.cc:3032] T 9e9899666b15498791f62dab6d9dda16 P c05f7620bb884392931d3243a0b47a9e [term 5 LEADER]: Stepping down as leader of term 5
I20240503 02:00:41.849479 15351 raft_consensus.cc:729] T 9e9899666b15498791f62dab6d9dda16 P c05f7620bb884392931d3243a0b47a9e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:41.849452 15089 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eade118a36da42e3ac08b96a13893a5c"
dest_uuid: "b602f6099cbb4758847e7e9550685372"
from {username='slave'} at 127.0.0.1:34018
I20240503 02:00:41.849612 15089 raft_consensus.cc:482] T eade118a36da42e3ac08b96a13893a5c P b602f6099cbb4758847e7e9550685372 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.849671 15089 raft_consensus.cc:3037] T eade118a36da42e3ac08b96a13893a5c P b602f6099cbb4758847e7e9550685372 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.850005 17154 mvcc.cc:205] Tried to move back new op lower bound from 7023417924998352896 to 7023417924986867712. Current Snapshot: MvccSnapshot[applied={T|T < 7023417924998352896}]
I20240503 02:00:41.850405 15024 leader_election.cc:302] T 5736c284a85540a1bbab2be6df672fc1 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b602f6099cbb4758847e7e9550685372, c05f7620bb884392931d3243a0b47a9e; no voters:
I20240503 02:00:41.850622 17176 raft_consensus.cc:2781] T 5736c284a85540a1bbab2be6df672fc1 P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 02:00:41.850694 17176 raft_consensus.cc:686] T 5736c284a85540a1bbab2be6df672fc1 P b602f6099cbb4758847e7e9550685372 [term 5 LEADER]: Becoming Leader. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.850804 15089 raft_consensus.cc:504] T eade118a36da42e3ac08b96a13893a5c P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:41.850836 17176 consensus_queue.cc:237] T 5736c284a85540a1bbab2be6df672fc1 P b602f6099cbb4758847e7e9550685372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.850728 14812 catalog_manager.cc:5484] T 9e9899666b15498791f62dab6d9dda16 P b602f6099cbb4758847e7e9550685372 reported cstate change: term changed from 5 to 6, leader changed from c05f7620bb884392931d3243a0b47a9e (127.0.99.132) to b602f6099cbb4758847e7e9550685372 (127.0.99.130). New cstate: current_term: 6 leader_uuid: "b602f6099cbb4758847e7e9550685372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:41.850919 15351 consensus_queue.cc:260] T 9e9899666b15498791f62dab6d9dda16 P c05f7620bb884392931d3243a0b47a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.10, Last appended by leader: 10, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:41.851075 15351 raft_consensus.cc:3037] T 9e9899666b15498791f62dab6d9dda16 P c05f7620bb884392931d3243a0b47a9e [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.851027 14812 catalog_manager.cc:5484] T 7b4f203f4048490ba599353a4cd92748 P b602f6099cbb4758847e7e9550685372 reported cstate change: term changed from 5 to 6, leader changed from dcfdfa62edc34dab9525e2712d487582 (127.0.99.129) to b602f6099cbb4758847e7e9550685372 (127.0.99.130). New cstate: current_term: 6 leader_uuid: "b602f6099cbb4758847e7e9550685372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:41.851145 15089 leader_election.cc:288] T eade118a36da42e3ac08b96a13893a5c P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 5 election: Requested vote from peers c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009), 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437)
I20240503 02:00:41.851217 14812 catalog_manager.cc:5484] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372 reported cstate change: term changed from 5 to 7, leader changed from c05f7620bb884392931d3243a0b47a9e (127.0.99.132) to b602f6099cbb4758847e7e9550685372 (127.0.99.130). New cstate: current_term: 7 leader_uuid: "b602f6099cbb4758847e7e9550685372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:41.851495 14812 catalog_manager.cc:5484] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372 reported cstate change: term changed from 4 to 6, leader changed from 620596b893d84291b88b90dc9e317612 (127.0.99.131) to b602f6099cbb4758847e7e9550685372 (127.0.99.130). New cstate: current_term: 6 leader_uuid: "b602f6099cbb4758847e7e9550685372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:41.852880 15351 raft_consensus.cc:2396] T 9e9899666b15498791f62dab6d9dda16 P c05f7620bb884392931d3243a0b47a9e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b602f6099cbb4758847e7e9550685372 for term 6 because replica has last-logged OpId of term: 5 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 9.
W20240503 02:00:41.853049 15176 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57618: Illegal state: replica 620596b893d84291b88b90dc9e317612 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.854508 15214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eade118a36da42e3ac08b96a13893a5c" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 5 candidate_status { last_received { term: 4 index: 22 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:41.854638 15214 raft_consensus.cc:3037] T eade118a36da42e3ac08b96a13893a5c P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.854601 15089 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4cdcdde158054cdaabc1856a90fe1302"
dest_uuid: "b602f6099cbb4758847e7e9550685372"
from {username='slave'} at 127.0.0.1:34018
I20240503 02:00:41.854679 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eade118a36da42e3ac08b96a13893a5c" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 5 candidate_status { last_received { term: 4 index: 22 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.854722 15089 raft_consensus.cc:482] T 4cdcdde158054cdaabc1856a90fe1302 P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.854773 15089 raft_consensus.cc:3037] T 4cdcdde158054cdaabc1856a90fe1302 P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.854780 15351 raft_consensus.cc:3032] T eade118a36da42e3ac08b96a13893a5c P c05f7620bb884392931d3243a0b47a9e [term 4 LEADER]: Stepping down as leader of term 4
I20240503 02:00:41.854818 15351 raft_consensus.cc:729] T eade118a36da42e3ac08b96a13893a5c P c05f7620bb884392931d3243a0b47a9e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:41.854969 15351 consensus_queue.cc:260] T eade118a36da42e3ac08b96a13893a5c P c05f7620bb884392931d3243a0b47a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 4.22, Last appended by leader: 22, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:41.855141 15351 raft_consensus.cc:3037] T eade118a36da42e3ac08b96a13893a5c P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.855803 15214 raft_consensus.cc:2454] T eade118a36da42e3ac08b96a13893a5c P 620596b893d84291b88b90dc9e317612 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 5.
I20240503 02:00:41.856140 15351 raft_consensus.cc:2454] T eade118a36da42e3ac08b96a13893a5c P c05f7620bb884392931d3243a0b47a9e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 5.
I20240503 02:00:41.856171 15024 leader_election.cc:302] T eade118a36da42e3ac08b96a13893a5c P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, b602f6099cbb4758847e7e9550685372; no voters:
I20240503 02:00:41.856561 17331 raft_consensus.cc:2781] T eade118a36da42e3ac08b96a13893a5c P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 02:00:41.856654 17331 raft_consensus.cc:686] T eade118a36da42e3ac08b96a13893a5c P b602f6099cbb4758847e7e9550685372 [term 5 LEADER]: Becoming Leader. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.856617 15089 raft_consensus.cc:504] T 4cdcdde158054cdaabc1856a90fe1302 P b602f6099cbb4758847e7e9550685372 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.856778 17331 consensus_queue.cc:237] T eade118a36da42e3ac08b96a13893a5c P b602f6099cbb4758847e7e9550685372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 4.22, Last appended by leader: 22, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:41.856977 15089 leader_election.cc:288] T 4cdcdde158054cdaabc1856a90fe1302 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 6 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:41.857357 15217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcdde158054cdaabc1856a90fe1302" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:41.857470 15217 raft_consensus.cc:3037] T 4cdcdde158054cdaabc1856a90fe1302 P 620596b893d84291b88b90dc9e317612 [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.857997 15084 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6d98ba8805024eb79bf5df7a8225d784"
dest_uuid: "b602f6099cbb4758847e7e9550685372"
from {username='slave'} at 127.0.0.1:34018
I20240503 02:00:41.858093 15084 raft_consensus.cc:482] T 6d98ba8805024eb79bf5df7a8225d784 P b602f6099cbb4758847e7e9550685372 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.858139 15084 raft_consensus.cc:3037] T 6d98ba8805024eb79bf5df7a8225d784 P b602f6099cbb4758847e7e9550685372 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.858479 15217 raft_consensus.cc:2396] T 4cdcdde158054cdaabc1856a90fe1302 P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b602f6099cbb4758847e7e9550685372 for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240503 02:00:41.859009 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcdde158054cdaabc1856a90fe1302" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.859149 15351 raft_consensus.cc:3037] T 4cdcdde158054cdaabc1856a90fe1302 P c05f7620bb884392931d3243a0b47a9e [term 5 FOLLOWER]: Advancing to term 6
W20240503 02:00:41.859247 15048 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33982: Illegal state: replica b602f6099cbb4758847e7e9550685372 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.859736 14816 catalog_manager.cc:5484] T 5736c284a85540a1bbab2be6df672fc1 P b602f6099cbb4758847e7e9550685372 reported cstate change: term changed from 3 to 5, leader changed from 620596b893d84291b88b90dc9e317612 (127.0.99.131) to b602f6099cbb4758847e7e9550685372 (127.0.99.130). New cstate: current_term: 5 leader_uuid: "b602f6099cbb4758847e7e9550685372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:41.860105 15084 raft_consensus.cc:504] T 6d98ba8805024eb79bf5df7a8225d784 P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.860065 14816 catalog_manager.cc:5484] T eade118a36da42e3ac08b96a13893a5c P b602f6099cbb4758847e7e9550685372 reported cstate change: term changed from 4 to 5, leader changed from c05f7620bb884392931d3243a0b47a9e (127.0.99.132) to b602f6099cbb4758847e7e9550685372 (127.0.99.130). New cstate: current_term: 5 leader_uuid: "b602f6099cbb4758847e7e9550685372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:41.860350 15219 raft_consensus.cc:2396] T 5736c284a85540a1bbab2be6df672fc1 P 620596b893d84291b88b90dc9e317612 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate b602f6099cbb4758847e7e9550685372 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20240503 02:00:41.860525 15084 leader_election.cc:288] T 6d98ba8805024eb79bf5df7a8225d784 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 5 election: Requested vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437)
I20240503 02:00:41.861282 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d98ba8805024eb79bf5df7a8225d784" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 5 candidate_status { last_received { term: 3 index: 7 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:41.861402 15084 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "42ceeb7b3c4146cd8a2e18b3e9ae987c"
dest_uuid: "b602f6099cbb4758847e7e9550685372"
from {username='slave'} at 127.0.0.1:34018
I20240503 02:00:41.861527 15084 raft_consensus.cc:482] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.861589 15084 raft_consensus.cc:3037] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.861470 14958 raft_consensus.cc:3032] T 6d98ba8805024eb79bf5df7a8225d784 P dcfdfa62edc34dab9525e2712d487582 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 02:00:41.861673 14958 raft_consensus.cc:729] T 6d98ba8805024eb79bf5df7a8225d784 P dcfdfa62edc34dab9525e2712d487582 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:41.861871 14958 consensus_queue.cc:260] T 6d98ba8805024eb79bf5df7a8225d784 P dcfdfa62edc34dab9525e2712d487582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.862035 14958 raft_consensus.cc:3037] T 6d98ba8805024eb79bf5df7a8225d784 P dcfdfa62edc34dab9525e2712d487582 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.862433 15084 raft_consensus.cc:504] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P b602f6099cbb4758847e7e9550685372 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:41.863003 14958 raft_consensus.cc:2396] T 6d98ba8805024eb79bf5df7a8225d784 P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate b602f6099cbb4758847e7e9550685372 for term 5 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 7.
I20240503 02:00:41.863065 15084 leader_election.cc:288] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 6 election: Requested vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437)
I20240503 02:00:41.863081 14954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ceeb7b3c4146cd8a2e18b3e9ae987c" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:41.863168 14954 raft_consensus.cc:3032] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P dcfdfa62edc34dab9525e2712d487582 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 02:00:41.863200 14954 raft_consensus.cc:729] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P dcfdfa62edc34dab9525e2712d487582 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:41.863329 14954 consensus_queue.cc:260] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P dcfdfa62edc34dab9525e2712d487582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:41.863471 14954 raft_consensus.cc:3037] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.863505 15217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ceeb7b3c4146cd8a2e18b3e9ae987c" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:41.863622 15217 raft_consensus.cc:3037] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P 620596b893d84291b88b90dc9e317612 [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.863714 15351 raft_consensus.cc:2454] T 4cdcdde158054cdaabc1856a90fe1302 P c05f7620bb884392931d3243a0b47a9e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 6.
I20240503 02:00:41.864078 15024 leader_election.cc:302] T 4cdcdde158054cdaabc1856a90fe1302 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b602f6099cbb4758847e7e9550685372, c05f7620bb884392931d3243a0b47a9e; no voters: 620596b893d84291b88b90dc9e317612
I20240503 02:00:41.864302 14954 raft_consensus.cc:2396] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P dcfdfa62edc34dab9525e2712d487582 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b602f6099cbb4758847e7e9550685372 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240503 02:00:41.864343 17331 raft_consensus.cc:2781] T 4cdcdde158054cdaabc1856a90fe1302 P b602f6099cbb4758847e7e9550685372 [term 6 FOLLOWER]: Leader election won for term 6
I20240503 02:00:41.864461 17331 raft_consensus.cc:686] T 4cdcdde158054cdaabc1856a90fe1302 P b602f6099cbb4758847e7e9550685372 [term 6 LEADER]: Becoming Leader. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.864564 15217 raft_consensus.cc:2454] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 6.
I20240503 02:00:41.864598 17331 consensus_queue.cc:237] T 4cdcdde158054cdaabc1856a90fe1302 P b602f6099cbb4758847e7e9550685372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.864853 15024 leader_election.cc:302] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 620596b893d84291b88b90dc9e317612, b602f6099cbb4758847e7e9550685372; no voters: dcfdfa62edc34dab9525e2712d487582
I20240503 02:00:41.865018 17347 raft_consensus.cc:2781] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P b602f6099cbb4758847e7e9550685372 [term 6 FOLLOWER]: Leader election won for term 6
I20240503 02:00:41.865100 17347 raft_consensus.cc:686] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P b602f6099cbb4758847e7e9550685372 [term 6 LEADER]: Becoming Leader. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.865219 17347 consensus_queue.cc:237] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P b602f6099cbb4758847e7e9550685372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:41.865072 15084 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6b0c8b72e1884529a8a9b83d2a734d71"
dest_uuid: "b602f6099cbb4758847e7e9550685372"
from {username='slave'} at 127.0.0.1:34018
I20240503 02:00:41.867449 15084 raft_consensus.cc:482] T 6b0c8b72e1884529a8a9b83d2a734d71 P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.867545 15084 raft_consensus.cc:3037] T 6b0c8b72e1884529a8a9b83d2a734d71 P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Advancing to term 6
W20240503 02:00:41.868177 15176 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57618: Illegal state: replica 620596b893d84291b88b90dc9e317612 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.868476 15084 raft_consensus.cc:504] T 6b0c8b72e1884529a8a9b83d2a734d71 P b602f6099cbb4758847e7e9550685372 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.868818 15084 leader_election.cc:288] T 6b0c8b72e1884529a8a9b83d2a734d71 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 6 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817)
I20240503 02:00:41.869243 14954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b0c8b72e1884529a8a9b83d2a734d71" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:41.869256 15219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d98ba8805024eb79bf5df7a8225d784" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 5 candidate_status { last_received { term: 3 index: 7 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:41.869369 15219 raft_consensus.cc:3037] T 6d98ba8805024eb79bf5df7a8225d784 P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.869369 14954 raft_consensus.cc:3037] T 6b0c8b72e1884529a8a9b83d2a734d71 P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.869990 14811 catalog_manager.cc:5484] T 4cdcdde158054cdaabc1856a90fe1302 P b602f6099cbb4758847e7e9550685372 reported cstate change: term changed from 4 to 6, leader changed from 620596b893d84291b88b90dc9e317612 (127.0.99.131) to b602f6099cbb4758847e7e9550685372 (127.0.99.130). New cstate: current_term: 6 leader_uuid: "b602f6099cbb4758847e7e9550685372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:41.870196 14811 catalog_manager.cc:5484] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P b602f6099cbb4758847e7e9550685372 reported cstate change: term changed from 5 to 6, leader changed from dcfdfa62edc34dab9525e2712d487582 (127.0.99.129) to b602f6099cbb4758847e7e9550685372 (127.0.99.130). New cstate: current_term: 6 leader_uuid: "b602f6099cbb4758847e7e9550685372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:41.870347 14954 raft_consensus.cc:2454] T 6b0c8b72e1884529a8a9b83d2a734d71 P dcfdfa62edc34dab9525e2712d487582 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 6.
I20240503 02:00:41.870347 15219 raft_consensus.cc:2454] T 6d98ba8805024eb79bf5df7a8225d784 P 620596b893d84291b88b90dc9e317612 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 5.
I20240503 02:00:41.870651 15217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b0c8b72e1884529a8a9b83d2a734d71" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:41.870723 15022 leader_election.cc:302] T 6b0c8b72e1884529a8a9b83d2a734d71 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b602f6099cbb4758847e7e9550685372, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:41.870754 15217 raft_consensus.cc:3037] T 6b0c8b72e1884529a8a9b83d2a734d71 P 620596b893d84291b88b90dc9e317612 [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.870891 16496 raft_consensus.cc:2781] T 6b0c8b72e1884529a8a9b83d2a734d71 P b602f6099cbb4758847e7e9550685372 [term 6 FOLLOWER]: Leader election won for term 6
I20240503 02:00:41.870958 16496 raft_consensus.cc:686] T 6b0c8b72e1884529a8a9b83d2a734d71 P b602f6099cbb4758847e7e9550685372 [term 6 LEADER]: Becoming Leader. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.871083 16496 consensus_queue.cc:237] T 6b0c8b72e1884529a8a9b83d2a734d71 P b602f6099cbb4758847e7e9550685372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.871704 15217 raft_consensus.cc:2396] T 6b0c8b72e1884529a8a9b83d2a734d71 P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b602f6099cbb4758847e7e9550685372 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20240503 02:00:41.872023 15024 leader_election.cc:302] T 6d98ba8805024eb79bf5df7a8225d784 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 620596b893d84291b88b90dc9e317612, b602f6099cbb4758847e7e9550685372; no voters: dcfdfa62edc34dab9525e2712d487582
I20240503 02:00:41.872176 16496 raft_consensus.cc:2781] T 6d98ba8805024eb79bf5df7a8225d784 P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 02:00:41.872252 16496 raft_consensus.cc:686] T 6d98ba8805024eb79bf5df7a8225d784 P b602f6099cbb4758847e7e9550685372 [term 5 LEADER]: Becoming Leader. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.872411 16496 consensus_queue.cc:237] T 6d98ba8805024eb79bf5df7a8225d784 P b602f6099cbb4758847e7e9550685372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.7, Last appended by leader: 7, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.873342 15084 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0989d98050744df6a1c699cb86ada505"
dest_uuid: "b602f6099cbb4758847e7e9550685372"
from {username='slave'} at 127.0.0.1:34018
I20240503 02:00:41.873471 15084 raft_consensus.cc:482] T 0989d98050744df6a1c699cb86ada505 P b602f6099cbb4758847e7e9550685372 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.873533 15084 raft_consensus.cc:3037] T 0989d98050744df6a1c699cb86ada505 P b602f6099cbb4758847e7e9550685372 [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:41.874521 15084 raft_consensus.cc:504] T 0989d98050744df6a1c699cb86ada505 P b602f6099cbb4758847e7e9550685372 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.874876 15084 leader_election.cc:288] T 0989d98050744df6a1c699cb86ada505 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 7 election: Requested vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
W20240503 02:00:41.875684 15311 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50880: Illegal state: replica c05f7620bb884392931d3243a0b47a9e is not leader of this config: current role FOLLOWER
I20240503 02:00:41.876521 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0989d98050744df6a1c699cb86ada505" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.876645 15351 raft_consensus.cc:3037] T 0989d98050744df6a1c699cb86ada505 P c05f7620bb884392931d3243a0b47a9e [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:41.876605 14816 catalog_manager.cc:5484] T 6b0c8b72e1884529a8a9b83d2a734d71 P b602f6099cbb4758847e7e9550685372 reported cstate change: term changed from 4 to 6, leader changed from 620596b893d84291b88b90dc9e317612 (127.0.99.131) to b602f6099cbb4758847e7e9550685372 (127.0.99.130). New cstate: current_term: 6 leader_uuid: "b602f6099cbb4758847e7e9550685372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:41.876875 14816 catalog_manager.cc:5484] T 6d98ba8805024eb79bf5df7a8225d784 P b602f6099cbb4758847e7e9550685372 reported cstate change: term changed from 4 to 5, leader changed from dcfdfa62edc34dab9525e2712d487582 (127.0.99.129) to b602f6099cbb4758847e7e9550685372 (127.0.99.130). New cstate: current_term: 5 leader_uuid: "b602f6099cbb4758847e7e9550685372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: HEALTHY } } }
W20240503 02:00:41.877902 15311 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50950: Illegal state: replica c05f7620bb884392931d3243a0b47a9e is not leader of this config: current role FOLLOWER
I20240503 02:00:41.877959 15351 raft_consensus.cc:2396] T 0989d98050744df6a1c699cb86ada505 P c05f7620bb884392931d3243a0b47a9e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b602f6099cbb4758847e7e9550685372 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 02:00:41.878616 15084 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d805ee04df39406ba30b3b3b9e349507"
dest_uuid: "b602f6099cbb4758847e7e9550685372"
from {username='slave'} at 127.0.0.1:34018
I20240503 02:00:41.879537 15084 raft_consensus.cc:482] T d805ee04df39406ba30b3b3b9e349507 P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.879622 15084 raft_consensus.cc:3037] T d805ee04df39406ba30b3b3b9e349507 P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.880089 14956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0989d98050744df6a1c699cb86ada505" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:41.880244 14956 raft_consensus.cc:3032] T 0989d98050744df6a1c699cb86ada505 P dcfdfa62edc34dab9525e2712d487582 [term 6 LEADER]: Stepping down as leader of term 6
I20240503 02:00:41.880317 14956 raft_consensus.cc:729] T 0989d98050744df6a1c699cb86ada505 P dcfdfa62edc34dab9525e2712d487582 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:41.881459 14956 consensus_queue.cc:260] T 0989d98050744df6a1c699cb86ada505 P dcfdfa62edc34dab9525e2712d487582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.881670 14956 raft_consensus.cc:3037] T 0989d98050744df6a1c699cb86ada505 P dcfdfa62edc34dab9525e2712d487582 [term 6 FOLLOWER]: Advancing to term 7
W20240503 02:00:41.882162 14915 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48020: Illegal state: replica dcfdfa62edc34dab9525e2712d487582 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.882781 14956 raft_consensus.cc:2396] T 0989d98050744df6a1c699cb86ada505 P dcfdfa62edc34dab9525e2712d487582 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b602f6099cbb4758847e7e9550685372 for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 02:00:41.883548 15022 leader_election.cc:302] T 0989d98050744df6a1c699cb86ada505 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b602f6099cbb4758847e7e9550685372; no voters: c05f7620bb884392931d3243a0b47a9e, dcfdfa62edc34dab9525e2712d487582
I20240503 02:00:41.883656 15084 raft_consensus.cc:504] T d805ee04df39406ba30b3b3b9e349507 P b602f6099cbb4758847e7e9550685372 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.883757 16496 raft_consensus.cc:2726] T 0989d98050744df6a1c699cb86ada505 P b602f6099cbb4758847e7e9550685372 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240503 02:00:41.884070 15084 leader_election.cc:288] T d805ee04df39406ba30b3b3b9e349507 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 6 election: Requested vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:41.884358 14956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d805ee04df39406ba30b3b3b9e349507" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 6 candidate_status { last_received { term: 3 index: 7 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:41.884575 14956 raft_consensus.cc:3032] T d805ee04df39406ba30b3b3b9e349507 P dcfdfa62edc34dab9525e2712d487582 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 02:00:41.884554 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d805ee04df39406ba30b3b3b9e349507" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 6 candidate_status { last_received { term: 3 index: 7 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.884672 15351 raft_consensus.cc:3037] T d805ee04df39406ba30b3b3b9e349507 P c05f7620bb884392931d3243a0b47a9e [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.884756 14956 raft_consensus.cc:729] T d805ee04df39406ba30b3b3b9e349507 P dcfdfa62edc34dab9525e2712d487582 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
W20240503 02:00:41.885318 15048 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33982: Illegal state: replica b602f6099cbb4758847e7e9550685372 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.885564 15351 raft_consensus.cc:2454] T d805ee04df39406ba30b3b3b9e349507 P c05f7620bb884392931d3243a0b47a9e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 6.
I20240503 02:00:41.886063 15084 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c74522fbb5c64d27b5c95f05896b0716"
dest_uuid: "b602f6099cbb4758847e7e9550685372"
from {username='slave'} at 127.0.0.1:34018
I20240503 02:00:41.886183 15084 raft_consensus.cc:482] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.886233 15084 raft_consensus.cc:3037] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.886026 14956 consensus_queue.cc:260] T d805ee04df39406ba30b3b3b9e349507 P dcfdfa62edc34dab9525e2712d487582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.8, Last appended by leader: 8, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.887038 14956 raft_consensus.cc:3037] T d805ee04df39406ba30b3b3b9e349507 P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.887332 15024 leader_election.cc:302] T d805ee04df39406ba30b3b3b9e349507 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b602f6099cbb4758847e7e9550685372, c05f7620bb884392931d3243a0b47a9e; no voters:
I20240503 02:00:41.887297 15084 raft_consensus.cc:504] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.887706 16496 raft_consensus.cc:2781] T d805ee04df39406ba30b3b3b9e349507 P b602f6099cbb4758847e7e9550685372 [term 6 FOLLOWER]: Leader election won for term 6
I20240503 02:00:41.887786 16496 raft_consensus.cc:686] T d805ee04df39406ba30b3b3b9e349507 P b602f6099cbb4758847e7e9550685372 [term 6 LEADER]: Becoming Leader. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.887928 16496 consensus_queue.cc:237] T d805ee04df39406ba30b3b3b9e349507 P b602f6099cbb4758847e7e9550685372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.888250 15084 leader_election.cc:288] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 6 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:41.888784 14956 raft_consensus.cc:2396] T d805ee04df39406ba30b3b3b9e349507 P dcfdfa62edc34dab9525e2712d487582 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b602f6099cbb4758847e7e9550685372 for term 6 because replica has last-logged OpId of term: 5 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 7.
I20240503 02:00:41.889139 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c74522fbb5c64d27b5c95f05896b0716" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 6 candidate_status { last_received { term: 3 index: 26 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.889731 15351 raft_consensus.cc:3037] T c74522fbb5c64d27b5c95f05896b0716 P c05f7620bb884392931d3243a0b47a9e [term 5 FOLLOWER]: Advancing to term 6
W20240503 02:00:41.890205 15176 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57688: Illegal state: replica 620596b893d84291b88b90dc9e317612 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.890400 15084 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2b757c430aaa42f382d174ca57f5fcfc"
dest_uuid: "b602f6099cbb4758847e7e9550685372"
from {username='slave'} at 127.0.0.1:34018
I20240503 02:00:41.890525 15084 raft_consensus.cc:482] T 2b757c430aaa42f382d174ca57f5fcfc P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.890592 15084 raft_consensus.cc:3037] T 2b757c430aaa42f382d174ca57f5fcfc P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.891506 15219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c74522fbb5c64d27b5c95f05896b0716" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 6 candidate_status { last_received { term: 3 index: 26 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:41.891566 15084 raft_consensus.cc:504] T 2b757c430aaa42f382d174ca57f5fcfc P b602f6099cbb4758847e7e9550685372 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.891669 15351 raft_consensus.cc:2454] T c74522fbb5c64d27b5c95f05896b0716 P c05f7620bb884392931d3243a0b47a9e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 6.
I20240503 02:00:41.891732 15219 raft_consensus.cc:3037] T c74522fbb5c64d27b5c95f05896b0716 P 620596b893d84291b88b90dc9e317612 [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.892067 15024 leader_election.cc:302] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b602f6099cbb4758847e7e9550685372, c05f7620bb884392931d3243a0b47a9e; no voters:
I20240503 02:00:41.892235 16496 raft_consensus.cc:2781] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372 [term 6 FOLLOWER]: Leader election won for term 6
I20240503 02:00:41.892305 16496 raft_consensus.cc:686] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372 [term 6 LEADER]: Becoming Leader. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.892465 16496 consensus_queue.cc:237] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 3.26, Last appended by leader: 26, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.893875 15214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b757c430aaa42f382d174ca57f5fcfc" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 6 candidate_status { last_received { term: 3 index: 7 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:41.894002 15214 raft_consensus.cc:3037] T 2b757c430aaa42f382d174ca57f5fcfc P 620596b893d84291b88b90dc9e317612 [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.894610 15219 raft_consensus.cc:2396] T c74522fbb5c64d27b5c95f05896b0716 P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b602f6099cbb4758847e7e9550685372 for term 6 because replica has last-logged OpId of term: 4 index: 26, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 26.
I20240503 02:00:41.894706 15214 raft_consensus.cc:2454] T 2b757c430aaa42f382d174ca57f5fcfc P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 6.
I20240503 02:00:41.895043 15024 leader_election.cc:302] T 2b757c430aaa42f382d174ca57f5fcfc P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, b602f6099cbb4758847e7e9550685372; no voters:
I20240503 02:00:41.895409 16496 raft_consensus.cc:2781] T 2b757c430aaa42f382d174ca57f5fcfc P b602f6099cbb4758847e7e9550685372 [term 6 FOLLOWER]: Leader election won for term 6
I20240503 02:00:41.895483 16496 raft_consensus.cc:686] T 2b757c430aaa42f382d174ca57f5fcfc P b602f6099cbb4758847e7e9550685372 [term 6 LEADER]: Becoming Leader. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.895604 16496 consensus_queue.cc:237] T 2b757c430aaa42f382d174ca57f5fcfc P b602f6099cbb4758847e7e9550685372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.897174 14811 catalog_manager.cc:5484] T d805ee04df39406ba30b3b3b9e349507 P b602f6099cbb4758847e7e9550685372 reported cstate change: term changed from 5 to 6, leader changed from dcfdfa62edc34dab9525e2712d487582 (127.0.99.129) to b602f6099cbb4758847e7e9550685372 (127.0.99.130). New cstate: current_term: 6 leader_uuid: "b602f6099cbb4758847e7e9550685372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: HEALTHY } } }
W20240503 02:00:41.897881 15311 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50950: Illegal state: replica c05f7620bb884392931d3243a0b47a9e is not leader of this config: current role FOLLOWER
I20240503 02:00:41.897910 15084 leader_election.cc:288] T 2b757c430aaa42f382d174ca57f5fcfc P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 6 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817)
W20240503 02:00:41.899163 15176 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57618: Illegal state: replica 620596b893d84291b88b90dc9e317612 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.899459 14956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b757c430aaa42f382d174ca57f5fcfc" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 6 candidate_status { last_received { term: 3 index: 7 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:41.899493 15084 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6b225db529f14e7386f21f42bf31493b"
dest_uuid: "b602f6099cbb4758847e7e9550685372"
from {username='slave'} at 127.0.0.1:34018
I20240503 02:00:41.899610 14956 raft_consensus.cc:3032] T 2b757c430aaa42f382d174ca57f5fcfc P dcfdfa62edc34dab9525e2712d487582 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 02:00:41.899619 15084 raft_consensus.cc:482] T 6b225db529f14e7386f21f42bf31493b P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.899657 14956 raft_consensus.cc:729] T 2b757c430aaa42f382d174ca57f5fcfc P dcfdfa62edc34dab9525e2712d487582 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:41.899677 15084 raft_consensus.cc:3037] T 6b225db529f14e7386f21f42bf31493b P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.899807 14956 consensus_queue.cc:260] T 2b757c430aaa42f382d174ca57f5fcfc P dcfdfa62edc34dab9525e2712d487582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.8, Last appended by leader: 8, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.899957 14956 raft_consensus.cc:3037] T 2b757c430aaa42f382d174ca57f5fcfc P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.900774 15084 raft_consensus.cc:504] T 6b225db529f14e7386f21f42bf31493b P b602f6099cbb4758847e7e9550685372 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.900977 14956 raft_consensus.cc:2396] T 2b757c430aaa42f382d174ca57f5fcfc P dcfdfa62edc34dab9525e2712d487582 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b602f6099cbb4758847e7e9550685372 for term 6 because replica has last-logged OpId of term: 5 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 7.
I20240503 02:00:41.901232 15084 leader_election.cc:288] T 6b225db529f14e7386f21f42bf31493b P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 6 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817)
I20240503 02:00:41.901890 15084 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ebd7cdd284a74741b338e9f4778502b5"
dest_uuid: "b602f6099cbb4758847e7e9550685372"
from {username='slave'} at 127.0.0.1:34018
I20240503 02:00:41.902027 15084 raft_consensus.cc:482] T ebd7cdd284a74741b338e9f4778502b5 P b602f6099cbb4758847e7e9550685372 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.902096 15084 raft_consensus.cc:3037] T ebd7cdd284a74741b338e9f4778502b5 P b602f6099cbb4758847e7e9550685372 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.902083 14956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b225db529f14e7386f21f42bf31493b" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 6 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:41.902192 14956 raft_consensus.cc:3032] T 6b225db529f14e7386f21f42bf31493b P dcfdfa62edc34dab9525e2712d487582 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 02:00:41.902226 14956 raft_consensus.cc:729] T 6b225db529f14e7386f21f42bf31493b P dcfdfa62edc34dab9525e2712d487582 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:41.902372 14956 consensus_queue.cc:260] T 6b225db529f14e7386f21f42bf31493b P dcfdfa62edc34dab9525e2712d487582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.9, Last appended by leader: 9, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.902494 14956 raft_consensus.cc:3037] T 6b225db529f14e7386f21f42bf31493b P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.902740 15219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b225db529f14e7386f21f42bf31493b" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 6 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:41.902864 15219 raft_consensus.cc:3037] T 6b225db529f14e7386f21f42bf31493b P 620596b893d84291b88b90dc9e317612 [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.903082 15084 raft_consensus.cc:504] T ebd7cdd284a74741b338e9f4778502b5 P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.903239 14811 catalog_manager.cc:5484] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372 reported cstate change: term changed from 4 to 6, leader changed from 620596b893d84291b88b90dc9e317612 (127.0.99.131) to b602f6099cbb4758847e7e9550685372 (127.0.99.130). New cstate: current_term: 6 leader_uuid: "b602f6099cbb4758847e7e9550685372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:41.903467 14956 raft_consensus.cc:2396] T 6b225db529f14e7386f21f42bf31493b P dcfdfa62edc34dab9525e2712d487582 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b602f6099cbb4758847e7e9550685372 for term 6 because replica has last-logged OpId of term: 5 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 8.
I20240503 02:00:41.903465 15084 leader_election.cc:288] T ebd7cdd284a74741b338e9f4778502b5 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 5 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:41.903491 14811 catalog_manager.cc:5484] T 2b757c430aaa42f382d174ca57f5fcfc P b602f6099cbb4758847e7e9550685372 reported cstate change: term changed from 5 to 6, leader changed from dcfdfa62edc34dab9525e2712d487582 (127.0.99.129) to b602f6099cbb4758847e7e9550685372 (127.0.99.130). New cstate: current_term: 6 leader_uuid: "b602f6099cbb4758847e7e9550685372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:41.904206 15084 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f85a361ce0254472b37526362d4edf0e"
dest_uuid: "b602f6099cbb4758847e7e9550685372"
from {username='slave'} at 127.0.0.1:34018
I20240503 02:00:41.904330 15084 raft_consensus.cc:482] T f85a361ce0254472b37526362d4edf0e P b602f6099cbb4758847e7e9550685372 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.904454 15084 raft_consensus.cc:3037] T f85a361ce0254472b37526362d4edf0e P b602f6099cbb4758847e7e9550685372 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.904608 15214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebd7cdd284a74741b338e9f4778502b5" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:41.904747 15214 raft_consensus.cc:3037] T ebd7cdd284a74741b338e9f4778502b5 P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.905371 15084 raft_consensus.cc:504] T f85a361ce0254472b37526362d4edf0e P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:41.905689 15214 raft_consensus.cc:2396] T ebd7cdd284a74741b338e9f4778502b5 P 620596b893d84291b88b90dc9e317612 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate b602f6099cbb4758847e7e9550685372 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20240503 02:00:41.905738 15084 leader_election.cc:288] T f85a361ce0254472b37526362d4edf0e P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 5 election: Requested vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437)
I20240503 02:00:41.906649 14956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85a361ce0254472b37526362d4edf0e" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 5 candidate_status { last_received { term: 4 index: 22 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:41.906769 14956 raft_consensus.cc:3032] T f85a361ce0254472b37526362d4edf0e P dcfdfa62edc34dab9525e2712d487582 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 02:00:41.906806 14956 raft_consensus.cc:729] T f85a361ce0254472b37526362d4edf0e P dcfdfa62edc34dab9525e2712d487582 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
W20240503 02:00:41.906890 14915 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48020: Illegal state: replica dcfdfa62edc34dab9525e2712d487582 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.906956 14956 consensus_queue.cc:260] T f85a361ce0254472b37526362d4edf0e P dcfdfa62edc34dab9525e2712d487582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 4.22, Last appended by leader: 22, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:41.907019 15214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85a361ce0254472b37526362d4edf0e" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 5 candidate_status { last_received { term: 4 index: 22 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:41.907106 14956 raft_consensus.cc:3037] T f85a361ce0254472b37526362d4edf0e P dcfdfa62edc34dab9525e2712d487582 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.907119 15214 raft_consensus.cc:3037] T f85a361ce0254472b37526362d4edf0e P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.907382 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebd7cdd284a74741b338e9f4778502b5" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.907495 15351 raft_consensus.cc:3037] T ebd7cdd284a74741b338e9f4778502b5 P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.908047 15214 raft_consensus.cc:2454] T f85a361ce0254472b37526362d4edf0e P 620596b893d84291b88b90dc9e317612 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 5.
I20240503 02:00:41.908156 14956 raft_consensus.cc:2454] T f85a361ce0254472b37526362d4edf0e P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 5.
I20240503 02:00:41.908545 15024 leader_election.cc:302] T f85a361ce0254472b37526362d4edf0e P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, b602f6099cbb4758847e7e9550685372; no voters:
I20240503 02:00:41.908846 15084 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "454a24fd6d964d73ae3d5057f90fe36d"
dest_uuid: "b602f6099cbb4758847e7e9550685372"
from {username='slave'} at 127.0.0.1:34018
I20240503 02:00:41.908878 17331 raft_consensus.cc:2781] T f85a361ce0254472b37526362d4edf0e P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 02:00:41.908984 17331 raft_consensus.cc:686] T f85a361ce0254472b37526362d4edf0e P b602f6099cbb4758847e7e9550685372 [term 5 LEADER]: Becoming Leader. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.908986 15084 raft_consensus.cc:482] T 454a24fd6d964d73ae3d5057f90fe36d P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.909083 15084 raft_consensus.cc:3037] T 454a24fd6d964d73ae3d5057f90fe36d P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.909129 17331 consensus_queue.cc:237] T f85a361ce0254472b37526362d4edf0e P b602f6099cbb4758847e7e9550685372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 4.22, Last appended by leader: 22, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:41.909312 15351 raft_consensus.cc:2454] T ebd7cdd284a74741b338e9f4778502b5 P c05f7620bb884392931d3243a0b47a9e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 5.
I20240503 02:00:41.910087 15084 raft_consensus.cc:504] T 454a24fd6d964d73ae3d5057f90fe36d P b602f6099cbb4758847e7e9550685372 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.910418 15084 leader_election.cc:288] T 454a24fd6d964d73ae3d5057f90fe36d P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 6 election: Requested vote from peers c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009), dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817)
I20240503 02:00:41.910946 14956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "454a24fd6d964d73ae3d5057f90fe36d" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:41.911068 14956 raft_consensus.cc:3032] T 454a24fd6d964d73ae3d5057f90fe36d P dcfdfa62edc34dab9525e2712d487582 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 02:00:41.911108 14956 raft_consensus.cc:729] T 454a24fd6d964d73ae3d5057f90fe36d P dcfdfa62edc34dab9525e2712d487582 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:41.911181 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "454a24fd6d964d73ae3d5057f90fe36d" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
W20240503 02:00:41.911264 15311 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50880: Illegal state: replica c05f7620bb884392931d3243a0b47a9e is not leader of this config: current role FOLLOWER
I20240503 02:00:41.911300 15351 raft_consensus.cc:3037] T 454a24fd6d964d73ae3d5057f90fe36d P c05f7620bb884392931d3243a0b47a9e [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.911384 14956 consensus_queue.cc:260] T 454a24fd6d964d73ae3d5057f90fe36d P dcfdfa62edc34dab9525e2712d487582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.911523 14956 raft_consensus.cc:3037] T 454a24fd6d964d73ae3d5057f90fe36d P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.911839 14811 catalog_manager.cc:5484] T f85a361ce0254472b37526362d4edf0e P b602f6099cbb4758847e7e9550685372 reported cstate change: term changed from 4 to 5, leader changed from dcfdfa62edc34dab9525e2712d487582 (127.0.99.129) to b602f6099cbb4758847e7e9550685372 (127.0.99.130). New cstate: current_term: 5 leader_uuid: "b602f6099cbb4758847e7e9550685372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:41.912292 15351 raft_consensus.cc:2454] T 454a24fd6d964d73ae3d5057f90fe36d P c05f7620bb884392931d3243a0b47a9e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 6.
I20240503 02:00:41.912544 14956 raft_consensus.cc:2396] T 454a24fd6d964d73ae3d5057f90fe36d P dcfdfa62edc34dab9525e2712d487582 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b602f6099cbb4758847e7e9550685372 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 02:00:41.912581 15024 leader_election.cc:302] T ebd7cdd284a74741b338e9f4778502b5 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b602f6099cbb4758847e7e9550685372, c05f7620bb884392931d3243a0b47a9e; no voters: 620596b893d84291b88b90dc9e317612
I20240503 02:00:41.912752 15084 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0e3bb299e61148fcb355641e0134b3df"
dest_uuid: "b602f6099cbb4758847e7e9550685372"
from {username='slave'} at 127.0.0.1:34018
I20240503 02:00:41.912817 15024 leader_election.cc:302] T 454a24fd6d964d73ae3d5057f90fe36d P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b602f6099cbb4758847e7e9550685372, c05f7620bb884392931d3243a0b47a9e; no voters:
I20240503 02:00:41.912868 15084 raft_consensus.cc:482] T 0e3bb299e61148fcb355641e0134b3df P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.912925 15084 raft_consensus.cc:3037] T 0e3bb299e61148fcb355641e0134b3df P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.912952 17331 raft_consensus.cc:2781] T ebd7cdd284a74741b338e9f4778502b5 P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 02:00:41.913066 17332 raft_consensus.cc:2781] T 454a24fd6d964d73ae3d5057f90fe36d P b602f6099cbb4758847e7e9550685372 [term 6 FOLLOWER]: Leader election won for term 6
I20240503 02:00:41.913122 17332 raft_consensus.cc:686] T 454a24fd6d964d73ae3d5057f90fe36d P b602f6099cbb4758847e7e9550685372 [term 6 LEADER]: Becoming Leader. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.913244 17332 consensus_queue.cc:237] T 454a24fd6d964d73ae3d5057f90fe36d P b602f6099cbb4758847e7e9550685372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.914101 15084 raft_consensus.cc:504] T 0e3bb299e61148fcb355641e0134b3df P b602f6099cbb4758847e7e9550685372 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.914340 17331 raft_consensus.cc:686] T ebd7cdd284a74741b338e9f4778502b5 P b602f6099cbb4758847e7e9550685372 [term 5 LEADER]: Becoming Leader. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.914460 15084 leader_election.cc:288] T 0e3bb299e61148fcb355641e0134b3df P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 6 election: Requested vote from peers c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009), dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817)
I20240503 02:00:41.914489 17331 consensus_queue.cc:237] T ebd7cdd284a74741b338e9f4778502b5 P b602f6099cbb4758847e7e9550685372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.915206 15084 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ca0d7d3e4d59496c9a9a59cde24b9247"
dest_uuid: "b602f6099cbb4758847e7e9550685372"
from {username='slave'} at 127.0.0.1:34018
I20240503 02:00:41.915221 14956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e3bb299e61148fcb355641e0134b3df" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 6 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:41.915334 15084 raft_consensus.cc:482] T ca0d7d3e4d59496c9a9a59cde24b9247 P b602f6099cbb4758847e7e9550685372 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.915395 15084 raft_consensus.cc:3037] T ca0d7d3e4d59496c9a9a59cde24b9247 P b602f6099cbb4758847e7e9550685372 [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:41.915581 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e3bb299e61148fcb355641e0134b3df" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 6 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.915705 15351 raft_consensus.cc:3037] T 0e3bb299e61148fcb355641e0134b3df P c05f7620bb884392931d3243a0b47a9e [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.916343 15084 raft_consensus.cc:504] T ca0d7d3e4d59496c9a9a59cde24b9247 P b602f6099cbb4758847e7e9550685372 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.916690 15084 leader_election.cc:288] T ca0d7d3e4d59496c9a9a59cde24b9247 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 7 election: Requested vote from peers c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009), dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817)
I20240503 02:00:41.916987 14811 catalog_manager.cc:5484] T ebd7cdd284a74741b338e9f4778502b5 P b602f6099cbb4758847e7e9550685372 reported cstate change: term changed from 3 to 5, leader changed from 620596b893d84291b88b90dc9e317612 (127.0.99.131) to b602f6099cbb4758847e7e9550685372 (127.0.99.130). New cstate: current_term: 5 leader_uuid: "b602f6099cbb4758847e7e9550685372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:41.917273 15351 raft_consensus.cc:2454] T 0e3bb299e61148fcb355641e0134b3df P c05f7620bb884392931d3243a0b47a9e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 6.
I20240503 02:00:41.917279 14811 catalog_manager.cc:5484] T 454a24fd6d964d73ae3d5057f90fe36d P b602f6099cbb4758847e7e9550685372 reported cstate change: term changed from 5 to 6, leader changed from dcfdfa62edc34dab9525e2712d487582 (127.0.99.129) to b602f6099cbb4758847e7e9550685372 (127.0.99.130). New cstate: current_term: 6 leader_uuid: "b602f6099cbb4758847e7e9550685372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:41.918056 14956 raft_consensus.cc:3032] T 0e3bb299e61148fcb355641e0134b3df P dcfdfa62edc34dab9525e2712d487582 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 02:00:41.918174 14956 raft_consensus.cc:729] T 0e3bb299e61148fcb355641e0134b3df P dcfdfa62edc34dab9525e2712d487582 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:41.918216 15084 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5da91b7af68e4cf597dc89a017ee0197"
dest_uuid: "b602f6099cbb4758847e7e9550685372"
from {username='slave'} at 127.0.0.1:34018
I20240503 02:00:41.918427 15084 raft_consensus.cc:482] T 5da91b7af68e4cf597dc89a017ee0197 P b602f6099cbb4758847e7e9550685372 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.918499 15084 raft_consensus.cc:3037] T 5da91b7af68e4cf597dc89a017ee0197 P b602f6099cbb4758847e7e9550685372 [term 4 FOLLOWER]: Advancing to term 5
W20240503 02:00:41.918547 15176 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57688: Illegal state: replica 620596b893d84291b88b90dc9e317612 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.918449 14956 consensus_queue.cc:260] T 0e3bb299e61148fcb355641e0134b3df P dcfdfa62edc34dab9525e2712d487582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.8, Last appended by leader: 8, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.918802 14956 raft_consensus.cc:3037] T 0e3bb299e61148fcb355641e0134b3df P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.918637 14954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca0d7d3e4d59496c9a9a59cde24b9247" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 7 candidate_status { last_received { term: 4 index: 8 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:41.919127 15024 leader_election.cc:302] T 0e3bb299e61148fcb355641e0134b3df P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b602f6099cbb4758847e7e9550685372, c05f7620bb884392931d3243a0b47a9e; no voters:
I20240503 02:00:41.919216 14954 raft_consensus.cc:3032] T ca0d7d3e4d59496c9a9a59cde24b9247 P dcfdfa62edc34dab9525e2712d487582 [term 6 LEADER]: Stepping down as leader of term 6
I20240503 02:00:41.919265 14954 raft_consensus.cc:729] T ca0d7d3e4d59496c9a9a59cde24b9247 P dcfdfa62edc34dab9525e2712d487582 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:41.919281 16496 raft_consensus.cc:2781] T 0e3bb299e61148fcb355641e0134b3df P b602f6099cbb4758847e7e9550685372 [term 6 FOLLOWER]: Leader election won for term 6
I20240503 02:00:41.919342 16496 raft_consensus.cc:686] T 0e3bb299e61148fcb355641e0134b3df P b602f6099cbb4758847e7e9550685372 [term 6 LEADER]: Becoming Leader. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.919409 14954 consensus_queue.cc:260] T ca0d7d3e4d59496c9a9a59cde24b9247 P dcfdfa62edc34dab9525e2712d487582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.10, Last appended by leader: 10, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.919443 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca0d7d3e4d59496c9a9a59cde24b9247" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 7 candidate_status { last_received { term: 4 index: 8 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.919463 16496 consensus_queue.cc:237] T 0e3bb299e61148fcb355641e0134b3df P b602f6099cbb4758847e7e9550685372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 3.8, Last appended by leader: 8, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.919550 14954 raft_consensus.cc:3037] T ca0d7d3e4d59496c9a9a59cde24b9247 P dcfdfa62edc34dab9525e2712d487582 [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:41.919569 15351 raft_consensus.cc:3037] T ca0d7d3e4d59496c9a9a59cde24b9247 P c05f7620bb884392931d3243a0b47a9e [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:41.920152 15084 raft_consensus.cc:504] T 5da91b7af68e4cf597dc89a017ee0197 P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.920500 15219 raft_consensus.cc:2454] T 6b225db529f14e7386f21f42bf31493b P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 6.
I20240503 02:00:41.920549 15351 raft_consensus.cc:2454] T ca0d7d3e4d59496c9a9a59cde24b9247 P c05f7620bb884392931d3243a0b47a9e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 7.
I20240503 02:00:41.920660 15084 leader_election.cc:288] T 5da91b7af68e4cf597dc89a017ee0197 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 5 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:41.921197 15024 leader_election.cc:302] T 6b225db529f14e7386f21f42bf31493b P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 620596b893d84291b88b90dc9e317612, b602f6099cbb4758847e7e9550685372; no voters: dcfdfa62edc34dab9525e2712d487582
I20240503 02:00:41.921355 17331 raft_consensus.cc:2781] T 6b225db529f14e7386f21f42bf31493b P b602f6099cbb4758847e7e9550685372 [term 6 FOLLOWER]: Leader election won for term 6
I20240503 02:00:41.921427 17331 raft_consensus.cc:686] T 6b225db529f14e7386f21f42bf31493b P b602f6099cbb4758847e7e9550685372 [term 6 LEADER]: Becoming Leader. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.921540 14954 raft_consensus.cc:2396] T ca0d7d3e4d59496c9a9a59cde24b9247 P dcfdfa62edc34dab9525e2712d487582 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b602f6099cbb4758847e7e9550685372 for term 7 because replica has last-logged OpId of term: 6 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 8.
I20240503 02:00:41.921423 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5da91b7af68e4cf597dc89a017ee0197" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.921651 15351 raft_consensus.cc:3037] T 5da91b7af68e4cf597dc89a017ee0197 P c05f7620bb884392931d3243a0b47a9e [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.921618 17331 consensus_queue.cc:237] T 6b225db529f14e7386f21f42bf31493b P b602f6099cbb4758847e7e9550685372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 3.8, Last appended by leader: 8, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.921741 15219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5da91b7af68e4cf597dc89a017ee0197" candidate_uuid: "b602f6099cbb4758847e7e9550685372" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:41.921860 15219 raft_consensus.cc:3037] T 5da91b7af68e4cf597dc89a017ee0197 P 620596b893d84291b88b90dc9e317612 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:00:41.922313 15024 leader_election.cc:302] T ca0d7d3e4d59496c9a9a59cde24b9247 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b602f6099cbb4758847e7e9550685372, c05f7620bb884392931d3243a0b47a9e; no voters: dcfdfa62edc34dab9525e2712d487582
I20240503 02:00:41.922757 17331 raft_consensus.cc:2781] T ca0d7d3e4d59496c9a9a59cde24b9247 P b602f6099cbb4758847e7e9550685372 [term 7 FOLLOWER]: Leader election won for term 7
I20240503 02:00:41.922837 17331 raft_consensus.cc:686] T ca0d7d3e4d59496c9a9a59cde24b9247 P b602f6099cbb4758847e7e9550685372 [term 7 LEADER]: Becoming Leader. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.922945 17331 consensus_queue.cc:237] T ca0d7d3e4d59496c9a9a59cde24b9247 P b602f6099cbb4758847e7e9550685372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.923029 15351 raft_consensus.cc:2454] T 5da91b7af68e4cf597dc89a017ee0197 P c05f7620bb884392931d3243a0b47a9e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b602f6099cbb4758847e7e9550685372 in term 5.
I20240503 02:00:41.923770 15024 leader_election.cc:302] T 5da91b7af68e4cf597dc89a017ee0197 P b602f6099cbb4758847e7e9550685372 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b602f6099cbb4758847e7e9550685372, c05f7620bb884392931d3243a0b47a9e; no voters:
I20240503 02:00:41.923951 17331 raft_consensus.cc:2781] T 5da91b7af68e4cf597dc89a017ee0197 P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 02:00:41.924120 17331 raft_consensus.cc:686] T 5da91b7af68e4cf597dc89a017ee0197 P b602f6099cbb4758847e7e9550685372 [term 5 LEADER]: Becoming Leader. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.924270 14956 raft_consensus.cc:2396] T 0e3bb299e61148fcb355641e0134b3df P dcfdfa62edc34dab9525e2712d487582 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b602f6099cbb4758847e7e9550685372 for term 6 because replica has last-logged OpId of term: 5 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 8.
I20240503 02:00:41.924247 17331 consensus_queue.cc:237] T 5da91b7af68e4cf597dc89a017ee0197 P b602f6099cbb4758847e7e9550685372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.924575 14811 catalog_manager.cc:5484] T 0e3bb299e61148fcb355641e0134b3df P b602f6099cbb4758847e7e9550685372 reported cstate change: term changed from 5 to 6, leader changed from dcfdfa62edc34dab9525e2712d487582 (127.0.99.129) to b602f6099cbb4758847e7e9550685372 (127.0.99.130). New cstate: current_term: 6 leader_uuid: "b602f6099cbb4758847e7e9550685372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:41.924844 14811 catalog_manager.cc:5484] T 6b225db529f14e7386f21f42bf31493b P b602f6099cbb4758847e7e9550685372 reported cstate change: term changed from 5 to 6, leader changed from dcfdfa62edc34dab9525e2712d487582 (127.0.99.129) to b602f6099cbb4758847e7e9550685372 (127.0.99.130). New cstate: current_term: 6 leader_uuid: "b602f6099cbb4758847e7e9550685372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:41.925768 15219 raft_consensus.cc:2396] T 5da91b7af68e4cf597dc89a017ee0197 P 620596b893d84291b88b90dc9e317612 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate b602f6099cbb4758847e7e9550685372 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 02:00:41.928231 15351 raft_consensus.cc:1261] T c74522fbb5c64d27b5c95f05896b0716 P c05f7620bb884392931d3243a0b47a9e [term 6 FOLLOWER]: Refusing update from remote peer b602f6099cbb4758847e7e9550685372: Log matching property violated. Preceding OpId in replica: term: 2 index: 25. Preceding OpId from leader: term: 6 index: 28. (index mismatch)
I20240503 02:00:41.928761 17331 consensus_queue.cc:1035] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 25, Time since last communication: 0.000s
I20240503 02:00:41.928831 15219 raft_consensus.cc:1261] T c74522fbb5c64d27b5c95f05896b0716 P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Refusing update from remote peer b602f6099cbb4758847e7e9550685372: Log matching property violated. Preceding OpId in replica: term: 4 index: 26. Preceding OpId from leader: term: 6 index: 28. (index mismatch)
I20240503 02:00:41.929237 15219 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "de805ef7511d4ee9b9bbed67e0a4186c"
dest_uuid: "620596b893d84291b88b90dc9e317612"
from {username='slave'} at 127.0.0.1:57680
I20240503 02:00:41.929489 17331 consensus_queue.cc:1035] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 25, Time since last communication: 0.000s
I20240503 02:00:41.929625 17331 consensus_queue.cc:1225] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372 [LEADER]: Peer 620596b893d84291b88b90dc9e317612 log is divergent from this leader: its last log entry 4.26 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 25
I20240503 02:00:41.930171 14816 catalog_manager.cc:5484] T ca0d7d3e4d59496c9a9a59cde24b9247 P b602f6099cbb4758847e7e9550685372 reported cstate change: term changed from 6 to 7, leader changed from dcfdfa62edc34dab9525e2712d487582 (127.0.99.129) to b602f6099cbb4758847e7e9550685372 (127.0.99.130). New cstate: current_term: 7 leader_uuid: "b602f6099cbb4758847e7e9550685372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:41.930382 15219 raft_consensus.cc:482] T de805ef7511d4ee9b9bbed67e0a4186c P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.930505 15219 raft_consensus.cc:3037] T de805ef7511d4ee9b9bbed67e0a4186c P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:41.930420 14816 catalog_manager.cc:5484] T 5da91b7af68e4cf597dc89a017ee0197 P b602f6099cbb4758847e7e9550685372 reported cstate change: term changed from 3 to 5, leader changed from 620596b893d84291b88b90dc9e317612 (127.0.99.131) to b602f6099cbb4758847e7e9550685372 (127.0.99.130). New cstate: current_term: 5 leader_uuid: "b602f6099cbb4758847e7e9550685372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } }
W20240503 02:00:41.930963 15311 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50950: Illegal state: replica c05f7620bb884392931d3243a0b47a9e is not leader of this config: current role FOLLOWER
I20240503 02:00:41.931562 15214 pending_rounds.cc:76] T c74522fbb5c64d27b5c95f05896b0716 P 620596b893d84291b88b90dc9e317612: Aborting all ops after (but not including) 25
I20240503 02:00:41.931607 15214 pending_rounds.cc:97] T c74522fbb5c64d27b5c95f05896b0716 P 620596b893d84291b88b90dc9e317612: Aborting uncommitted NO_OP operation due to leader change: 4.26
I20240503 02:00:41.931636 15214 raft_consensus.cc:2866] T c74522fbb5c64d27b5c95f05896b0716 P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 02:00:41.932232 15219 raft_consensus.cc:504] T de805ef7511d4ee9b9bbed67e0a4186c P 620596b893d84291b88b90dc9e317612 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.932669 15219 leader_election.cc:288] T de805ef7511d4ee9b9bbed67e0a4186c P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 7 election: Requested vote from peers c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
I20240503 02:00:41.933897 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de805ef7511d4ee9b9bbed67e0a4186c" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.934177 15351 raft_consensus.cc:3037] T de805ef7511d4ee9b9bbed67e0a4186c P c05f7620bb884392931d3243a0b47a9e [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:41.934629 15214 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7b4f203f4048490ba599353a4cd92748"
dest_uuid: "620596b893d84291b88b90dc9e317612"
from {username='slave'} at 127.0.0.1:57680
I20240503 02:00:41.934748 15214 raft_consensus.cc:482] T 7b4f203f4048490ba599353a4cd92748 P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.934803 15214 raft_consensus.cc:3037] T 7b4f203f4048490ba599353a4cd92748 P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:41.935276 15351 raft_consensus.cc:2454] T de805ef7511d4ee9b9bbed67e0a4186c P c05f7620bb884392931d3243a0b47a9e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 7.
I20240503 02:00:41.935551 15155 leader_election.cc:302] T de805ef7511d4ee9b9bbed67e0a4186c P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, c05f7620bb884392931d3243a0b47a9e; no voters:
W20240503 02:00:41.935639 14915 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47950: Illegal state: replica dcfdfa62edc34dab9525e2712d487582 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.935770 15214 raft_consensus.cc:504] T 7b4f203f4048490ba599353a4cd92748 P 620596b893d84291b88b90dc9e317612 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.936133 15214 leader_election.cc:288] T 7b4f203f4048490ba599353a4cd92748 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 7 election: Requested vote from peers b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207), dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817)
I20240503 02:00:41.936305 16264 raft_consensus.cc:2781] T de805ef7511d4ee9b9bbed67e0a4186c P 620596b893d84291b88b90dc9e317612 [term 7 FOLLOWER]: Leader election won for term 7
I20240503 02:00:41.936405 16264 raft_consensus.cc:686] T de805ef7511d4ee9b9bbed67e0a4186c P 620596b893d84291b88b90dc9e317612 [term 7 LEADER]: Becoming Leader. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:41.936487 15084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de805ef7511d4ee9b9bbed67e0a4186c" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:41.936535 16264 consensus_queue.cc:237] T de805ef7511d4ee9b9bbed67e0a4186c P 620596b893d84291b88b90dc9e317612 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.936601 15084 raft_consensus.cc:3032] T de805ef7511d4ee9b9bbed67e0a4186c P b602f6099cbb4758847e7e9550685372 [term 6 LEADER]: Stepping down as leader of term 6
I20240503 02:00:41.936635 15084 raft_consensus.cc:729] T de805ef7511d4ee9b9bbed67e0a4186c P b602f6099cbb4758847e7e9550685372 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.936774 15084 consensus_queue.cc:260] T de805ef7511d4ee9b9bbed67e0a4186c P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.936899 15084 raft_consensus.cc:3037] T de805ef7511d4ee9b9bbed67e0a4186c P b602f6099cbb4758847e7e9550685372 [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:41.937175 15087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4f203f4048490ba599353a4cd92748" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:41.937275 15087 raft_consensus.cc:3032] T 7b4f203f4048490ba599353a4cd92748 P b602f6099cbb4758847e7e9550685372 [term 6 LEADER]: Stepping down as leader of term 6
I20240503 02:00:41.937309 15087 raft_consensus.cc:729] T 7b4f203f4048490ba599353a4cd92748 P b602f6099cbb4758847e7e9550685372 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.937418 15087 consensus_queue.cc:260] T 7b4f203f4048490ba599353a4cd92748 P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:41.937556 15087 raft_consensus.cc:3037] T 7b4f203f4048490ba599353a4cd92748 P b602f6099cbb4758847e7e9550685372 [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:41.938814 15214 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "21d92d412a5247e8907c5c41d253ad2e"
dest_uuid: "620596b893d84291b88b90dc9e317612"
from {username='slave'} at 127.0.0.1:57680
I20240503 02:00:41.938952 15214 raft_consensus.cc:482] T 21d92d412a5247e8907c5c41d253ad2e P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.939009 15214 raft_consensus.cc:3037] T 21d92d412a5247e8907c5c41d253ad2e P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:41.939965 15214 raft_consensus.cc:504] T 21d92d412a5247e8907c5c41d253ad2e P 620596b893d84291b88b90dc9e317612 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.940780 15349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21d92d412a5247e8907c5c41d253ad2e" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.940910 15349 raft_consensus.cc:3037] T 21d92d412a5247e8907c5c41d253ad2e P c05f7620bb884392931d3243a0b47a9e [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:41.941256 15214 leader_election.cc:288] T 21d92d412a5247e8907c5c41d253ad2e P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 7 election: Requested vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:41.941627 15084 raft_consensus.cc:2396] T de805ef7511d4ee9b9bbed67e0a4186c P b602f6099cbb4758847e7e9550685372 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 620596b893d84291b88b90dc9e317612 for term 7 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
I20240503 02:00:41.941916 15214 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "830da5cb4972453cb9bf2145a0839ced"
dest_uuid: "620596b893d84291b88b90dc9e317612"
from {username='slave'} at 127.0.0.1:57680
I20240503 02:00:41.941987 15349 raft_consensus.cc:2396] T 21d92d412a5247e8907c5c41d253ad2e P c05f7620bb884392931d3243a0b47a9e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 620596b893d84291b88b90dc9e317612 for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
I20240503 02:00:41.942030 15214 raft_consensus.cc:482] T 830da5cb4972453cb9bf2145a0839ced P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.942080 15214 raft_consensus.cc:3037] T 830da5cb4972453cb9bf2145a0839ced P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:41.942130 15087 raft_consensus.cc:2396] T 7b4f203f4048490ba599353a4cd92748 P b602f6099cbb4758847e7e9550685372 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 620596b893d84291b88b90dc9e317612 for term 7 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 02:00:41.942840 14954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21d92d412a5247e8907c5c41d253ad2e" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:41.942981 14954 raft_consensus.cc:3037] T 21d92d412a5247e8907c5c41d253ad2e P dcfdfa62edc34dab9525e2712d487582 [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:41.943353 14811 catalog_manager.cc:5484] T de805ef7511d4ee9b9bbed67e0a4186c P 620596b893d84291b88b90dc9e317612 reported cstate change: term changed from 6 to 7, leader changed from b602f6099cbb4758847e7e9550685372 (127.0.99.130) to 620596b893d84291b88b90dc9e317612 (127.0.99.131). New cstate: current_term: 7 leader_uuid: "620596b893d84291b88b90dc9e317612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:41.943449 15214 raft_consensus.cc:504] T 830da5cb4972453cb9bf2145a0839ced P 620596b893d84291b88b90dc9e317612 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.943840 15214 leader_election.cc:288] T 830da5cb4972453cb9bf2145a0839ced P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 7 election: Requested vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
I20240503 02:00:41.944020 14954 raft_consensus.cc:2454] T 21d92d412a5247e8907c5c41d253ad2e P dcfdfa62edc34dab9525e2712d487582 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 7.
I20240503 02:00:41.944332 14958 raft_consensus.cc:1261] T f85a361ce0254472b37526362d4edf0e P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Refusing update from remote peer b602f6099cbb4758847e7e9550685372: Log matching property violated. Preceding OpId in replica: term: 4 index: 22. Preceding OpId from leader: term: 5 index: 24. (index mismatch)
I20240503 02:00:41.944967 17331 consensus_queue.cc:1035] T f85a361ce0254472b37526362d4edf0e P b602f6099cbb4758847e7e9550685372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20240503 02:00:41.945364 15153 leader_election.cc:302] T 21d92d412a5247e8907c5c41d253ad2e P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 620596b893d84291b88b90dc9e317612, dcfdfa62edc34dab9525e2712d487582; no voters: c05f7620bb884392931d3243a0b47a9e
I20240503 02:00:41.945441 15087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "830da5cb4972453cb9bf2145a0839ced" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 7 candidate_status { last_received { term: 6 index: 10 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:41.945525 17011 raft_consensus.cc:2781] T 21d92d412a5247e8907c5c41d253ad2e P 620596b893d84291b88b90dc9e317612 [term 7 FOLLOWER]: Leader election won for term 7
I20240503 02:00:41.945551 15087 raft_consensus.cc:3032] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372 [term 6 LEADER]: Stepping down as leader of term 6
I20240503 02:00:41.945591 15087 raft_consensus.cc:729] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.945611 17011 raft_consensus.cc:686] T 21d92d412a5247e8907c5c41d253ad2e P 620596b893d84291b88b90dc9e317612 [term 7 LEADER]: Becoming Leader. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:41.945531 15217 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9e9899666b15498791f62dab6d9dda16"
dest_uuid: "620596b893d84291b88b90dc9e317612"
from {username='slave'} at 127.0.0.1:57680
I20240503 02:00:41.945712 15087 consensus_queue.cc:260] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 6.10, Last appended by leader: 10, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.945772 15217 raft_consensus.cc:482] T 9e9899666b15498791f62dab6d9dda16 P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.945735 17011 consensus_queue.cc:237] T 21d92d412a5247e8907c5c41d253ad2e P 620596b893d84291b88b90dc9e317612 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.945822 15217 raft_consensus.cc:3037] T 9e9899666b15498791f62dab6d9dda16 P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:41.945909 15087 raft_consensus.cc:3037] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372 [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:41.946785 15217 raft_consensus.cc:504] T 9e9899666b15498791f62dab6d9dda16 P 620596b893d84291b88b90dc9e317612 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:41.946902 15087 raft_consensus.cc:2454] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 7.
I20240503 02:00:41.947144 15217 leader_election.cc:288] T 9e9899666b15498791f62dab6d9dda16 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 7 election: Requested vote from peers b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:41.947400 15214 raft_consensus.cc:1261] T f85a361ce0254472b37526362d4edf0e P 620596b893d84291b88b90dc9e317612 [term 5 FOLLOWER]: Refusing update from remote peer b602f6099cbb4758847e7e9550685372: Log matching property violated. Preceding OpId in replica: term: 4 index: 22. Preceding OpId from leader: term: 5 index: 24. (index mismatch)
I20240503 02:00:41.947639 15349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e9899666b15498791f62dab6d9dda16" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 7 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.947763 15349 raft_consensus.cc:3037] T 9e9899666b15498791f62dab6d9dda16 P c05f7620bb884392931d3243a0b47a9e [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:41.948885 15349 raft_consensus.cc:2396] T 9e9899666b15498791f62dab6d9dda16 P c05f7620bb884392931d3243a0b47a9e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 620596b893d84291b88b90dc9e317612 for term 7 because replica has last-logged OpId of term: 5 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 8.
I20240503 02:00:41.949488 15154 leader_election.cc:302] T 830da5cb4972453cb9bf2145a0839ced P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, b602f6099cbb4758847e7e9550685372; no voters:
I20240503 02:00:41.949749 16264 raft_consensus.cc:2781] T 830da5cb4972453cb9bf2145a0839ced P 620596b893d84291b88b90dc9e317612 [term 7 FOLLOWER]: Leader election won for term 7
I20240503 02:00:41.949812 16264 raft_consensus.cc:686] T 830da5cb4972453cb9bf2145a0839ced P 620596b893d84291b88b90dc9e317612 [term 7 LEADER]: Becoming Leader. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:41.949936 16264 consensus_queue.cc:237] T 830da5cb4972453cb9bf2145a0839ced P 620596b893d84291b88b90dc9e317612 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 6.10, Last appended by leader: 10, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:41.951610 15087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e9899666b15498791f62dab6d9dda16" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 7 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:41.951821 15087 raft_consensus.cc:3032] T 9e9899666b15498791f62dab6d9dda16 P b602f6099cbb4758847e7e9550685372 [term 6 LEADER]: Stepping down as leader of term 6
I20240503 02:00:41.951872 15087 raft_consensus.cc:729] T 9e9899666b15498791f62dab6d9dda16 P b602f6099cbb4758847e7e9550685372 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:41.952024 15087 consensus_queue.cc:260] T 9e9899666b15498791f62dab6d9dda16 P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.10, Last appended by leader: 10, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:41.952162 15087 raft_consensus.cc:3037] T 9e9899666b15498791f62dab6d9dda16 P b602f6099cbb4758847e7e9550685372 [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:41.953722 15219 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "50c5abd860ae4b09bb289e9c7550c84e"
dest_uuid: "620596b893d84291b88b90dc9e317612"
from {username='slave'} at 127.0.0.1:57680
I20240503 02:00:41.953894 15219 raft_consensus.cc:482] T 50c5abd860ae4b09bb289e9c7550c84e P 620596b893d84291b88b90dc9e317612 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.953965 15219 raft_consensus.cc:3037] T 50c5abd860ae4b09bb289e9c7550c84e P 620596b893d84291b88b90dc9e317612 [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.954737 15087 raft_consensus.cc:2396] T 9e9899666b15498791f62dab6d9dda16 P b602f6099cbb4758847e7e9550685372 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 620596b893d84291b88b90dc9e317612 for term 7 because replica has last-logged OpId of term: 6 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 8.
I20240503 02:00:41.955070 15154 leader_election.cc:302] T 9e9899666b15498791f62dab6d9dda16 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 620596b893d84291b88b90dc9e317612; no voters: b602f6099cbb4758847e7e9550685372, c05f7620bb884392931d3243a0b47a9e
I20240503 02:00:41.955195 15219 raft_consensus.cc:504] T 50c5abd860ae4b09bb289e9c7550c84e P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.955333 17011 raft_consensus.cc:2726] T 9e9899666b15498791f62dab6d9dda16 P 620596b893d84291b88b90dc9e317612 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240503 02:00:41.955679 16496 consensus_queue.cc:1035] T f85a361ce0254472b37526362d4edf0e P b602f6099cbb4758847e7e9550685372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20240503 02:00:41.957170 15349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50c5abd860ae4b09bb289e9c7550c84e" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 6 candidate_status { last_received { term: 3 index: 7 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.957299 15349 raft_consensus.cc:3037] T 50c5abd860ae4b09bb289e9c7550c84e P c05f7620bb884392931d3243a0b47a9e [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.958205 17179 mvcc.cc:205] Tried to move back new op lower bound from 7023417925397377024 to 7023417925261852672. Current Snapshot: MvccSnapshot[applied={T|T < 7023417925397377024}]
I20240503 02:00:41.958926 14811 catalog_manager.cc:5484] T 21d92d412a5247e8907c5c41d253ad2e P 620596b893d84291b88b90dc9e317612 reported cstate change: term changed from 5 to 7, leader changed from c05f7620bb884392931d3243a0b47a9e (127.0.99.132) to 620596b893d84291b88b90dc9e317612 (127.0.99.131). New cstate: current_term: 7 leader_uuid: "620596b893d84291b88b90dc9e317612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:41.959216 14811 catalog_manager.cc:5484] T 830da5cb4972453cb9bf2145a0839ced P 620596b893d84291b88b90dc9e317612 reported cstate change: term changed from 6 to 7, leader changed from b602f6099cbb4758847e7e9550685372 (127.0.99.130) to 620596b893d84291b88b90dc9e317612 (127.0.99.131). New cstate: current_term: 7 leader_uuid: "620596b893d84291b88b90dc9e317612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:41.960175 15349 raft_consensus.cc:2396] T 50c5abd860ae4b09bb289e9c7550c84e P c05f7620bb884392931d3243a0b47a9e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 620596b893d84291b88b90dc9e317612 for term 6 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 7.
W20240503 02:00:41.960281 15311 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50880: Illegal state: replica c05f7620bb884392931d3243a0b47a9e is not leader of this config: current role FOLLOWER
W20240503 02:00:41.946647 14915 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48020: Illegal state: replica dcfdfa62edc34dab9525e2712d487582 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.944660 14956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4f203f4048490ba599353a4cd92748" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:41.960762 14956 raft_consensus.cc:3037] T 7b4f203f4048490ba599353a4cd92748 P dcfdfa62edc34dab9525e2712d487582 [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:41.956851 14954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50c5abd860ae4b09bb289e9c7550c84e" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 6 candidate_status { last_received { term: 3 index: 7 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:41.961277 14954 raft_consensus.cc:3037] T 50c5abd860ae4b09bb289e9c7550c84e P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:41.961834 14956 raft_consensus.cc:2396] T 7b4f203f4048490ba599353a4cd92748 P dcfdfa62edc34dab9525e2712d487582 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 620596b893d84291b88b90dc9e317612 for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 02:00:41.962149 15153 leader_election.cc:302] T 7b4f203f4048490ba599353a4cd92748 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 620596b893d84291b88b90dc9e317612; no voters: b602f6099cbb4758847e7e9550685372, dcfdfa62edc34dab9525e2712d487582
I20240503 02:00:41.962410 17011 raft_consensus.cc:2726] T 7b4f203f4048490ba599353a4cd92748 P 620596b893d84291b88b90dc9e317612 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240503 02:00:41.945637 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "830da5cb4972453cb9bf2145a0839ced" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 7 candidate_status { last_received { term: 6 index: 10 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:41.962509 14958 raft_consensus.cc:3037] T 830da5cb4972453cb9bf2145a0839ced P dcfdfa62edc34dab9525e2712d487582 [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:41.963378 14958 raft_consensus.cc:2454] T 830da5cb4972453cb9bf2145a0839ced P dcfdfa62edc34dab9525e2712d487582 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 7.
I20240503 02:00:41.964226 14954 raft_consensus.cc:2454] T 50c5abd860ae4b09bb289e9c7550c84e P dcfdfa62edc34dab9525e2712d487582 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 6.
I20240503 02:00:41.960624 15219 leader_election.cc:288] T 50c5abd860ae4b09bb289e9c7550c84e P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 6 election: Requested vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:41.964555 15153 leader_election.cc:302] T 50c5abd860ae4b09bb289e9c7550c84e P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 620596b893d84291b88b90dc9e317612, dcfdfa62edc34dab9525e2712d487582; no voters: c05f7620bb884392931d3243a0b47a9e
I20240503 02:00:41.964833 17011 raft_consensus.cc:2781] T 50c5abd860ae4b09bb289e9c7550c84e P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Leader election won for term 6
I20240503 02:00:41.964915 17011 raft_consensus.cc:686] T 50c5abd860ae4b09bb289e9c7550c84e P 620596b893d84291b88b90dc9e317612 [term 6 LEADER]: Becoming Leader. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:41.965049 17011 consensus_queue.cc:237] T 50c5abd860ae4b09bb289e9c7550c84e P 620596b893d84291b88b90dc9e317612 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.965299 15219 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7fc3330dfd0644f6a303659b960124f4"
dest_uuid: "620596b893d84291b88b90dc9e317612"
from {username='slave'} at 127.0.0.1:57680
I20240503 02:00:41.965395 15219 raft_consensus.cc:482] T 7fc3330dfd0644f6a303659b960124f4 P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:41.965441 15219 raft_consensus.cc:3037] T 7fc3330dfd0644f6a303659b960124f4 P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:41.966434 15214 raft_consensus.cc:1261] T eade118a36da42e3ac08b96a13893a5c P 620596b893d84291b88b90dc9e317612 [term 5 FOLLOWER]: Refusing update from remote peer b602f6099cbb4758847e7e9550685372: Log matching property violated. Preceding OpId in replica: term: 4 index: 22. Preceding OpId from leader: term: 5 index: 24. (index mismatch)
I20240503 02:00:41.966928 14816 catalog_manager.cc:5484] T 50c5abd860ae4b09bb289e9c7550c84e P 620596b893d84291b88b90dc9e317612 reported cstate change: term changed from 4 to 6, leader changed from c05f7620bb884392931d3243a0b47a9e (127.0.99.132) to 620596b893d84291b88b90dc9e317612 (127.0.99.131). New cstate: current_term: 6 leader_uuid: "620596b893d84291b88b90dc9e317612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:41.967113 15349 raft_consensus.cc:1261] T eade118a36da42e3ac08b96a13893a5c P c05f7620bb884392931d3243a0b47a9e [term 5 FOLLOWER]: Refusing update from remote peer b602f6099cbb4758847e7e9550685372: Log matching property violated. Preceding OpId in replica: term: 4 index: 22. Preceding OpId from leader: term: 5 index: 24. (index mismatch)
I20240503 02:00:41.967316 15219 raft_consensus.cc:504] T 7fc3330dfd0644f6a303659b960124f4 P 620596b893d84291b88b90dc9e317612 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.967968 14954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fc3330dfd0644f6a303659b960124f4" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 7 candidate_status { last_received { term: 4 index: 13 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:41.968099 14954 raft_consensus.cc:3037] T 7fc3330dfd0644f6a303659b960124f4 P dcfdfa62edc34dab9525e2712d487582 [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:41.969135 14954 raft_consensus.cc:2454] T 7fc3330dfd0644f6a303659b960124f4 P dcfdfa62edc34dab9525e2712d487582 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 7.
I20240503 02:00:41.969099 16496 consensus_queue.cc:1035] T eade118a36da42e3ac08b96a13893a5c P b602f6099cbb4758847e7e9550685372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20240503 02:00:41.969779 15153 leader_election.cc:302] T 7fc3330dfd0644f6a303659b960124f4 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:41.970630 17011 raft_consensus.cc:2781] T 7fc3330dfd0644f6a303659b960124f4 P 620596b893d84291b88b90dc9e317612 [term 7 FOLLOWER]: Leader election won for term 7
I20240503 02:00:41.970718 17011 raft_consensus.cc:686] T 7fc3330dfd0644f6a303659b960124f4 P 620596b893d84291b88b90dc9e317612 [term 7 LEADER]: Becoming Leader. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:41.970835 17011 consensus_queue.cc:237] T 7fc3330dfd0644f6a303659b960124f4 P 620596b893d84291b88b90dc9e317612 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 4.13, Last appended by leader: 13, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:41.971308 16496 consensus_queue.cc:1035] T eade118a36da42e3ac08b96a13893a5c P b602f6099cbb4758847e7e9550685372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20240503 02:00:41.973305 14811 catalog_manager.cc:5484] T 7fc3330dfd0644f6a303659b960124f4 P 620596b893d84291b88b90dc9e317612 reported cstate change: term changed from 5 to 7, leader changed from c05f7620bb884392931d3243a0b47a9e (127.0.99.132) to 620596b893d84291b88b90dc9e317612 (127.0.99.131). New cstate: current_term: 7 leader_uuid: "620596b893d84291b88b90dc9e317612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:41.974038 15348 raft_consensus.cc:1261] T 21d92d412a5247e8907c5c41d253ad2e P c05f7620bb884392931d3243a0b47a9e [term 7 FOLLOWER]: Refusing update from remote peer 620596b893d84291b88b90dc9e317612: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20240503 02:00:41.974112 14954 raft_consensus.cc:1261] T 21d92d412a5247e8907c5c41d253ad2e P dcfdfa62edc34dab9525e2712d487582 [term 7 FOLLOWER]: Refusing update from remote peer 620596b893d84291b88b90dc9e317612: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20240503 02:00:41.974525 17224 consensus_queue.cc:1035] T 21d92d412a5247e8907c5c41d253ad2e P 620596b893d84291b88b90dc9e317612 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:41.974658 17224 consensus_queue.cc:1225] T 21d92d412a5247e8907c5c41d253ad2e P 620596b893d84291b88b90dc9e317612 [LEADER]: Peer c05f7620bb884392931d3243a0b47a9e log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20240503 02:00:41.974896 17224 consensus_queue.cc:1035] T 21d92d412a5247e8907c5c41d253ad2e P 620596b893d84291b88b90dc9e317612 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:41.975630 15348 pending_rounds.cc:76] T 21d92d412a5247e8907c5c41d253ad2e P c05f7620bb884392931d3243a0b47a9e: Aborting all ops after (but not including) 6
I20240503 02:00:41.975693 15348 pending_rounds.cc:97] T 21d92d412a5247e8907c5c41d253ad2e P c05f7620bb884392931d3243a0b47a9e: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20240503 02:00:41.975728 15348 raft_consensus.cc:2866] T 21d92d412a5247e8907c5c41d253ad2e P c05f7620bb884392931d3243a0b47a9e [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 02:00:41.977116 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fc3330dfd0644f6a303659b960124f4" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 7 candidate_status { last_received { term: 4 index: 13 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:41.977977 15351 raft_consensus.cc:3037] T 7fc3330dfd0644f6a303659b960124f4 P c05f7620bb884392931d3243a0b47a9e [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:41.979794 15351 raft_consensus.cc:2396] T 7fc3330dfd0644f6a303659b960124f4 P c05f7620bb884392931d3243a0b47a9e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 620596b893d84291b88b90dc9e317612 for term 7 because replica has last-logged OpId of term: 5 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 13.
W20240503 02:00:41.989284 15048 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34032: Illegal state: replica b602f6099cbb4758847e7e9550685372 is not leader of this config: current role FOLLOWER
I20240503 02:00:41.993680 15087 raft_consensus.cc:1261] T 830da5cb4972453cb9bf2145a0839ced P b602f6099cbb4758847e7e9550685372 [term 7 FOLLOWER]: Refusing update from remote peer 620596b893d84291b88b90dc9e317612: Log matching property violated. Preceding OpId in replica: term: 6 index: 10. Preceding OpId from leader: term: 7 index: 12. (index mismatch)
I20240503 02:00:41.994077 14954 raft_consensus.cc:1261] T 830da5cb4972453cb9bf2145a0839ced P dcfdfa62edc34dab9525e2712d487582 [term 7 FOLLOWER]: Refusing update from remote peer 620596b893d84291b88b90dc9e317612: Log matching property violated. Preceding OpId in replica: term: 6 index: 10. Preceding OpId from leader: term: 7 index: 12. (index mismatch)
I20240503 02:00:41.994144 17011 consensus_queue.cc:1035] T 830da5cb4972453cb9bf2145a0839ced P 620596b893d84291b88b90dc9e317612 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:00:41.994457 17224 consensus_queue.cc:1035] T 830da5cb4972453cb9bf2145a0839ced P 620596b893d84291b88b90dc9e317612 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:00:41.997468 15219 leader_election.cc:288] T 7fc3330dfd0644f6a303659b960124f4 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 7 election: Requested vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:41.999948 15219 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5736c284a85540a1bbab2be6df672fc1"
dest_uuid: "620596b893d84291b88b90dc9e317612"
from {username='slave'} at 127.0.0.1:57680
I20240503 02:00:42.000245 15219 raft_consensus.cc:482] T 5736c284a85540a1bbab2be6df672fc1 P 620596b893d84291b88b90dc9e317612 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:42.000365 15219 raft_consensus.cc:3037] T 5736c284a85540a1bbab2be6df672fc1 P 620596b893d84291b88b90dc9e317612 [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:42.001538 15219 raft_consensus.cc:504] T 5736c284a85540a1bbab2be6df672fc1 P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:42.002566 15087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5736c284a85540a1bbab2be6df672fc1" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:42.002697 15087 raft_consensus.cc:3032] T 5736c284a85540a1bbab2be6df672fc1 P b602f6099cbb4758847e7e9550685372 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 02:00:42.002739 15087 raft_consensus.cc:729] T 5736c284a85540a1bbab2be6df672fc1 P b602f6099cbb4758847e7e9550685372 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:42.002873 15087 consensus_queue.cc:260] T 5736c284a85540a1bbab2be6df672fc1 P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:42.002966 17160 mvcc.cc:205] Tried to move back new op lower bound from 7023417925600993280 to 7023417925428346880. Current Snapshot: MvccSnapshot[applied={T|T < 7023417925600993280}]
I20240503 02:00:42.003031 15087 raft_consensus.cc:3037] T 5736c284a85540a1bbab2be6df672fc1 P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:42.004105 15087 raft_consensus.cc:2396] T 5736c284a85540a1bbab2be6df672fc1 P b602f6099cbb4758847e7e9550685372 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 620596b893d84291b88b90dc9e317612 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240503 02:00:42.004998 15349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5736c284a85540a1bbab2be6df672fc1" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:42.005293 15349 raft_consensus.cc:3037] T 5736c284a85540a1bbab2be6df672fc1 P c05f7620bb884392931d3243a0b47a9e [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:42.006471 15349 raft_consensus.cc:2454] T 5736c284a85540a1bbab2be6df672fc1 P c05f7620bb884392931d3243a0b47a9e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 6.
I20240503 02:00:42.006824 15219 leader_election.cc:288] T 5736c284a85540a1bbab2be6df672fc1 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 6 election: Requested vote from peers b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:42.007637 15214 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f85a361ce0254472b37526362d4edf0e"
dest_uuid: "620596b893d84291b88b90dc9e317612"
from {username='slave'} at 127.0.0.1:57680
I20240503 02:00:42.007784 15214 raft_consensus.cc:482] T f85a361ce0254472b37526362d4edf0e P 620596b893d84291b88b90dc9e317612 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:42.007946 15214 raft_consensus.cc:3037] T f85a361ce0254472b37526362d4edf0e P 620596b893d84291b88b90dc9e317612 [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:42.008872 15214 raft_consensus.cc:504] T f85a361ce0254472b37526362d4edf0e P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:42.009282 15214 leader_election.cc:288] T f85a361ce0254472b37526362d4edf0e P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 6 election: Requested vote from peers b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207), dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817)
I20240503 02:00:42.007949 15155 leader_election.cc:302] T 5736c284a85540a1bbab2be6df672fc1 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 620596b893d84291b88b90dc9e317612, c05f7620bb884392931d3243a0b47a9e; no voters: b602f6099cbb4758847e7e9550685372
I20240503 02:00:42.010224 17224 raft_consensus.cc:2781] T 5736c284a85540a1bbab2be6df672fc1 P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Leader election won for term 6
I20240503 02:00:42.010309 17224 raft_consensus.cc:686] T 5736c284a85540a1bbab2be6df672fc1 P 620596b893d84291b88b90dc9e317612 [term 6 LEADER]: Becoming Leader. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:42.010479 17224 consensus_queue.cc:237] T 5736c284a85540a1bbab2be6df672fc1 P 620596b893d84291b88b90dc9e317612 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:42.010715 15214 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eade118a36da42e3ac08b96a13893a5c"
dest_uuid: "620596b893d84291b88b90dc9e317612"
from {username='slave'} at 127.0.0.1:57680
I20240503 02:00:42.010818 15214 raft_consensus.cc:482] T eade118a36da42e3ac08b96a13893a5c P 620596b893d84291b88b90dc9e317612 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:42.010864 15214 raft_consensus.cc:3037] T eade118a36da42e3ac08b96a13893a5c P 620596b893d84291b88b90dc9e317612 [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:42.011904 15214 raft_consensus.cc:504] T eade118a36da42e3ac08b96a13893a5c P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:42.012291 15214 leader_election.cc:288] T eade118a36da42e3ac08b96a13893a5c P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 6 election: Requested vote from peers b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:42.012656 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85a361ce0254472b37526362d4edf0e" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 6 candidate_status { last_received { term: 5 index: 24 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:42.012782 14958 raft_consensus.cc:3037] T f85a361ce0254472b37526362d4edf0e P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:42.013830 14958 raft_consensus.cc:2396] T f85a361ce0254472b37526362d4edf0e P dcfdfa62edc34dab9525e2712d487582 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 620596b893d84291b88b90dc9e317612 for term 6 because replica has last-logged OpId of term: 5 index: 26, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 24.
I20240503 02:00:42.013831 15349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eade118a36da42e3ac08b96a13893a5c" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 6 candidate_status { last_received { term: 5 index: 24 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:42.013953 15349 raft_consensus.cc:3037] T eade118a36da42e3ac08b96a13893a5c P c05f7620bb884392931d3243a0b47a9e [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:42.014811 15214 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4cdcdde158054cdaabc1856a90fe1302"
dest_uuid: "620596b893d84291b88b90dc9e317612"
from {username='slave'} at 127.0.0.1:57680
I20240503 02:00:42.014950 15214 raft_consensus.cc:482] T 4cdcdde158054cdaabc1856a90fe1302 P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:42.015013 15214 raft_consensus.cc:3037] T 4cdcdde158054cdaabc1856a90fe1302 P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:42.012645 15084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eade118a36da42e3ac08b96a13893a5c" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 6 candidate_status { last_received { term: 5 index: 24 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:42.015619 15084 raft_consensus.cc:3032] T eade118a36da42e3ac08b96a13893a5c P b602f6099cbb4758847e7e9550685372 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 02:00:42.015678 15084 raft_consensus.cc:729] T eade118a36da42e3ac08b96a13893a5c P b602f6099cbb4758847e7e9550685372 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:42.015816 14958 raft_consensus.cc:1226] T f85a361ce0254472b37526362d4edf0e P dcfdfa62edc34dab9525e2712d487582 [term 6 FOLLOWER]: Rejecting Update request from peer b602f6099cbb4758847e7e9550685372 for earlier term 5. Current term is 6. Ops: [5.27-5.27]
I20240503 02:00:42.015833 15084 consensus_queue.cc:260] T eade118a36da42e3ac08b96a13893a5c P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 5.24, Last appended by leader: 24, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:42.016019 15084 raft_consensus.cc:3037] T eade118a36da42e3ac08b96a13893a5c P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:42.016049 15214 raft_consensus.cc:504] T 4cdcdde158054cdaabc1856a90fe1302 P 620596b893d84291b88b90dc9e317612 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:42.016863 15348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcdde158054cdaabc1856a90fe1302" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:42.016984 15348 raft_consensus.cc:3037] T 4cdcdde158054cdaabc1856a90fe1302 P c05f7620bb884392931d3243a0b47a9e [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:42.017045 15084 raft_consensus.cc:2454] T eade118a36da42e3ac08b96a13893a5c P b602f6099cbb4758847e7e9550685372 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 6.
I20240503 02:00:42.017426 15349 raft_consensus.cc:2454] T eade118a36da42e3ac08b96a13893a5c P c05f7620bb884392931d3243a0b47a9e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 6.
I20240503 02:00:42.017743 15155 leader_election.cc:302] T eade118a36da42e3ac08b96a13893a5c P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, c05f7620bb884392931d3243a0b47a9e; no voters:
I20240503 02:00:42.017884 15348 raft_consensus.cc:2454] T 4cdcdde158054cdaabc1856a90fe1302 P c05f7620bb884392931d3243a0b47a9e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 7.
I20240503 02:00:42.018229 15155 leader_election.cc:302] T 4cdcdde158054cdaabc1856a90fe1302 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, c05f7620bb884392931d3243a0b47a9e; no voters:
I20240503 02:00:42.018250 15084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcdde158054cdaabc1856a90fe1302" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:42.018378 15084 raft_consensus.cc:3032] T 4cdcdde158054cdaabc1856a90fe1302 P b602f6099cbb4758847e7e9550685372 [term 6 LEADER]: Stepping down as leader of term 6
I20240503 02:00:42.018412 17224 raft_consensus.cc:2781] T eade118a36da42e3ac08b96a13893a5c P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Leader election won for term 6
I20240503 02:00:42.018424 15084 raft_consensus.cc:729] T 4cdcdde158054cdaabc1856a90fe1302 P b602f6099cbb4758847e7e9550685372 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:42.018498 17224 raft_consensus.cc:686] T eade118a36da42e3ac08b96a13893a5c P 620596b893d84291b88b90dc9e317612 [term 6 LEADER]: Becoming Leader. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:42.018534 16264 raft_consensus.cc:2781] T 4cdcdde158054cdaabc1856a90fe1302 P 620596b893d84291b88b90dc9e317612 [term 7 FOLLOWER]: Leader election won for term 7
I20240503 02:00:42.018576 15084 consensus_queue.cc:260] T 4cdcdde158054cdaabc1856a90fe1302 P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:42.018669 16264 raft_consensus.cc:686] T 4cdcdde158054cdaabc1856a90fe1302 P 620596b893d84291b88b90dc9e317612 [term 7 LEADER]: Becoming Leader. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:42.018630 17224 consensus_queue.cc:237] T eade118a36da42e3ac08b96a13893a5c P 620596b893d84291b88b90dc9e317612 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 5.24, Last appended by leader: 24, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:42.018707 15084 raft_consensus.cc:3037] T 4cdcdde158054cdaabc1856a90fe1302 P b602f6099cbb4758847e7e9550685372 [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:42.018770 16264 consensus_queue.cc:237] T 4cdcdde158054cdaabc1856a90fe1302 P 620596b893d84291b88b90dc9e317612 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:42.020617 17347 consensus_queue.cc:1046] T f85a361ce0254472b37526362d4edf0e P b602f6099cbb4758847e7e9550685372 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 }, Status: INVALID_TERM, Last received: 5.26, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 02:00:42.020685 15214 leader_election.cc:288] T 4cdcdde158054cdaabc1856a90fe1302 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 7 election: Requested vote from peers b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:42.020818 15084 raft_consensus.cc:2396] T 4cdcdde158054cdaabc1856a90fe1302 P b602f6099cbb4758847e7e9550685372 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 620596b893d84291b88b90dc9e317612 for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240503 02:00:42.021020 17332 raft_consensus.cc:3032] T f85a361ce0254472b37526362d4edf0e P b602f6099cbb4758847e7e9550685372 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 02:00:42.021080 17332 raft_consensus.cc:729] T f85a361ce0254472b37526362d4edf0e P b602f6099cbb4758847e7e9550685372 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:42.021205 17332 consensus_queue.cc:260] T f85a361ce0254472b37526362d4edf0e P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 5.27, Last appended by leader: 27, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:42.021354 17332 raft_consensus.cc:3037] T f85a361ce0254472b37526362d4edf0e P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:42.021773 15214 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6d98ba8805024eb79bf5df7a8225d784"
dest_uuid: "620596b893d84291b88b90dc9e317612"
from {username='slave'} at 127.0.0.1:57680
I20240503 02:00:42.021895 15214 raft_consensus.cc:482] T 6d98ba8805024eb79bf5df7a8225d784 P 620596b893d84291b88b90dc9e317612 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:42.022845 15214 raft_consensus.cc:3037] T 6d98ba8805024eb79bf5df7a8225d784 P 620596b893d84291b88b90dc9e317612 [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:42.012212 15087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85a361ce0254472b37526362d4edf0e" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 6 candidate_status { last_received { term: 5 index: 24 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:42.024631 15087 raft_consensus.cc:2396] T f85a361ce0254472b37526362d4edf0e P b602f6099cbb4758847e7e9550685372 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 620596b893d84291b88b90dc9e317612 for term 6 because replica has last-logged OpId of term: 5 index: 27, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 24.
I20240503 02:00:42.024916 15154 leader_election.cc:302] T f85a361ce0254472b37526362d4edf0e P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 620596b893d84291b88b90dc9e317612; no voters: b602f6099cbb4758847e7e9550685372, dcfdfa62edc34dab9525e2712d487582
I20240503 02:00:42.025199 17011 raft_consensus.cc:2726] T f85a361ce0254472b37526362d4edf0e P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240503 02:00:42.025465 15219 raft_consensus.cc:1226] T f85a361ce0254472b37526362d4edf0e P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Rejecting Update request from peer b602f6099cbb4758847e7e9550685372 for earlier term 5. Current term is 6. Ops: [5.25-5.25]
I20240503 02:00:42.024293 14816 catalog_manager.cc:5484] T 5736c284a85540a1bbab2be6df672fc1 P 620596b893d84291b88b90dc9e317612 reported cstate change: term changed from 5 to 6, leader changed from b602f6099cbb4758847e7e9550685372 (127.0.99.130) to 620596b893d84291b88b90dc9e317612 (127.0.99.131). New cstate: current_term: 6 leader_uuid: "620596b893d84291b88b90dc9e317612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:42.025642 14816 catalog_manager.cc:5484] T 4cdcdde158054cdaabc1856a90fe1302 P 620596b893d84291b88b90dc9e317612 reported cstate change: term changed from 6 to 7, leader changed from b602f6099cbb4758847e7e9550685372 (127.0.99.130) to 620596b893d84291b88b90dc9e317612 (127.0.99.131). New cstate: current_term: 7 leader_uuid: "620596b893d84291b88b90dc9e317612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:42.025856 14816 catalog_manager.cc:5484] T eade118a36da42e3ac08b96a13893a5c P 620596b893d84291b88b90dc9e317612 reported cstate change: term changed from 5 to 6, leader changed from b602f6099cbb4758847e7e9550685372 (127.0.99.130) to 620596b893d84291b88b90dc9e317612 (127.0.99.131). New cstate: current_term: 6 leader_uuid: "620596b893d84291b88b90dc9e317612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:42.026672 15214 raft_consensus.cc:504] T 6d98ba8805024eb79bf5df7a8225d784 P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:42.027052 15214 leader_election.cc:288] T 6d98ba8805024eb79bf5df7a8225d784 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 6 election: Requested vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
I20240503 02:00:42.027343 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d98ba8805024eb79bf5df7a8225d784" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 6 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:42.027489 14958 raft_consensus.cc:3037] T 6d98ba8805024eb79bf5df7a8225d784 P dcfdfa62edc34dab9525e2712d487582 [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:42.028216 15084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d98ba8805024eb79bf5df7a8225d784" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 6 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:42.028355 15084 raft_consensus.cc:3032] T 6d98ba8805024eb79bf5df7a8225d784 P b602f6099cbb4758847e7e9550685372 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 02:00:42.028429 15084 raft_consensus.cc:729] T 6d98ba8805024eb79bf5df7a8225d784 P b602f6099cbb4758847e7e9550685372 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:42.028631 14958 raft_consensus.cc:2396] T 6d98ba8805024eb79bf5df7a8225d784 P dcfdfa62edc34dab9525e2712d487582 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 620596b893d84291b88b90dc9e317612 for term 6 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 6.
I20240503 02:00:42.028609 15084 consensus_queue.cc:260] T 6d98ba8805024eb79bf5df7a8225d784 P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.8, Last appended by leader: 8, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:42.028775 15084 raft_consensus.cc:3037] T 6d98ba8805024eb79bf5df7a8225d784 P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:42.032219 15214 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "42ceeb7b3c4146cd8a2e18b3e9ae987c"
dest_uuid: "620596b893d84291b88b90dc9e317612"
from {username='slave'} at 127.0.0.1:57680
I20240503 02:00:42.032891 15214 raft_consensus.cc:482] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:42.032982 15214 raft_consensus.cc:3037] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:42.034024 15214 raft_consensus.cc:504] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P 620596b893d84291b88b90dc9e317612 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:42.034374 15214 leader_election.cc:288] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 7 election: Requested vote from peers b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207), dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817)
I20240503 02:00:42.034731 15089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ceeb7b3c4146cd8a2e18b3e9ae987c" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 7 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:42.034730 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ceeb7b3c4146cd8a2e18b3e9ae987c" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 7 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:42.034847 15089 raft_consensus.cc:3032] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P b602f6099cbb4758847e7e9550685372 [term 6 LEADER]: Stepping down as leader of term 6
I20240503 02:00:42.034847 14958 raft_consensus.cc:3037] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P dcfdfa62edc34dab9525e2712d487582 [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:42.034888 15089 raft_consensus.cc:729] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P b602f6099cbb4758847e7e9550685372 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:42.035032 15089 consensus_queue.cc:260] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:42.035166 15089 raft_consensus.cc:3037] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P b602f6099cbb4758847e7e9550685372 [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:42.035737 15084 raft_consensus.cc:2396] T 6d98ba8805024eb79bf5df7a8225d784 P b602f6099cbb4758847e7e9550685372 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 620596b893d84291b88b90dc9e317612 for term 6 because replica has last-logged OpId of term: 5 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 6.
I20240503 02:00:42.035941 14958 raft_consensus.cc:2396] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P dcfdfa62edc34dab9525e2712d487582 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 620596b893d84291b88b90dc9e317612 for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20240503 02:00:42.036234 15089 raft_consensus.cc:2396] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P b602f6099cbb4758847e7e9550685372 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 620596b893d84291b88b90dc9e317612 for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20240503 02:00:42.036900 15154 leader_election.cc:302] T 6d98ba8805024eb79bf5df7a8225d784 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 620596b893d84291b88b90dc9e317612; no voters: b602f6099cbb4758847e7e9550685372, dcfdfa62edc34dab9525e2712d487582
I20240503 02:00:42.037285 17011 raft_consensus.cc:2726] T 6d98ba8805024eb79bf5df7a8225d784 P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240503 02:00:42.037508 15154 leader_election.cc:302] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 620596b893d84291b88b90dc9e317612; no voters: b602f6099cbb4758847e7e9550685372, dcfdfa62edc34dab9525e2712d487582
I20240503 02:00:42.037634 17011 raft_consensus.cc:2726] T 42ceeb7b3c4146cd8a2e18b3e9ae987c P 620596b893d84291b88b90dc9e317612 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240503 02:00:42.037932 15219 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6b0c8b72e1884529a8a9b83d2a734d71"
dest_uuid: "620596b893d84291b88b90dc9e317612"
from {username='slave'} at 127.0.0.1:57680
I20240503 02:00:42.038057 15219 raft_consensus.cc:482] T 6b0c8b72e1884529a8a9b83d2a734d71 P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:42.038105 15219 raft_consensus.cc:3037] T 6b0c8b72e1884529a8a9b83d2a734d71 P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:42.045823 15219 raft_consensus.cc:504] T 6b0c8b72e1884529a8a9b83d2a734d71 P 620596b893d84291b88b90dc9e317612 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
W20240503 02:00:42.046542 15311 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50950: Illegal state: replica c05f7620bb884392931d3243a0b47a9e is not leader of this config: current role FOLLOWER
I20240503 02:00:42.046833 14956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b0c8b72e1884529a8a9b83d2a734d71" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:42.046962 14956 raft_consensus.cc:3037] T 6b0c8b72e1884529a8a9b83d2a734d71 P dcfdfa62edc34dab9525e2712d487582 [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:42.048228 14956 raft_consensus.cc:2454] T 6b0c8b72e1884529a8a9b83d2a734d71 P dcfdfa62edc34dab9525e2712d487582 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 7.
I20240503 02:00:42.048712 15153 leader_election.cc:302] T 6b0c8b72e1884529a8a9b83d2a734d71 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:42.048949 17420 raft_consensus.cc:2781] T 6b0c8b72e1884529a8a9b83d2a734d71 P 620596b893d84291b88b90dc9e317612 [term 7 FOLLOWER]: Leader election won for term 7
I20240503 02:00:42.049078 17420 raft_consensus.cc:686] T 6b0c8b72e1884529a8a9b83d2a734d71 P 620596b893d84291b88b90dc9e317612 [term 7 LEADER]: Becoming Leader. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:42.049290 17420 consensus_queue.cc:237] T 6b0c8b72e1884529a8a9b83d2a734d71 P 620596b893d84291b88b90dc9e317612 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:42.049896 15219 leader_election.cc:288] T 6b0c8b72e1884529a8a9b83d2a734d71 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 7 election: Requested vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
I20240503 02:00:42.050374 15087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b0c8b72e1884529a8a9b83d2a734d71" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:42.050520 15087 raft_consensus.cc:3032] T 6b0c8b72e1884529a8a9b83d2a734d71 P b602f6099cbb4758847e7e9550685372 [term 6 LEADER]: Stepping down as leader of term 6
I20240503 02:00:42.050561 15087 raft_consensus.cc:729] T 6b0c8b72e1884529a8a9b83d2a734d71 P b602f6099cbb4758847e7e9550685372 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:42.050740 15087 consensus_queue.cc:260] T 6b0c8b72e1884529a8a9b83d2a734d71 P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:42.050925 15087 raft_consensus.cc:3037] T 6b0c8b72e1884529a8a9b83d2a734d71 P b602f6099cbb4758847e7e9550685372 [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:42.051939 15087 raft_consensus.cc:2396] T 6b0c8b72e1884529a8a9b83d2a734d71 P b602f6099cbb4758847e7e9550685372 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 620596b893d84291b88b90dc9e317612 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20240503 02:00:42.053109 15214 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9a74cca2988a4f219f6fd9b07ade42d8"
dest_uuid: "620596b893d84291b88b90dc9e317612"
from {username='slave'} at 127.0.0.1:57680
I20240503 02:00:42.053241 15214 raft_consensus.cc:482] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:42.053297 15214 raft_consensus.cc:3037] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:42.054096 14816 catalog_manager.cc:5484] T 6b0c8b72e1884529a8a9b83d2a734d71 P 620596b893d84291b88b90dc9e317612 reported cstate change: term changed from 6 to 7, leader changed from b602f6099cbb4758847e7e9550685372 (127.0.99.130) to 620596b893d84291b88b90dc9e317612 (127.0.99.131). New cstate: current_term: 7 leader_uuid: "620596b893d84291b88b90dc9e317612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:42.054307 15214 raft_consensus.cc:504] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
W20240503 02:00:42.054378 14915 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48020: Illegal state: replica dcfdfa62edc34dab9525e2712d487582 is not leader of this config: current role FOLLOWER
I20240503 02:00:42.054692 15214 leader_election.cc:288] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 7 election: Requested vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:42.055240 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a74cca2988a4f219f6fd9b07ade42d8" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 7 candidate_status { last_received { term: 4 index: 8 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:42.055390 15351 raft_consensus.cc:3037] T 9a74cca2988a4f219f6fd9b07ade42d8 P c05f7620bb884392931d3243a0b47a9e [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:42.055639 15214 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c74522fbb5c64d27b5c95f05896b0716"
dest_uuid: "620596b893d84291b88b90dc9e317612"
from {username='slave'} at 127.0.0.1:57680
I20240503 02:00:42.055753 15214 raft_consensus.cc:482] T c74522fbb5c64d27b5c95f05896b0716 P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:42.055809 15214 raft_consensus.cc:3037] T c74522fbb5c64d27b5c95f05896b0716 P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:42.055697 14956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a74cca2988a4f219f6fd9b07ade42d8" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 7 candidate_status { last_received { term: 4 index: 8 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:42.055946 14956 raft_consensus.cc:3037] T 9a74cca2988a4f219f6fd9b07ade42d8 P dcfdfa62edc34dab9525e2712d487582 [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:42.056603 15351 raft_consensus.cc:2396] T 9a74cca2988a4f219f6fd9b07ade42d8 P c05f7620bb884392931d3243a0b47a9e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 620596b893d84291b88b90dc9e317612 for term 7 because replica has last-logged OpId of term: 5 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 8.
I20240503 02:00:42.056742 15214 raft_consensus.cc:504] T c74522fbb5c64d27b5c95f05896b0716 P 620596b893d84291b88b90dc9e317612 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:42.057119 15214 leader_election.cc:288] T c74522fbb5c64d27b5c95f05896b0716 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 7 election: Requested vote from peers c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
I20240503 02:00:42.057410 14956 raft_consensus.cc:2454] T 9a74cca2988a4f219f6fd9b07ade42d8 P dcfdfa62edc34dab9525e2712d487582 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 7.
I20240503 02:00:42.057658 15087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c74522fbb5c64d27b5c95f05896b0716" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 7 candidate_status { last_received { term: 6 index: 31 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:42.057781 15087 raft_consensus.cc:3032] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372 [term 6 LEADER]: Stepping down as leader of term 6
I20240503 02:00:42.057757 15214 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2b757c430aaa42f382d174ca57f5fcfc"
dest_uuid: "620596b893d84291b88b90dc9e317612"
from {username='slave'} at 127.0.0.1:57680
I20240503 02:00:42.057824 15087 raft_consensus.cc:729] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:42.057861 15214 raft_consensus.cc:482] T 2b757c430aaa42f382d174ca57f5fcfc P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:42.057907 15214 raft_consensus.cc:3037] T 2b757c430aaa42f382d174ca57f5fcfc P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:42.057931 15153 leader_election.cc:302] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 620596b893d84291b88b90dc9e317612, dcfdfa62edc34dab9525e2712d487582; no voters: c05f7620bb884392931d3243a0b47a9e
I20240503 02:00:42.057993 15087 consensus_queue.cc:260] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 6.31, Last appended by leader: 31, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:42.058164 15087 raft_consensus.cc:3037] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372 [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:42.058209 17420 raft_consensus.cc:2781] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612 [term 7 FOLLOWER]: Leader election won for term 7
I20240503 02:00:42.058357 17420 raft_consensus.cc:686] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612 [term 7 LEADER]: Becoming Leader. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:42.058480 17420 consensus_queue.cc:237] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:42.058939 15214 raft_consensus.cc:504] T 2b757c430aaa42f382d174ca57f5fcfc P 620596b893d84291b88b90dc9e317612 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:42.059152 15087 raft_consensus.cc:2454] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 7.
I20240503 02:00:42.059305 15214 leader_election.cc:288] T 2b757c430aaa42f382d174ca57f5fcfc P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 7 election: Requested vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
I20240503 02:00:42.059876 14956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b757c430aaa42f382d174ca57f5fcfc" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 7 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:42.060057 14956 raft_consensus.cc:3037] T 2b757c430aaa42f382d174ca57f5fcfc P dcfdfa62edc34dab9525e2712d487582 [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:42.060125 15154 leader_election.cc:302] T c74522fbb5c64d27b5c95f05896b0716 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, b602f6099cbb4758847e7e9550685372; no voters:
I20240503 02:00:42.060164 15214 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5d448658ed3b460fb1725a26a7c743e8"
dest_uuid: "620596b893d84291b88b90dc9e317612"
from {username='slave'} at 127.0.0.1:57680
I20240503 02:00:42.060252 15214 raft_consensus.cc:482] T 5d448658ed3b460fb1725a26a7c743e8 P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:42.060304 15214 raft_consensus.cc:3037] T 5d448658ed3b460fb1725a26a7c743e8 P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:42.060492 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c74522fbb5c64d27b5c95f05896b0716" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 7 candidate_status { last_received { term: 6 index: 31 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:42.060606 15351 raft_consensus.cc:3037] T c74522fbb5c64d27b5c95f05896b0716 P c05f7620bb884392931d3243a0b47a9e [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:42.061357 14816 catalog_manager.cc:5484] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612 reported cstate change: term changed from 5 to 7, leader changed from c05f7620bb884392931d3243a0b47a9e (127.0.99.132) to 620596b893d84291b88b90dc9e317612 (127.0.99.131). New cstate: current_term: 7 leader_uuid: "620596b893d84291b88b90dc9e317612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:42.061511 15351 raft_consensus.cc:2454] T c74522fbb5c64d27b5c95f05896b0716 P c05f7620bb884392931d3243a0b47a9e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 7.
I20240503 02:00:42.061681 15214 raft_consensus.cc:504] T 5d448658ed3b460fb1725a26a7c743e8 P 620596b893d84291b88b90dc9e317612 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:42.062117 15087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b757c430aaa42f382d174ca57f5fcfc" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 7 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:42.062228 15087 raft_consensus.cc:3032] T 2b757c430aaa42f382d174ca57f5fcfc P b602f6099cbb4758847e7e9550685372 [term 6 LEADER]: Stepping down as leader of term 6
I20240503 02:00:42.062268 15087 raft_consensus.cc:729] T 2b757c430aaa42f382d174ca57f5fcfc P b602f6099cbb4758847e7e9550685372 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:42.062520 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d448658ed3b460fb1725a26a7c743e8" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 7 candidate_status { last_received { term: 4 index: 9 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:42.062628 14958 raft_consensus.cc:3032] T 5d448658ed3b460fb1725a26a7c743e8 P dcfdfa62edc34dab9525e2712d487582 [term 6 LEADER]: Stepping down as leader of term 6
I20240503 02:00:42.062665 14958 raft_consensus.cc:729] T 5d448658ed3b460fb1725a26a7c743e8 P dcfdfa62edc34dab9525e2712d487582 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:42.062827 14958 consensus_queue.cc:260] T 5d448658ed3b460fb1725a26a7c743e8 P dcfdfa62edc34dab9525e2712d487582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 6.10, Last appended by leader: 10, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:42.062959 14958 raft_consensus.cc:3037] T 5d448658ed3b460fb1725a26a7c743e8 P dcfdfa62edc34dab9525e2712d487582 [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:42.063975 14958 raft_consensus.cc:2396] T 5d448658ed3b460fb1725a26a7c743e8 P dcfdfa62edc34dab9525e2712d487582 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 620596b893d84291b88b90dc9e317612 for term 7 because replica has last-logged OpId of term: 6 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 9.
I20240503 02:00:42.064244 15349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d448658ed3b460fb1725a26a7c743e8" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 7 candidate_status { last_received { term: 4 index: 9 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:42.064416 14954 raft_consensus.cc:1261] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582 [term 7 FOLLOWER]: Refusing update from remote peer b602f6099cbb4758847e7e9550685372: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20240503 02:00:42.064410 15214 leader_election.cc:288] T 5d448658ed3b460fb1725a26a7c743e8 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 7 election: Requested vote from peers c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009), dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817)
I20240503 02:00:42.064440 15349 raft_consensus.cc:3037] T 5d448658ed3b460fb1725a26a7c743e8 P c05f7620bb884392931d3243a0b47a9e [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:42.064826 17418 consensus_queue.cc:1035] T 590736f5f1ca4703821c409d882c2f77 P b602f6099cbb4758847e7e9550685372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:42.064940 17418 consensus_queue.cc:1225] T 590736f5f1ca4703821c409d882c2f77 P b602f6099cbb4758847e7e9550685372 [LEADER]: Peer dcfdfa62edc34dab9525e2712d487582 log is divergent from this leader: its last log entry 6.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20240503 02:00:42.065150 15351 raft_consensus.cc:1261] T 590736f5f1ca4703821c409d882c2f77 P c05f7620bb884392931d3243a0b47a9e [term 7 FOLLOWER]: Refusing update from remote peer b602f6099cbb4758847e7e9550685372: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20240503 02:00:42.065392 15349 raft_consensus.cc:2396] T 5d448658ed3b460fb1725a26a7c743e8 P c05f7620bb884392931d3243a0b47a9e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 620596b893d84291b88b90dc9e317612 for term 7 because replica has last-logged OpId of term: 5 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 9.
I20240503 02:00:42.065526 17418 consensus_queue.cc:1035] T 590736f5f1ca4703821c409d882c2f77 P b602f6099cbb4758847e7e9550685372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:00:42.065827 15155 leader_election.cc:302] T 5d448658ed3b460fb1725a26a7c743e8 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 620596b893d84291b88b90dc9e317612; no voters: c05f7620bb884392931d3243a0b47a9e, dcfdfa62edc34dab9525e2712d487582
I20240503 02:00:42.066165 17011 raft_consensus.cc:2781] T c74522fbb5c64d27b5c95f05896b0716 P 620596b893d84291b88b90dc9e317612 [term 7 FOLLOWER]: Leader election won for term 7
I20240503 02:00:42.066244 17011 raft_consensus.cc:686] T c74522fbb5c64d27b5c95f05896b0716 P 620596b893d84291b88b90dc9e317612 [term 7 LEADER]: Becoming Leader. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:42.066278 15214 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6798a9ed0cf441aca7b8ae1b05c001f3"
dest_uuid: "620596b893d84291b88b90dc9e317612"
from {username='slave'} at 127.0.0.1:57680
I20240503 02:00:42.066363 17011 consensus_queue.cc:237] T c74522fbb5c64d27b5c95f05896b0716 P 620596b893d84291b88b90dc9e317612 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 6.31, Last appended by leader: 31, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:42.066417 15214 raft_consensus.cc:482] T 6798a9ed0cf441aca7b8ae1b05c001f3 P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:42.066432 14954 pending_rounds.cc:76] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582: Aborting all ops after (but not including) 7
I20240503 02:00:42.066471 15214 raft_consensus.cc:3037] T 6798a9ed0cf441aca7b8ae1b05c001f3 P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:42.066471 14954 pending_rounds.cc:97] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582: Aborting uncommitted NO_OP operation due to leader change: 6.8
I20240503 02:00:42.066502 14954 raft_consensus.cc:2866] T 590736f5f1ca4703821c409d882c2f77 P dcfdfa62edc34dab9525e2712d487582 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 02:00:42.066865 17011 raft_consensus.cc:2726] T 5d448658ed3b460fb1725a26a7c743e8 P 620596b893d84291b88b90dc9e317612 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240503 02:00:42.067203 15087 consensus_queue.cc:260] T 2b757c430aaa42f382d174ca57f5fcfc P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:42.067368 15087 raft_consensus.cc:3037] T 2b757c430aaa42f382d174ca57f5fcfc P b602f6099cbb4758847e7e9550685372 [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:42.067485 15214 raft_consensus.cc:504] T 6798a9ed0cf441aca7b8ae1b05c001f3 P 620596b893d84291b88b90dc9e317612 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:42.068118 15214 leader_election.cc:288] T 6798a9ed0cf441aca7b8ae1b05c001f3 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 7 election: Requested vote from peers c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009), dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817)
I20240503 02:00:42.069262 14811 catalog_manager.cc:5484] T c74522fbb5c64d27b5c95f05896b0716 P 620596b893d84291b88b90dc9e317612 reported cstate change: term changed from 6 to 7, leader changed from b602f6099cbb4758847e7e9550685372 (127.0.99.130) to 620596b893d84291b88b90dc9e317612 (127.0.99.131). New cstate: current_term: 7 leader_uuid: "620596b893d84291b88b90dc9e317612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:42.074805 15087 raft_consensus.cc:2396] T 2b757c430aaa42f382d174ca57f5fcfc P b602f6099cbb4758847e7e9550685372 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 620596b893d84291b88b90dc9e317612 for term 7 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 6.
I20240503 02:00:42.075309 15214 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ebd7cdd284a74741b338e9f4778502b5"
dest_uuid: "620596b893d84291b88b90dc9e317612"
from {username='slave'} at 127.0.0.1:57680
I20240503 02:00:42.077811 15214 raft_consensus.cc:482] T ebd7cdd284a74741b338e9f4778502b5 P 620596b893d84291b88b90dc9e317612 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:42.077989 15214 raft_consensus.cc:3037] T ebd7cdd284a74741b338e9f4778502b5 P 620596b893d84291b88b90dc9e317612 [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:42.079084 15214 raft_consensus.cc:504] T ebd7cdd284a74741b338e9f4778502b5 P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:42.079504 15214 leader_election.cc:288] T ebd7cdd284a74741b338e9f4778502b5 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 6 election: Requested vote from peers b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:42.079914 15089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebd7cdd284a74741b338e9f4778502b5" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:42.080065 15089 raft_consensus.cc:3032] T ebd7cdd284a74741b338e9f4778502b5 P b602f6099cbb4758847e7e9550685372 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 02:00:42.080106 15089 raft_consensus.cc:729] T ebd7cdd284a74741b338e9f4778502b5 P b602f6099cbb4758847e7e9550685372 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:42.080245 15089 consensus_queue.cc:260] T ebd7cdd284a74741b338e9f4778502b5 P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:42.080432 15089 raft_consensus.cc:3037] T ebd7cdd284a74741b338e9f4778502b5 P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:42.080662 15214 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6b225db529f14e7386f21f42bf31493b"
dest_uuid: "620596b893d84291b88b90dc9e317612"
from {username='slave'} at 127.0.0.1:57680
I20240503 02:00:42.080785 15214 raft_consensus.cc:482] T 6b225db529f14e7386f21f42bf31493b P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:42.080834 15214 raft_consensus.cc:3037] T 6b225db529f14e7386f21f42bf31493b P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:42.081593 15089 raft_consensus.cc:2396] T ebd7cdd284a74741b338e9f4778502b5 P b602f6099cbb4758847e7e9550685372 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 620596b893d84291b88b90dc9e317612 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20240503 02:00:42.081733 15214 raft_consensus.cc:504] T 6b225db529f14e7386f21f42bf31493b P 620596b893d84291b88b90dc9e317612 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:42.082252 15214 leader_election.cc:288] T 6b225db529f14e7386f21f42bf31493b P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 7 election: Requested vote from peers b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207), dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817)
I20240503 02:00:42.082553 15348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebd7cdd284a74741b338e9f4778502b5" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:42.082677 15348 raft_consensus.cc:3037] T ebd7cdd284a74741b338e9f4778502b5 P c05f7620bb884392931d3243a0b47a9e [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:42.083612 15348 raft_consensus.cc:2454] T ebd7cdd284a74741b338e9f4778502b5 P c05f7620bb884392931d3243a0b47a9e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 6.
I20240503 02:00:42.083848 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b225db529f14e7386f21f42bf31493b" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 7 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:42.083968 14958 raft_consensus.cc:3037] T 6b225db529f14e7386f21f42bf31493b P dcfdfa62edc34dab9525e2712d487582 [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:42.084761 15089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b225db529f14e7386f21f42bf31493b" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 7 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:42.084936 15155 leader_election.cc:302] T ebd7cdd284a74741b338e9f4778502b5 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 620596b893d84291b88b90dc9e317612, c05f7620bb884392931d3243a0b47a9e; no voters: b602f6099cbb4758847e7e9550685372
I20240503 02:00:42.085150 14954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6798a9ed0cf441aca7b8ae1b05c001f3" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:42.085194 17420 raft_consensus.cc:2781] T ebd7cdd284a74741b338e9f4778502b5 P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Leader election won for term 6
I20240503 02:00:42.085263 14954 raft_consensus.cc:3032] T 6798a9ed0cf441aca7b8ae1b05c001f3 P dcfdfa62edc34dab9525e2712d487582 [term 6 LEADER]: Stepping down as leader of term 6
I20240503 02:00:42.085299 14954 raft_consensus.cc:729] T 6798a9ed0cf441aca7b8ae1b05c001f3 P dcfdfa62edc34dab9525e2712d487582 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:42.085439 14954 consensus_queue.cc:260] T 6798a9ed0cf441aca7b8ae1b05c001f3 P dcfdfa62edc34dab9525e2712d487582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:42.085510 17420 raft_consensus.cc:686] T ebd7cdd284a74741b338e9f4778502b5 P 620596b893d84291b88b90dc9e317612 [term 6 LEADER]: Becoming Leader. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:42.085568 14954 raft_consensus.cc:3037] T 6798a9ed0cf441aca7b8ae1b05c001f3 P dcfdfa62edc34dab9525e2712d487582 [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:42.085644 17420 consensus_queue.cc:237] T ebd7cdd284a74741b338e9f4778502b5 P 620596b893d84291b88b90dc9e317612 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:42.084882 15089 raft_consensus.cc:3032] T 6b225db529f14e7386f21f42bf31493b P b602f6099cbb4758847e7e9550685372 [term 6 LEADER]: Stepping down as leader of term 6
I20240503 02:00:42.087024 15089 raft_consensus.cc:729] T 6b225db529f14e7386f21f42bf31493b P b602f6099cbb4758847e7e9550685372 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:42.087167 15089 consensus_queue.cc:260] T 6b225db529f14e7386f21f42bf31493b P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.9, Last appended by leader: 9, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:42.087327 15089 raft_consensus.cc:3037] T 6b225db529f14e7386f21f42bf31493b P b602f6099cbb4758847e7e9550685372 [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:42.086496 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6798a9ed0cf441aca7b8ae1b05c001f3" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:42.087752 15351 raft_consensus.cc:3037] T 6798a9ed0cf441aca7b8ae1b05c001f3 P c05f7620bb884392931d3243a0b47a9e [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:42.089298 15351 raft_consensus.cc:2396] T 6798a9ed0cf441aca7b8ae1b05c001f3 P c05f7620bb884392931d3243a0b47a9e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 620596b893d84291b88b90dc9e317612 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 02:00:42.091214 14816 catalog_manager.cc:5484] T ebd7cdd284a74741b338e9f4778502b5 P 620596b893d84291b88b90dc9e317612 reported cstate change: term changed from 5 to 6, leader changed from b602f6099cbb4758847e7e9550685372 (127.0.99.130) to 620596b893d84291b88b90dc9e317612 (127.0.99.131). New cstate: current_term: 6 leader_uuid: "620596b893d84291b88b90dc9e317612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:42.091843 15214 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "11275e009edd4354b0807b7f015023be"
dest_uuid: "620596b893d84291b88b90dc9e317612"
from {username='slave'} at 127.0.0.1:57680
I20240503 02:00:42.091991 15214 raft_consensus.cc:482] T 11275e009edd4354b0807b7f015023be P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:42.092051 15214 raft_consensus.cc:3037] T 11275e009edd4354b0807b7f015023be P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:42.093114 15214 raft_consensus.cc:504] T 11275e009edd4354b0807b7f015023be P 620596b893d84291b88b90dc9e317612 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:42.093516 15214 leader_election.cc:288] T 11275e009edd4354b0807b7f015023be P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 7 election: Requested vote from peers c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009), dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817)
I20240503 02:00:42.093561 14958 raft_consensus.cc:2396] T 6b225db529f14e7386f21f42bf31493b P dcfdfa62edc34dab9525e2712d487582 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 620596b893d84291b88b90dc9e317612 for term 7 because replica has last-logged OpId of term: 5 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 7.
I20240503 02:00:42.094494 15089 raft_consensus.cc:2396] T 6b225db529f14e7386f21f42bf31493b P b602f6099cbb4758847e7e9550685372 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 620596b893d84291b88b90dc9e317612 for term 7 because replica has last-logged OpId of term: 6 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 7.
I20240503 02:00:42.094520 15214 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5da91b7af68e4cf597dc89a017ee0197"
dest_uuid: "620596b893d84291b88b90dc9e317612"
from {username='slave'} at 127.0.0.1:57680
I20240503 02:00:42.094638 15214 raft_consensus.cc:482] T 5da91b7af68e4cf597dc89a017ee0197 P 620596b893d84291b88b90dc9e317612 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:42.094700 15214 raft_consensus.cc:3037] T 5da91b7af68e4cf597dc89a017ee0197 P 620596b893d84291b88b90dc9e317612 [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:42.094833 15154 leader_election.cc:302] T 6b225db529f14e7386f21f42bf31493b P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 620596b893d84291b88b90dc9e317612; no voters: b602f6099cbb4758847e7e9550685372, dcfdfa62edc34dab9525e2712d487582
I20240503 02:00:42.095063 17011 raft_consensus.cc:2726] T 6b225db529f14e7386f21f42bf31493b P 620596b893d84291b88b90dc9e317612 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240503 02:00:42.095074 14954 raft_consensus.cc:2396] T 6798a9ed0cf441aca7b8ae1b05c001f3 P dcfdfa62edc34dab9525e2712d487582 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 620596b893d84291b88b90dc9e317612 for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 02:00:42.095660 15153 leader_election.cc:302] T 6798a9ed0cf441aca7b8ae1b05c001f3 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 620596b893d84291b88b90dc9e317612; no voters: c05f7620bb884392931d3243a0b47a9e, dcfdfa62edc34dab9525e2712d487582
I20240503 02:00:42.095860 17224 raft_consensus.cc:2726] T 6798a9ed0cf441aca7b8ae1b05c001f3 P 620596b893d84291b88b90dc9e317612 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240503 02:00:42.096801 15349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11275e009edd4354b0807b7f015023be" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:42.096923 15349 raft_consensus.cc:3037] T 11275e009edd4354b0807b7f015023be P c05f7620bb884392931d3243a0b47a9e [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:42.097143 14956 raft_consensus.cc:2396] T 2b757c430aaa42f382d174ca57f5fcfc P dcfdfa62edc34dab9525e2712d487582 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 620596b893d84291b88b90dc9e317612 for term 7 because replica has last-logged OpId of term: 5 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 6.
I20240503 02:00:42.097465 15153 leader_election.cc:302] T 2b757c430aaa42f382d174ca57f5fcfc P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 620596b893d84291b88b90dc9e317612; no voters: b602f6099cbb4758847e7e9550685372, dcfdfa62edc34dab9525e2712d487582
I20240503 02:00:42.097671 17224 raft_consensus.cc:2726] T 2b757c430aaa42f382d174ca57f5fcfc P 620596b893d84291b88b90dc9e317612 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240503 02:00:42.097986 15349 raft_consensus.cc:2454] T 11275e009edd4354b0807b7f015023be P c05f7620bb884392931d3243a0b47a9e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 7.
I20240503 02:00:42.098320 15155 leader_election.cc:302] T 11275e009edd4354b0807b7f015023be P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620596b893d84291b88b90dc9e317612, c05f7620bb884392931d3243a0b47a9e; no voters:
I20240503 02:00:42.098515 17224 raft_consensus.cc:2781] T 11275e009edd4354b0807b7f015023be P 620596b893d84291b88b90dc9e317612 [term 7 FOLLOWER]: Leader election won for term 7
I20240503 02:00:42.098574 17224 raft_consensus.cc:686] T 11275e009edd4354b0807b7f015023be P 620596b893d84291b88b90dc9e317612 [term 7 LEADER]: Becoming Leader. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:42.098699 17224 consensus_queue.cc:237] T 11275e009edd4354b0807b7f015023be P 620596b893d84291b88b90dc9e317612 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:42.104848 15214 raft_consensus.cc:504] T 5da91b7af68e4cf597dc89a017ee0197 P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:42.104967 14954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11275e009edd4354b0807b7f015023be" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:42.105276 15214 leader_election.cc:288] T 5da91b7af68e4cf597dc89a017ee0197 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 6 election: Requested vote from peers b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207), c05f7620bb884392931d3243a0b47a9e (127.0.99.132:43009)
I20240503 02:00:42.105554 14954 raft_consensus.cc:3032] T 11275e009edd4354b0807b7f015023be P dcfdfa62edc34dab9525e2712d487582 [term 6 LEADER]: Stepping down as leader of term 6
I20240503 02:00:42.105683 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5da91b7af68e4cf597dc89a017ee0197" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
I20240503 02:00:42.105803 15351 raft_consensus.cc:3037] T 5da91b7af68e4cf597dc89a017ee0197 P c05f7620bb884392931d3243a0b47a9e [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:42.106139 15087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5da91b7af68e4cf597dc89a017ee0197" candidate_uuid: "620596b893d84291b88b90dc9e317612" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:42.106249 15087 raft_consensus.cc:3032] T 5da91b7af68e4cf597dc89a017ee0197 P b602f6099cbb4758847e7e9550685372 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 02:00:42.106287 15087 raft_consensus.cc:729] T 5da91b7af68e4cf597dc89a017ee0197 P b602f6099cbb4758847e7e9550685372 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:42.106424 15087 consensus_queue.cc:260] T 5da91b7af68e4cf597dc89a017ee0197 P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:42.106566 15087 raft_consensus.cc:3037] T 5da91b7af68e4cf597dc89a017ee0197 P b602f6099cbb4758847e7e9550685372 [term 5 FOLLOWER]: Advancing to term 6
I20240503 02:00:42.107265 14811 catalog_manager.cc:5484] T 11275e009edd4354b0807b7f015023be P 620596b893d84291b88b90dc9e317612 reported cstate change: term changed from 6 to 7, leader changed from dcfdfa62edc34dab9525e2712d487582 (127.0.99.129) to 620596b893d84291b88b90dc9e317612 (127.0.99.131). New cstate: current_term: 7 leader_uuid: "620596b893d84291b88b90dc9e317612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:42.107596 15087 raft_consensus.cc:2396] T 5da91b7af68e4cf597dc89a017ee0197 P b602f6099cbb4758847e7e9550685372 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 620596b893d84291b88b90dc9e317612 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 02:00:42.108868 15351 raft_consensus.cc:2454] T 5da91b7af68e4cf597dc89a017ee0197 P c05f7620bb884392931d3243a0b47a9e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 620596b893d84291b88b90dc9e317612 in term 6.
I20240503 02:00:42.110872 15155 leader_election.cc:302] T 5da91b7af68e4cf597dc89a017ee0197 P 620596b893d84291b88b90dc9e317612 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 620596b893d84291b88b90dc9e317612, c05f7620bb884392931d3243a0b47a9e; no voters: b602f6099cbb4758847e7e9550685372
I20240503 02:00:42.111117 17224 raft_consensus.cc:2781] T 5da91b7af68e4cf597dc89a017ee0197 P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Leader election won for term 6
I20240503 02:00:42.111192 17224 raft_consensus.cc:686] T 5da91b7af68e4cf597dc89a017ee0197 P 620596b893d84291b88b90dc9e317612 [term 6 LEADER]: Becoming Leader. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:42.111301 17224 consensus_queue.cc:237] T 5da91b7af68e4cf597dc89a017ee0197 P 620596b893d84291b88b90dc9e317612 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:42.112800 14954 raft_consensus.cc:729] T 11275e009edd4354b0807b7f015023be P dcfdfa62edc34dab9525e2712d487582 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: dcfdfa62edc34dab9525e2712d487582, State: Running, Role: LEADER
I20240503 02:00:42.112994 14954 consensus_queue.cc:260] T 11275e009edd4354b0807b7f015023be P dcfdfa62edc34dab9525e2712d487582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } }
I20240503 02:00:42.113143 14954 raft_consensus.cc:3037] T 11275e009edd4354b0807b7f015023be P dcfdfa62edc34dab9525e2712d487582 [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:42.114207 14816 catalog_manager.cc:5484] T 5da91b7af68e4cf597dc89a017ee0197 P 620596b893d84291b88b90dc9e317612 reported cstate change: term changed from 5 to 6, leader changed from b602f6099cbb4758847e7e9550685372 (127.0.99.130) to 620596b893d84291b88b90dc9e317612 (127.0.99.131). New cstate: current_term: 6 leader_uuid: "620596b893d84291b88b90dc9e317612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:42.115285 14954 raft_consensus.cc:2396] T 11275e009edd4354b0807b7f015023be P dcfdfa62edc34dab9525e2712d487582 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 620596b893d84291b88b90dc9e317612 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20240503 02:00:42.123881 15351 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4cdcdde158054cdaabc1856a90fe1302"
dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
from {username='slave'} at 127.0.0.1:50864
I20240503 02:00:42.124135 15351 raft_consensus.cc:482] T 4cdcdde158054cdaabc1856a90fe1302 P c05f7620bb884392931d3243a0b47a9e [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:42.124262 15351 raft_consensus.cc:3037] T 4cdcdde158054cdaabc1856a90fe1302 P c05f7620bb884392931d3243a0b47a9e [term 7 FOLLOWER]: Advancing to term 8
I20240503 02:00:42.127992 15351 raft_consensus.cc:504] T 4cdcdde158054cdaabc1856a90fe1302 P c05f7620bb884392931d3243a0b47a9e [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:42.143918 15351 leader_election.cc:288] T 4cdcdde158054cdaabc1856a90fe1302 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 8 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
I20240503 02:00:42.144322 15089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcdde158054cdaabc1856a90fe1302" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 8 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:42.144474 15089 raft_consensus.cc:3037] T 4cdcdde158054cdaabc1856a90fe1302 P b602f6099cbb4758847e7e9550685372 [term 7 FOLLOWER]: Advancing to term 8
I20240503 02:00:42.144785 15351 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "de805ef7511d4ee9b9bbed67e0a4186c"
dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
from {username='slave'} at 127.0.0.1:50864
I20240503 02:00:42.144899 15351 raft_consensus.cc:482] T de805ef7511d4ee9b9bbed67e0a4186c P c05f7620bb884392931d3243a0b47a9e [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:42.144948 15351 raft_consensus.cc:3037] T de805ef7511d4ee9b9bbed67e0a4186c P c05f7620bb884392931d3243a0b47a9e [term 7 FOLLOWER]: Advancing to term 8
I20240503 02:00:42.145584 15089 raft_consensus.cc:2396] T 4cdcdde158054cdaabc1856a90fe1302 P b602f6099cbb4758847e7e9550685372 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate c05f7620bb884392931d3243a0b47a9e for term 8 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 02:00:42.147917 15351 raft_consensus.cc:504] T de805ef7511d4ee9b9bbed67e0a4186c P c05f7620bb884392931d3243a0b47a9e [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:42.149283 15214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcdde158054cdaabc1856a90fe1302" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 8 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:42.149426 15214 raft_consensus.cc:3032] T 4cdcdde158054cdaabc1856a90fe1302 P 620596b893d84291b88b90dc9e317612 [term 7 LEADER]: Stepping down as leader of term 7
I20240503 02:00:42.149466 15214 raft_consensus.cc:729] T 4cdcdde158054cdaabc1856a90fe1302 P 620596b893d84291b88b90dc9e317612 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:42.149645 15214 consensus_queue.cc:260] T 4cdcdde158054cdaabc1856a90fe1302 P 620596b893d84291b88b90dc9e317612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:42.149787 15214 raft_consensus.cc:3037] T 4cdcdde158054cdaabc1856a90fe1302 P 620596b893d84291b88b90dc9e317612 [term 7 FOLLOWER]: Advancing to term 8
I20240503 02:00:42.151515 15214 raft_consensus.cc:2396] T 4cdcdde158054cdaabc1856a90fe1302 P 620596b893d84291b88b90dc9e317612 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate c05f7620bb884392931d3243a0b47a9e for term 8 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 02:00:42.151871 15286 leader_election.cc:302] T 4cdcdde158054cdaabc1856a90fe1302 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c05f7620bb884392931d3243a0b47a9e; no voters: 620596b893d84291b88b90dc9e317612, b602f6099cbb4758847e7e9550685372
I20240503 02:00:42.152170 17163 raft_consensus.cc:2726] T 4cdcdde158054cdaabc1856a90fe1302 P c05f7620bb884392931d3243a0b47a9e [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240503 02:00:42.160146 15351 leader_election.cc:288] T de805ef7511d4ee9b9bbed67e0a4186c P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 8 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
I20240503 02:00:42.160907 15089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de805ef7511d4ee9b9bbed67e0a4186c" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 8 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:42.161048 15089 raft_consensus.cc:3037] T de805ef7511d4ee9b9bbed67e0a4186c P b602f6099cbb4758847e7e9550685372 [term 7 FOLLOWER]: Advancing to term 8
I20240503 02:00:42.162171 15089 raft_consensus.cc:2396] T de805ef7511d4ee9b9bbed67e0a4186c P b602f6099cbb4758847e7e9550685372 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate c05f7620bb884392931d3243a0b47a9e for term 8 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 6.
I20240503 02:00:42.164532 15349 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "21d92d412a5247e8907c5c41d253ad2e"
dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
from {username='slave'} at 127.0.0.1:50864
I20240503 02:00:42.164741 15349 raft_consensus.cc:482] T 21d92d412a5247e8907c5c41d253ad2e P c05f7620bb884392931d3243a0b47a9e [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:42.164804 15349 raft_consensus.cc:3037] T 21d92d412a5247e8907c5c41d253ad2e P c05f7620bb884392931d3243a0b47a9e [term 7 FOLLOWER]: Advancing to term 8
I20240503 02:00:42.166198 15214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de805ef7511d4ee9b9bbed67e0a4186c" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 8 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:42.166857 15214 raft_consensus.cc:3032] T de805ef7511d4ee9b9bbed67e0a4186c P 620596b893d84291b88b90dc9e317612 [term 7 LEADER]: Stepping down as leader of term 7
I20240503 02:00:42.166980 15214 raft_consensus.cc:729] T de805ef7511d4ee9b9bbed67e0a4186c P 620596b893d84291b88b90dc9e317612 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:42.167585 15214 consensus_queue.cc:260] T de805ef7511d4ee9b9bbed67e0a4186c P 620596b893d84291b88b90dc9e317612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:42.168735 15214 raft_consensus.cc:3037] T de805ef7511d4ee9b9bbed67e0a4186c P 620596b893d84291b88b90dc9e317612 [term 7 FOLLOWER]: Advancing to term 8
I20240503 02:00:42.165933 15349 raft_consensus.cc:504] T 21d92d412a5247e8907c5c41d253ad2e P c05f7620bb884392931d3243a0b47a9e [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:42.169203 15348 raft_consensus.cc:1226] T 21d92d412a5247e8907c5c41d253ad2e P c05f7620bb884392931d3243a0b47a9e [term 8 FOLLOWER]: Rejecting Update request from peer 620596b893d84291b88b90dc9e317612 for earlier term 7. Current term is 8. Ops: []
I20240503 02:00:42.169986 15214 raft_consensus.cc:2396] T de805ef7511d4ee9b9bbed67e0a4186c P 620596b893d84291b88b90dc9e317612 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate c05f7620bb884392931d3243a0b47a9e for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 6.
I20240503 02:00:42.170567 15286 leader_election.cc:302] T de805ef7511d4ee9b9bbed67e0a4186c P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c05f7620bb884392931d3243a0b47a9e; no voters: 620596b893d84291b88b90dc9e317612, b602f6099cbb4758847e7e9550685372
I20240503 02:00:42.170840 17163 raft_consensus.cc:2726] T de805ef7511d4ee9b9bbed67e0a4186c P c05f7620bb884392931d3243a0b47a9e [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240503 02:00:42.171541 17224 consensus_queue.cc:1046] T 21d92d412a5247e8907c5c41d253ad2e P 620596b893d84291b88b90dc9e317612 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 }, Status: INVALID_TERM, Last received: 7.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.000s
I20240503 02:00:42.172466 17224 raft_consensus.cc:3032] T 21d92d412a5247e8907c5c41d253ad2e P 620596b893d84291b88b90dc9e317612 [term 7 LEADER]: Stepping down as leader of term 7
I20240503 02:00:42.172602 17224 raft_consensus.cc:729] T 21d92d412a5247e8907c5c41d253ad2e P 620596b893d84291b88b90dc9e317612 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:42.172824 15349 leader_election.cc:288] T 21d92d412a5247e8907c5c41d253ad2e P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 8 election: Requested vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437)
I20240503 02:00:42.172833 14956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21d92d412a5247e8907c5c41d253ad2e" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 8 candidate_status { last_received { term: 7 index: 16 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:42.172859 17224 consensus_queue.cc:260] T 21d92d412a5247e8907c5c41d253ad2e P 620596b893d84291b88b90dc9e317612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 7.17, Last appended by leader: 17, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:42.173460 17224 raft_consensus.cc:3037] T 21d92d412a5247e8907c5c41d253ad2e P 620596b893d84291b88b90dc9e317612 [term 7 FOLLOWER]: Advancing to term 8
I20240503 02:00:42.173768 15349 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9e9899666b15498791f62dab6d9dda16"
dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
from {username='slave'} at 127.0.0.1:50864
I20240503 02:00:42.173914 15349 raft_consensus.cc:482] T 9e9899666b15498791f62dab6d9dda16 P c05f7620bb884392931d3243a0b47a9e [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:42.173987 15349 raft_consensus.cc:3037] T 9e9899666b15498791f62dab6d9dda16 P c05f7620bb884392931d3243a0b47a9e [term 7 FOLLOWER]: Advancing to term 8
I20240503 02:00:42.174829 15214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21d92d412a5247e8907c5c41d253ad2e" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 8 candidate_status { last_received { term: 7 index: 16 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:42.174980 15214 raft_consensus.cc:2396] T 21d92d412a5247e8907c5c41d253ad2e P 620596b893d84291b88b90dc9e317612 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate c05f7620bb884392931d3243a0b47a9e for term 8 because replica has last-logged OpId of term: 7 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 16.
I20240503 02:00:42.175294 15286 leader_election.cc:302] T 21d92d412a5247e8907c5c41d253ad2e P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c05f7620bb884392931d3243a0b47a9e; no voters: 620596b893d84291b88b90dc9e317612, dcfdfa62edc34dab9525e2712d487582
I20240503 02:00:42.175539 17163 raft_consensus.cc:2726] T 21d92d412a5247e8907c5c41d253ad2e P c05f7620bb884392931d3243a0b47a9e [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240503 02:00:42.175935 15349 raft_consensus.cc:504] T 9e9899666b15498791f62dab6d9dda16 P c05f7620bb884392931d3243a0b47a9e [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:42.176327 15349 leader_election.cc:288] T 9e9899666b15498791f62dab6d9dda16 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 8 election: Requested vote from peers b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207), 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437)
I20240503 02:00:42.176793 15089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e9899666b15498791f62dab6d9dda16" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 8 candidate_status { last_received { term: 5 index: 10 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:42.176950 15089 raft_consensus.cc:3037] T 9e9899666b15498791f62dab6d9dda16 P b602f6099cbb4758847e7e9550685372 [term 7 FOLLOWER]: Advancing to term 8
I20240503 02:00:42.177165 15214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e9899666b15498791f62dab6d9dda16" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 8 candidate_status { last_received { term: 5 index: 10 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:42.177316 15214 raft_consensus.cc:3037] T 9e9899666b15498791f62dab6d9dda16 P 620596b893d84291b88b90dc9e317612 [term 7 FOLLOWER]: Advancing to term 8
I20240503 02:00:42.177320 15351 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "50c5abd860ae4b09bb289e9c7550c84e"
dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
from {username='slave'} at 127.0.0.1:50864
I20240503 02:00:42.177562 15351 raft_consensus.cc:482] T 50c5abd860ae4b09bb289e9c7550c84e P c05f7620bb884392931d3243a0b47a9e [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:42.177664 15351 raft_consensus.cc:3037] T 50c5abd860ae4b09bb289e9c7550c84e P c05f7620bb884392931d3243a0b47a9e [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:42.178249 15089 raft_consensus.cc:2396] T 9e9899666b15498791f62dab6d9dda16 P b602f6099cbb4758847e7e9550685372 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate c05f7620bb884392931d3243a0b47a9e for term 8 because replica has last-logged OpId of term: 6 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 10.
I20240503 02:00:42.178738 15351 raft_consensus.cc:504] T 50c5abd860ae4b09bb289e9c7550c84e P c05f7620bb884392931d3243a0b47a9e [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:42.178856 15214 raft_consensus.cc:2454] T 9e9899666b15498791f62dab6d9dda16 P 620596b893d84291b88b90dc9e317612 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 8.
I20240503 02:00:42.179147 15286 leader_election.cc:302] T 9e9899666b15498791f62dab6d9dda16 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 620596b893d84291b88b90dc9e317612, c05f7620bb884392931d3243a0b47a9e; no voters: b602f6099cbb4758847e7e9550685372
I20240503 02:00:42.179400 17163 raft_consensus.cc:2781] T 9e9899666b15498791f62dab6d9dda16 P c05f7620bb884392931d3243a0b47a9e [term 8 FOLLOWER]: Leader election won for term 8
I20240503 02:00:42.179492 15351 leader_election.cc:288] T 50c5abd860ae4b09bb289e9c7550c84e P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 7 election: Requested vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437)
I20240503 02:00:42.179478 14954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50c5abd860ae4b09bb289e9c7550c84e" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:42.179701 14954 raft_consensus.cc:3037] T 50c5abd860ae4b09bb289e9c7550c84e P dcfdfa62edc34dab9525e2712d487582 [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:42.179500 17163 raft_consensus.cc:686] T 9e9899666b15498791f62dab6d9dda16 P c05f7620bb884392931d3243a0b47a9e [term 8 LEADER]: Becoming Leader. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:42.179860 17163 consensus_queue.cc:237] T 9e9899666b15498791f62dab6d9dda16 P c05f7620bb884392931d3243a0b47a9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.10, Last appended by leader: 10, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:42.180186 15351 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7fc3330dfd0644f6a303659b960124f4"
dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
from {username='slave'} at 127.0.0.1:50864
I20240503 02:00:42.180282 15351 raft_consensus.cc:482] T 7fc3330dfd0644f6a303659b960124f4 P c05f7620bb884392931d3243a0b47a9e [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:42.180327 15351 raft_consensus.cc:3037] T 7fc3330dfd0644f6a303659b960124f4 P c05f7620bb884392931d3243a0b47a9e [term 7 FOLLOWER]: Advancing to term 8
I20240503 02:00:42.180753 14954 raft_consensus.cc:2454] T 50c5abd860ae4b09bb289e9c7550c84e P dcfdfa62edc34dab9525e2712d487582 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 7.
I20240503 02:00:42.181152 15284 leader_election.cc:302] T 50c5abd860ae4b09bb289e9c7550c84e P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c05f7620bb884392931d3243a0b47a9e, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:42.181327 17163 raft_consensus.cc:2781] T 50c5abd860ae4b09bb289e9c7550c84e P c05f7620bb884392931d3243a0b47a9e [term 7 FOLLOWER]: Leader election won for term 7
I20240503 02:00:42.181428 17163 raft_consensus.cc:686] T 50c5abd860ae4b09bb289e9c7550c84e P c05f7620bb884392931d3243a0b47a9e [term 7 LEADER]: Becoming Leader. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:42.181543 17163 consensus_queue.cc:237] T 50c5abd860ae4b09bb289e9c7550c84e P c05f7620bb884392931d3243a0b47a9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:42.181679 14816 catalog_manager.cc:5484] T 9e9899666b15498791f62dab6d9dda16 P c05f7620bb884392931d3243a0b47a9e reported cstate change: term changed from 6 to 8, leader changed from b602f6099cbb4758847e7e9550685372 (127.0.99.130) to c05f7620bb884392931d3243a0b47a9e (127.0.99.132). New cstate: current_term: 8 leader_uuid: "c05f7620bb884392931d3243a0b47a9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:42.182194 15351 raft_consensus.cc:504] T 7fc3330dfd0644f6a303659b960124f4 P c05f7620bb884392931d3243a0b47a9e [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:42.182845 14954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fc3330dfd0644f6a303659b960124f4" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 8 candidate_status { last_received { term: 5 index: 13 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:42.182914 15351 leader_election.cc:288] T 7fc3330dfd0644f6a303659b960124f4 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 8 election: Requested vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437)
I20240503 02:00:42.182967 14954 raft_consensus.cc:3037] T 7fc3330dfd0644f6a303659b960124f4 P dcfdfa62edc34dab9525e2712d487582 [term 7 FOLLOWER]: Advancing to term 8
I20240503 02:00:42.183606 15351 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c74522fbb5c64d27b5c95f05896b0716"
dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
from {username='slave'} at 127.0.0.1:50864
I20240503 02:00:42.183717 15351 raft_consensus.cc:482] T c74522fbb5c64d27b5c95f05896b0716 P c05f7620bb884392931d3243a0b47a9e [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:42.183760 15351 raft_consensus.cc:3037] T c74522fbb5c64d27b5c95f05896b0716 P c05f7620bb884392931d3243a0b47a9e [term 7 FOLLOWER]: Advancing to term 8
I20240503 02:00:42.184173 14954 raft_consensus.cc:2454] T 7fc3330dfd0644f6a303659b960124f4 P dcfdfa62edc34dab9525e2712d487582 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 8.
I20240503 02:00:42.184527 15284 leader_election.cc:302] T 7fc3330dfd0644f6a303659b960124f4 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c05f7620bb884392931d3243a0b47a9e, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:42.184721 17162 raft_consensus.cc:2781] T 7fc3330dfd0644f6a303659b960124f4 P c05f7620bb884392931d3243a0b47a9e [term 8 FOLLOWER]: Leader election won for term 8
I20240503 02:00:42.184828 17162 raft_consensus.cc:686] T 7fc3330dfd0644f6a303659b960124f4 P c05f7620bb884392931d3243a0b47a9e [term 8 LEADER]: Becoming Leader. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:42.184959 17162 consensus_queue.cc:237] T 7fc3330dfd0644f6a303659b960124f4 P c05f7620bb884392931d3243a0b47a9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 5.13, Last appended by leader: 13, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:42.185127 15351 raft_consensus.cc:504] T c74522fbb5c64d27b5c95f05896b0716 P c05f7620bb884392931d3243a0b47a9e [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:42.185348 15219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fc3330dfd0644f6a303659b960124f4" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 8 candidate_status { last_received { term: 5 index: 13 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:42.185468 15219 raft_consensus.cc:3032] T 7fc3330dfd0644f6a303659b960124f4 P 620596b893d84291b88b90dc9e317612 [term 7 LEADER]: Stepping down as leader of term 7
I20240503 02:00:42.185526 15219 raft_consensus.cc:729] T 7fc3330dfd0644f6a303659b960124f4 P 620596b893d84291b88b90dc9e317612 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:42.185662 15219 consensus_queue.cc:260] T 7fc3330dfd0644f6a303659b960124f4 P 620596b893d84291b88b90dc9e317612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 7.14, Last appended by leader: 14, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:42.185711 15217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c74522fbb5c64d27b5c95f05896b0716" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 8 candidate_status { last_received { term: 6 index: 31 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:42.185801 15219 raft_consensus.cc:3037] T 7fc3330dfd0644f6a303659b960124f4 P 620596b893d84291b88b90dc9e317612 [term 7 FOLLOWER]: Advancing to term 8
I20240503 02:00:42.185822 15217 raft_consensus.cc:3032] T c74522fbb5c64d27b5c95f05896b0716 P 620596b893d84291b88b90dc9e317612 [term 7 LEADER]: Stepping down as leader of term 7
I20240503 02:00:42.185861 15217 raft_consensus.cc:729] T c74522fbb5c64d27b5c95f05896b0716 P 620596b893d84291b88b90dc9e317612 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:42.186028 15217 consensus_queue.cc:260] T c74522fbb5c64d27b5c95f05896b0716 P 620596b893d84291b88b90dc9e317612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 7.32, Last appended by leader: 32, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:42.186170 15217 raft_consensus.cc:3037] T c74522fbb5c64d27b5c95f05896b0716 P 620596b893d84291b88b90dc9e317612 [term 7 FOLLOWER]: Advancing to term 8
I20240503 02:00:42.186803 15219 raft_consensus.cc:2396] T 7fc3330dfd0644f6a303659b960124f4 P 620596b893d84291b88b90dc9e317612 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate c05f7620bb884392931d3243a0b47a9e for term 8 because replica has last-logged OpId of term: 7 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 13.
I20240503 02:00:42.187121 15217 raft_consensus.cc:2396] T c74522fbb5c64d27b5c95f05896b0716 P 620596b893d84291b88b90dc9e317612 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate c05f7620bb884392931d3243a0b47a9e for term 8 because replica has last-logged OpId of term: 7 index: 32, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 31.
I20240503 02:00:42.187935 15089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c74522fbb5c64d27b5c95f05896b0716" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 8 candidate_status { last_received { term: 6 index: 31 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:42.188076 15089 raft_consensus.cc:3037] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372 [term 7 FOLLOWER]: Advancing to term 8
I20240503 02:00:42.188405 15214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50c5abd860ae4b09bb289e9c7550c84e" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:42.189136 15089 raft_consensus.cc:2454] T c74522fbb5c64d27b5c95f05896b0716 P b602f6099cbb4758847e7e9550685372 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 8.
I20240503 02:00:42.189429 15285 leader_election.cc:302] T c74522fbb5c64d27b5c95f05896b0716 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b602f6099cbb4758847e7e9550685372, c05f7620bb884392931d3243a0b47a9e; no voters: 620596b893d84291b88b90dc9e317612
I20240503 02:00:42.189196 15214 raft_consensus.cc:3032] T 50c5abd860ae4b09bb289e9c7550c84e P 620596b893d84291b88b90dc9e317612 [term 6 LEADER]: Stepping down as leader of term 6
I20240503 02:00:42.189666 17163 raft_consensus.cc:2781] T c74522fbb5c64d27b5c95f05896b0716 P c05f7620bb884392931d3243a0b47a9e [term 8 FOLLOWER]: Leader election won for term 8
I20240503 02:00:42.189713 15214 raft_consensus.cc:729] T 50c5abd860ae4b09bb289e9c7550c84e P 620596b893d84291b88b90dc9e317612 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:42.189743 17163 raft_consensus.cc:686] T c74522fbb5c64d27b5c95f05896b0716 P c05f7620bb884392931d3243a0b47a9e [term 8 LEADER]: Becoming Leader. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:42.189886 15214 consensus_queue.cc:260] T 50c5abd860ae4b09bb289e9c7550c84e P 620596b893d84291b88b90dc9e317612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:42.189899 17163 consensus_queue.cc:237] T c74522fbb5c64d27b5c95f05896b0716 P c05f7620bb884392931d3243a0b47a9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 6.31, Last appended by leader: 31, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:42.190038 15214 raft_consensus.cc:3037] T 50c5abd860ae4b09bb289e9c7550c84e P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:42.190724 15351 leader_election.cc:288] T c74522fbb5c64d27b5c95f05896b0716 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 8 election: Requested vote from peers 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
I20240503 02:00:42.190647 14816 catalog_manager.cc:5484] T 50c5abd860ae4b09bb289e9c7550c84e P c05f7620bb884392931d3243a0b47a9e reported cstate change: term changed from 6 to 7, leader changed from 620596b893d84291b88b90dc9e317612 (127.0.99.131) to c05f7620bb884392931d3243a0b47a9e (127.0.99.132). New cstate: current_term: 7 leader_uuid: "c05f7620bb884392931d3243a0b47a9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:42.191056 14816 catalog_manager.cc:5484] T 7fc3330dfd0644f6a303659b960124f4 P c05f7620bb884392931d3243a0b47a9e reported cstate change: term changed from 7 to 8, leader changed from 620596b893d84291b88b90dc9e317612 (127.0.99.131) to c05f7620bb884392931d3243a0b47a9e (127.0.99.132). New cstate: current_term: 8 leader_uuid: "c05f7620bb884392931d3243a0b47a9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:42.191459 15351 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6d2a1296a54a4b92a03c12954de0cc34"
dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
from {username='slave'} at 127.0.0.1:50864
I20240503 02:00:42.191619 15214 raft_consensus.cc:2396] T 50c5abd860ae4b09bb289e9c7550c84e P 620596b893d84291b88b90dc9e317612 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate c05f7620bb884392931d3243a0b47a9e for term 7 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
I20240503 02:00:42.191716 15351 raft_consensus.cc:482] T 6d2a1296a54a4b92a03c12954de0cc34 P c05f7620bb884392931d3243a0b47a9e [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:42.191787 15351 raft_consensus.cc:3037] T 6d2a1296a54a4b92a03c12954de0cc34 P c05f7620bb884392931d3243a0b47a9e [term 7 FOLLOWER]: Advancing to term 8
I20240503 02:00:42.192744 15351 raft_consensus.cc:504] T 6d2a1296a54a4b92a03c12954de0cc34 P c05f7620bb884392931d3243a0b47a9e [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:42.193543 14954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2a1296a54a4b92a03c12954de0cc34" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 8 candidate_status { last_received { term: 5 index: 11 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:42.193671 14954 raft_consensus.cc:3037] T 6d2a1296a54a4b92a03c12954de0cc34 P dcfdfa62edc34dab9525e2712d487582 [term 7 FOLLOWER]: Advancing to term 8
I20240503 02:00:42.194715 15089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2a1296a54a4b92a03c12954de0cc34" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 8 candidate_status { last_received { term: 5 index: 11 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:42.194849 15089 raft_consensus.cc:3032] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372 [term 7 LEADER]: Stepping down as leader of term 7
I20240503 02:00:42.194888 15089 raft_consensus.cc:729] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:42.194917 14954 raft_consensus.cc:2454] T 6d2a1296a54a4b92a03c12954de0cc34 P dcfdfa62edc34dab9525e2712d487582 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 8.
I20240503 02:00:42.195241 15284 leader_election.cc:302] T 6d2a1296a54a4b92a03c12954de0cc34 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c05f7620bb884392931d3243a0b47a9e, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:42.195428 17163 raft_consensus.cc:2781] T 6d2a1296a54a4b92a03c12954de0cc34 P c05f7620bb884392931d3243a0b47a9e [term 8 FOLLOWER]: Leader election won for term 8
I20240503 02:00:42.195516 17163 raft_consensus.cc:686] T 6d2a1296a54a4b92a03c12954de0cc34 P c05f7620bb884392931d3243a0b47a9e [term 8 LEADER]: Becoming Leader. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:42.195647 17163 consensus_queue.cc:237] T 6d2a1296a54a4b92a03c12954de0cc34 P c05f7620bb884392931d3243a0b47a9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 5.11, Last appended by leader: 11, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:42.195905 14811 catalog_manager.cc:5484] T c74522fbb5c64d27b5c95f05896b0716 P c05f7620bb884392931d3243a0b47a9e reported cstate change: term changed from 7 to 8, leader changed from 620596b893d84291b88b90dc9e317612 (127.0.99.131) to c05f7620bb884392931d3243a0b47a9e (127.0.99.132). New cstate: current_term: 8 leader_uuid: "c05f7620bb884392931d3243a0b47a9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:42.196328 15089 consensus_queue.cc:260] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 7.12, Last appended by leader: 12, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:42.196631 15089 raft_consensus.cc:3037] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372 [term 7 FOLLOWER]: Advancing to term 8
I20240503 02:00:42.197645 16669 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:42.197862 15351 leader_election.cc:288] T 6d2a1296a54a4b92a03c12954de0cc34 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 8 election: Requested vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
I20240503 02:00:42.198303 15089 raft_consensus.cc:2396] T 6d2a1296a54a4b92a03c12954de0cc34 P b602f6099cbb4758847e7e9550685372 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate c05f7620bb884392931d3243a0b47a9e for term 8 because replica has last-logged OpId of term: 7 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 11.
I20240503 02:00:42.199033 15351 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5736c284a85540a1bbab2be6df672fc1"
dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
from {username='slave'} at 127.0.0.1:50864
I20240503 02:00:42.199194 15351 raft_consensus.cc:482] T 5736c284a85540a1bbab2be6df672fc1 P c05f7620bb884392931d3243a0b47a9e [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:42.199291 15351 raft_consensus.cc:3037] T 5736c284a85540a1bbab2be6df672fc1 P c05f7620bb884392931d3243a0b47a9e [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:42.200271 15351 raft_consensus.cc:504] T 5736c284a85540a1bbab2be6df672fc1 P c05f7620bb884392931d3243a0b47a9e [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:42.200277 14811 catalog_manager.cc:5484] T 6d2a1296a54a4b92a03c12954de0cc34 P c05f7620bb884392931d3243a0b47a9e reported cstate change: term changed from 7 to 8, leader changed from b602f6099cbb4758847e7e9550685372 (127.0.99.130) to c05f7620bb884392931d3243a0b47a9e (127.0.99.132). New cstate: current_term: 8 leader_uuid: "c05f7620bb884392931d3243a0b47a9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:42.201035 15089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5736c284a85540a1bbab2be6df672fc1" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 7 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:42.201416 15089 raft_consensus.cc:3037] T 5736c284a85540a1bbab2be6df672fc1 P b602f6099cbb4758847e7e9550685372 [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:42.202464 15089 raft_consensus.cc:2396] T 5736c284a85540a1bbab2be6df672fc1 P b602f6099cbb4758847e7e9550685372 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate c05f7620bb884392931d3243a0b47a9e for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20240503 02:00:42.202718 15351 leader_election.cc:288] T 5736c284a85540a1bbab2be6df672fc1 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 7 election: Requested vote from peers b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207), 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437)
I20240503 02:00:42.203630 15214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5736c284a85540a1bbab2be6df672fc1" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 7 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:42.203766 15214 raft_consensus.cc:3032] T 5736c284a85540a1bbab2be6df672fc1 P 620596b893d84291b88b90dc9e317612 [term 6 LEADER]: Stepping down as leader of term 6
I20240503 02:00:42.203804 15214 raft_consensus.cc:729] T 5736c284a85540a1bbab2be6df672fc1 P 620596b893d84291b88b90dc9e317612 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:42.203917 15351 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eade118a36da42e3ac08b96a13893a5c"
dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
from {username='slave'} at 127.0.0.1:50864
I20240503 02:00:42.203977 15214 consensus_queue.cc:260] T 5736c284a85540a1bbab2be6df672fc1 P 620596b893d84291b88b90dc9e317612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:42.204146 15214 raft_consensus.cc:3037] T 5736c284a85540a1bbab2be6df672fc1 P 620596b893d84291b88b90dc9e317612 [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:42.204145 15351 raft_consensus.cc:482] T eade118a36da42e3ac08b96a13893a5c P c05f7620bb884392931d3243a0b47a9e [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:42.204252 15351 raft_consensus.cc:3037] T eade118a36da42e3ac08b96a13893a5c P c05f7620bb884392931d3243a0b47a9e [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:42.205367 15351 raft_consensus.cc:504] T eade118a36da42e3ac08b96a13893a5c P c05f7620bb884392931d3243a0b47a9e [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:42.205425 15214 raft_consensus.cc:2396] T 5736c284a85540a1bbab2be6df672fc1 P 620596b893d84291b88b90dc9e317612 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate c05f7620bb884392931d3243a0b47a9e for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20240503 02:00:42.205704 15286 leader_election.cc:302] T 5736c284a85540a1bbab2be6df672fc1 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c05f7620bb884392931d3243a0b47a9e; no voters: 620596b893d84291b88b90dc9e317612, b602f6099cbb4758847e7e9550685372
I20240503 02:00:42.205891 17162 raft_consensus.cc:2726] T 5736c284a85540a1bbab2be6df672fc1 P c05f7620bb884392931d3243a0b47a9e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240503 02:00:42.206140 15089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eade118a36da42e3ac08b96a13893a5c" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 7 candidate_status { last_received { term: 5 index: 24 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:42.206262 15089 raft_consensus.cc:3037] T eade118a36da42e3ac08b96a13893a5c P b602f6099cbb4758847e7e9550685372 [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:42.207186 15089 raft_consensus.cc:2454] T eade118a36da42e3ac08b96a13893a5c P b602f6099cbb4758847e7e9550685372 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 7.
I20240503 02:00:42.207469 15351 leader_election.cc:288] T eade118a36da42e3ac08b96a13893a5c P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 7 election: Requested vote from peers b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207), 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437)
I20240503 02:00:42.207536 15285 leader_election.cc:302] T eade118a36da42e3ac08b96a13893a5c P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b602f6099cbb4758847e7e9550685372, c05f7620bb884392931d3243a0b47a9e; no voters:
I20240503 02:00:42.207808 17162 raft_consensus.cc:2781] T eade118a36da42e3ac08b96a13893a5c P c05f7620bb884392931d3243a0b47a9e [term 7 FOLLOWER]: Leader election won for term 7
I20240503 02:00:42.207895 17162 raft_consensus.cc:686] T eade118a36da42e3ac08b96a13893a5c P c05f7620bb884392931d3243a0b47a9e [term 7 LEADER]: Becoming Leader. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:42.208042 17162 consensus_queue.cc:237] T eade118a36da42e3ac08b96a13893a5c P c05f7620bb884392931d3243a0b47a9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 5.24, Last appended by leader: 24, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } }
I20240503 02:00:42.208550 15351 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9a74cca2988a4f219f6fd9b07ade42d8"
dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
from {username='slave'} at 127.0.0.1:50864
I20240503 02:00:42.208659 15351 raft_consensus.cc:482] T 9a74cca2988a4f219f6fd9b07ade42d8 P c05f7620bb884392931d3243a0b47a9e [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:42.208669 16800 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:42.208709 15351 raft_consensus.cc:3037] T 9a74cca2988a4f219f6fd9b07ade42d8 P c05f7620bb884392931d3243a0b47a9e [term 7 FOLLOWER]: Advancing to term 8
I20240503 02:00:42.209839 15351 raft_consensus.cc:504] T 9a74cca2988a4f219f6fd9b07ade42d8 P c05f7620bb884392931d3243a0b47a9e [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:42.210516 15351 leader_election.cc:288] T 9a74cca2988a4f219f6fd9b07ade42d8 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 8 election: Requested vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), 620596b893d84291b88b90dc9e317612 (127.0.99.131:42437)
I20240503 02:00:42.210546 14954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a74cca2988a4f219f6fd9b07ade42d8" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:42.210676 14954 raft_consensus.cc:3037] T 9a74cca2988a4f219f6fd9b07ade42d8 P dcfdfa62edc34dab9525e2712d487582 [term 7 FOLLOWER]: Advancing to term 8
I20240503 02:00:42.211199 15351 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d805ee04df39406ba30b3b3b9e349507"
dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
from {username='slave'} at 127.0.0.1:50864
I20240503 02:00:42.211306 15351 raft_consensus.cc:482] T d805ee04df39406ba30b3b3b9e349507 P c05f7620bb884392931d3243a0b47a9e [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:42.211350 15351 raft_consensus.cc:3037] T d805ee04df39406ba30b3b3b9e349507 P c05f7620bb884392931d3243a0b47a9e [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:42.211962 14954 raft_consensus.cc:2454] T 9a74cca2988a4f219f6fd9b07ade42d8 P dcfdfa62edc34dab9525e2712d487582 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f7620bb884392931d3243a0b47a9e in term 8.
I20240503 02:00:42.212311 15284 leader_election.cc:302] T 9a74cca2988a4f219f6fd9b07ade42d8 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c05f7620bb884392931d3243a0b47a9e, dcfdfa62edc34dab9525e2712d487582; no voters:
I20240503 02:00:42.212548 17162 raft_consensus.cc:2781] T 9a74cca2988a4f219f6fd9b07ade42d8 P c05f7620bb884392931d3243a0b47a9e [term 8 FOLLOWER]: Leader election won for term 8
I20240503 02:00:42.212611 17162 raft_consensus.cc:686] T 9a74cca2988a4f219f6fd9b07ade42d8 P c05f7620bb884392931d3243a0b47a9e [term 8 LEADER]: Becoming Leader. State: Replica: c05f7620bb884392931d3243a0b47a9e, State: Running, Role: LEADER
I20240503 02:00:42.212720 17162 consensus_queue.cc:237] T 9a74cca2988a4f219f6fd9b07ade42d8 P c05f7620bb884392931d3243a0b47a9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:42.213727 15351 raft_consensus.cc:504] T d805ee04df39406ba30b3b3b9e349507 P c05f7620bb884392931d3243a0b47a9e [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:42.214538 14954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d805ee04df39406ba30b3b3b9e349507" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 7 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "dcfdfa62edc34dab9525e2712d487582"
I20240503 02:00:42.214668 14954 raft_consensus.cc:3037] T d805ee04df39406ba30b3b3b9e349507 P dcfdfa62edc34dab9525e2712d487582 [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:42.214742 15089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d805ee04df39406ba30b3b3b9e349507" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 7 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "b602f6099cbb4758847e7e9550685372"
I20240503 02:00:42.214860 15089 raft_consensus.cc:3032] T d805ee04df39406ba30b3b3b9e349507 P b602f6099cbb4758847e7e9550685372 [term 6 LEADER]: Stepping down as leader of term 6
I20240503 02:00:42.214895 15089 raft_consensus.cc:729] T d805ee04df39406ba30b3b3b9e349507 P b602f6099cbb4758847e7e9550685372 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: b602f6099cbb4758847e7e9550685372, State: Running, Role: LEADER
I20240503 02:00:42.215075 15089 consensus_queue.cc:260] T d805ee04df39406ba30b3b3b9e349507 P b602f6099cbb4758847e7e9550685372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } } peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } }
I20240503 02:00:42.215214 15089 raft_consensus.cc:3037] T d805ee04df39406ba30b3b3b9e349507 P b602f6099cbb4758847e7e9550685372 [term 6 FOLLOWER]: Advancing to term 7
I20240503 02:00:42.215759 14954 raft_consensus.cc:2396] T d805ee04df39406ba30b3b3b9e349507 P dcfdfa62edc34dab9525e2712d487582 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate c05f7620bb884392931d3243a0b47a9e for term 7 because replica has last-logged OpId of term: 5 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 6.
I20240503 02:00:42.215965 15217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a74cca2988a4f219f6fd9b07ade42d8" candidate_uuid: "c05f7620bb884392931d3243a0b47a9e" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: true dest_uuid: "620596b893d84291b88b90dc9e317612"
I20240503 02:00:42.216094 15217 raft_consensus.cc:3032] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612 [term 7 LEADER]: Stepping down as leader of term 7
I20240503 02:00:42.216130 15217 raft_consensus.cc:729] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 620596b893d84291b88b90dc9e317612, State: Running, Role: LEADER
I20240503 02:00:42.216277 15217 consensus_queue.cc:260] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } }
I20240503 02:00:42.216420 15217 raft_consensus.cc:3037] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612 [term 7 FOLLOWER]: Advancing to term 8
I20240503 02:00:42.216689 14811 catalog_manager.cc:5484] T eade118a36da42e3ac08b96a13893a5c P c05f7620bb884392931d3243a0b47a9e reported cstate change: term changed from 6 to 7, leader changed from 620596b893d84291b88b90dc9e317612 (127.0.99.131) to c05f7620bb884392931d3243a0b47a9e (127.0.99.132). New cstate: current_term: 7 leader_uuid: "c05f7620bb884392931d3243a0b47a9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b602f6099cbb4758847e7e9550685372" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05f7620bb884392931d3243a0b47a9e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42437 } health_report { overall_health: UNKNOWN } } }
I20240503 02:00:42.217401 15217 raft_consensus.cc:2396] T 9a74cca2988a4f219f6fd9b07ade42d8 P 620596b893d84291b88b90dc9e317612 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate c05f7620bb884392931d3243a0b47a9e for term 8 because replica has last-logged OpId of term: 7 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 8.
I20240503 02:00:42.218731 15089 raft_consensus.cc:2396] T d805ee04df39406ba30b3b3b9e349507 P b602f6099cbb4758847e7e9550685372 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate c05f7620bb884392931d3243a0b47a9e for term 7 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 6.
I20240503 02:00:42.219043 15351 leader_election.cc:288] T d805ee04df39406ba30b3b3b9e349507 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 7 election: Requested vote from peers dcfdfa62edc34dab9525e2712d487582 (127.0.99.129:38817), b602f6099cbb4758847e7e9550685372 (127.0.99.130:38207)
I20240503 02:00:42.219141 15285 leader_election.cc:302] T d805ee04df39406ba30b3b3b9e349507 P c05f7620bb884392931d3243a0b47a9e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c05f7620bb884392931d3243a0b47a9e; no voters: b602f6099cbb4758847e7e9550685372, dcfdfa62edc34dab9525e2712d487582
I20240503 02:00:42.219553 17163 raft_consensus.cc:2726] T d805ee04df39406ba30b3b3b9e349507 P c05f7620bb884392931d3243a0b47a9e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240503 02:00:42.220930 15351 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6798a9ed0cf441aca7b8ae1b05c001f3"
dest_uuid: "c05f7620bb884392931d3243a0b47a9e"
from {username='slave'} at 127.0.0.1:50864
I20240503 02:00:42.221077 15351 raft_consensus.cc:482] T 6798a9ed0cf441aca7b8ae1b05c001f3 P c05f7620bb884392931d3243a0b47a9e [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:42.221135 15351 raft_consensus.cc:3037] T 6798a9ed0cf441aca7b8ae1b05c001f3 P c05f7620bb884392931d3243a0b47a9e [term 7 FOLLOWER]: Advancing to term 8
I20240503 02:00:42.222175 15329 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:00:42.222231 14811 catalog_manager.cc:5484] T 9a74cca2988a4f219f6fd9b07ade42d8 P c05f7620bb884392931d3243a0b47a9e reported cstate change: term changed from 7 to 8, leader changed from 620596b893d84291b88b90dc9e317612 (127.0.99.131) to c05f7620bb884392931d3243a0b47a9e (127.0.99.132). New cstate: current_term: 8 leader_uuid: "c05f7620bb884392931d3243a0b47a9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcfdfa62edc34dab9525e2712d487582" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "620596b893d84291b88b90dc9e317612" member_type: VOTER last_known_addr { host: "127.0.99.131"